Use the NuRec gRPC API Server#

To interface with driving simulators, or to generate arbitrary novel views, NuRec implements a gRPC server you can use to serve rendered images for given camera and actor posing.

There are three aspects to rendering with the grpc server:

Downloading the gRPC protobuf package for the client
Launching the server
Querying it as the client

Tip

Before you begin, make sure you have a NuRec-trained .usdz scene. To get one, either download the NVIDIA Nurec-trained Physical AI dataset from HuggingFace, or Use NuRec for Autonomous Vehicles to train scenes from real-world data.

Downloading the gRPC protobuf package#

To download the gRPC protobuf package for the client, run the following command:

   ngc registry resource download-version "nvidia/nre/nre_grpc_protos:25.11"

Note: You must have the NGC CLI installed to run the command.

Launching the gRPC Server#

Launch the gRPC server by running the docker image:

    docker run --shm-size=64g -it --rm --gpus all \
        --net=host \
        --privileged \
        --volume /path/to/output/folder:/workdir/output \
        nvcr.io/nvidia/nre/nre:latest \
        serve-grpc \
        --artifact-glob /workdir/output/<RUN-ID>/usd-out/last.usdz

Command Line Parameters#

Required Parameters#

--artifact-glob: Glob expression to find artifacts. Must end in .usdz to find relevant files.

Optional Parameters#

Server Configuration:

--host: GRPC server host (default: “localhost”)
--port: GRPC server port (default: 8080)
--max-workers: Maximum number of workers for the grpc server (default: 1)

Scene Management:

--test-scenes-are-valid/--no-test-scenes-are-valid: Try to load each detected scene before coming online (default: False)
--cache-size: Maximum number of models to cache with count-based LRU. If OOM occurs during loading, spare backends are automatically evicted and load is retried (default: 10)
--download-cache-dir: Directory for downloaded scene files (default: “~/.cache/nre/downloaded_scenes”)
--download-cache-size: Maximum number of downloaded scenes to keep in cache (default: 5)

Rendering Options:

--enable-nrend/--no-enable-nrend: Use nrend (NuRec rendering) for faster rendering if supported by the model (default: True)
--ray-chunk-size: Maximum number of rays processed in a single forward pass. Set if seeing GPU OOMs (default: 2^62)
--enable-editing-actors: Enable the ability to edit the actors in a scene by using poses specified through RGBRenderRequest. If this is disabled, NuRec uses the original actor poses specified in the USDZ. (default: False)
--egocar-hood-dir: Directory with egocar hood images (default: None)

DiFix (Fixer) Options:

--enable-difix: Use Difix postprocessing (default: False)
--difix-url: URL of Difix checkpoint (default: NGC cosmos_3dgut.pt)
--difix-cache: Full path to local Difix cache dir (default: “~/.cache/nre/difix”)
--difix-model-filename: Filename of Difix checkpoint (default: “cosmos_3dgut.pt”)
--difix-resolution: Resolution for Difix processing (default: (576, 1024))

Performance Profiling:

--enable-timing: Enable timing of the different parts of the rendering pipeline (default: False)
--timing-verbosity: Verbosity level for timing output: NONE, SUMMARY, BASIC, DETAILS (default: “BASIC”)
--timing-logfile: File to write timing results to (e.g., timing.log) (default: None)
--timing-synchronize: Synchronize GPU before timing measurements (default: False)
--profiling-backend: Profiling backend to use: NONE, TRACY, or NVTX (default: “NONE”)
--metrics-output-dir: Directory to save render time metrics. If not specified, metrics collection is disabled (default: None)

Data Structures#

Pose and Transformation Types

    Pose: grpc_types.Pose
        vec: Vec3  # 3D position vector
            x: float  # X coordinate
            y: float  # Y coordinate
            z: float  # Z coordinate
        quat: Quat  # Quaternion representing rotation
            x: float  # X component of quaternion
            y: float  # Y component of quaternion
            z: float  # Z component of quaternion
            w: float  # W component of quaternion

    PosePair: PosePair
        start_pose: Pose  # Initial pose of the sensor
        end_pose: Pose  # Final pose of the sensor (must be different from start_pose)

Camera Types

    CameraSpec: CameraSpec
        logical_id: str  # Unique identifier for the camera
        trajectory_idx: int  # Index of the trajectory this camera belongs to
        resolution_w: int  # Width of the camera's resolution
        resolution_h: int  # Height of the camera's resolution
        shutter_type: enum  # Type of shutter mechanism
        camera_param: One of:
            ftheta_param: FthetaCameraParam  # F-theta camera parameters
                principal_point_x: float  # X coordinate of principal point
                principal_point_y: float  # Y coordinate of principal point
                reference_poly: enum  # Reference polynomial type
                pixeldist_to_angle_poly: List[float]  # Polynomial coefficients for pixel to angle conversion
                angle_to_pixeldist_poly: List[float]  # Polynomial coefficients for angle to pixel conversion
                max_angle: float  # Maximum angle supported by the camera

Dynamic Object Types

    DynamicObject: DynamicObject
        track_id: str  # Unique identifier for the track
        pose_pair: PosePair  # Start and end poses of the object

API Endpoints#

Server Information

    get_version()
    # Returns: VersionId (version_id, git_hash, grpc_api_version)

Server Configuration

    get_server_config()
    # Returns: ServerConfig (server_config: map<string, string>)
    # Current server configuration as string key-value pairs. Use to detect unexpected
    # default changes or configuration parameter mismatches between client and server.

Scene Management

    get_available_scenes()
    # Returns: AvailableScenesReturn (scene_ids: List[str])

    shut_down()
    # Shuts down the gRPC server
    # Returns: Empty

Camera Management

    get_available_cameras(scene_id: str)
    # Returns: AvailableCamerasReturn (available_cameras: List[AvailableCamera])

    get_available_trajectories(scene_id: str)
    # Returns: AvailableTrajectoriesReturn (available_trajectories: List[AvailableTrajectory])

Asset Management

    get_external_asset_objects(scene_id: str)
    # Get list of track IDs for external asset objects in a scene
    # Returns: ExternalAssetObjectsReturn (track_ids: List[str])

    get_dynamic_objects(scene_id: str)
    # Get all controllable dynamic objects in a scene
    # Returns: AvailableDynamicObjectsReturn (dynamic_objects: List[DynamicObjectTrack])

    edit_assets(scene_id: str, replace: List[ReplaceAssetAction], insert: List[DynamicObjectTrack])
    # Replace and insert assets in the scene
    # Returns: EditAssetsResponse (success: bool, message: str)

    restore_model_parameters(scene_id: str)
    # Restore the original training parameters, undoing asset edits
    # Returns: Empty

Ego Vehicle

    get_available_ego_masks()
    # Get available ego vehicle mask metadata
    # Returns: AvailableEgoMasksReturn (ego_mask_metadata)

Rendering

    render_rgb(request: RGBRenderRequest)
    # Render an RGB image from the specified camera pose
    # Returns: RGBRenderReturn (image_bytes)

    batch_render_rgb(request: BatchRGBRenderRequest)
    # Renders multiple cameras in one request. Phase 1: common setup (backend init, actor updates) once.
    # Phase 2: each camera rendered sequentially. Request has repeated BatchRGBRenderRequestItem,
    # each with camera_name (identifier) and request (RGBRenderRequest). Returns BatchRGBRenderReturn
    # with one BatchRGBRenderReturnItem per camera: camera_name, result (RGBRenderReturn), success, error_message.

    render_lidar(request: LidarRenderRequest)
    # Render a LiDAR point cloud from the specified sensor pose
    # Returns: LidarRenderReturn (point_xyzs_buffer, point_intensities_buffer, num_points)

Note:

The nre.grpc.protos package contains all the protobuf messages for the gRPC server / client.
The server supports automatic OOM (Out of Memory) recovery by evicting spare cached models when GPU memory is exhausted during model loading.
The server includes health check support through the gRPC health checking protocol at the standard health service endpoint.
Dynamic scene downloading is supported through custom HTTP headers (x-nre-scene-url and x-nre-scene-id).

Coordinate Conversion Utilities#

The nre.grpc.serve module provides utility functions for converting between coordinate representations:

from nre.grpc.serve import (
    grpc_pose_to_tquat,      # gRPC Pose -> 7D tquat [x,y,z,qx,qy,qz,qw]
    tquat_to_grpc_pose,      # 7D tquat -> gRPC Pose
    grpc_pose_to_torch_se3,  # gRPC Pose -> SE3 4x4 torch.Tensor
    se3_to_grpc_pose,        # SE3 4x4 matrix -> gRPC Pose
)

Sending gRPC Render Requests#

To send gRPC render requests, launch the docker container with the following command:

    docker run --shm-size=64g -it --rm --gpus all \
        --network host \
        --volume /path/to/output/folder:/workdir/output \
        nvcr.io/nvidia/nre/nre:latest \
        render-grpc \
        --artifact-path /workdir/output/<RUN-ID>/usd-out/last.usdz \
        --output-dir /path/to/render/directory \
        --camera-id <CAMERA-ID>

Command Line Parameters#

Required Parameters#

--artifact-path TEXT: Path to the NuRec artifact last.usdz
--output-dir TEXT: Path to the output rendered image

Optional Parameters#

--host TEXT: gRPC server host (default: localhost)
--port INTEGER: Port to run the gRPC server on (default: 8080)
--height INTEGER: Height of the image (default: 300)
--camera-id TEXT: Camera ID (default: camera_front_wide_120fov)
--image-format [png|jpeg]: Image format for the output, options are PNG or JPEG (default: jpeg)
--frame-step INTEGER: Step size in frames (default: 1)
--disable-rolling-shutter: Disable rolling shutter by applying the frame-end timestamps to full frames, which is useful for debugging
--demo-actor-transform: Apply a precomputed transformation to actor poses to generate demos
--shutdown-server-on-completion: Shutdown the server on completion
--rig-name TEXT: Rig name for the inpainted ego hood (e.g. hyperion8.0 or hyperion8.1). Set to None to disable inpainting.

Example Usage#

Here’s a step-by-step guide to render scenes from USDZ artifacts using the NuRec gRPC API:

Connect to the Server:

    import grpc.aio
    from nre.grpc.protos.sensorsim_pb2_grpc import SensorsimServiceStub

    channel = grpc.aio.insecure_channel("localhost:8080")
    client_service = SensorsimServiceStub(channel)

Discover Available Scenes:

    from nre.grpc.protos.common_pb2 import Empty
    response = await client_service.get_available_scenes(Empty())
    scene_ids = response.scene_ids

(Optional) Get server configuration:

    from nre.grpc.protos.common_pb2 import Empty
    config_response = await client_service.get_server_config(Empty())
    # config_response.server_config is a dict mapping config key -> string value
    # Use to verify expected defaults or detect parameter mismatches
    for key, value in config_response.server_config.items():
        print(f"{key}: {value}")

Get Scene Information:

    from nre.grpc.protos.sensorsim_pb2 import AvailableTrajectoriesRequest, AvailableCamerasRequest

    # Get trajectories
    trajectories = await client_service.get_available_trajectories(
        AvailableTrajectoriesRequest(scene_id="your_scene_id")
    )

    # Get cameras
    cameras = await client_service.get_available_cameras(
        AvailableCamerasRequest(scene_id="your_scene_id")
    )

Render an Image:

    from nre.grpc.protos.sensorsim_pb2 import RGBRenderRequest, ImageFormat, PosePair
    from nre.grpc.serve import se3_to_grpc_pose

    request = RGBRenderRequest(
        scene_id="your_scene_id",
        resolution_h=300,
        resolution_w=400,
        camera_intrinsics=front_wide_camera.intrinsics,
        frame_start_us=middle_timestamp,
        frame_end_us=middle_timestamp + 1,
        sensor_pose=PosePair(
            start_pose=se3_to_grpc_pose(pose),
            end_pose=se3_to_grpc_pose(pose)
        ),
        image_format=ImageFormat.JPEG,
        image_quality=95
    )

    response = await client_service.render_rgb(request)

    # Save the image
    with open("output.jpg", "wb") as f:
        f.write(response.image_bytes)

Render Multiple Cameras (Batch):

    from nre.grpc.protos.sensorsim_pb2 import (
        BatchRGBRenderRequest,
        BatchRGBRenderRequestItem,
        RGBRenderRequest,
        ImageFormat,
        PosePair,
    )
    from nre.grpc.serve import se3_to_grpc_pose

    # Build one BatchRGBRenderRequestItem per camera: camera_name + full RGBRenderRequest
    items = []
    for cam in cameras:
        rgb_request = RGBRenderRequest(
            scene_id="your_scene_id",
            resolution_h=300,
            resolution_w=400,
            camera_intrinsics=cam.intrinsics,
            frame_start_us=middle_timestamp,
            frame_end_us=middle_timestamp + 1,
            sensor_pose=PosePair(
                start_pose=se3_to_grpc_pose(pose),
                end_pose=se3_to_grpc_pose(pose),
            ),
            image_format=ImageFormat.JPEG,
            image_quality=95,
        )
        items.append(BatchRGBRenderRequestItem(camera_name=cam.logical_id, request=rgb_request))

    batch_request = BatchRGBRenderRequest(items=items)
    response = await client_service.batch_render_rgb(batch_request)

    for item in response.items:
        if not item.success:
            # Per-camera error (item.error_message is set)
            continue
        with open(f"output_{item.camera_name}.jpg", "wb") as f:
            f.write(item.result.image_bytes)