Get Started with NuRec: For Simulation Platforms#

NVIDIA Omniverse NuRec is available for use in your own simulation platform. Follow the steps in this guide to integrate it.

Step 1: Set Up#

Check that you have the right hardware and software setup to run NuRec.

Step 2: Get Compatible Data#

The NCore data format is a standardized format for sensor data recordings from various sources, including camera and lidar sensors. NuRec requires input data to be in the NCore format.

You can get pre-converted sample data that is already in the NCore format from the NVIDIA NCore-converted Physical AI dataset on HuggingFace and use that to generate NuRec-trained scenes.

For reference, this NCore-formatted sample data leverages sample data from the NVIDIA Physical AI raw dataset converted using the NVIDIA Physical AI Data Converter.

Step 3: Generate Reconstructed Scenes#

Reconstruction converts real-world data into a 3D scene, output as a USDZ file.

The USDZ file is a zipped package that includes the following files:

  • XODR file: A driveable map for use in the simulation.

  • USDA files: Default file that defines the scene using Universal Scene Description (USD) and includes mapping, domelight, sequence tracks (cuboid tracks), and rig trajectories (logs of all trajectories).

  • Checkpoint: This is the actual AI-trained reconstruction data, including GS positions, auxiliary data.

  • JSON: Sequence track and rig trajectories are also available in this format.

To create these reconstructed scenes, follow the steps outlined in Use the NuRec Model for Autonomous Vehicles.

Asset Harvester is a system of 5 models that converts actors and objects from the dataset to 3D assets. To generate standalone assets from NCore-formatted data, follow the steps to Get 3D Assets with Asset Harvester. To use the generated assets in NuRec reconstructions, follow the steps in Use Asset Harvester Output in Reconstructions.

Step 4: Render Reconstructions in a Simulation#

Rendering for NuRec reconstructions is powered by gsplat.

To integrate NuRec rendering in your simulation platform, use the NuRec gRPC API.

On this page