How NuRec Works#
NVIDIA Omniverse NuRec refers to the neural reconstruction and rendering models and services from NVIDIA that support the seamless ingestion of real-world camera and lidar data to create a simulated 3D environment suitable for training and testing Physical AI Agents, including robotics and autonomous driving systems.
Reconstruction and Rendering with NuRec#
NuRec encompasses multiple sub-components that work together to provide the core NuRec service.
NuRec overview
Data Compatibility with NuRec#
The NCore data format is a standardized format for sensor data recordings from various sources, including camera and lidar sensors. NuRec also requires an auxiliary dataset to reconstruct scenes when you convert and input your own real-world data. To use your own real-world data, you would need to convert it to the NCore format and then generate auxiliary data for NuRec.
If you want to use NuRec with a sample dataset, explore the options in Prepare Data for Use with NuRec. To learn more about NCore, see the NCore documentation.
Reconstruction#
Reconstruction converts real-world data into a 3D scene, output as a USDZ file. The reconstruction engine is inside the main NuRec container. The USDZ file is a zipped package that includes the following files:
XODR file: A driveable map for use in the simulation.
USDA files: Default file that defines the scene using Universal Scene Description (USD) and includes mapping, domelight, sequence tracks (cuboid tracks), and rig trajectories (logs of all trajectories).
Checkpoint: This is the actual AI-trained reconstruction data, including GS positions, auxiliary data.
JSON: Sequence track and rig trajectories are also available in this format.
Asset Harvester is a system of 5 models (Mask2Former, C-RADIO, Sana Multiview, LGM, Fixer) that converts actors and objects from the dataset to 3D assets. This is available as a model on HuggingFace.
Rendering#
Rendering leverages gsplat. Find the rendering engine inside the main NuRec container.
You can use the NuRec gRPC API to neurally render scenes in your simulation platform. If you use the NVIDIA Physical AI dataset, follow the instructions in Render the Physical AI dataset with NuRec.
Fixer (Difix3D+) refines and smooths the rendering of the scene. The current model is based on the text-to-image framework Sana, but soon will transition to Cosmos. The Stable Diffusion version is available as a model on HuggingFace.
Further Reading#
Learn more about the technology powering NuRec in these research papers from NVIDIA Research: