DeepStream 3D Depth Camera App#

The deepstream-3d-depth-camera sample application is provided at app/sample_apps/deepstream-3d-depth-camera for your reference. This example demonstrates two different pipelines which bring depth camera from 2D into 3D world. The depth and color data capture component is produced by Intel^® RealSense^™ Depth Camera D435 series. For more details, see https://www.intelrealsense.com/depth-camera-d435/.

Prerequisites#

You must have the following development packages installed:

GStreamer-1.0

GStreamer-1.0 Base Plugins

GStreamer-1.0 gstrtspserver

X11 client-side library

libyaml-cpp-dev

RealSense SDK

To install these packages, execute the following command:

sudo apt-get install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev \
libgstrtspserver-1.0-dev libx11-dev libyaml-cpp-dev

Follow the RealSense SDK documentation to add RealSense public key and list of repositories. For x86, refer to IntelRealSense/librealsense. For Jetson platforms, refer to IntelRealSense/librealsense. You may use Ubuntu 20.04 source path into apt-repository to install dependencies.
```
sudo add-apt-repository "deb https://librealsense.intel.com/Debian/apt-repo bionic main" -u
```
Then install librealsense2 package to build and test.
```
sudo apt-get install librealsense2-utils librealsense2-dev
```
Make sure RealSense SDK version is not earlier than 2.48.
Plugin in Intel RealSense Depth D400 series USB camera. To verify the camera and upgrade firmware, run realsense-viewer and update from the UI. Navigate to the folder sources/apps/sample_apps/deepstream-3d-depth-camera.
If you are running these commands inside deepstream container, make sure camera devices are mounted in the docker run command. For example:
```
docker run --rm -it --gpus '"device=0"' --device /dev/video0 --device /dev/video1 --device /dev/video2 ...
```
Export DISPLAY environment to the correct display. for example, export DISPLAY=:0.0.

Depth Color Capture to 2D Rendering Pipeline Overview#

This is the first pipeline from the depth capture to 2D based depthmap to direct rendering with linear colors configured by the user.

The pipeline is setup by ds_3d_realsense_depth_capture_render.yaml. It has 2 components, ds3d::dataloader for depth/color and datarender for GLES 2D render.

ds3d::dataloader loads custom lib libnvds_3d_dataloader_realsense.so and creates a RealSense dataloader through the createRealsenseDataloader function. This specific loader is configured for output streams of [color, depth]. Gst-appsrc connects the dataloader into the deepstream pipeline.
```
name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
```
ds3d::datarender loads custom lib libnvds_3d_gl_datarender.so and create a GLES render through the createDepthStreamDataRender function. This specific loader is configured for 2D depth and color images. Gst-appsink connects the datarender into the deepstream pipeline.
```
name: depth-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender
```
realsense_dataloader captures color and depth streams in ds3d/datamap, then delivers the data to ds3d::datarender component depth-render.

Depth Color Capture to 3D Point Cloud Processing and Rendering#

The 2nd pipeline is from depth capture, 3D point cloud processing and alignment, to the 3D point cloud rendering with colors.

The 2nd pipeline is setup by the ds_3d_realsense_depth_to_point_cloud.yaml. It has 3 components: ds3d::dataloader for depth/color capture, ds3d::datafilter for depth-to-point-cloud processing, and datarender for GLES 3D points render.

ds3d::dataloader is the same as the 1st pipeline for depth and color capture. It also outputs the intrinsic parameters of the color/depth sensors, and extrinsic parameters from depth to color sensor module.
```
name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader
```
ds3d::datafilter loads custom lib libnvds_3d_depth2point_datafilter.so and creates a depth-to-3D-point processing filter through the createDepth2PointFilter function. This specific filter generates the 3D point-cloud dataset from depth, and calculates a UV-coordinate map for correspondence between points and color position. It requires sensors’ intrinsic and extrinsic data from dataloader to make the alignment.
```
name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter
```
ds3d::datafilter is loaded by the nvds3dfilter Gst-plugin which accepts in_caps as sink_caps and out_caps as src_caps. It creates a custom ds3d::datafilter instance and processess data as ds3d/datamap.
ds3d::datarender loads custom lib libnvds_3d_gl_datarender.so and create a GLES render through the createPointCloudDataRender function . This specific loader is configured for 3D points and colors rendering. It also supports 3D scene rotation based on mouse-drag.
```
name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender
```
realsense_dataloader captures color and depth streams along with the intrinsic and extrinsic parameters of the sensor in ds3d/datamap. It then delivers the data to the next component point2cloud_datafilter. This component generates 3D point-cloud data and UV coordination map into new output data ds3d/datamap. Finally, the data is delivered to ds3d::datarender component point-render for 3D rendering.

Inside the configuration files, in_caps and out_caps correspond to Gstreamer’s sink_caps and src_caps.

Getting Started#

Run RealSense Camera for Depth Capture and 2D Rendering Examples#

Run the depth capture 2D render pipeline:

$ deepstream-3d-depth-camera -c ds_3d_realsense_depth_capture_render.yaml

This sets up a realsense dataloader.

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

It then streams ds3d/datamap to final the rendering component: depth-render.

name: depth-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createDepthStreamDataRender

The depth-render custom lib shall display the depth data and color data together in same window. Update min_depth/max_depth to remove foreground and background objects in the depth rendering. From RealSense specification, D435 Camera depth range is between 0.3 and 3 meters.

min_depth: 0.3 # in meters
max_depth: 2.0 # in meters

Update min_depth_color/min_depth_color [R, G, B] values to visualize color map of depth.

     min_depth_color: [255, 128, 0] # RGB color value for mininum depth
     max_depth_color: [0, 128, 255] # RGB color value for maximum depth

The other colors are linearly interpolated between ``min_depth_color`` and ``max_depth_color``

Run 3D Depth Capture, Point Cloud filter, and 3D Points Rendering Examples#

Run the depth capture and 3D processing pipeline:

$ deepstream-3d-depth-camera -c ds_3d_realsense_depth_to_point_cloud.yaml

This sets up a realsense dataloader (same as 2D depth). It then streams ds3d/datamap to the downstream datafilter component point2cloud_datafilter.

name: realsense_dataloader
type: ds3d::dataloader
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_dataloader_realsense.so
custom_create_function: createRealsenseDataloader

It streams ds3d/datamap to the nvds3dfilter Gst-plugin which loads point2cloud_datafilter to convert 2D depth into 3D points. For more details on nvds3dfilter Gst-plugin, See Gst-nvds3dfilter.

name: point2cloud_datafilter
type: ds3d::datafilter
in_caps: ds3d/datamap
out_caps: ds3d/datamap
custom_lib_path: libnvds_3d_depth2point_datafilter.so
custom_create_function: createDepth2PointFilter

Finally the data stream as ds3d/datamap is delivered to the render component point-render.

name: point-render
type: ds3d::datarender
in_caps: ds3d/datamap
custom_lib_path: libnvds_3d_gl_datarender.so
custom_create_function: createPointCloudDataRender

Fields of view_* are the eyes of the view position, similar to the OpenGL gluLookAt().

view_position: [0, 0, -1] # view position in xyz coordinates
view_target: [0, 0, 1] # view target which is the direction pointing to
view_up: [0, -1.0, 0] # view up direction

Fields of near/far/fov are the perspective range of the eye, similar to the OpenGL gluPerspective().

near: 0.01 # nearest points of perspective
far: 10.0 # farmost points of perspective
fov: 40.0 # FOV of perspective