Gst-nvmultiurisrcbin

Note

The DeepStream 7.0 introduces nvmultiurisrcbin with REST API support for dynamic sensor add/remove capability.

This bin integrates three major DeepStream components viz. nvds_rest_server, nvurisrcbin, and nvstreammux (Gst-nvstreammux) into a single GstBin. For more details on nvurisrcbin, please run gst-inspect-1.0 nvurisrcbin on a machine/docker container with DeepStream installed.

The bin allows users to create simple gstreamer pipelines for AI multimedia analytics. Users can add or remove sensors over REST API calls made to a HTTP endpoint. A sensor is defined as a streaming entity with a valid unique Sensor ID, and URI.

The below High Level Overview diagram and table illustrate how nvmultiurisrcbin integrates nvds_rest_server, nvurisrcbin, and nvstreammux.

Gst-nvmultiurisrcbin High Level Overview

Gst-nvmultiurisrcbin High Level Overview of integrated components
Component Name	Source path in DeepStreamSDK Package for more info
REST API Server; nvds_rest_server	/opt/nvidia/deepstream/deepstream/sources/libs/nvds_rest_server/
Bin Manipulation API Library	/opt/nvidia/deepstream/deepstream/sources/libs/gstnvdscustomhelper/
nvmultiurisrcbin	/opt/nvidia/deepstream/deepstream/sources/gst-plugins/gst-nvmultiurisrcbin/
deepstream-test5-app Demonstrating usage of nvmultiurisrcbin with nvmsgconv and nvmsgbroker	/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-test5/
nvurisrcbin	Source not available. For more info please run gst-inspect-1.0 nvurisrcbin on a machine/docker container with DeepStream installed.
nvstreammux	Source not available. For more info please run gst-inspect-1.0 nvstreammux

1. Introduction

This GstBin is a GStreamer source bin. The bin exposes only one source pad that provide batched buffers from an nvstreammux instance.

Users can add/remove streams to the bin for batching over the REST APIs defined in the Section REST API Payload definitions.

Users can also provide a static list of streams and their sensor IDs to start the pipeline with. This can be done using the Gst properties uri-list and sensor-id-list described below.

The nvmultiurisrcbin component output batched buffers from an nvstreammux instance.

2. Features

The following table summarizes the features of the bin.

Gst-nvmultiurisrcbin features
Feature	Description	Release
nvmultiurisrcbin integrating REST API Server, nvurisrcbin, and nvstreammux	Introducing nvmultiurisrcbin	DS 6.2
REST API to add and remove sensors runtime	Dynamically add/remove sensors	DS 6.2

3. How to use nvmultiurisrcbin in a pipeline

Below pipeline is a sample gstreamer pipeline which is configured to accept upto 10 input sources and the pipline starts up with two sources:

gst-launch-1.0 nvmultiurisrcbin \
port=9000 ip-address=localhost \
batched-push-timeout=33333 max-batch-size=10 \
drop-pipeline-eos=1 live-source=1 \
uri-list=file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4,file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 width=1920 height=1080 \
! nvmultistreamtiler ! nveglglessink

Below diagram illustrates a similar pipeline (with different plugins like nvinfer, nvmsgconv, and nvmsgbroker downstream).

Gst-nvmultiurisrcbin sample video pipeline

REST API payload definitions and sample curl commands for reference

Note:

Currently only few fields from the JSON schema described below are used/mandatory. The remaining fields are optional and ignored.

List of mandatory (and used) fields are:

value/camera_id
value/camera_url
value/change

ADD a new stream to a DeepStream pipeline

Note

DS-7.0 supports REST API version /api/v1. Refer below schema details.

Payload definition and the sample curl command:

1.curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
  "key": "sensor",
  "value": {
     "camera_id": "uniqueSensorID1",
     "camera_name": "front_door",
     "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
     "change": "camera_add",
     "metadata": {
         "resolution": "1920 x1080",
         "codec": "h264",
         "framerate": 30
     }
 },
 "headers": {
     "source": "vst",
     "created_at": "2021-06-01T14:34:13.417Z"
 }
}'

2.curl -XPOST 'http://localhost:9000/api/v1/stream/add' -d '{
 "key": "sensor",
 "event": {
     "camera_id": "uniqueSensorID1",
     "camera_name": "front_door",
     "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
     "change": "camera_streaming",
     "metadata": {
         "resolution": "1920 x1080",
         "codec": "h264",
         "framerate": 30
     }
 },
 "headers": {
     "source": "vst",
     "created_at": "2021-06-01T14:34:13.417Z"
 }
}'

REMOVE a new stream to a DeepStream pipeline

Payload definition and the sample curl command:

curl -XPOST 'http://localhost:9000/api/v1/stream/remove' -d '{
    "key": "sensor",
    "value": {
        "camera_id": "uniqueSensorID1",
        "camera_name": "front_door",
        "camera_url": "file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4",
        "change": "camera_remove",
        "metadata": {
            "resolution": "1920 x1080",
            "codec": "h264",
            "framerate": 30
        }
    },
    "headers": {
        "source": "vst",
        "created_at": "2021-06-01T14:34:13.417Z"
    }
}'

4. Gst Properties

Gst Properties directly configuring nvmultiurisrcbin

Gst-nvmultiurisrcbin gstreamer properties directly configuring the bin
Property	Meaning	Type and Range	Example Notes
uri-list	comma separated URI list of sources; URI of the file or rtsp source	string	uri-list=file:///opt/file.mp4,file:///opt/file.mp4
sensor-id-list	comma separated list of unique source sensor IDs; this vector is one to one mapped with the uri-list	string	sensor-id-list=uniqueSensorID1,uniqueSensorID2
sensor-name-list	OPTIONAL comma separated list of source sensor names; this vector is one to one mapped with the uri-list	string	sensor-name-list=SensorName1,SensorName2
mode	Video-only or Audio-only modes available	0: for video-only 1: for audio-only	mode=0 (default)
ip-address	Set REST API HTTP IP Address	string	ip-address=localhost (default)
port	Set REST API HTTP Port number; Note: User may pass 0 to disable REST API Server	string	port=9000 (default)
max-batch-size	Set the maximum batch size to be used for nvstreammux; Maximum number of sources to be supported with this instance of nvmultiurisrcbin	Integer, 1 to 4,294,967,295	max-batch-size=30 (default)

Gst Properties to configure each instance of nvurisrcbin created inside this bin

Gst-nvmultiurisrcbin nvurisrcbin specific properties
Property	Meaning	Type and Range	Example Notes
num-extra-surfaces	Set extra decoder surfaces; Number of surfaces in addition to minimum decode surfaces given by the decoder	Integer, 1 to 4,294,967,295	num-extra-surfaces=1 (default)
gpu-id	Set GPU Device ID used by nvurisrcbin components like decoder (nvv4l2decoder) and nvvideoconvert	Integer, 1 to 4,294,967,295	gpu-id=0 (default)
cudadec-memtype	Memory type for cuda decoder buffers, Memory type for CUDA decoder buffers. Represented internally by enum CudaDecMemType. 0 (memtype_device): Device 1 (memtype_pinned): Host Pinned 2 (memtype_unified): Unified	Integer, 0, 1, or 2	cudadec-memtype=2 (default)
drop-frame-interval	Interval to drop the frames, e.g. a value of 5 means the decoder outputs every fifth frame, and others are dropped.	Integer, 1 to 30	Default: 0, dGPU / Jetson
dec-skip-frames	Type of frames to skip during decoding. Represented internally by enum SkipFrame. 0 (decode_all): decode all frames 1 (decode_non_ref): skips non-ref frames (Applicable only on Jetson platform) 2 (decode_key): decode key frames	Integer, 0, 1, or 2	skip-frames=0 Default: 0 ,dGPU / Jetson
select-rtp-protocol	Transport Protocol to use for RTP	Enum RtpProtocol Default: 0, rtp-multi (0): rtp-multi - UDP + UDP Multicast + TCP (4): rtp-tcp - TCP Only	select-rtp-protocol=0 (default)
file-loop	Loop file sources after EOS. Src type must be source-type-uri and uri starting with ‘file:/’	boolean	file-loop=false (default)
rtsp-reconnect-interval	Timeout in seconds to wait since last data was received from an RTSP source before forcing a reconnection. 0=disable timeout	Integer, 1 to 4,294,967,295	rtsp-reconnect-interval=0
latency	Jitterbuffer size in milliseconds; applicable only for RTSP streams.	Integer, 1 to 4,294,967,295	latency=100 (default)
udp-buffer-size	UDP Buffer Size in bytes; applicable only for RTSP streams.	Integer, 1 to 4,294,967,295	udp-buffer-size=524288 (default)
smart-record	Enable Smart Record and choose the type of events to respond to. Sources must be of type source-type-rtsp	(0):smart-rec-disable - Disable Smart Record (1):smart-rec-cloud - Trigger Smart Record through cloud messages only (2):smart-rec-multi - Trigger Smart Record through cloud and local events	smart-record=0 (default)
smart-rec-dir-path	Path of directory to save the recorded file.	string	Absolute or relative path
smart-rec-file-prefix	By default, Smart_Record is the prefix. For unique file names every source must be provided with a unique prefix	string	Absolute or relative path
smart-rec-video-cache	Size of video cache in seconds. DEPRECATED: Use ‘smart-rec-cache’ instead	Integer, 1 to 4,294,967,295	smart-rec-video-cache=0 (default)
smart-rec-cache	Size of cache in seconds, applies to both audio and video cache	Integer, 1 to 4,294,967,295	smart-rec-cache=0 (default)
smart-rec-container	Container format of recorded video. MP4 and MKV containers are supported. Sources must be of type source-type-rtsp	(0): smart-rec-mp4 - MP4 container (1): smart-rec-mkv - MKV container	smart-rec-container=0 (default)
smart-rec-mode	Smart record mode	(0):smart-rec-mode-av - Record audio and video if available (1):smart-rec-mode-video - Record video only if available (2):smart-rec-mode-audio - Record audio only if available	smart-rec-mode=0 (default)
smart-rec-default-duration	In case a Stop event is not generated. This parameter will ensure the recording is stopped after a predefined default duration.	Integer, 1 to 4,294,967,295	smart-rec-default-duration=20 (default)
disable-passthrough	To disable the passhthrough mode of nvvideoconvert used inside the nvurisrcbin	boolean	disable-passhtorugh=false(default)
drop-pipeline-eos	Boolean property so that EOS is not propagated downstream when all source pads are at EOS.	boolean	drop-pipeline-eos=false(default)

Gst Properties to configure the instance of nvstreammux created inside this bin

All properties except “batch-size” can be configured using the same property names as defined for nvstreammux here.

5. Important Notes

nvmultiurisrcbin config recommendations and notes on expected behavior

Gst-nvmultiurisrcbin recommendations
SL	Expected Behavior and Recommendation
1	nvstreammux, live-source property. live-source=1 live-source need to be turned ON to allow batching of available buffers when number of sources is < max-batch-size configuration.
2	REST API stream/add or stream/remove payload value of the change key must contain substring add or remove respectively. Expected Behavior: If ‘add’ is in the change field value, the sensorId and uri will be used to create a new stream. If ‘remove’ is in the change field value, the sensorId and uri will be used to find and remove the matching running stream.
3	Boolean property to inform muxer to skip sending EOS downstream is recommended to be set to 1. When using nvmultiurisrcbin drop-pipeline-eos=1 is preferred default to allow pipeline to stay alive for new sensor add requests after the last added sensor EOS. drop-pipeline-eos=1

6. Limitations

Gst-nvmultiurisrcbin known issues
SL	Limitations
1	Usage of new nvstreammux for video and audio usecases are not supported at this time and might not work.