DeepStream 3D Action Recognition App

The deepstream-3d-action-recognition sample application is provided at app/sample_apps/deepstream-3d-action-recognition for your reference. This example demonstrates a sequence batching based 3D or 2D model inference pipeline for action recognition. The image below shows the architecture of this reference app.

Gst-nvdspreprocess plugin re-processses the input tensors for Gst-nvinfer plugin. Gst-nvdspreprocess loads a custom_sequence_preprocess lib (subfolder) to perform temporal sequence batching and ROI spatial batching. It delivers the preprocessed batched tensor buffers into downstream plugin Gst-nvinfer for inference. This application probes the tensor data and action classification result, converts them into display metadata to print on screen. This 3D/2D model is pretrained by NVIDIA TAO toolkit. The 3D model has NCDHW (NCSHW) input and the 2D model has NSHW shapes.

N: Max batch size of total number of ROIs in all streams, value > 0.
C: Channel numbers, must be 3.
D/S: sequence length of consecutive frames, value > 1
H: height, value > 0
W: width, value > 0
2D S: channels x sequence_length, reshaped from [C, D]

A custom sequence preprocessing lib: libnvds_custom_sequence_preprocess.so is also provided at sources/apps/sample_apps/deepstream-3d-action-recognition/custom_sequence_preprocess to demonstrate how to implement a sequence batching and preprocessing methods with Gst-nvdspreprocess plugin. This custom lib normalizes each incoming ROI cropped image and accumulates the data into buffer sequence for temporal batching. When temporal batching is ready, it continues to do spacial batching on multi-ROIs and multi-streams. Finally it returns the temporal and spacial batched buffer(tensor) to Gst-nvdspreprocess plugin which would attach the buffer as preprocess input metadata and deliver to downstream Gst-nvinfer plugin to do inference.

Getting Started

Prerequisites

Go to the folder sources/apps/sample_apps/deepstream-3d-action-recognition.
Search and Download 3D and 2D RGB based tao_iva_action_recognition_pretrained models from NGC https://ngc.nvidia.com/catalog/models/nvidia:tao:actionrecognitionnet (Version 5):
- resnet18_3d_rgb_hmdb5_32
- resnet18_2d_rgb_hmdb5_32
These Models support following classes : push; fall_floor; walk; run; ride_bike.

Update source streams uri-list in action recognition config file: deepstream_action_recognition_config.txt.

uri-list=file:///path/to/sample_action1.mov;file:///path/to/sample_action2.mov;file:///path/to/sample_action3.mov;file:///path/to/sample_action4.mov;

Export DISPLAY environment to correct display. e.g. export DISPLAY=:0.0.

Run 3D Action Recognition Examples

Make sure 3D preprocess config and 3D inference config are enabled in deepstream_action_recognition_config.txt.

# Enable 3D preprocess and inference
preprocess-config=config_preprocess_3d_custom.txt
infer-config=config_infer_primary_3d_action.txt

Run the following command:

$ deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt

Run with DS-Triton, update application config file deepstream_triton_action_recognition_config.txt.

preprocess-config=config_preprocess_3d_custom.txt
triton-infer-config=config_triton_infer_primary_3d_action.txt

Run 3D test with DS-Triton:
$ ./deepstream-3d-action-recognition -c deepstream_triton_action_recognition_config.txt
Check sources/TritonOnnxYolo/README for more details how to switch action recognition DS-Triton tests between CAPI and gRPC.

Run 2D Action Recognition Examples

Make sure 2D preprocess config and 2D inference config are enabled in deepstream_action_recognition_config.txt.

# Enable 2D preprocess and inference
preprocess-config=config_preprocess_2d_custom.txt
infer-config=config_infer_primary_2d_action.txt

Run the following command:

$ deepstream-3d-action-recognition -c deepstream_action_recognition_config.txt

Run with DS-Triton, update application config file deepstream_triton_action_recognition_config.txt.

preprocess-config=config_preprocess_2d_custom.txt
triton-infer-config=config_triton_infer_primary_2d_action.txt

Run 2D test with DS-Triton:
$ ./deepstream-3d-action-recognition -c deepstream_triton_action_recognition_config.txt
Check sources/TritonOnnxYolo/README for more details how to switch action recognition DS-Triton tests between CAPI and gRPC.