Gst-nvmsgconv#

The Gst-nvmsgconv plugin is responsible for converting metadata into message payloads based on schema. DeepStream 5.0 supports two variations of the schema, full and minimal. The Gst-nvmsgconv plugin can be configured to use either one of the schemas. By default, the plugin uses the full DeepStream schema to generate the payload in JSON format. The full schema supports elaborate semantics for object detection, analytics modules, events, location, sensor and 3D tracking. Each payload has information about a single object. You can use the minimal variation of the schema to communicate minimal information with the back end. This provides a small footprint for the payload to be transmitted from DeepStream to a message broker. Each payload can have information for multiple objects in the frame.

Gst-nvmsgconv plugin interface with a low level nvmsgconv library which provides APIs for payload generation using below metadata types:

NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type metadata attached to the buffer as user metadata of frame meta. For the batched buffer, metadata of all objects of a frame must be under the corresponding frame meta. This is the default option.
By parsing the NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type in Gst buffer and available fields in these metadata types are used to create message payload based on schema type. To use this option, set gst property msg2p-newapi = true.

This API also supports attaching a custom user message blob(a json formatted string) for each payload if the msg blob is added as user metadata of NvDsFrameMeta. Additionally, with this API there’s also an other gst property you can use to set the frame interval at which each payload is generated. frame-interval = <interval>. If this is not set, the default frame interval at which frames are generates is every 30 frames

The generated payload (NvDsPayload) in both the above cases are attached back to the input buffer as NVDS_PAYLOAD_META type user metadata.

Additionally, Gst-nvmsgconv plugin provides optional properties: - debug-payload-dir : Absolute path of the directory to dump payloads for debugging - multiple-payloads : Generate multiple message payloads

Inputs and Outputs#

Inputs
- Gst Buffer with NvDsEventMsgMeta or NvDsFrameMeta/NvDsObjectMeta
Control parameters
- config
- msg2p-lib
- payload-type
- comp-id
- debug-payload-dir (optional)
- multiple-payloads (optional)
- msg2p-newapi (optional)
- frame-interval (optional)
Output
- Same Gst Buffer with additional NvDsPayload metadata. This metadata contains information about the payload generated by the plugin.

Features#

The following table summarizes the features of the plugin.

Gst-nvmsgconv plugin features#
Feature	Description	Release
Payload in JSON format	Message payload is generated in JSON format	DS 3.0
Supports DeepStream schema specification	DeepStream schema spec implementation for messages	DS 3.0
Custom schema specification	Provision for custom schemas for messages	DS 3.0
Key-value file parsing for static properties	Read static properties of sensor/place/module in the form of key-value pair from a text file	DS 3.0
CSV file parsing for static properties	Read static properties of sensor/place/module from a CSV file	DS 3.0
DeepStream 4.0.1 minimalistic schema	Minimal variation of the DeepStream message schema	DS 4.0
New msgconv api	Supports generation of deepStream message schema payloads(full & minimal) directly using Gst buffer frame & object metadata	DS 6.0
Payload generation interval	Set frame intervals at which payloads are generated	DS 6.0

Gst Properties#

The following table describes the Gst-nvmsgconv plugin’s Gst properties.

Gst-nvmsgconv plugin gst properties#
Property	Meaning	Type and Range	Example / Notes	Platforms
config	Absolute pathname of a configuration file that defines static properties of various sensors, places, and modules.	String	config=msgconv_config.txt or config=msgconv_config.csv	dGPU Jetson
msg2p-lib	Absolute pathname of the library containing a custom implementation of the nvds_msg2p_* interface for custom payload generation.	String	msg2p-lib=libnvds_msgconv_custom.so	dGPU Jetson
payload-type	Type of schema payload to be generated. Possible values are: PAYLOAD_DEEPSTREAM: Payload using DeepStream schema. PAYLOAD_DEEPSTREAM_MINIMAL: Payload using minimal DeepStream schema. PAYLOAD_CUSTOM: Payload using custom schemas.	Integer, 0 to 4,294,967,295	payload-type=0 or payload-type=257	dGPU Jetson
comp-id	Component ID of the plugin from which metadata is to be processed.	Integer, 0 to 4,294,967,295	comp-id=2 Default is NvDsEventMsgMeta	dGPU Jetson
debug-payload-dir	Directory to dump payload	String	debug-payload-dir=<absolute path> Default is NULL	dGPU Jetson
multiple-payloads	Generate multiple message payloads	Boolean	multiple-payloads=1 Default is 0	dGPU Jetson
msg2p-newapi	Generate payloads using Gst buffer frame/object metadata	Boolean	msg2p-newapi=1 Default is 0	dGPU Jetson
frame-interval	Frame interval at which payload is generated	Integer, 1 to 4,294,967,295	frame-interval=25 Default is 30	dGPU Jetson
dummy-payload	By default payload is generated if NVDS_EVENT_MSG_META is attached to buffer. With this dummy payload can be generated if there is no NVDS_EVENT_MSG_META attached to buffer	Boolean	dummy-payload=true Default is false	dGPU Jetson

NvMsgConv low level library API’s#

Gst-nvmsgconv plugin interface with a low level nvmsgconv library which provides APIs for payload generation:

nvds_msg2p_generate to generate message payload using NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type
nvds_msg2p_generate_new to generate message payload using GST buffers(NVDS_FRAME_META (NvDsFrameMeta) and NVDS_OBJECT_META (NvDsObjectMeta) type)

Deepstream sample test apps have options to use either of API’s to generate message payload

deepstream-test4-app

Use commandline option –msg2p-meta to choose the metadata type to create payload.
0=Event Msg meta(default), Create NVDS_EVENT_MSG_META type of meta and attach to buffer
1=nvdsmeta, Use the fields within NvDsFrameMeta/NvDsObjectMeta to populate payload

deepstream-test5-app

Config options present as below and default option creates payload using NvdsEventMsgMeta
#(0): Create payload using NvdsEventMsgMeta
#(1): New Api to create payload using NvDsFrameMeta
msg-conv-msg2p-new-api=0

For payload generation using NVDS_EVENT_MSG_META type, users need:

Create NVDS_EVENT_MSG_META type of meta and attach to buffer.
Use NVDS_EVENT_MSG_META for different types of objects e.g. vehicle, person etc.
Provide copy / free functions if meta data is extended through “extMsg” field.

Whereas using the new NvMsgConv low level library API to generate payload, users do not need to worry about creating/destroying any payload object in their sample test application. The nvmsgconv low level library directly uses GST buffers(NVDS_FRAME_META / NVDS_OBJECT_META) to create payload.

Schema Customization#

This plugin can be used to implement a custom schema in two ways:

By modifying the payload generator library: To perform a simple customization of DeepStream schema fields, modify the low-level payload generation library files inside directory sources/libs/nvmsgconv. Payload generated using NVDS_EVENT_MSG_META (NvDsEventMsgMeta) type can be customized by modifying sources/libs/nvmsgconv/deepstream_schema/eventmsg_payload.cpp. If payload is generated using metadata within DS GST buffers(NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type), you can customize payload genertion by modifying the file sources/libs/nvmsgconv/deepstream_schema/dsmeta_payload.cpp.
By implementing the nvds_msg2p interface: If a library that implements the custom schema needs to be integrated with the DeepStream SDK, wrap the library in the nvds_msg2p interface and set the plugin’s msg2p-lib property to the library’s name. Set the payload-type property to PAYLOAD_CUSTOM.

See sources/libs/nvmsgconv/nvmsgconv.cpp for an example implementation of the nvds_msg2p interface.

Payload with Custom Objects#

You can add a group of custom objects to the NvDsEventMsgMeta structure in the extMsg field and specify their size in the extMsgSize field. The meta copy (copy_func) and free (release_func) functions must handle the custom fields accordingly. The payload generator library handles some standard types of objects (Vehicle, Person, Face, etc.) and generates the payload according to the schema selected. To handle custom object types, you must modify the payload generator library nvmsgconv.cpp. See deepstream-test4 for details about adding custom objects as NVDS_EVENT_MSG_META user metadata with buffers for generating a custom payload to send to back end.

If using Deepstream GST buffers(NVDS_FRAME_META (NvDsFrameMeta) type and NVDS_OBJECT_META (NvDsObjectMeta) type) to generate payload instead of NvDsEventMsgMeta, You can attach a custom message blob(string) to payload using NVDS_CUSTOM_MSG_BLOB type by creating NvDsCustomMsgInfo object and specifying the fields message (custom message blob) and len (size of the custom message). The low-level payload generation library parses every frame metadata and checks for NVDS_CUSTOM_MSG_BLOB type and if present attaches it to the payload. This option is available when gst property msg2p-newapi = true (Using the new msgconv API).

Similar to generating custom objects to the NvDsEventMsgMeta structure, refer deepstream-test4 for details about adding custom objects, you can use the meta copy (copy_func) and free (release_func) functions to handle the custom fields and modify the application to attach a custom message using NVDS_CUSTOM_MSG_BLOB (and NvDsCustomMsgInfo object) type instead of NvDsEventMsgMeta type.