Developing Extensions for DeepStream#
This section provides an overview of developing extensions with a focus on developing DeepStream based components. The usage of various interfaces and components is explained through sample code snippets.
For the sake of simplicity, the component definition and implementation of its methods are shown together in the sample code. However, this can be split between a header file and .cpp source file as required.
Note
The interfaces.hpp header provided as part of
NvDsInterfaceExtcontains the basic DeepStream interface definitions.The definition of some concrete component types like the I/Os,
NvDsProbeConnectorare included in header files provided as part ofNvDsBaseExt.The definitions of component specific Action, Signal, PropertyController, and Configuration types are included in header files provided as part of the extension the component belongs to.
All DeepStream types are defined under the namespace
nvidia::deepstream. Implementations should take care of this if not using the same namespace.
Extension and component factory registration boilerplate#
The following provides a boilerplate on registering extensions and components
#include "gxf/std/extension_factory_helper.hpp"
#include "sample_runtime_source_manipulator.hpp"
...
GXF_EXT_FACTORY_BEGIN()
GXF_EXT_FACTORY_SET_INFO(0x44a711e485194a68, 0x81e8be7ee4af3ff0,
"NvDsSampleExt",
"Sample components for demonstrating usage of various "
"DeepStream interfaces and components",
"NVIDIA", "0.0.1", "Proprietary");
...
GXF_EXT_FACTORY_ADD(
0x717b2c432f104fe8, 0xb96165e408ece299,
nvidia::deepstream::NvDsSimpleComponent,
nvidia::deepstream::INvDsComponent,
"Description of a simple component");
...
GXF_EXT_FACTORY_END()
Include the header file
gxf/std/extension_factory_helper.hppcontaining the helper macros. Include any other header file containing the definitions of components to be registered.GXF_EXT_FACTORY_BEGIN()andGXF_EXT_FACTORY_END()mark the beginning and end of the block of code containing registration code.Within the factory block, call
GXF_EXT_FACTORY_SET_INFO()to set the extension information including the UUID for the extension, the extension name, description, author, version and license.For each of the component to be registered as part of the extension, call
GXF_EXT_FACTORY_ADD()to add the component along with the component information including the UUID for the component, the component type, the component’s base type and a description of the component.The UUIDs are unique 128-bit identifiers. This must be unique across all extensions and component registered. In the code, two 64-bit unsigned integers represent the upper 64 bits and lower 64bits of the 128-bit UUID.
A simple DeepStream component#
#include "extensions/nvdsinterface/interfaces.hpp"
namespace nvidia {
namespace deepstream {
class NvDsSimpleComponent : INvDsComponent {
public:
// Public methods using which other components can interact with this
// component via its handle.
void simpleComponentMethod() {
//
}
private:
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
nvidia::gxf::Expected<void> result;
result &= registrar->parameter(
simple_param_, // Parameter member variable
"simple-param-key", // Parameter name(key). This is used to set
// parameter value in a graph
"Simple Parameter", // Parameter head line
"Description of the simple parameter", // A description of the
// parameter
100UL, // A default value for the parameter
GXF_PARAMETER_FLAGS_OPTIONAL // Parameter flags marking it
);
result &= registrar->parameter(handle_param_, "handle-parameter",
"Handle Parameter",
"Description of the handle parameter",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
return nvidia::gxf::ToResultCode(result);
}
gxf_result_t initialize() override {
// This method can be used to initialize the component.
...
// Check if parameter is set
if (simple_param_.try_get() != std::nullopt) {
uint64_t simple_param_value = simple_param_.try_get().value();
...
}
return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error
}
gxf_result_t deinitialize() override {
// This method can be used to deinitialize the component.
...
return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error
}
gxf_result_t start() override {
// Start the component. The underlying DeepStream pipeline and other
// components are already initialized. It is safe to call methods of other components
// via their handles.
...
// Check if any component is attached to the parameter
if (handle_param.try_get() != std::nullopt) {
SampleOtherComponent *other_comp = handle_param.try_get().value();
other_comp->otherComponentMethod();
...
}
return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error
}
gxf_result_t stop() override {
// Pipeline has been stopped. All the components would be deinitialized
// after this. Add any logic for stopping the component.
return GXF_SUCCESS; // return GXF_FAILURE in case of any fatal error
}
nvidia::gxf::Parameter<uint64_t> simple_param_;
nvidia::gxf::Parameter<nvidia::gxf::Handle<SampleOtherComponent>>
handle_param_;
};
} // namespace deepstream
Implementation of INvDsInPlaceDataHandler#
#include "extensions/nvdsbase/nvds_probe_connector.hpp"
#include "extensions/nvdsinterface/interfaces.hpp"
namespace nvidia {
namespace deepstream {
class NvDsSampleInPlaceDataHandler : public INvDsInPlaceDataHandler {
nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsProbeConnector>>
probe_connector_;
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) {
...
// Must register a handle parameter of type NvDsProbeConnector
result &= registrar->parameter(
probe_connector_, "probe-connector", "Probe Connector",
"Handle to a nvidia::deepstream::NvDsProbeConnector component",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
...
}
gxf_result_t initialize() override {
...
// Need to set the handler and flags which tells NvDsProbe component
// what type of data this component wants to handle. This must be done
// in the initialize method.
if (probe_connector_.try_get() != std::nullopt) {
// The pointer to self (this) can be passed since the component
// implements the INvDsInPlaceDataHandler interface.
probe_connector_.try_get().value()->set_handler(this);
probe_connector_.try_get().value()->set_flags(static_cast<NvDsProbeFlags>(
NvDsProbeFlags::BUFFER | NvDsProbeFlags::EVENT));
}
...
return GXF_SUCCESS;
}
bool handle_buffer(GstPad *pad, nvidia::gxf::Entity buffer_data) override {
// Check for presence of NvDsBatchMeta and NvBufSurface in the buffer_data
// entity since this implementation requires it. Other implementations may
// want to check for presence of other data components.
if (!buffer_data.get<NvDsBatchMetaHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)) {
return true;
}
if (!buffer_data.get<NvBufSurfaceHandle>(BUF_DATA_KEY_NVBUFSURFACE)) {
return true;
}
NvDsBatchMeta *batch_meta =
*(buffer_data.get<NvDsBatchMetaHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)
.value());
NvBufSurface *surf =
*(buffer_data.get<NvBufSurfaceHandle>(BUF_DATA_KEY_VIDEO_BATCH_META)
.value());
// Use batch_meta and surf
...
return true; // allows buffer to pass through. return false to drop the buffer
}
bool handle_event(GstPad *pad, GstEvent *event) override {
// Use the event
...
return true; // allows buffer to pass through. return false to drop the event
}
};
} // namespace deepstream
} // namespace nvidia
The component NvDsSampleProbeMessageMetaCreation provided as part of the sample extension is a complete sample implementation of INvDsInPlaceDataHandler interface.
Controlling Properties#
This sample code demonstrates the usage of an INvDsPropertyController (NvDsOSDPropertyController in this case) component by a custom component to control properties of an INvDsElement based component (NvDsOSD in this case).
// Provided as part of the NvDsVisualizationExt extension which contains the
// NvDsOSD component. This header contains the definition of
// NvDsOSDPropertyController component which provides methods to control
// properties of NvDsOSD component.
#include "nvdsvisualization_prop_controllers.hpp"
namespace nvidia {
namespace deepstream {
class NvDsSamplePropertyControl : public INvDsComponent {
nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsOSDPropertyController>>
osd_property_controller_;
NvDsOSDPropertyController *osd_property_controller = nullptr;
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
...
// Must register a handle parameter of type NvDsOSDPropertyController
result &= registrar->parameter(
osd_property_controller_, "nvdsosd-prop-controller",
"NvDsOSD Property Controller",
"Handle to a nvidia::deepstream::NvDsOSDPropertyController "
"component.",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
...
};
void sample_thread_func() {
while (1) {
bool display_text;
// Get the current property value
osd_property_controller->get_display_text(&display_text);
// Toggle the property value
osd_property_controller->set_display_text(!display_text);
sleep(1);
// Break when signalled to stop
}
}
gxf_result_t start() override {
...
// Check if the property controller component is attached
if (osd_property_controller_.try_get() != std::nullopt) {
osd_property_controller = osd_property_controller_.try_get().value();
osd_property_controller->set_display_text(true);
// Start a thread which periodically toggles the display-text property
}
...
}
gxf_result_t stop() override {
// Signal samplethread to stop
}
};
} // namespace deepstream
} // namespace nvidia
Triggering Actions#
This sample code demonstrates the usage of an INvDsAction (NvDsSourceManipulationAction in this case) component by a custom component to trigger actions of an INvDsElement based component (NvDsMultiSrcInput in this case).
// Provided as part of the NvDsSourceExt extension which contains the
// NvDsMultiSrcInput component. This header contains the definition of
// NvDsSourceManipulationAction component which provides methods to trigger
// add/remove source actions of NvDsMultiSrcInput component.
#include "nvdsinputsrc_signals.hpp"
namespace nvidia {
namespace deepstream {
class NvDsSampleSourceManipulator : public INvDsComponent {
nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsSourceManipulationAction>>
source_manip_action_;
NvDsSourceManipulationAction *source_manip_action = nullptr;
bool add = true;
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
...
// Must register a handle parameter of type NvDsSourceManipulationAction
result &= registrar->parameter(
source_manip_action_, "source-manip-action", "Source Manipulation Action",
"Handle to a nvidia::deepstream::NvDsSourceManipulationAction "
"component",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
...
}
void sample_thread_func() {
while (1) {
// Call methods of the action component via it's handle to trigger the add/remove actions.
if (add) {
if (!source_manip_action->add_source("file:///some/file/path.mp4", 2)) {
GXF_LOG_WARNING("Failed to add source");
}
} else {
if (!source_manip_action->remove_source("file:///some/file/path.mp4", 2)) {
GXF_LOG_WARNING("Failed to remove source");
}
}
add = !add;
sleep(10);
// Break when signalled to stop
}
}
gxf_result_t start() override {
...
// Check if the action component is attached
if (source_manip_action_.try_get() != std::nullopt) {
source_manip_action = source_manip_action_.try_get().value();
// Start a thread to periodically add / remove a source
}
...
}
gxf_result_t stop() override {
// Stop the thread
}
};
} // namespace deepstream
} // namespace nvidia
The component NvDsSampleSourceManipulator provided as part of the sample extension is a complete sample implementation demonstrating this.
Handling signal callbacks#
This sample code demonstrates the usage of an INvDsSignal (NvDsModelUpdatedSignal in this case) component by a custom component to handle signals emitted by an INvDsElement based component (NvDsInferVideo in this case).
// Provided as part of the NvDsInferenceExt extension which contains the
// NvDsInferVideo component. This header contains the definition of
// NvDsModelUpdatedSignal component which provides the Handler interface and
// callback method prototype and a way to set the Handler instance.
#include "nvdsinference_signals.hpp"
namespace nvidia {
namespace deepstream {
class NvDsSampleSignalHandler : public INvDsComponent,
public NvDsModelUpdatedSignal::Handler {
nvidia::gxf::Parameter<nvidia::gxf::Handle<NvDsModelUpdatedSignal>> model_update_signal_;
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
...
// Must register a handle parameter of type NvDsModelUpdatedSignal
result &= registrar->parameter(
model_update_signal_, "model-update-signal",
"Model Updated Signal",
"Handle to a nvidia::deepstream::NvDsModelUpdatedSignal "
"component.",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
...
};
// Implement the methods of the NvDsModelUpdatedSignal::Handler interface
void on_model_updated(int errorCode, gchar *configFilePath) override {
// Handle the on-the-fly model update status.
}
gxf_result_t start() override {
// Start the component
...
// Check if the signal component is attached, set the signal handler to self
if (model_update_signal_.try_get() != std::nullopt) {
// The pointer to self (this) can be passed since the component
// implements the NvDsModelUpdatedSignal::Handler interface
model_update_signal_.try_get().value()->set_handler(this);
}
...
}
gxf_result_t stop() override {
// Stop the component
}
};
} // namespace deepstream
} // namespace nvidia
Implementation of an Configuration Provider component#
This sample code demonstrates the implementation of a configuration provider interface (INvDsInferModelConfigComponent in this case) component by a custom component to provide a configuration (to NvDsInferVideo in this case).
// Provided as part of the NvDsInferenceExt extension which contains the
// NvDsInferVideo component. This header contains the definition of
// INvDsInferModelConfigComponent interface which acts as a model configuration
// provider to the NvDsInferVideo component.
#include "nvdsinference_config.hpp"
namespace nvidia {
namespace deepstream {
class SampleModel : public INvDsInferModelConfigComponent {
gxf_result_t fill_config(NvDsInferModelConfig *config) override {
// config_infer_primary.txt is packaged alongside the extension binary,
// along with caffemodel, prototxt and labels file. Use get_absolute_path()
// method to convert a path relative to the extension binary to an absolute
// path at runtime.
config->config_file_path = get_absolute_path("config_infer_primary.txt");
if (engine_file_.try_get() != std::nullopt) {
// The configuration provider component itself can have parameters which
// can be used to populate the configuration.
config->model_engine_path =
get_absolute_path(engine_file_.try_get().value());
}
return GXF_SUCCESS;
}
gxf_result_t registerInterface(nvidia::gxf::Registrar *registrar) override {
nvidia::gxf::Expected<void> result;
result &= registrar->parameter(engine_file_, "model-engine-file",
"Model Engine File",
"Path to the model engine file. Absolute or "
"relative to the extension directoy.",
std::nullopt, GXF_PARAMETER_FLAGS_OPTIONAL);
return nvidia::gxf::ToResultCode(result);
}
nvidia::gxf::Parameter<std::string> engine_file_;
};
} // namespace deepstream
} // namespace nvidia
NvDsSampleAudioTemplateLib and NvDsSampleVideoTemplateLib are other examples of components acting as configuration providers. These components are part of the sample extension.