Detailed Description

This is a header file for pre-processing cuda kernels with normalization and mean subtraction required by nvdsinfer.

Namespaces
	triton

Data Structures
class	BaseBackend
	Base class of inference backend processing. More...

class	BaseBatchArray
	The base class for array of batch buffers. More...

class	BaseBatchBuffer
	The base class for batch buffers. More...

class	BasePostprocessor
	Base post-processor class. More...

class	BasePreprocessor
	Base preprocessor class. More...

class	BatchSurfaceBuffer

class	BufOptions

class	ClassificationOutput

class	ClassifyPostprocessor
	Post processor class for classification output. More...

class	CpuMem
	Allocates and manages host memory. More...

class	CropSurfaceConverter
	Preprocessor for cropping, scaling and padding the inference input to required height, width. More...

class	CudaDeviceMem
	Allocates and manages CUDA device memory. More...

class	CudaEvent
	Wrapper class for CUDA events. More...

class	CudaHostMem
	Allocates and manages CUDA pinned memory. More...

class	CudaStream
	Wrapper class for CUDA streams. More...

class	CudaTensorBuf
	A batch buffer with CUDA memory allocation. More...

class	DetectionOutput

class	DetectPostprocessor
	Post processor class for detection output. More...

class	IBackend

class	IBatchArray
	Interface class for an array of batch buffers. More...

class	IBatchBuffer
	Interface class for a batch buffer. More...

class	IInferCustomProcessor
	Interface of Custom processor which is created and loaded at runtime through CreateCustomProcessorFunc. More...

class	ImageAlignBuffer

class	InferBaseContext
	The base class for handling the inference context. More...

struct	InferBatchDims
	Holds full dimensions (including batch size) for a layer. More...

struct	InferBufferDescription
	Holds the information about a inference buffer. More...

class	InferCudaContext
	Class for inference context CUDA processing. More...

struct	InferDims
	Holds the information about the dimensions of a neural network layer. More...

class	InferExtraProcessor
	: Extra processing pre/post inference. More...

class	InferGrpcClient
	Wrapper class for the gRPC client of the Triton Inference Server, interfaces with the Triton client library. More...

class	InferGrpcContext
	Inference context class accessing Triton Inference Server in gRPC mode. More...

class	InferSimpleContext

class	InferTrtISContext
	Inference context accessing Triton Inference Server in C-API mode. More...

class	IOptions

class	IPostprocessor
	Post-processor interface class. More...

class	IPreprocessor
	Preprocessor interface class. More...

struct	LayerDescription
	Stores the information of a layer in the inference model. More...

class	LstmController

class	NetworkPreprocessor
	Preprocessor for scaling and normalization of the input and conversion to network media format. More...

class	OtherPostprocessor
	Post processor class for tensor output for custom post processing. More...

class	Postprocessor
	A generic post processor class. More...

class	RefBatchBuffer
	A batch buffer with allocated memory. More...

class	SegmentationOutput

class	SegmentPostprocessor
	Post processor class for segmentation output. More...

class	StreamManager

class	SurfaceBuffer

class	SysMem
	Base class for managing memory allocation. More...

class	ThreadCudaPostprocessor
	A CUDA post processor thread queue template class. More...

class	ThreadHostPostprocessor
	A host post processor thread queue template class. More...

class	ThreadPreprocessor
	Preprocessor thread queue class template. More...

class	TritonGrpcBackend
	Triton gRPC mode backend processing class. More...

class	TritonGrpcRequest
	Triton gRPC inference request class holding data associated with one inference request. More...

class	TritonSimpleRuntime

class	TrtISBackend
	Triton backend processing class. More...

class	TrtIsClassifier
	Post processor class for Triton Classification option. More...

class	TrtISServer
	Wrapper class for creating Triton Inference Server instance. More...

class	TrtServerAllocator
	Wrapper class for Triton server output memory allocator. More...

class	TrtServerRequest
	Wrapper class for Triton inference request. More...

class	TrtServerResponse
	Wrapper class for Triton output parsing. More...

class	WrapCBatchBuffer

Typedefs
using	SharedIBatchBuffer = std::shared_ptr< IBatchBuffer >

using	SharedIBatchArray = std::shared_ptr< IBatchArray >

using	SharedIOptions = std::shared_ptr< IOptions >

using	SharedBufOptions = std::shared_ptr< BufOptions >

using	UniqBackend = std::unique_ptr< BaseBackend >

using	InferCompleted = std::function< void(NvDsInferStatus, SharedBatchArray)>

using	TensorMapPool = MapBufferPool< std::string, UniqCudaTensorBuf >

using	TensorMapPoolPtr = std::unique_ptr< TensorMapPool >

typedef std::map< std::string, std::string >	Headers

using	SharedGrpcRequest = std::shared_ptr< TritonGrpcRequest >

using	TritonGrpcAsyncDone = std::function< void(NvDsInferStatus, SharedBatchArray)>

using	LayerDescriptionList = std::vector< LayerDescription >

using	SharedClassOutput = std::shared_ptr< ClassificationOutput >

using	UniqSurfaceBuf = std::unique_ptr< SurfaceBuffer >

using	SharedBatchSurface = std::shared_ptr< BatchSurfaceBuffer >

using	SharedRequest = std::shared_ptr< TrtServerRequest >

using	UniqResponse = std::unique_ptr< TrtServerResponse >

using	SharedResponse = std::shared_ptr< TrtServerResponse >

using	TritonInferAsyncDone = std::function< void(SharedRequest, UniqResponse)>

Enumerations
enum	InferTensorOrder : int { InferTensorOrder::kNone = 0, InferTensorOrder::kLinear = 1, InferTensorOrder::kNHWC = 2, InferTensorOrder::kNone = 0, InferTensorOrder::kLinear = 1, InferTensorOrder::kNHWC = 2 }
	The type of tensor order. More...

enum	InferMemType : int { InferMemType::kNone = 0, InferMemType::kGpuCuda = 1, InferMemType::kCpu = 2, InferMemType::kCpuCuda = 3, InferMemType::kNvSurface = 5, InferMemType::kNvSurfaceArray = 6, InferMemType::kNone = 0, InferMemType::kGpuCuda = 1, InferMemType::kCpu = 2, InferMemType::kCpuCuda = 3, InferMemType::kNvSurface = 5, InferMemType::kNvSurfaceArray = 6 }
	The memory types of inference buffers. More...

enum	InferDataType : int { InferDataType::kFp32 = FLOAT, InferDataType::kFp16 = HALF, InferDataType::kInt8 = INT8, InferDataType::kInt32 = INT32, InferDataType::kInt16 = 7, InferDataType::kUint8, InferDataType::kUint16, InferDataType::kUint32, InferDataType::kFp64, InferDataType::kInt64, InferDataType::kUint64, InferDataType::kString, InferDataType::kBool, InferDataType::kNone = -1, InferDataType::kFp32 = FLOAT, InferDataType::kFp16 = HALF, InferDataType::kInt8 = INT8, InferDataType::kInt32 = INT32, InferDataType::kInt16 = 7, InferDataType::kUint8, InferDataType::kUint16, InferDataType::kUint32, InferDataType::kFp64, InferDataType::kInt64, InferDataType::kUint64, InferDataType::kString, InferDataType::kBool, InferDataType::kNone = -1 }
	Datatype of the tensor buffer. More...

enum	InferPostprocessType : int { InferPostprocessType::kDetector = 0, InferPostprocessType::kClassifier = 1, InferPostprocessType::kSegmentation = 2, InferPostprocessType::kTrtIsClassifier = 3, InferPostprocessType::kOther = 100, InferPostprocessType::kDetector = 0, InferPostprocessType::kClassifier = 1, InferPostprocessType::kSegmentation = 2, InferPostprocessType::kTrtIsClassifier = 3, InferPostprocessType::kOther = 100 }
	Inference post processing types. More...

enum	InferMediaFormat : int { InferMediaFormat::kRGB = 0, InferMediaFormat::kBGR, InferMediaFormat::kGRAY, InferMediaFormat::kRGBA, InferMediaFormat::kBGRx, InferMediaFormat::kUnknown = -1, InferMediaFormat::kRGB = 0, InferMediaFormat::kBGR, InferMediaFormat::kGRAY, InferMediaFormat::kRGBA, InferMediaFormat::kBGRx, InferMediaFormat::kUnknown = -1 }
	Image formats. More...

enum	OptionType : int { OptionType::oBool = 0, OptionType::oDouble, OptionType::oInt, OptionType::oUint, OptionType::oString, OptionType::oObject, OptionType::oArray, OptionType::oNone = -1, OptionType::oBool = 0, OptionType::oDouble, OptionType::oInt, OptionType::oUint, OptionType::oString, OptionType::oObject, OptionType::oArray, OptionType::oNone = -1 }

enum	InferTensorOrder : int { InferTensorOrder::kNone = 0, InferTensorOrder::kLinear = 1, InferTensorOrder::kNHWC = 2, InferTensorOrder::kNone = 0, InferTensorOrder::kLinear = 1, InferTensorOrder::kNHWC = 2 }
	The type of tensor order. More...

enum	InferMemType : int { InferMemType::kNone = 0, InferMemType::kGpuCuda = 1, InferMemType::kCpu = 2, InferMemType::kCpuCuda = 3, InferMemType::kNvSurface = 5, InferMemType::kNvSurfaceArray = 6, InferMemType::kNone = 0, InferMemType::kGpuCuda = 1, InferMemType::kCpu = 2, InferMemType::kCpuCuda = 3, InferMemType::kNvSurface = 5, InferMemType::kNvSurfaceArray = 6 }
	The memory types of inference buffers. More...

enum	InferDataType : int { InferDataType::kFp32 = FLOAT, InferDataType::kFp16 = HALF, InferDataType::kInt8 = INT8, InferDataType::kInt32 = INT32, InferDataType::kInt16 = 7, InferDataType::kUint8, InferDataType::kUint16, InferDataType::kUint32, InferDataType::kFp64, InferDataType::kInt64, InferDataType::kUint64, InferDataType::kString, InferDataType::kBool, InferDataType::kNone = -1, InferDataType::kFp32 = FLOAT, InferDataType::kFp16 = HALF, InferDataType::kInt8 = INT8, InferDataType::kInt32 = INT32, InferDataType::kInt16 = 7, InferDataType::kUint8, InferDataType::kUint16, InferDataType::kUint32, InferDataType::kFp64, InferDataType::kInt64, InferDataType::kUint64, InferDataType::kString, InferDataType::kBool, InferDataType::kNone = -1 }
	Datatype of the tensor buffer. More...

enum	InferPostprocessType : int { InferPostprocessType::kDetector = 0, InferPostprocessType::kClassifier = 1, InferPostprocessType::kSegmentation = 2, InferPostprocessType::kTrtIsClassifier = 3, InferPostprocessType::kOther = 100, InferPostprocessType::kDetector = 0, InferPostprocessType::kClassifier = 1, InferPostprocessType::kSegmentation = 2, InferPostprocessType::kTrtIsClassifier = 3, InferPostprocessType::kOther = 100 }
	Inference post processing types. More...

enum	InferMediaFormat : int { InferMediaFormat::kRGB = 0, InferMediaFormat::kBGR, InferMediaFormat::kGRAY, InferMediaFormat::kRGBA, InferMediaFormat::kBGRx, InferMediaFormat::kUnknown = -1, InferMediaFormat::kRGB = 0, InferMediaFormat::kBGR, InferMediaFormat::kGRAY, InferMediaFormat::kRGBA, InferMediaFormat::kBGRx, InferMediaFormat::kUnknown = -1 }
	Image formats. More...

enum	OptionType : int { OptionType::oBool = 0, OptionType::oDouble, OptionType::oInt, OptionType::oUint, OptionType::oString, OptionType::oObject, OptionType::oArray, OptionType::oNone = -1, OptionType::oBool = 0, OptionType::oDouble, OptionType::oInt, OptionType::oUint, OptionType::oString, OptionType::oObject, OptionType::oArray, OptionType::oNone = -1 }

Functions
template<>
void	BufOptions::D::BasicValue::setV< std::string > (const std::string &v, OptionType t)

void	normalizeDims (InferDims &dims)

UniqCudaTensorBuf	createTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name, InferMemType mt, int devId, bool initCuEvent)
	Create a tensor buffer of the specified memory type, dimensions on the given device. More...

UniqCudaTensorBuf	createGpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false)
	Create a CUDA device memory tensor buffer of specified dimensions on the given device. More...

UniqCudaTensorBuf	createCpuTensorBuf (const InferDims &dims, InferDataType dt, int batchSize, const std::string &name="", int devId=0, bool initCuEvent=false)
	Create a CUDA pinned memory tensor buffer of specified dimensions on the given device. More...

NvDsInferStatus	syncAllCudaEvents (const SharedBatchArray &bufList)
	Synchronize on all events associated with the batch buffer array. More...

InferDataType	dataTypeFromDsProto (ic::TensorDataType dt)

InferTensorOrder	tensorOrderFromDsProto (ic::TensorOrder o)

InferMediaFormat	mediaFormatFromDsProto (ic::MediaFormat f)

InferMemType	memTypeFromDsProto (ic::MemoryType t)

NvBufSurfTransform_Compute	computeHWFromDsProto (ic::FrameScalingHW h)

NvBufSurfTransform_Inter	scalingFilterFromDsProto (uint32_t filter)

bool	validateProtoConfig (ic::InferenceConfig &c, const std::string &path)

bool	compareModelRepo (const ic::TritonModelRepo &repoA, const ic::TritonModelRepo &repoB)

bool	hasTriton (const ic::BackendParams &params)

const ic::TritonParams &	getTritonParam (const ic::BackendParams &params)

ic::TritonParams *	mutableTriton (ic::BackendParams &params)

SharedBufPool< UniqSurfaceBuf >	createSurfaceBufPool (int width, int height, InferMediaFormat color, int batchSize, int gpuId, int poolSize)

InferDataType	DataTypeFromTriton (TRITONSERVER_DataType type)
	Maps the TRITONSERVER_DataType to the InferDataType. More...

TRITONSERVER_DataType	DataTypeToTriton (InferDataType type)
	Maps the InferDataType to TRITONSERVER_DataType. More...

InferDataType	DataTypeFromTritonPb (ni::DataType type)
	Maps the data type from Triton model configuration proto definition to InferDataType. More...

InferTensorOrder	TensorOrderFromTritonPb (ni::ModelInput::Format order)
	Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type. More...

InferTensorOrder	TensorOrderFromTritonMeta (const std::string &format)
	Maps the tensor order from Triton metadata string to the InferTensorOrder type. More...

TRITONSERVER_MemoryType	MemTypeToTriton (InferMemType type)
	Maps the InferMemType to the TRITONSERVER_MemoryType. More...

InferMemType	MemTypeFromTriton (TRITONSERVER_MemoryType type)
	Maps the TRITONSERVER_MemoryType to the InferMemType. More...

template<typename VecDims >
InferDims	DimsFromTriton (const VecDims &shape)
	Converts the input shape vector from Triton to InferDims type. More...

const char *	TritonControlModeToStr (int32_t mode)
	Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll. More...

Typedef Documentation

◆ Headers

typedef std::map< std::string, std::string > nvdsinferserver::Headers

Definition at line 42 of file sources/libs/nvdsinferserver/infer_grpc_client.h.

◆ InferCompleted

typedef std::function< void(NvDsInferStatus, SharedBatchArray)> nvdsinferserver::InferCompleted

Definition at line 45 of file sources/libs/nvdsinferserver/infer_base_context.h.

◆ InferCustomProcessorPtr

typedef std::shared_ptr< IInferCustomProcessor > nvdsinferserver::InferCustomProcessorPtr

Definition at line 141 of file sources/libs/nvdsinferserver/infer_common.h.

◆ LayerDescriptionList

typedef std::vector< LayerDescription > nvdsinferserver::LayerDescriptionList

Definition at line 58 of file sources/libs/nvdsinferserver/infer_ibackend.h.

◆ SharedBatchArray

typedef std::shared_ptr< BaseBatchArray > nvdsinferserver::SharedBatchArray

Definition at line 75 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedBatchBuf

typedef std::shared_ptr< BaseBatchBuffer > nvdsinferserver::SharedBatchBuf

Common buffer interfaces (internal).

Definition at line 71 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedBatchSurface

typedef std::shared_ptr< BatchSurfaceBuffer > nvdsinferserver::SharedBatchSurface

Definition at line 198 of file sources/libs/nvdsinferserver/infer_surface_bufs.h.

◆ SharedBufOptions

typedef std::shared_ptr< BufOptions > nvdsinferserver::SharedBufOptions

Definition at line 41 of file sources/includes/nvdsinferserver/infer_options.h.

◆ SharedClassOutput

typedef std::shared_ptr< ClassificationOutput > nvdsinferserver::SharedClassOutput

Definition at line 132 of file sources/libs/nvdsinferserver/infer_postproc_buf.h.

◆ SharedCudaTensorBuf

typedef std::shared_ptr< CudaTensorBuf > nvdsinferserver::SharedCudaTensorBuf

Definition at line 91 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedCuEvent

typedef std::shared_ptr< CudaEvent > nvdsinferserver::SharedCuEvent

Definition at line 86 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedCuStream

typedef std::shared_ptr< CudaStream > nvdsinferserver::SharedCuStream

Cuda based pointers.

Definition at line 84 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedDllHandle

typedef std::shared_ptr< DlLibHandle > nvdsinferserver::SharedDllHandle

Definition at line 111 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedGrpcRequest

typedef std::shared_ptr< TritonGrpcRequest > nvdsinferserver::SharedGrpcRequest

Definition at line 46 of file sources/libs/nvdsinferserver/infer_grpc_client.h.

◆ SharedIBatchArray

typedef std::shared_ptr< IBatchArray > nvdsinferserver::SharedIBatchArray

Definition at line 205 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ SharedIBatchBuffer

typedef std::shared_ptr< IBatchBuffer > nvdsinferserver::SharedIBatchBuffer

Definition at line 204 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ SharedIOptions

typedef std::shared_ptr< IOptions > nvdsinferserver::SharedIOptions

Definition at line 206 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ SharedOptions

typedef std::shared_ptr< IOptions > nvdsinferserver::SharedOptions

Definition at line 73 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedRefBatchBuf

typedef std::shared_ptr< RefBatchBuffer > nvdsinferserver::SharedRefBatchBuf

Definition at line 77 of file sources/libs/nvdsinferserver/infer_common.h.

◆ SharedRequest

typedef std::shared_ptr< TrtServerRequest > nvdsinferserver::SharedRequest

Definition at line 59 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ SharedResponse

typedef std::shared_ptr< TrtServerResponse > nvdsinferserver::SharedResponse

Definition at line 61 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ SharedSysMem

typedef std::shared_ptr< SysMem > nvdsinferserver::SharedSysMem

Definition at line 88 of file sources/libs/nvdsinferserver/infer_common.h.

◆ ShrTritonAllocator

typedef std::shared_ptr< TrtServerAllocator > nvdsinferserver::ShrTritonAllocator

Definition at line 124 of file sources/libs/nvdsinferserver/infer_common.h.

◆ ShrTritonT

template<typename T >

using nvdsinferserver::ShrTritonT = typedef std::shared_ptr<T>

Definition at line 117 of file sources/libs/nvdsinferserver/infer_common.h.

◆ TensorMapPool

typedef MapBufferPool< std::string, UniqCudaTensorBuf > nvdsinferserver::TensorMapPool

Definition at line 31 of file sources/libs/nvdsinferserver/infer_extra_processor.h.

◆ TensorMapPoolPtr

typedef std::unique_ptr< TensorMapPool > nvdsinferserver::TensorMapPoolPtr

Definition at line 32 of file sources/libs/nvdsinferserver/infer_extra_processor.h.

◆ TritonGrpcAsyncDone

typedef std::function< void(NvDsInferStatus, SharedBatchArray)> nvdsinferserver::TritonGrpcAsyncDone

Definition at line 48 of file sources/libs/nvdsinferserver/infer_grpc_client.h.

◆ TritonInferAsyncDone

typedef std::function< void(SharedRequest, UniqResponse)> nvdsinferserver::TritonInferAsyncDone

Definition at line 63 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ TrtServerPtr

typedef std::shared_ptr< TrtISServer > nvdsinferserver::TrtServerPtr

Definition at line 121 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqBackend

typedef std::unique_ptr< BaseBackend > nvdsinferserver::UniqBackend

Definition at line 212 of file sources/libs/nvdsinferserver/infer_base_backend.h.

◆ UniqBatchArray

typedef std::unique_ptr< BaseBatchArray > nvdsinferserver::UniqBatchArray

Definition at line 76 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqBatchBuf

typedef std::unique_ptr< BaseBatchBuffer > nvdsinferserver::UniqBatchBuf

Definition at line 72 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqCudaTensorBuf

typedef std::unique_ptr< CudaTensorBuf > nvdsinferserver::UniqCudaTensorBuf

Definition at line 90 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqCuEvent

typedef std::unique_ptr< CudaEvent > nvdsinferserver::UniqCuEvent

Definition at line 87 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqCuStream

typedef std::unique_ptr< CudaStream > nvdsinferserver::UniqCuStream

Definition at line 85 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqInferExtraProcessor

typedef std::unique_ptr< InferExtraProcessor > nvdsinferserver::UniqInferExtraProcessor

Definition at line 139 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqLock

typedef std::unique_lock< std::mutex > nvdsinferserver::UniqLock

Miscellaneous declarations.

Definition at line 108 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqLstmController

typedef std::unique_ptr< LstmController > nvdsinferserver::UniqLstmController

Definition at line 128 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqPostprocessor

typedef std::unique_ptr< BasePostprocessor > nvdsinferserver::UniqPostprocessor

Processor interfaces.

Definition at line 98 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqPreprocessor

typedef std::unique_ptr< BasePreprocessor > nvdsinferserver::UniqPreprocessor

Definition at line 99 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqResponse

typedef std::unique_ptr< TrtServerResponse > nvdsinferserver::UniqResponse

Definition at line 60 of file sources/libs/nvdsinferserver/infer_trtis_server.h.

◆ UniqStreamManager

typedef std::unique_ptr< StreamManager > nvdsinferserver::UniqStreamManager

Definition at line 131 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqSurfaceBuf

typedef std::unique_ptr< SurfaceBuffer > nvdsinferserver::UniqSurfaceBuf

Definition at line 44 of file sources/libs/nvdsinferserver/infer_surface_bufs.h.

◆ UniqSysMem

typedef std::unique_ptr< SysMem > nvdsinferserver::UniqSysMem

Definition at line 89 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqTritonAllocator

typedef std::unique_ptr< TrtServerAllocator > nvdsinferserver::UniqTritonAllocator

Definition at line 123 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqTritonT

template<typename T >

using nvdsinferserver::UniqTritonT = typedef std::unique_ptr<T, std::function<void(T*)> >

Definition at line 114 of file sources/libs/nvdsinferserver/infer_common.h.

◆ UniqTrtISBackend

typedef std::unique_ptr< TrtISBackend > nvdsinferserver::UniqTrtISBackend

Definition at line 101 of file sources/libs/nvdsinferserver/infer_common.h.

◆ WeakTritonAllocator

typedef std::weak_ptr< TrtServerAllocator > nvdsinferserver::WeakTritonAllocator

Definition at line 125 of file sources/libs/nvdsinferserver/infer_common.h.

Enumeration Type Documentation

◆ InferDataType [1/2]

enum nvdsinferserver::InferDataType : int

strong

Datatype of the tensor buffer.

Enumerator
kFp32
kFp16
kInt8
kInt32
kInt16
kUint8
kUint16
kUint32
kFp64
kInt64
kUint64
kString
kBool
kNone
kFp32
kFp16
kInt8
kInt32
kInt16
kUint8
kUint16
kUint32
kFp64
kInt64
kUint64
kString
kBool
kNone

Definition at line 83 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferDataType [2/2]

enum nvdsinferserver::InferDataType : int

strong

Datatype of the tensor buffer.

Enumerator
kFp32
kFp16
kInt8
kInt32
kInt16
kUint8
kUint16
kUint32
kFp64
kInt64
kUint64
kString
kBool
kNone
kFp32
kFp16
kInt8
kInt32
kInt16
kUint8
kUint16
kUint32
kFp64
kInt64
kUint64
kString
kBool
kNone

Definition at line 83 of file 9.0/sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferMediaFormat [1/2]

enum nvdsinferserver::InferMediaFormat : int

strong

Image formats.

Enumerator
kRGB	24-bit interleaved R-G-B
kBGR	24-bit interleaved B-G-R
kGRAY	8-bit Luma
kRGBA	32-bit interleaved R-G-B-A
kBGRx	32-bit interleaved B-G-R-x
kUnknown
kRGB	24-bit interleaved R-G-B
kBGR	24-bit interleaved B-G-R
kGRAY	8-bit Luma
kRGBA	32-bit interleaved R-G-B-A
kBGRx	32-bit interleaved B-G-R-x
kUnknown

Definition at line 129 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferMediaFormat [2/2]

enum nvdsinferserver::InferMediaFormat : int

strong

Image formats.

Enumerator
kRGB	24-bit interleaved R-G-B
kBGR	24-bit interleaved B-G-R
kGRAY	8-bit Luma
kRGBA	32-bit interleaved R-G-B-A
kBGRx	32-bit interleaved B-G-R-x
kUnknown
kRGB	24-bit interleaved R-G-B
kBGR	24-bit interleaved B-G-R
kGRAY	8-bit Luma
kRGBA	32-bit interleaved R-G-B-A
kBGRx	32-bit interleaved B-G-R-x
kUnknown

Definition at line 129 of file 9.0/sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferMemType [1/2]

enum nvdsinferserver::InferMemType : int

strong

The memory types of inference buffers.

Enumerator
kNone
kGpuCuda	GPU CUDA memory.
kCpu	Host (CPU) memory.
kCpuCuda	CUDA pinned memory.
kNvSurface	NVRM surface memory.
kNvSurfaceArray	NVRM surface array memory.
kNone
kGpuCuda	GPU CUDA memory.
kCpu	Host (CPU) memory.
kCpuCuda	CUDA pinned memory.
kNvSurface	NVRM surface memory.
kNvSurfaceArray	NVRM surface array memory.

Definition at line 56 of file 9.0/sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferMemType [2/2]

enum nvdsinferserver::InferMemType : int

strong

The memory types of inference buffers.

Enumerator
kNone
kGpuCuda	GPU CUDA memory.
kCpu	Host (CPU) memory.
kCpuCuda	CUDA pinned memory.
kNvSurface	NVRM surface memory.
kNvSurfaceArray	NVRM surface array memory.
kNone
kGpuCuda	GPU CUDA memory.
kCpu	Host (CPU) memory.
kCpuCuda	CUDA pinned memory.
kNvSurface	NVRM surface memory.
kNvSurfaceArray	NVRM surface array memory.

Definition at line 56 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferPostprocessType [1/2]

enum nvdsinferserver::InferPostprocessType : int

strong

Inference post processing types.

Enumerator
kDetector	Post processing for object detection.
kClassifier	Post processing for object classification.
kSegmentation	Post processing for image segmentation.
kTrtIsClassifier	Post processing using Triton Classifier.
kOther	Custom post processing.
kDetector	Post processing for object detection.
kClassifier	Post processing for object classification.
kSegmentation	Post processing for image segmentation.
kTrtIsClassifier	Post processing using Triton Classifier.
kOther	Custom post processing.

Definition at line 103 of file 9.0/sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferPostprocessType [2/2]

enum nvdsinferserver::InferPostprocessType : int

strong

Inference post processing types.

Enumerator
kDetector	Post processing for object detection.
kClassifier	Post processing for object classification.
kSegmentation	Post processing for image segmentation.
kTrtIsClassifier	Post processing using Triton Classifier.
kOther	Custom post processing.
kDetector	Post processing for object detection.
kClassifier	Post processing for object classification.
kSegmentation	Post processing for image segmentation.
kTrtIsClassifier	Post processing using Triton Classifier.
kOther	Custom post processing.

Definition at line 103 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferTensorOrder [1/2]

enum nvdsinferserver::InferTensorOrder : int

strong

The type of tensor order.

Enumerator
kNone
kLinear	NCHW (batch-channels-height-width) tensor order.
kNHWC	NHWC (batch-height-width-channels) tensor order.
kNone
kLinear	NCHW (batch-channels-height-width) tensor order.
kNHWC	NHWC (batch-height-width-channels) tensor order.

Definition at line 41 of file sources/includes/nvdsinferserver/infer_datatypes.h.

◆ InferTensorOrder [2/2]

enum nvdsinferserver::InferTensorOrder : int

strong

The type of tensor order.

Enumerator
kNone
kLinear	NCHW (batch-channels-height-width) tensor order.
kNHWC	NHWC (batch-height-width-channels) tensor order.
kNone
kLinear	NCHW (batch-channels-height-width) tensor order.
kNHWC	NHWC (batch-height-width-channels) tensor order.

Definition at line 41 of file 9.0/sources/includes/nvdsinferserver/infer_datatypes.h.

◆ OptionType [1/2]

enum nvdsinferserver::OptionType : int

strong

Enumerator
oBool
oDouble
oInt
oUint
oString
oObject
oArray
oNone
oBool
oDouble
oInt
oUint
oString
oObject
oArray
oNone

Definition at line 28 of file sources/includes/nvdsinferserver/infer_ioptions.h.

◆ OptionType [2/2]

enum nvdsinferserver::OptionType : int

strong

Enumerator
oBool
oDouble
oInt
oUint
oString
oObject
oArray
oNone
oBool
oDouble
oInt
oUint
oString
oObject
oArray
oNone

Definition at line 28 of file 9.0/sources/includes/nvdsinferserver/infer_ioptions.h.

Function Documentation

◆ BufOptions::D::BasicValue::setV< std::string >()

template<>

void nvdsinferserver::BufOptions::D::BasicValue::setV< std::string >	(	const std::string &	v,
		OptionType	t
	)

inline

Definition at line 183 of file sources/includes/nvdsinferserver/infer_options.h.

References oString.

◆ compareModelRepo()

bool nvdsinferserver::compareModelRepo	(	const ic::TritonModelRepo &	repoA,
		const ic::TritonModelRepo &	repoB
	)

◆ computeHWFromDsProto()

NvBufSurfTransform_Compute nvdsinferserver::computeHWFromDsProto ( ic::FrameScalingHW h )

◆ createCpuTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createCpuTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name = `""`,
		int	devId = `0`,
		bool	initCuEvent = `false`
	)

Create a CUDA pinned memory tensor buffer of specified dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ createGpuTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createGpuTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name = `""`,
		int	devId = `0`,
		bool	initCuEvent = `false`
	)

Create a CUDA device memory tensor buffer of specified dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ createSurfaceBufPool()

SharedBufPool< UniqSurfaceBuf > nvdsinferserver::createSurfaceBufPool	(	int	width,
		int	height,
		InferMediaFormat	color,
		int	batchSize,
		int	gpuId,
		int	poolSize
	)

◆ createTensorBuf()

UniqCudaTensorBuf nvdsinferserver::createTensorBuf	(	const InferDims &	dims,
		InferDataType	dt,
		int	batchSize,
		const std::string &	name,
		InferMemType	mt,
		int	devId,
		bool	initCuEvent
	)

Create a tensor buffer of the specified memory type, dimensions on the given device.

Parameters

[in]	dims	Dimensions of the tensor.
[in]	dt	Datatype.
[in]	batchSize	Batch size.
[in]	name	Name of the buffer.
[in]	mt	Memory type.
[in]	devId	Device ID for the memory allocation.
[in]	initCuEvent	Flag to create an associated CUDA event.

Returns: Pointer to the newly created tensor buffer, null if the buffer could not be created.

◆ dataTypeFromDsProto()

InferDataType nvdsinferserver::dataTypeFromDsProto ( ic::TensorDataType dt )

◆ DataTypeFromTriton()

InferDataType nvdsinferserver::DataTypeFromTriton ( TRITONSERVER_DataType type )

Maps the TRITONSERVER_DataType to the InferDataType.

Parameters

type	TRITONSERVER_DataType

Returns: InferDataType corresponding to the input type.

◆ DataTypeFromTritonPb()

InferDataType nvdsinferserver::DataTypeFromTritonPb ( ni::DataType type )

Maps the data type from Triton model configuration proto definition to InferDataType.

Parameters

type	The protobuf datatype.

Returns: InferDataType corresponding to the protobuf type.

◆ DataTypeToTriton()

TRITONSERVER_DataType nvdsinferserver::DataTypeToTriton ( InferDataType type )

Maps the InferDataType to TRITONSERVER_DataType.

Parameters

type	InferDataType

Returns: TRITONSERVER_DataType corresponding to the input type.

◆ DimsFromTriton()

template<typename VecDims >

InferDims nvdsinferserver::DimsFromTriton ( const VecDims & shape )

Converts the input shape vector from Triton to InferDims type.

This functions provides a InferDims structure created based on the input dimensions vector. Any negative dimensions are mapped to -1. The numElements value is updated if there is no dynamic size dimension.

Template Parameters

VecDims Input dimensions vector type.

Parameters

shape Input dimensions vector.

Returns: The InferDims structure object created based on the input.

Definition at line 131 of file sources/libs/nvdsinferserver/infer_trtis_utils.h.

References nvdsinfer::hasWildcard(), normalizeDims(), and NVDSINFER_MAX_DIMS.

◆ getTritonParam()

const ic::TritonParams & nvdsinferserver::getTritonParam ( const ic::BackendParams & params )

inline

Definition at line 61 of file sources/libs/nvdsinferserver/infer_proto_utils.h.

◆ hasTriton()

bool nvdsinferserver::hasTriton ( const ic::BackendParams & params )

inline

Definition at line 56 of file sources/libs/nvdsinferserver/infer_proto_utils.h.

◆ mediaFormatFromDsProto()

InferMediaFormat nvdsinferserver::mediaFormatFromDsProto ( ic::MediaFormat f )

◆ memTypeFromDsProto()

InferMemType nvdsinferserver::memTypeFromDsProto ( ic::MemoryType t )

◆ MemTypeFromTriton()

InferMemType nvdsinferserver::MemTypeFromTriton ( TRITONSERVER_MemoryType type )

Maps the TRITONSERVER_MemoryType to the InferMemType.

◆ MemTypeToTriton()

TRITONSERVER_MemoryType nvdsinferserver::MemTypeToTriton ( InferMemType type )

Maps the InferMemType to the TRITONSERVER_MemoryType.

◆ mutableTriton()

ic::TritonParams * nvdsinferserver::mutableTriton ( ic::BackendParams & params )

inline

Definition at line 71 of file sources/libs/nvdsinferserver/infer_proto_utils.h.

◆ normalizeDims()

void nvdsinferserver::normalizeDims ( InferDims & dims )

Referenced by DimsFromTriton(), and nvdsinferserver::RefBatchBuffer::RefBatchBuffer().

◆ scalingFilterFromDsProto()

NvBufSurfTransform_Inter nvdsinferserver::scalingFilterFromDsProto ( uint32_t filter )

◆ syncAllCudaEvents()

NvDsInferStatus nvdsinferserver::syncAllCudaEvents ( const SharedBatchArray & bufList )

Synchronize on all events associated with the batch buffer array.

Parameters

[in] bufList Array of the batch buffers.

Returns: NVDSINFER_SUCCESS or NVDSINFER_CUDA_ERROR.

◆ tensorOrderFromDsProto()

InferTensorOrder nvdsinferserver::tensorOrderFromDsProto ( ic::TensorOrder o )

◆ TensorOrderFromTritonMeta()

InferTensorOrder nvdsinferserver::TensorOrderFromTritonMeta ( const std::string & format )

Maps the tensor order from Triton metadata string to the InferTensorOrder type.

Parameters

[in] format The Triton metadata tensor order string.

Returns: InferTensorOrder type corresponding to the protobuf order.

◆ TensorOrderFromTritonPb()

InferTensorOrder nvdsinferserver::TensorOrderFromTritonPb ( ni::ModelInput::Format order )

Maps the tensor order from Triton model configuration proto definition to the InferTensorOrder type.

Parameters

[in] order The protobuf tensor order type.

Returns: InferTensorOrder type corresponding to the protobuf order.

◆ TritonControlModeToStr()

const char * nvdsinferserver::TritonControlModeToStr ( int32_t mode )

Returns a string describing the TRITONSERVER_ModelControlMode: none, explicit or poll.

◆ validateProtoConfig()

bool nvdsinferserver::validateProtoConfig	(	ic::InferenceConfig &	c,
		const std::string &	path
	)


NVIDIA DeepStream SDK API Reference	8.0 Release

NVIDIA DeepStream SDK API Reference

8.0 Release

Detailed Description

Namespaces

Data Structures

Typedefs

Enumerations

Functions

Typedef Documentation

◆ Headers

◆ InferCompleted

◆ InferCustomProcessorPtr

◆ LayerDescriptionList

◆ SharedBatchArray

◆ SharedBatchBuf

◆ SharedBatchSurface

◆ SharedBufOptions

◆ SharedClassOutput

◆ SharedCudaTensorBuf

◆ SharedCuEvent

◆ SharedCuStream

◆ SharedDllHandle

◆ SharedGrpcRequest

◆ SharedIBatchArray

◆ SharedIBatchBuffer

◆ SharedIOptions

◆ SharedOptions

◆ SharedRefBatchBuf

◆ SharedRequest

◆ SharedResponse

◆ SharedSysMem

◆ ShrTritonAllocator

◆ ShrTritonT

◆ TensorMapPool

◆ TensorMapPoolPtr

◆ TritonGrpcAsyncDone

◆ TritonInferAsyncDone

◆ TrtServerPtr

◆ UniqBackend

◆ UniqBatchArray

◆ UniqBatchBuf

◆ UniqCudaTensorBuf

◆ UniqCuEvent

◆ UniqCuStream

◆ UniqInferExtraProcessor

◆ UniqLock

◆ UniqLstmController

◆ UniqPostprocessor

◆ UniqPreprocessor

◆ UniqResponse

◆ UniqStreamManager

◆ UniqSurfaceBuf

◆ UniqSysMem

◆ UniqTritonAllocator

◆ UniqTritonT

◆ UniqTrtISBackend

◆ WeakTritonAllocator

Enumeration Type Documentation

◆ InferDataType [1/2]

◆ InferDataType [2/2]

◆ InferMediaFormat [1/2]

◆ InferMediaFormat [2/2]

◆ InferMemType [1/2]

◆ InferMemType [2/2]

◆ InferPostprocessType [1/2]

◆ InferPostprocessType [2/2]

◆ InferTensorOrder [1/2]

◆ InferTensorOrder [2/2]

◆ OptionType [1/2]

◆ OptionType [2/2]

Function Documentation

◆ BufOptions::D::BasicValue::setV< std::string >()

◆ compareModelRepo()

◆ computeHWFromDsProto()

◆ createCpuTensorBuf()

◆ createGpuTensorBuf()

◆ createSurfaceBufPool()

◆ createTensorBuf()

◆ dataTypeFromDsProto()

◆ DataTypeFromTriton()