### Install and Configure Kafka

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Installs Java, downloads Kafka, formats the Kafka cluster, and starts the Kafka server. Ensure Java 17+ is installed.

```bash
# Kafka requires Java 17+. Check your Java version.
# If you see "Command 'java' not found" or it is older than 17, please install openjdk-17-jdk.
java -version
sudo apt install openjdk-17-jdk 

# Get Kafka
wget https://dlcdn.apache.org/kafka/4.2.0/kafka_2.13-4.2.0.tgz
tar -xzf kafka_2.13-4.2.0.tgz
cd kafka_2.13-4.2.0

# Start the Kafka environment
export KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties
# [Expected output] You should see `Formatting dynamic metadata voter directory /tmp/kraft-combined-logs with metadata.version 4.0-IV3.`

bin/kafka-server-start.sh config/server.properties
# [Expected output] You should see `Kafka Server started.` and it will keep logging `INFO` messages.
```

--------------------------------

### Start Kafka Broker

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Starts the Kafka broker using the configuration file. This command is used for subsequent starts after the initial setup.

```bash
bin/kafka-server-start.sh config/server.properties
# [Expected output] It is expected to see DUPLICATE_BROKER_REGISTRATION in the logs. As long as the broker keeps running and logging INFO messages, you can proceed.
```

--------------------------------

### Setup and Build 3D Body Pose Estimation App

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Installs prerequisites, builds the custom parser and application, and downloads pretrained models for 3D body pose estimation.

```bash
sudo apt install libeigen3-dev
cd /usr/include && sudo ln -sf eigen3/Eigen Eigen

export BODYPOSE3D_HOME=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/3d-bodypose-deepstream

cd $BODYPOSE3D_HOME/sources/nvdsinfer_custom_impl_BodyPose3DNet && make
cd $BODYPOSE3D_HOME/sources && make
# Produces: deepstream-pose-estimation-app
#           nvdsinfer_custom_impl_BodyPose3DNet/libnvdsinfer_custom_impl_BodyPose3DNet.so

cd $BODYPOSE3D_HOME && bash ./download_models.sh
```

--------------------------------

### Setup Inference Builder Virtual Environment

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Installs protobuf-compiler, deactivates the existing 'mv3dt_venv', creates a new virtual environment 'ib_venv', and installs prerequisites for the Inference Builder.

```bash
# Install required deb packages
sudo apt install protobuf-compiler

# Deactivate the mv3dt_venv, and create a new virtual environment named ib_venv for inference builder
deactivate
python -m venv ib_venv
source ib_venv/bin/activate
pip3 install -r requirements.txt
```

--------------------------------

### Run Prerequisites Setup Script

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

Execute the setup script to install all necessary prerequisites for the DeepStream Container. This script may require sudo access and GitHub credentials. It can be re-run to check or complete the setup.

```bash
./scripts/setup_prerequisites.sh
```

--------------------------------

### Setup Python Virtual Environment and Dependencies

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Creates a Python virtual environment named 'mv3dt_venv', installs necessary deb packages, and then installs Python dependencies from 'requirements.txt'. Activate the environment before proceeding.

```bash
cd <path to current repo, i.e. deepstream-tracker-3d-multi-view>

# Install required deb packages
sudo apt update
sudo apt install python3-tk python3.12-venv python3.12-dev

# Create a python virtual enviornment named `mv3dt_venv` and install required python packages
python3 -m venv mv3dt_venv
source mv3dt_venv/bin/activate

pip install -r requirements.txt
```

--------------------------------

### Setup DeepStream Multi-View 3D Tracking

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Installs necessary dependencies like Git LFS and Docker images, clones the repository, and pulls LFS files. Use `USE_INFERENCE_BUILDER=true` for Inference Builder support.

```bash
sudo apt install git-lfs && git lfs install
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream-tracker-3d-multi-view
git lfs pull
```

```bash
# DeepStream Container only
./scripts/setup_prerequisites.sh
```

```bash
# With Inference Builder support
USE_INFERENCE_BUILDER=true ./scripts/setup_prerequisites.sh
```

--------------------------------

### Pull DeepStream Container and Clone Repos

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Initial setup involves pulling the DeepStream container and cloning necessary repositories. Ensure Docker is installed and configured.

```bash
export DS_IMG_NAME="nvcr.io/nvidia/deepstream:9.0-triton-multiarch"
docker pull $DS_IMG_NAME

git clone https://github.com/NVIDIA-AI-IOT/deepstream_tools.git
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream-masktracker
```

--------------------------------

### Install Mosquitto MQTT Broker

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Install the Mosquitto MQTT broker and client tools. This is a prerequisite for applications that use MQTT for communication.

```bash
sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa
sudo apt update
sudo apt install mosquitto mosquitto-clients
```

--------------------------------

### DS3D Sensor Fusion: Setup and Run BEVFusion

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Prepare custom NuScenes data for the demo scene and set up the BEVFusion pipeline. This involves running a Python helper script and ensuring DeepStream SDK 7.0+, BEVFusion TensorRT engine, and PyTriton are installed.

```bash
# Step 3: Setup and run BEVFusion pipeline
# Follow DeepStream SDK sample (source installed with SDK):
cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/

# Collect custom NuScenes data for the demo scene
python3 python/triton_lmm/helper/nuscene_data_setup.py \
  --data_dir=dataset/nuscene \
  --ds3d_fusion_workspace=$(pwd) \
  --print_calibration

# Full setup documentation:
# https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_3D_MultiModal_Lidar_Sensor_Fusion.html
# https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_3D_MultiModal_Lidar_Camera_BEVFusion.html

# Requirements: DeepStream SDK 7.0+, BEVFusion TensorRT engine, PyTriton
```

--------------------------------

### Install DeepStream Container and Dependencies

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-vllm-plugin/README.md

Launches the DeepStream container and installs the plugin's dependencies within it. Ensure you have Docker and NVIDIA Container Runtime installed.

```bash
# Launch DeepStream container
sudo docker run -it --rm --runtime=nvidia --gpus all --network=host \
  -v $(pwd):/home/vllm_ds_plugin \
  nvcr.io/nvidia/deepstream:9.0-triton-multiarch

# Inside DeepStream container install dependencies
cd /home/vllm_ds_plugin/deepstream-vllm-plugin
./install.sh
```

--------------------------------

### Install torchvision

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/pyservicemaker_sample_apps/README.md

Install the torchvision library using pip3.

```bash
pip3 install torchvision
```

--------------------------------

### VLM Configuration Example (config.yaml)

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

A complete configuration example for a VLM pipeline, detailing model parameters, segmentation settings, inference prompts, and pipeline queueing.

```yaml
# model:
#   path: "nvidia/Cosmos-Reason2-8B"
#   max_model_len: 20480
#   gpu_memory_utilization: 0.7
#   trust_remote_code: true
#   gpu_id: 0
#   video_mode: 1          # 1=native video, 0=multi-image
#   tensor_format: "pytorch"
#
# segment:
#   length_sec: 30
#   overlap_sec: 0
#   subsample_interval: 1
#   selection_fps: 30
#
# inference:
#   user_prompt: "These are {num_frames} images from stream {stream_id} at {timestamps}. Describe the scene."
#   system_prompt: "Provide captions with timestamps: <start> <end> caption."
#   max_tokens: 2048
#   temperature: 0.7
#   stream_prompts:        # per-stream overrides
#     0:
#       user_prompt: "Stream {stream_id} at {timestamps}: Monitor for security threats."
#       system_prompt: "You are a security analyst."
#     1:
#       user_prompt: "Analyze traffic flow."
#
# pipeline:
#   queue_maxsize: 20
#   max_wait_timeout: 300
```

--------------------------------

### Install pyyaml

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/pyservicemaker_sample_apps/README.md

Install the pyyaml library using pip3.

```bash
pip3 install pyyaml
```

--------------------------------

### Clone and Setup Inference Builder

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Clones the Inference Builder repository and initializes its submodules. This is an optional step for Option 2: Inference Builder.

```bash
git clone https://github.com/NVIDIA-AI-IOT/inference_builder.git
cd inference_builder
git submodule update --init --recursive
```

--------------------------------

### Build and Install dsdirection Plugin

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Builds and installs the custom GStreamer dsdirection plugin. Ensure you are in the plugin's directory and have the necessary build tools.

```bash
cd /path/to/deepstream_reference_apps/anomaly/plugins/gst-dsdirection/
sudo make && sudo make install
```

--------------------------------

### VLLM DeepStream Plugin Configuration Example

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-vllm-plugin/README.md

A complete configuration file example for the nvvllmvlm plugin. Adjust parameters like model path, memory utilization, and processing modes based on your environment and requirements.

```yaml
# Model Configuration
model:
  path: "nvidia/Cosmos-Reason2-8B"
  max_model_len: 20480          # Max context length
  gpu_memory_utilization: 0.7   # GPU memory fraction to use. Update depending on platform and availabe gpu memory.
  trust_remote_code: true
  gpu_id: 0                     # GPU device ID (-1 for auto)

  # Video processing mode
  video_mode: 1  # 1=video metadata (native video support), 0=multi-image mode

  # Tensor format for image inputs
  tensor_format: "pytorch"  # pytorch/pil/numpy

# Segment Processing
segment:
  length_sec: 30                # Segment length in seconds
  overlap_sec: 0                # Segment overlap in seconds
  subsample_interval: 1         # Keep every Nth frame
  selection_fps: 30             # Target FPS (0 to disable FPS-based sampling)
```

--------------------------------

### Save Output Video and Pose Keypoints

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-bodypose-3d/README.md

This example processes an input video, saves the output video with skeleton overlay, and also saves the skeleton's keypoints to a JSON file. It includes options for focal length, width, height, and FPS display.

```bash
./deepstream-pose-estimation-app --input file://$BODYPOSE3D_HOME/streams/bodypose.mp4 --output $BODYPOSE3D_HOME/streams/bodypose_3dbp.mp4 --focal 800.0 --width 1280 --height 720 --fps --save-pose $BODYPOSE3D_HOME/streams/bodypose_3dbp.json
```

--------------------------------

### Example: Server with Local File, Multiple Clients

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Demonstrates the server generating a URL from a local MP4 file and multiple clients connecting to play the stream. The domain socket path is used for IPC.

```bash
    $ ./deepstream-ipc-test-app server file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 /tmp/test1
    $ ./deepstream-ipc-test-app client /tmp/test1
    $ ./deepstream-ipc-test-app client /tmp/test1
```

--------------------------------

### Metamux Group Configuration Example

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream_parallel_inference_app/README.md

Specifies the configuration file for the gst-dsmetamux plugin. Ensure the 'enable' property is set to 1 to activate the plugin.

```yaml
meta-mux:
  key1: value1
  key2: value2
```

--------------------------------

### Install Eigen Development Packages

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-bodypose-3d/README.md

Installs Eigen development packages and creates a symbolic link for DeepStream SDK versions older than 6.2.

```bash
sudo apt install libeigen3-dev
cd /usr/include
sudo ln -sf eigen3/Eigen Eigen
```

--------------------------------

### Example: Server with RTSP URL, Client

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Shows the server generating a URL from an RTSP stream and a client connecting to play it. The domain socket path facilitates IPC communication.

```bash
    $ ./deepstream-ipc-test-app server rtsp://127.0.0.1/video1 /tmp/test1
    $ ./deepstream-ipc-test-app client /tmp/test1
```

--------------------------------

### Install Dependencies and Convert SAM2 Model Inside Container

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Installs additional dependencies and converts the SAM2 model to a TensorRT engine within the running Docker container. This is a prerequisite for running the tracker.

```bash
cd /opt/nvidia/deepstream/deepstream/ 
bash user_additional_install.sh

export TRACKER_MODEL_DIR="/opt/nvidia/deepstream/deepstream/samples/models/Tracker"
mkdir -p $TRACKER_MODEL_DIR
cd /opt/nvidia/deepstream/deepstream/sources/tracker_ReID/sam2-onnx-tensorrt
bash run.sh  # downloads + converts SAM2 to TensorRT engine
```

--------------------------------

### Run 4-Camera Inference Builder Pipeline

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

Quickly starts the 4-camera DeepStream Inference Builder pipeline. Ensure INFERENCE_BUILDER_DIR is exported if BASE_DIR was changed.

```bash
# sudo xhost + # give container access to display

# If you changed BASE_DIR in the prerequisites setup, you need to export INFERENCE_BUILDER_DIR to the correct path
# export INFERENCE_BUILDER_DIR=<path-to-inference-builder-repo>

# chmod +x scripts/test_4cam_ib.sh
./scripts/test_4cam_ib.sh
```

--------------------------------

### Example: Server with Multiple URLs, Client with Multiple Sockets

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Illustrates the server generating two URLs from different RTSP streams and a client connecting to both using separate domain socket paths. This allows the client to process multiple streams concurrently.

```bash
    $ ./deepstream-ipc-test-app server rtsp://127.0.0.1/video1 /tmp/test1 rtsp://127.0.0.1/video2 /tmp/test2
    $ ./deepstream-ipc-test-app client /tmp/test1 /tmp/test2
```

--------------------------------

### Compile dsdirection Plugin

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/anomaly/README.md

Compile and install the dsdirection plugin. Ensure you are in the correct directory before running.

```bash
cd plugins/gst-dsdirection/
sudo make && sudo make install
```

--------------------------------

### Find nuscene_data_setup.py Script

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-3d-sensor-fusion/README.md

Use the find command to locate the nuscene_data_setup.py script within the DeepStream SDK installation directory. This is useful if the script is not in the expected location.

```bash
find /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/python -name nuscene_data_setup.py
```

--------------------------------

### Check Python Virtual Environment

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Verifies the existence of the 'mv3dt_venv' virtual environment and lists installed Python packages, ensuring 'kafka-python' and 'protobuf' are present.

```bash
ls -d mv3dt_venv
# [Expected output] You should see "mv3dt_venv" printed. If you see "No such file or directory", please check the previous step "python3 -m venv mv3dt_venv".

pip list
# [Expected output] You should see kafka-python, protobuf in the list
```

--------------------------------

### Run 12-Camera DeepStream Pipeline

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

Launches the 12-camera DeepStream pipeline. For detailed window explanations and notes, refer to the 4-camera quick start section.

```bash
# sudo xhost + # give container access to display

# chmod +x scripts/test_12cam_ds.sh
./scripts/test_12cam_ds.sh
```

--------------------------------

### Camera Calibration File Format (camInfo.yml)

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Example structure of the `camInfo.yml` file, which contains camera calibration parameters including the projection matrix and model information for 3D tracking.

```yaml
# projectionMatrix_3x4:  # row-major 3x4 world-to-pixel matrix
#   - 2582.5691623002185
#   - -485.10283397043617
#   - 650.27745033162591
#   - -89466.605755471101
#   - ...
# modelInfo:
#   height: 205   # cylinder height in mm
```

--------------------------------

### User Prompt with Timestamps

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-vllm-plugin/README.md

Example of a user prompt that includes placeholders for the number of frames, stream ID, and timestamps. This configuration ensures that timestamp information is passed to the model.

```yaml
inference:
  user_prompt: "These are {num_frames} images from stream {stream_id} sampled at timestamps {timestamps}. Describe the scene."

```

--------------------------------

### Launch BEV Visualizer for MV3DT

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Launches the Bird-Eye View visualizer before starting the MV3DT application. Requires activating the `mv3dt_venv` environment and setting dataset and experiment directories.

```bash
source mv3dt_venv/bin/activate
python utils/kafka_bev_visualizer.py \
  --dataset-path=$DATASET_DIR \
  --msgconv-config=$EXPERIMENT_DIR/config_msgconv.txt \
  --show-ids \
  --average-multi-cam
```

--------------------------------

### Generate Inference Builder Configuration Files

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-inference-builder.md

Use the auto-configurator script to generate configuration files for inference and tracking, including overrides for multi-camera setups. Ensure the Python environment is activated before running.

```bash
# Activate the Python environment
source mv3dt_venv/bin/activate

# Generate configs with 4-camera overrides
python utils/inference_builder_auto_configurator.py \
    --dataset-dir=$DATASET_DIR \
    --config-overrides=override_tracker_4cam.yml \
    --output-dir=$EXPERIMENT_DIR
# [Expected output] You should see 
#   Generated files:
#     - ds_mv3dt.yaml (inference config with max_batch_size: 4)
#     - config_tracker.yml (3D tracker config)
#     - source_list_static.yaml (source configuration)
#     - nvdsinfer_config.yaml (inference engine config with batch_size: 4)
#     - config_msgconv.txt (message converter config)
#     - pub_sub_info_config_0.yml (communication config)

# Copy the generated nvdsinfer config to the model directory
cp $EXPERIMENT_DIR/nvdsinfer_config.yaml $MODEL_REPO/PeopleNetTransformer/
```

--------------------------------

### Generate Super Resolution Model

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Install necessary Python packages, clone the PyTorch tutorials repository, reset to a specific commit, update the opset_version, and run the script to generate the ONNX model. Copy the generated model files to the application directory.

```bash
  $ pip install onnx onnxruntime torch torchvision onnxscript
  $ git clone --shallow-since=2025-07-1  https://github.com/pytorch/tutorials.git
  $ cd tutorials && git reset --hard `git rev-list --max-parents=0 HEAD`
  #update opset_version to 18 in advanced_source/super_resolution_with_onnxruntime.py
  $ python3 advanced_source/super_resolution_with_onnxruntime.py
  # copy the generated super_resolution.onnx and super_resolution.onnx.data to deepstream-ipc-test-sr.
  $ cp advanced_source/super_resolution.onnx* /path/to/your/deepstream-ipc-test-sr
```

--------------------------------

### Clone DeepStream Container and Repos for 3D Tracking

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Initial setup for the 3D tracking application, including pulling the DeepStream container and cloning the reference applications repository.

```bash
export DS_IMG_NAME="nvcr.io/nvidia/deepstream:9.0-triton-multiarch"
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream-tracker-3d
```

--------------------------------

### Run DeepStream IPC Test SR Application (Server Mode)

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Start the application in server mode, providing a URL for the video stream and a domain socket path for IPC. The server decodes the stream and listens for client connections.

```bash
    # server
    $ ./deepstream-ipc-test-app server <url> <domain_socket_path>
```

--------------------------------

### Launch Real-time BEV Visualization

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-deepstream.md

Optionally, start the Bird's-Eye View (BEV) visualizer to see real-time 3D tracking results. This script requires the dataset path and message converter configuration. It can be run in a separate terminal or in the background.

```bash
python utils/kafka_bev_visualizer.py \
    --dataset-path=$DATASET_DIR \
    --msgconv-config=$EXPERIMENT_DIR/config_msgconv.txt \
    --average-multi-cam \
    --show-ids
```

--------------------------------

### Set Up Environment and Directories

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-inference-builder.md

Prepare the environment by setting necessary variables and creating directories for datasets, experiments, and models.

```bash
export DATASET_DIR=$PWD/datasets/mtmc_4cam/ 
export EXPERIMENT_DIR=$PWD/experiments/inference_builder/4cam
export MODEL_REPO=$PWD/models

mkdir -p $EXPERIMENT_DIR/infer-kitti-dump
mkdir -p $EXPERIMENT_DIR/tracker-kitti-dump
```

--------------------------------

### Set Up Environment and Directories for 4-Camera Dataset

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-deepstream.md

Prepare the necessary directories and set environment variables for the 4-camera dataset experiment. This includes defining paths for datasets, experiments, and models, and creating subdirectories for inference and tracking output.

```bash
export DATASET_DIR=$PWD/datasets/mtmc_4cam/
export EXPERIMENT_DIR=$PWD/experiments/deepstream/4cam
export MODEL_REPO=$PWD/models

mkdir -p $EXPERIMENT_DIR/infer-kitti-dump
mkdir -p $EXPERIMENT_DIR/tracker-kitti-dump
mkdir -p $EXPERIMENT_DIR/outVideos
```

--------------------------------

### Set Custom Base Directory for Installations

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

Customize the base directory for Kafka and Inference Builder installations by setting the BASE_DIR environment variable before running the setup script. Optionally, enable Inference Builder setup.

```bash
export BASE_DIR=/path/to/your/preferred/base/directory
# If you want to use Inference Builder (Option 2), uncomment the lines below
# export USE_INFERENCE_BUILDER=true
# export INFERENCE_BUILDER_DIR="$BASE_DIR/inference_builder"
./scripts/setup_prerequisites.sh
```

--------------------------------

### Run Custom Tiler Sample with Screen Display

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-custom-tile-config/README.md

Execute the custom tiler sample with four input video files, enabling direct screen display.

```bash
./deepstream-custom-tile-config -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4  file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4  file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4
```

--------------------------------

### Manually Start Mosquitto Broker

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

If the Mosquitto broker service does not start automatically or fails, manually start it on port 1883. The broker can be run in the foreground or background.

```bash
mosquitto -p 1883
# [Expected output] You should see "mosquitto version <version_number> running" printed.
# You need to keep it running in a separate terminal window. To avoid this, you can use the following command to start it in the background:
# mosquitto -p 1883 -d
# [Expected output] Nothing will be printed. Use the mosquitto_test.sh script to verify the broker is running.
```

--------------------------------

### Set Up Environment and Directories for 12-Camera Dataset

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-deepstream.md

Configure environment variables and directories for a 12-camera dataset experiment. The auto-configurator will automatically detect the number of cameras and generate appropriate configuration files.

```bash
export DATASET_DIR=$PWD/datasets/mtmc_12cam/
export EXPERIMENT_DIR=$PWD/experiments/deepstream/12cam

python utils/deepstream_auto_configurator.py \
    --dataset-dir=$DATASET_DIR \
    --enable-msg-broker \
    --enable-osd \
    --output-dir=$EXPERIMENT_DIR
```

--------------------------------

### Display Application Help Options

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-bodypose-3d/README.md

View all available command-line options for the `deepstream-pose-estimation-app` by running the application with the -h flag.

```bash
./deepstream-pose-estimation-app -h
```

--------------------------------

### Install Additional Prerequisites Inside Container

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d/README.md

Installs additional user-specific prerequisites within the DeepStream container. This script is typically found in the DeepStream installation directory.

```bash
cd /opt/nvidia/deepstream/deepstream/ 
bash user_additional_install.sh
```

--------------------------------

### Navigate to Sample Application Directory

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-3d-sensor-fusion/README.md

Change the current directory to the location of the deepstream-3d-lidar-sensor-fusion sample application.

```bash
cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion/
```

--------------------------------

### Install Additional Tools for Jetson

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream_parallel_inference_app/README.md

Installs necessary development libraries and tools on Jetson devices, such as libjson-glib-dev and libgstrtspserver-1.0-dev, required for certain DeepStream functionalities.

```bash
apt-get install -y libjson-glib-dev libgstrtspserver-1.0-dev
/opt/nvidia/deepstream/deepstream/samples/triton_backend_setup.sh
```

--------------------------------

### Run 4-Camera Pipeline with Inference Builder

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Launches a quick-start 4-camera pipeline with Inference Builder support, using a simpler YAML configuration.

```bash
./scripts/test_4cam_ib.sh
```

--------------------------------

### Install Prerequisites Inside Container

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-masktracker/README.md

Installs additional prerequisites required by DeepStream applications within the running container. This script should be executed after entering the container.

```bash
# Install prerequisites
cd /opt/nvidia/deepstream/deepstream/ 
bash user_additional_install.sh
```

--------------------------------

### Clone Repository and Install Git LFS

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Clone the DeepStream reference applications repository and install Git Large File Storage (LFS). This is necessary for handling large model files.

```bash
sudo apt install git-lfs
git lfs install

git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream-tracker-3d-multi-view
git lfs pull  # In case repo is already cloned before installing git-lfs
```

--------------------------------

### Run Prerequisites Setup with Inference Builder

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

To include Inference Builder in the setup, set the USE_INFERENCE_BUILDER environment variable to true before running the prerequisites script. This option is specific to the DeepStream Container.

```bash
USE_INFERENCE_BUILDER=true ./scripts/setup_prerequisites.sh
```

--------------------------------

### Clone and Build DeepStream Custom Tiler Sample

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-custom-tile-config/README.md

Clone the reference application repository and build the custom tiler sample.

```bash
git clone https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps.git
cd deepstream_reference_apps/deepstream-custom-tile-config
make
```

--------------------------------

### Start IPC Server for Latency Measurement

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-ipc-test-sr/README.md

Run the DeepStream IPC test application in server mode. This command starts the server and specifies the input video file and the IPC socket path. Ensure IPC_SR_PERF_MODE is set to 1 for performance mode.

```shell
$ IPC_SR_PERF_MODE=1 ./deepstream-ipc-test-app server file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4 /tmp/test1
```

--------------------------------

### Pull DeepStream Container Image

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/manual-setup.md

Download the latest DeepStream container image required for the application. Ensure you have Docker installed and configured.

```bash
docker pull nvcr.io/nvidia/deepstream:9.0-triton-multiarch
```

--------------------------------

### Run 4-Camera Inference Builder Pipeline with RT-DETR Detector

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/README.md

Starts the 4-camera DeepStream Inference Builder pipeline using the RT-DETR detector.

```bash
# To use RT-DETR detector instead of the default PeopleNetTransformer:
# DETECTOR_MODEL=RTDETR ./scripts/test_4cam_ib.sh
```

--------------------------------

### Run Custom Tiler Sample with MP4 Output

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-custom-tile-config/README.md

Execute the custom tiler sample with four input video files and generate an MP4 output video. The --no-display flag is used to disable screen display.

```bash
./deepstream-custom-tile-config -i file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4  file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4  file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_720p.mp4 --no-display
```

--------------------------------

### DS3D Sensor Fusion: Download NuScenes Dataset

Source: https://context7.com/nvidia-ai-iot/deepstream_reference_apps/llms.txt

Download and checkout the NuScenes dataset using git-lfs. Ensure git-lfs is installed and configured.

```bash
# Step 1: Download NuScenes subset dataset via git-lfs
git lfs install && git lfs fetch && git lfs checkout
# data/nuscene.tar.gz is now available (CC BY-NC-SA 4.0 license)
```

--------------------------------

### KITTI Format Output Sample

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d/README.md

This sample demonstrates the output format for object metadata in the KITTI format, typically saved per frame. It includes object label, unique ID, bounding box coordinates, confidence, and optional visibility and foot position. Note that visibility and foot position may be absent for objects tracked across frames.

```txt
person 1 0.0 0 0.0 1365.907227 -196.875290 1500.249146 93.554581 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.890186 0.229870 1458.166992 81.585060
person 6 0.0 0 0.0 1419.655151 72.774818 1647.446167 561.540894 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.841420 0.408513 1575.264160 531.851746
person 0 0.0 0 0.0 1008.387817 36.228714 1202.886353 421.609314 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.632065 0.504738 1148.117188 399.992645
...
```

--------------------------------

### Default Inference Configuration

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-vllm-plugin/README.md

Sets up the default user and system prompts, including placeholders for dynamic content like frame counts and timestamps. Advanced sampling parameters are commented out but available for tuning.

```yaml
inference:
  # User prompt with optional placeholders: {num_frames}, {stream_id}, {timestamps}
  # Control timestamp inclusion by using or omitting {timestamps} placeholder
  user_prompt: "These are {num_frames} images from stream {stream_id} sampled at timestamps {timestamps}. Describe the scene in detail."

  # System prompt (optional - omit for no system prompt)
  system_prompt: |
    Provide captions with timestamps using format:
    <start time> <end time> caption of event.

  # Sampling parameters (all optional)
  max_tokens: 2048             # Max tokens to generate
  temperature: 0.7              # Sampling temperature (0-1)

  # Advanced sampling parameters (optional)
  # top_p: 0.9                  # Top-p nucleus sampling
  # top_k: 100                  # Top-k sampling
  # repetition_penalty: 1.1     # Repetition penalty

  # Per-stream prompt overrides (multi-stream mode)
  stream_prompts:
    0:  # Stream 0
      user_prompt: "Stream {stream_id} at {timestamps}: Monitor for security threats."
      system_prompt: "You are a security analyst."
    1:  # Stream 1
      user_prompt: "Analyze traffic flow."  # No {timestamps} = no timestamps

```

--------------------------------

### Branch Group Configuration Example

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream_parallel_inference_app/README.md

Defines sources for inference branches using source IDs. The PGIE unique-id identifies the inferencing branch.

```yaml
branch0:
  key1: value1
  key2: value2
```

--------------------------------

### Launch Inference Container

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-tracker-3d-multi-view/docs/step-by-step-inference-builder.md

Run the `inference-builder-mv3dt:latest` Docker container with necessary volume mounts for models, datasets, and experiments. Ensure the container has access to the display.

```bash
sudo xhost + # give container access to display

docker run --privileged --rm -it --net=host --runtime=nvidia \
-v $INFERENCE_BUILDER_DIR/builder/samples/mv3dt_app/deepstream-app:/mv3dt_app \
-v $MODEL_REPO:/workspace/models \
-v $DATASET_DIR:/workspace/inputs \
-v $EXPERIMENT_DIR:/workspace/experiments \
-v /tmp/.X11-unix/:/tmp/.X11-unix \
-e DISPLAY=$DISPLAY \
-w /mv3dt_app \
inference-builder-mv3dt:latest \
python3 __main__.py --source-config /workspace/experiments/source_list_static.yaml -s /dev/null

# [Expected output] You should see a window named "python3" pop up and will display 4 camera views in a grid.
# Run this command to quit early: `docker ps -q --filter "ancestor=inference-builder-mv3dt" | xargs docker stop`.
# By default, the application waits up to **1000 seconds** if there is no data being streamed before exiting gracefully. You should see "Inference completed" as the last line from the logs.
```

--------------------------------

### Collect NuScenes Dataset for Demo Scene

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-3d-sensor-fusion/README.md

Execute a Python script to collect and set up the NuScenes dataset specifically for this demo. This script handles data directory configuration and prints calibration information.

```bash
cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion
python3 python/triton_lmm/helper/nuscene_data_setup.py  --data_dir=dataset/nuscene \
--ds3d_fusion_workspace=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-3d-lidar-sensor-fusion --print_calibration
```

--------------------------------

### Test Direction Calculation (Two Videos, dGPU)

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/anomaly/README.md

Test direction calculation using optical flow on two video inputs on dGPU. This pipeline processes two streams concurrently.

```bash
cd /opt/nvidia/deepstream/deepstream/
gst-launch-1.0 filesrc location = samples/streams/sample_1080p_h264.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 \
nvstreammux name=m batch-size=2 width=1920 height=1080 ! nvinfer config-file-path= samples/configs/deepstream-app/config_infer_primary.txt ! \
nvof ! tee name=t ! queue ! nvofvisual ! nvmultistreamtiler width=1920 height=540 !  nveglglessink t. ! queue ! dsdirection ! \
nvmultistreamtiler width=1920 height=540 ! nvvideoconvert ! nvdsosd ! nveglglessink filesrc location = samples/streams/sample_1080p_h264.mp4 ! \
qtdemux ! h264parse ! nvv4l2decoder ! m.sink_1  --gst-debug=3
```

--------------------------------

### Per-Stream Custom Prompts

Source: https://github.com/nvidia-ai-iot/deepstream_reference_apps/blob/master/deepstream-vllm-plugin/README.md

Demonstrates how to apply custom user prompts on a per-stream basis. Stream 0 includes timestamps for security analysis, while Stream 1 uses a minimal prompt for vehicle analysis without timestamps.

```yaml
inference:
  stream_prompts:
    0:
      user_prompt: "Stream {stream_id} at {timestamps}: Security analysis."
    1:
      user_prompt: "Analyse vehicles."  # Minimal, no placeholders

```