NVIDIA Dynamo

https://docs.nvidia.com/dynamo
Admin
NVIDIA Dynamo is an open-source, high-throughput, low-latency distributed inference framework...
Tokens:1,072,161
Snippets:9,087
Trust Score:10
Update:6 days ago
Show doc for...
Tokens:
Raw
### Start local LTX-2 example

Source: https://docs.nvidia.com/dynamo/diffusion/text-to-video

Starts the local example with an LTX-2 model, setting the model and worker extra args for attention backend and FP4 quantization.

```bash
cd <dynamo-root>/examples/diffusers/local

MODEL=FastVideo/LTX2-Distilled-Diffusers \
WORKER_EXTRA_ARGS="--attention-backend TORCH_SDPA --fp4-quantization" \
./run_local.sh
```

--------------------------------

### Start Minikube with GPU support and enable required addons

Source: https://docs.nvidia.com/dynamo/cli/installation/minikube-setup

Starts a local Minikube cluster with Docker driver, GPU support, and resource limits. Then enables the istio-provisioner, istio, and storage-provisioner-rancher addons. Run this after installing Minikube and configuring GPU support if needed.

```bash
# Start Minikube with GPU support (if configured)
minikube start --driver docker --container-runtime docker --gpus all --memory=16000mb --cpus=8

# Enable required addons
minikube addons enable istio-provisioner
minikube addons enable istio
minikube addons enable storage-provisioner-rancher
```

--------------------------------

### Run backend locally with pip install and my-backend

Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-unified-backends

Install the package and start the backend with NATS and etcd reachable. The --model-name must be a valid HuggingFace repo or local path; the framework fetches tokenizer and chat template on startup. Use a small public repo for smoke tests.

```bash
pip install -e .

# Ensure NATS + etcd are reachable (NATS_SERVER, ETCD_ENDPOINTS).
# --model-name must be a valid HuggingFace repo (or local path); the
# framework fetches the tokenizer + chat template from it on startup.
# Pick a small public repo for smoke tests.
my-backend --model-name Qwen/Qwen3-0.6B --namespace dynamo

# In another shell, start the Dynamo frontend:
python -m dynamo.frontend --http-port 8000
```

--------------------------------

### Example 1: Data parallel load balanced setup

Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-python-workers

Shows two instances of the same model (qwen3-32b) with the same namespace/component/endpoint but different instance IDs, allowing the router to spread traffic.

```text
Node 1: namespace: qwen3-32b, component: backend, endpoint: generate, model: /data/Qwen3-32B --tensor-parallel-size 2 --base-gpu-id 0
Node 2: namespace: qwen3-32b, component: backend, endpoint: generate model: /data/Qwen3-32B --tensor-parallel-size 2 --base-gpu-id 2
```

--------------------------------

### Example: High-ISL/OSL configuration for streaming workloads

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/kserve-grpc-frontend

Set the HTTP/2 flow control window sizes before starting the server. This example configures 16MB connection window and 1MB stream window for 128 concurrent 15k-token requests. If not set, tonic defaults (64KB) are used.

```bash
# For 128 concurrent 15k-token requests
export DYN_GRPC_INITIAL_CONNECTION_WINDOW_SIZE=16777216  # 16MB
export DYN_GRPC_INITIAL_STREAM_WINDOW_SIZE=1048576      # 1MB
python -m dynamo.frontend --kserve-grpc-server
```

--------------------------------

### Install dependencies and run the local resource monitor

Source: https://docs.nvidia.com/dynamo/reference/observability/local-resource-monitor-local

Run these commands on the host machine to install required Python packages and start the monitor. The monitor exposes Prometheus metrics at /metrics and a WebSocket dashboard at / when dependencies are installed.

```bash
pip install psutil nvidia-ml-py prometheus-client
python3 dev/observability/dynamo_local_resource_monitor.py --host 0.0.0.0 --port 8051
```

--------------------------------

### Install system libraries on macOS

Source: https://docs.nvidia.com/dynamo/advanced-customizations/building-from-source

Installs Homebrew if needed and then installs cmake and protobuf via brew.

```bash
# Install Homebrew if needed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install cmake protobuf
```

--------------------------------

### Two-Replica Setup Commands

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer

Commands to start two indexer replicas. Replica A runs without peers; Replica B uses --peers to recover state from A on startup. Both subscribe to the same workers.

```bash
# Replica A (first instance, no peers)
python -m dynamo.indexer --port 8090 --block-size 16 \
  --workers "1=tcp://worker1:5557,2=tcp://worker2:5558"

# Replica B (recovers from A on startup)
python -m dynamo.indexer --port 8091 --block-size 16 \
  --workers "1=tcp://worker1:5557,2=tcp://worker2:5558" \
  --peers "http://localhost:8090"
```

--------------------------------

### Launch two backend engines for KvRouter setup

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/router-examples

Starts two vLLM backend engines with unique system ports and KV event endpoints, required for the router to have a meaningful selection. Uses Qwen/Qwen3-0.6B model with block size 64 and ZMQ KV events.

```bash
export PYTHONHASHSEED=0

DYN_SYSTEM_PORT=8081 CUDA_VISIBLE_DEVICES=0 python -m dynamo.vllm \
  --model Qwen/Qwen3-0.6B \
  --block-size 64 \
  --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20080","enable_kv_cache_events":true}' &

DYN_SYSTEM_PORT=8082 CUDA_VISIBLE_DEVICES=1 python -m dynamo.vllm \
  --model Qwen/Qwen3-0.6B \
  --block-size 64 \
  --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' &
```

--------------------------------

### Run aggregated LoRA launch script

Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang

Executes the aggregated LoRA launch script. Requires the setup from the launch/lora README, MinIO at http://localhost:9000, and the my-loras bucket populated. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter.

```bash
./launch/lora/agg_lora.sh
```

--------------------------------

### Verify Minikube installation and Istio/storage setup

Source: https://docs.nvidia.com/dynamo/cli/installation/minikube-setup

Checks Minikube status, verifies Istio pods in the istio-system namespace, and confirms the storage class is available. Run these commands after starting Minikube to ensure the environment is ready.

```bash
# Check Minikube status
minikube status

# Verify Istio installation
kubectl get pods -n istio-system

# Verify storage class
kubectl get storageclass
```

--------------------------------

### GET /workers response example

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer

Example JSON response from GET /workers showing a worker with instance_id, source, status, model_name, routing_group, block_size, endpoints, and listeners. The source is always "zmq" for ZMQ-managed workers.

```json
[
  {
    "instance_id": 1,
    "source": "zmq",
    "status": "active",
    "model_name": "llama-3-8b",
    "routing_group": "default",
    "block_size": 16,
    "endpoints": {
      "0": "tcp://127.0.0.1:5557",
      "1": "tcp://127.0.0.1:5558"
    },
    "listeners": {
      "0": {
        "endpoint": "tcp://127.0.0.1:5557",
        "status": "active"
      },
      "1": {
        "endpoint": "tcp://127.0.0.1:5558",
        "status": "active"
      }
    }
  }
]
```

--------------------------------

### Launch Multimodal E/PD Server (Bash)

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal

Command to launch the multimodal E/PD server from the examples directory.

```bash
cd $DYNAMO_HOME/examples/backends/sglang
./launch/multimodal_epd.sh
```

--------------------------------

### GET /peers - Example response

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer

Example JSON response from the /peers endpoint, showing a list of registered peer URLs.

```json
["http://peer:8091"]
```

--------------------------------

### tilt-settings.yaml Configuration Example

Source: https://docs.nvidia.com/dynamo/advanced-customizations/developing-with-tilt

Example configuration file with all available settings. All settings are optional; the Tiltfile defines defaults. The registry can also be set via the REGISTRY env var, which takes precedence. The file is gitignored to avoid leaking personal values.

```yaml
# Kubernetes contexts Tilt is allowed to connect to.
# Safety guard: prevents accidental deployments to production clusters.
allowed_contexts:
  - my-cluster-context

# Container registry for the operator image.
# Can also be set via the REGISTRY env var (env var takes precedence).
registry: docker.io/myuser

# Skip running `make generate && make manifests` before applying CRDs.
# Set to true when you haven't changed API types (faster iteration).
skip_codegen: true

# Target architecture for the operator binaries.
# Use arm64 for an ARM-based kind cluster.
goarch: amd64

# Target namespace for the operator and related resources.
# namespace: dynamo-system

# Subchart toggles
# enable_nats: true            # Only for NATS-based transports or features
# enable_etcd: false           # Only if discoveryBackend is "etcd"
# enable_kai_scheduler: false  # GPU-aware scheduling for multi-node
# enable_grove: false          # PodClique-based multi-node orchestration

# Extra Helm value overrides (applied on top of subchart toggles)
# helm_values:
#   dynamo-operator.discoveryBackend: kubernetes
#   dynamo-operator.natsAddr: "nats://external-nats:4222"
```

--------------------------------

### Prepare SGLang LoRA adapter with MinIO

Source: https://docs.nvidia.com/dynamo/cli/operations/lo-ra-adapters

Start MinIO and upload the example adapter for the SGLang LoRA example. See the linked example for scripts and configurable environment variables.

```bash
cd examples/backends/sglang/launch/lora
./setup_minio.sh
```

--------------------------------

### GET /engine/control/start_profile

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/reference-guide

Starts PyTorch profiling on the SGLang worker.

```APIDOC
## GET /engine/control/start_profile

### Description
Starts PyTorch profiling on the SGLang worker.

### Method
GET

### Endpoint
/engine/control/start_profile

```

--------------------------------

### Build runtime bindings and install Dynamo

Source: https://docs.nvidia.com/dynamo/knowledge-base/concepts/simulation/simulation-runs

Run these commands from the repository root to build the runtime bindings and install Dynamo into the project virtual environment. Use a release build because simulation is CPU-bound.

```bash
.venv/bin/maturin develop --release -m lib/bindings/python/Cargo.toml
uv pip install -e .
```

--------------------------------

### Verify Dynamo platform installation

Source: https://docs.nvidia.com/dynamo/kubernetes/installation/managed-kubernetes/gcp/gke-setup

Expected output of 'kubectl get pods' after installing the Dynamo Kubernetes Platform, showing running pods.

```bash
kubectl get pods
NAME                                                              READY   STATUS             RESTARTS   AGE
dynamo-platform-dynamo-operator-controller-manager-69b9794fpgv9   2/2     Running            0          4m27s
dynamo-platform-etcd-0                                            1/1     Running            0          4m27s
dynamo-platform-nats-0                                            2/2     Running            0          4m27s
```

--------------------------------

### Launch Image and Video Inputs Script

Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang

Serves a vision-language model for image and video prompts. Default model is Qwen/Qwen3-VL-2B-Instruct and requires 1 GPU. Configure via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, DYN_HTTP_PORT, and DYN_SYSTEM_PORT. Use --model-path, --chat-template, --page-size, --enable-otel, and --frontend-decoding flags. Additional unrecognized arguments are forwarded to the backend worker command.

```bash
./launch/agg_vision.sh
```

--------------------------------

### Launch frontend with load tracking and policy configuration

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/deficit-round-robin

Command to start the frontend with load-aware routing and the policy configuration file. Use after starting the backend worker.

```bash
python -m dynamo.frontend \
    --router-mode kv \
    --load-aware \
    --router-policy-config ./policy-classes.yaml
```

--------------------------------

### Start Dynamo Frontend with KV Router Mode

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/offloading-support-matrix

Starts the Dynamo frontend on HTTP port 8000 with KV router mode enabled. This is the common setup for all offloading combinations.

```bash
python -m dynamo.frontend --http-port 8000 --router-mode kv
```

--------------------------------

### Launch aggregated vLLM tracing example

Source: https://docs.nvidia.com/dynamo/cli/operations/observability

Run the aggregated vLLM tracing example from the examples/backends/vllm/launch directory. The script enables JSONL logging and OTLP export over gRPC and assigns service names to Dynamo processes.

```bash
cd examples/backends/vllm/launch
./agg_tracing.sh
```

--------------------------------

### Start observability stack and launch SGLang disaggregated with tracing

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/observability

Starts the Docker Compose observability stack and launches the SGLang disaggregated example with tracing enabled via the --enable-otel flag.

```bash
docker compose -f dev/docker-compose.yml up -d
docker compose -f dev/docker-observability.yml up -d

# Launch SGLang disaggregated with tracing
cd examples/backends/sglang/launch
./disagg.sh --enable-otel
```

--------------------------------

### GET /dump - Example response

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer

Example JSON response from the /dump endpoint, showing the radix tree state for two model:routing_group keys. Each entry contains a block_size and an events array of RouterEvent objects.

```json
{
  "llama-3-8b:default": {
    "block_size": 16,
    "events": [<RouterEvent>, ...]
  },
  "mistral-7b:customer-a": {
    "block_size": 16,
    "events": [<RouterEvent>, ...]
  }
}
```

--------------------------------

### Start with environment variable and call the added route

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/python-route-extensions

Uses the equivalent environment-variable form to start the Frontend with the hello-world extension, then calls the added route with curl. The response confirms the handler returns a JSON-serializable body wrapped in HTTP 200.

```bash
DYN_FRONTEND_ROUTE_EXTENSIONS="hello-world" python -m dynamo.frontend

```

```bash
curl localhost:8000/hello_world

```

```json
{"message": "hello world!"}

```

--------------------------------

### ExampleProvider class implementing SweepConfigProvider contract

Source: https://docs.nvidia.com/dynamo/knowledge-base/concepts/simulation/ai-simulate/sweeper/sweep-configuration-providers

Defines a provider with name 'example.policy' and api_version 1. The generate_search_space method returns an AdapterSearchPlan with a SearchSpaceFragment containing choices for the 'agg' branch. The materialize_replay method returns an AdapterReplaySpec with a config using the selected mode.

```python
class ExampleProvider:
    name = "example.policy"
    api_version = 1

    def generate_search_space(self, search_spec, context):
        return AdapterSearchPlan(
            fragment=SearchSpaceFragment(
                choices_by_branch={
                    "agg": {"mode": list(search_spec["mode"])}
                }
            )
        )

    def materialize_replay(self, plan, selection, context):
        return AdapterReplaySpec(config={"mode": selection["mode"]})
```

--------------------------------

### Run MinIO setup script for local LoRA

Source: https://docs.nvidia.com/dynamo/cli/operations/lo-ra-adapters

Sets up MinIO for S3-compatible local storage. Run from the examples directory.

```bash
cd examples/backends/vllm/launch/lora
./setup_minio.sh
```

--------------------------------

### Set environment variables for EKS setup

Source: https://docs.nvidia.com/dynamo/kubernetes/installation/managed-kubernetes/eks/eks-setup

Exports environment variables used throughout the guide. Modify AWS_REGION to use a different region.

```bash
export AWS_REGION="us-east-1"
export CLUSTER_NAME="ai-dynamo"
export DYNAMO_NAMESPACE="dynamo-system"
export DYNAMO_RELEASE_VERSION="1.2.1"
```

--------------------------------

### Create EPP crate and add dependencies

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/write-custom-routing-strategies

Initializes a new EPP binary crate and adds required dependencies: anyhow for error handling, tokio for async runtime, the policy catalog, the kv-router with standalone-selection feature, and the standard EPP runner.

```bash
# Create the custom EPP binary crate.
cargo init --bin --name acme-epp "$POLICY_DIR/epp"

# Add error handling and the asynchronous runtime.
cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" anyhow
cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" tokio@=1.48.0 --features macros,rt-multi-thread

# Add the policy catalog and worker-selection API.
cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$POLICY_DIR/catalog" acme-routing-catalog
cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$DYNAMO_DIR/lib/kv-router" --features standalone-selection dynamo-kv-router

# Add the standard EPP runner.
cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$DYNAMO_DIR/deploy/inference-gateway/ext-proc" dynamo-ext-proc
```

--------------------------------

### Launch included custom encoder example

Source: https://docs.nvidia.com/dynamo/multimodal/custom-vision-encoders

Runs the aggregated path with the Qwen3.5-2B model and Qwen35VisionEncoder. The encoder loads the vision tower, preprocesses images, and returns projected features via the native TokensPrompt path. Favors readability over speed.

```bash
bash examples/custom_encoder/launch/agg_qwen3_5_native.sh --gpu 0
```

--------------------------------

### Launch script: text-to-video-diffusion.sh

Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang

Starts a video-generation worker for Wan models. Default model is Wan-AI/Wan2.1-T2V-1.3B-Diffusers; --wan-size 1b uses one GPU, --wan-size 14b uses two GPUs.

```bash
./launch/text-to-video-diffusion.sh
```

--------------------------------

### Define and register a Python route extension provider

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/python-route-extensions

Defines a route provider that returns a FrontendRoute with a synchronous handler, registers it under the dynamo.frontend.routes entry-point group in pyproject.toml, installs the package, then starts the Frontend with the extension selected by name. Requires the package to be installable so the entry point is discoverable.

```python
# hello_routes.py
from dynamo.llm import FrontendExtensionContext, FrontendRoute


def _hello(ctx: FrontendExtensionContext):
    return {"message": "hello world!"}


def hello_world_routes():
    return [FrontendRoute("GET", "/hello_world", _hello)]

```

```toml
# pyproject.toml
[project.entry-points."dynamo.frontend.routes"]
hello-world = "hello_routes:hello_world_routes"

```

```bash
pip install -e .

```

```bash
python -m dynamo.frontend --frontend-route-extension hello-world

```

--------------------------------

### Quick Start: Create tilt-settings.yaml and Launch Tilt

Source: https://docs.nvidia.com/dynamo/advanced-customizations/developing-with-tilt

Commands to set up a personal Tilt settings file and start the development environment. The settings file is gitignored. Tilt opens a terminal UI and web dashboard at http://localhost:10350. Press Space to open the web UI, Ctrl-C to shut down (resources remain deployed), and run 'tilt down' to tear down.

```bash
cd deploy/operator

# Create your personal settings file (gitignored)
cat > tilt-settings.yaml <<EOF
allowed_contexts:
  - my-cluster-context
registry: docker.io/myuser
skip_codegen: true
EOF

# Launch
tilt up
```

--------------------------------

### GET /v1/models/{model_id} returns 404

Source: https://docs.nvidia.com/dynamo/digest/agentic-harnesses

Example of a request to the model details endpoint that returned HTTP/1.1 404 Not Found during startup, illustrating a bug where Dynamo did not serve that endpoint.

```http
GET /v1/models/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
HTTP/1.1 404 Not Found
```

--------------------------------

### Start local preview server with fern docs dev

Source: https://docs.nvidia.com/dynamo/contributing/documentation/building-and-publishing

Builds the site and serves it locally with hot-reload, allowing you to see exactly how pages will look on the live site.

```bash
cd docs/fern
fern docs dev
```

--------------------------------

### Expected NIXL backend initialization output

Source: https://docs.nvidia.com/dynamo/kubernetes/disaggregated-serving/overview

Example log output indicating NIXL successfully instantiated a backend (UCX). This confirms the transfer path is working.

```text
NIXL INFO Backend UCX was instantiated
```

--------------------------------

### Enable conditional disaggregation with --router-conditional-disagg

Source: https://docs.nvidia.com/dynamo/advanced-customizations/conditional-disaggregation

Starts the Dynamo frontend with conditional disaggregation enabled, using KV routing mode. This is the basic setup for the hybrid routing behavior.

```bash
python -m dynamo.frontend \
    --router-mode kv \
    --router-conditional-disagg
```

--------------------------------

### Implement start() method returning EngineConfig

Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-unified-backends

Performs heavy initialization (loading weights, building scheduler, warming up CUDA) and returns EngineConfig metadata. After this returns, generate() must be ready for concurrent calls. worker_id is opaque and can be used for stable cluster-wide keys.

```python
async def start(self, worker_id: int) -> EngineConfig:
    # ... load weights, build scheduler, warm up CUDA, etc.
    # Heavy: may take minutes. Emit logger.info checkpoints so
    # operators see progress (Worker logs around start() but not
    # inside it).
    self._engine = await heavy_init(self.model_name)

    return EngineConfig(
        model=self.model_name,
        served_model_name=self.model_name,
        context_length=8192,
        kv_cache_block_size=16,    # None if no block-structured KV
        total_kv_blocks=1024,
        max_num_seqs=64,
        max_num_batched_tokens=8192,
    )
```

--------------------------------

### Launch single-worker text generation with ./launch/agg.sh

Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang

Starts the frontend and one SGLang worker handling prefill and decode. Default model is Qwen/Qwen3-0.6B, requires 1 GPU. Configure via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, DYN_HTTP_PORT, DYN_SYSTEM_PORT, and flags --model-path and --enable-otel.

```bash
./launch/agg.sh
```

--------------------------------

### Run router with policy-class queues config

Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/configuration-and-tuning

Command to start the Dynamo frontend in KV router mode with a policy-class queues YAML configuration. The example uses the sample policy file from the repository.

```bash
python -m dynamo.frontend \
    --router-mode kv \
    --router-policy-config examples/router/policy-class-queues.yaml
```

--------------------------------

### Dry-run install of media decoders for vLLM

Source: https://docs.nvidia.com/dynamo/multimodal/additional-media-decoders

Shows what the installer would do without making changes. Use this to preview the actions before running the actual install.

```bash
python -m dynamo.common.utils.install_media_decoders vllm --dry-run
```
Tokens:
Raw