### Start local LTX-2 example Source: https://docs.nvidia.com/dynamo/diffusion/text-to-video Starts the local example with an LTX-2 model, setting the model and worker extra args for attention backend and FP4 quantization. ```bash cd /examples/diffusers/local MODEL=FastVideo/LTX2-Distilled-Diffusers \ WORKER_EXTRA_ARGS="--attention-backend TORCH_SDPA --fp4-quantization" \ ./run_local.sh ``` -------------------------------- ### Start Minikube with GPU support and enable required addons Source: https://docs.nvidia.com/dynamo/cli/installation/minikube-setup Starts a local Minikube cluster with Docker driver, GPU support, and resource limits. Then enables the istio-provisioner, istio, and storage-provisioner-rancher addons. Run this after installing Minikube and configuring GPU support if needed. ```bash # Start Minikube with GPU support (if configured) minikube start --driver docker --container-runtime docker --gpus all --memory=16000mb --cpus=8 # Enable required addons minikube addons enable istio-provisioner minikube addons enable istio minikube addons enable storage-provisioner-rancher ``` -------------------------------- ### Run backend locally with pip install and my-backend Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-unified-backends Install the package and start the backend with NATS and etcd reachable. The --model-name must be a valid HuggingFace repo or local path; the framework fetches tokenizer and chat template on startup. Use a small public repo for smoke tests. ```bash pip install -e . # Ensure NATS + etcd are reachable (NATS_SERVER, ETCD_ENDPOINTS). # --model-name must be a valid HuggingFace repo (or local path); the # framework fetches the tokenizer + chat template from it on startup. # Pick a small public repo for smoke tests. my-backend --model-name Qwen/Qwen3-0.6B --namespace dynamo # In another shell, start the Dynamo frontend: python -m dynamo.frontend --http-port 8000 ``` -------------------------------- ### Example 1: Data parallel load balanced setup Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-python-workers Shows two instances of the same model (qwen3-32b) with the same namespace/component/endpoint but different instance IDs, allowing the router to spread traffic. ```text Node 1: namespace: qwen3-32b, component: backend, endpoint: generate, model: /data/Qwen3-32B --tensor-parallel-size 2 --base-gpu-id 0 Node 2: namespace: qwen3-32b, component: backend, endpoint: generate model: /data/Qwen3-32B --tensor-parallel-size 2 --base-gpu-id 2 ``` -------------------------------- ### Example: High-ISL/OSL configuration for streaming workloads Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/kserve-grpc-frontend Set the HTTP/2 flow control window sizes before starting the server. This example configures 16MB connection window and 1MB stream window for 128 concurrent 15k-token requests. If not set, tonic defaults (64KB) are used. ```bash # For 128 concurrent 15k-token requests export DYN_GRPC_INITIAL_CONNECTION_WINDOW_SIZE=16777216 # 16MB export DYN_GRPC_INITIAL_STREAM_WINDOW_SIZE=1048576 # 1MB python -m dynamo.frontend --kserve-grpc-server ``` -------------------------------- ### Install dependencies and run the local resource monitor Source: https://docs.nvidia.com/dynamo/reference/observability/local-resource-monitor-local Run these commands on the host machine to install required Python packages and start the monitor. The monitor exposes Prometheus metrics at /metrics and a WebSocket dashboard at / when dependencies are installed. ```bash pip install psutil nvidia-ml-py prometheus-client python3 dev/observability/dynamo_local_resource_monitor.py --host 0.0.0.0 --port 8051 ``` -------------------------------- ### Install system libraries on macOS Source: https://docs.nvidia.com/dynamo/advanced-customizations/building-from-source Installs Homebrew if needed and then installs cmake and protobuf via brew. ```bash # Install Homebrew if needed /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" brew install cmake protobuf ``` -------------------------------- ### Two-Replica Setup Commands Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer Commands to start two indexer replicas. Replica A runs without peers; Replica B uses --peers to recover state from A on startup. Both subscribe to the same workers. ```bash # Replica A (first instance, no peers) python -m dynamo.indexer --port 8090 --block-size 16 \ --workers "1=tcp://worker1:5557,2=tcp://worker2:5558" # Replica B (recovers from A on startup) python -m dynamo.indexer --port 8091 --block-size 16 \ --workers "1=tcp://worker1:5557,2=tcp://worker2:5558" \ --peers "http://localhost:8090" ``` -------------------------------- ### Launch two backend engines for KvRouter setup Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/router-examples Starts two vLLM backend engines with unique system ports and KV event endpoints, required for the router to have a meaningful selection. Uses Qwen/Qwen3-0.6B model with block size 64 and ZMQ KV events. ```bash export PYTHONHASHSEED=0 DYN_SYSTEM_PORT=8081 CUDA_VISIBLE_DEVICES=0 python -m dynamo.vllm \ --model Qwen/Qwen3-0.6B \ --block-size 64 \ --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20080","enable_kv_cache_events":true}' & DYN_SYSTEM_PORT=8082 CUDA_VISIBLE_DEVICES=1 python -m dynamo.vllm \ --model Qwen/Qwen3-0.6B \ --block-size 64 \ --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:20081","enable_kv_cache_events":true}' & ``` -------------------------------- ### Run aggregated LoRA launch script Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang Executes the aggregated LoRA launch script. Requires the setup from the launch/lora README, MinIO at http://localhost:9000, and the my-loras bucket populated. Run launch/lora/setup_minio.sh --start first to start MinIO and upload the example adapter. ```bash ./launch/lora/agg_lora.sh ``` -------------------------------- ### Verify Minikube installation and Istio/storage setup Source: https://docs.nvidia.com/dynamo/cli/installation/minikube-setup Checks Minikube status, verifies Istio pods in the istio-system namespace, and confirms the storage class is available. Run these commands after starting Minikube to ensure the environment is ready. ```bash # Check Minikube status minikube status # Verify Istio installation kubectl get pods -n istio-system # Verify storage class kubectl get storageclass ``` -------------------------------- ### GET /workers response example Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer Example JSON response from GET /workers showing a worker with instance_id, source, status, model_name, routing_group, block_size, endpoints, and listeners. The source is always "zmq" for ZMQ-managed workers. ```json [ { "instance_id": 1, "source": "zmq", "status": "active", "model_name": "llama-3-8b", "routing_group": "default", "block_size": 16, "endpoints": { "0": "tcp://127.0.0.1:5557", "1": "tcp://127.0.0.1:5558" }, "listeners": { "0": { "endpoint": "tcp://127.0.0.1:5557", "status": "active" }, "1": { "endpoint": "tcp://127.0.0.1:5558", "status": "active" } } } ] ``` -------------------------------- ### Launch Multimodal E/PD Server (Bash) Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/sglang-multimodal Command to launch the multimodal E/PD server from the examples directory. ```bash cd $DYNAMO_HOME/examples/backends/sglang ./launch/multimodal_epd.sh ``` -------------------------------- ### GET /peers - Example response Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer Example JSON response from the /peers endpoint, showing a list of registered peer URLs. ```json ["http://peer:8091"] ``` -------------------------------- ### tilt-settings.yaml Configuration Example Source: https://docs.nvidia.com/dynamo/advanced-customizations/developing-with-tilt Example configuration file with all available settings. All settings are optional; the Tiltfile defines defaults. The registry can also be set via the REGISTRY env var, which takes precedence. The file is gitignored to avoid leaking personal values. ```yaml # Kubernetes contexts Tilt is allowed to connect to. # Safety guard: prevents accidental deployments to production clusters. allowed_contexts: - my-cluster-context # Container registry for the operator image. # Can also be set via the REGISTRY env var (env var takes precedence). registry: docker.io/myuser # Skip running `make generate && make manifests` before applying CRDs. # Set to true when you haven't changed API types (faster iteration). skip_codegen: true # Target architecture for the operator binaries. # Use arm64 for an ARM-based kind cluster. goarch: amd64 # Target namespace for the operator and related resources. # namespace: dynamo-system # Subchart toggles # enable_nats: true # Only for NATS-based transports or features # enable_etcd: false # Only if discoveryBackend is "etcd" # enable_kai_scheduler: false # GPU-aware scheduling for multi-node # enable_grove: false # PodClique-based multi-node orchestration # Extra Helm value overrides (applied on top of subchart toggles) # helm_values: # dynamo-operator.discoveryBackend: kubernetes # dynamo-operator.natsAddr: "nats://external-nats:4222" ``` -------------------------------- ### Prepare SGLang LoRA adapter with MinIO Source: https://docs.nvidia.com/dynamo/cli/operations/lo-ra-adapters Start MinIO and upload the example adapter for the SGLang LoRA example. See the linked example for scripts and configurable environment variables. ```bash cd examples/backends/sglang/launch/lora ./setup_minio.sh ``` -------------------------------- ### GET /engine/control/start_profile Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/reference-guide Starts PyTorch profiling on the SGLang worker. ```APIDOC ## GET /engine/control/start_profile ### Description Starts PyTorch profiling on the SGLang worker. ### Method GET ### Endpoint /engine/control/start_profile ``` -------------------------------- ### Build runtime bindings and install Dynamo Source: https://docs.nvidia.com/dynamo/knowledge-base/concepts/simulation/simulation-runs Run these commands from the repository root to build the runtime bindings and install Dynamo into the project virtual environment. Use a release build because simulation is CPU-bound. ```bash .venv/bin/maturin develop --release -m lib/bindings/python/Cargo.toml uv pip install -e . ``` -------------------------------- ### Verify Dynamo platform installation Source: https://docs.nvidia.com/dynamo/kubernetes/installation/managed-kubernetes/gcp/gke-setup Expected output of 'kubectl get pods' after installing the Dynamo Kubernetes Platform, showing running pods. ```bash kubectl get pods NAME READY STATUS RESTARTS AGE dynamo-platform-dynamo-operator-controller-manager-69b9794fpgv9 2/2 Running 0 4m27s dynamo-platform-etcd-0 1/1 Running 0 4m27s dynamo-platform-nats-0 2/2 Running 0 4m27s ``` -------------------------------- ### Launch Image and Video Inputs Script Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang Serves a vision-language model for image and video prompts. Default model is Qwen/Qwen3-VL-2B-Instruct and requires 1 GPU. Configure via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, DYN_HTTP_PORT, and DYN_SYSTEM_PORT. Use --model-path, --chat-template, --page-size, --enable-otel, and --frontend-decoding flags. Additional unrecognized arguments are forwarded to the backend worker command. ```bash ./launch/agg_vision.sh ``` -------------------------------- ### Launch frontend with load tracking and policy configuration Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/deficit-round-robin Command to start the frontend with load-aware routing and the policy configuration file. Use after starting the backend worker. ```bash python -m dynamo.frontend \ --router-mode kv \ --load-aware \ --router-policy-config ./policy-classes.yaml ``` -------------------------------- ### Start Dynamo Frontend with KV Router Mode Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/offloading-support-matrix Starts the Dynamo frontend on HTTP port 8000 with KV router mode enabled. This is the common setup for all offloading combinations. ```bash python -m dynamo.frontend --http-port 8000 --router-mode kv ``` -------------------------------- ### Launch aggregated vLLM tracing example Source: https://docs.nvidia.com/dynamo/cli/operations/observability Run the aggregated vLLM tracing example from the examples/backends/vllm/launch directory. The script enables JSONL logging and OTLP export over gRPC and assigns service names to Dynamo processes. ```bash cd examples/backends/vllm/launch ./agg_tracing.sh ``` -------------------------------- ### Start observability stack and launch SGLang disaggregated with tracing Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/backends/sg-lang/observability Starts the Docker Compose observability stack and launches the SGLang disaggregated example with tracing enabled via the --enable-otel flag. ```bash docker compose -f dev/docker-compose.yml up -d docker compose -f dev/docker-observability.yml up -d # Launch SGLang disaggregated with tracing cd examples/backends/sglang/launch ./disagg.sh --enable-otel ``` -------------------------------- ### ExampleProvider class implementing SweepConfigProvider contract Source: https://docs.nvidia.com/dynamo/knowledge-base/concepts/simulation/ai-simulate/sweeper/sweep-configuration-providers Defines a provider with name 'example.policy' and api_version 1. The generate_search_space method returns an AdapterSearchPlan with a SearchSpaceFragment containing choices for the 'agg' branch. The materialize_replay method returns an AdapterReplaySpec with a config using the selected mode. ```python class ExampleProvider: name = "example.policy" api_version = 1 def generate_search_space(self, search_spec, context): return AdapterSearchPlan( fragment=SearchSpaceFragment( choices_by_branch={ "agg": {"mode": list(search_spec["mode"])} } ) ) def materialize_replay(self, plan, selection, context): return AdapterReplaySpec(config={"mode": selection["mode"]}) ``` -------------------------------- ### GET /dump - Example response Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/standalone-indexer Example JSON response from the /dump endpoint, showing the radix tree state for two model:routing_group keys. Each entry contains a block_size and an events array of RouterEvent objects. ```json { "llama-3-8b:default": { "block_size": 16, "events": [, ...] }, "mistral-7b:customer-a": { "block_size": 16, "events": [, ...] } } ``` -------------------------------- ### Run MinIO setup script for local LoRA Source: https://docs.nvidia.com/dynamo/cli/operations/lo-ra-adapters Sets up MinIO for S3-compatible local storage. Run from the examples directory. ```bash cd examples/backends/vllm/launch/lora ./setup_minio.sh ``` -------------------------------- ### Start with environment variable and call the added route Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/python-route-extensions Uses the equivalent environment-variable form to start the Frontend with the hello-world extension, then calls the added route with curl. The response confirms the handler returns a JSON-serializable body wrapped in HTTP 200. ```bash DYN_FRONTEND_ROUTE_EXTENSIONS="hello-world" python -m dynamo.frontend ``` ```bash curl localhost:8000/hello_world ``` ```json {"message": "hello world!"} ``` -------------------------------- ### Set environment variables for EKS setup Source: https://docs.nvidia.com/dynamo/kubernetes/installation/managed-kubernetes/eks/eks-setup Exports environment variables used throughout the guide. Modify AWS_REGION to use a different region. ```bash export AWS_REGION="us-east-1" export CLUSTER_NAME="ai-dynamo" export DYNAMO_NAMESPACE="dynamo-system" export DYNAMO_RELEASE_VERSION="1.2.1" ``` -------------------------------- ### Create EPP crate and add dependencies Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/write-custom-routing-strategies Initializes a new EPP binary crate and adds required dependencies: anyhow for error handling, tokio for async runtime, the policy catalog, the kv-router with standalone-selection feature, and the standard EPP runner. ```bash # Create the custom EPP binary crate. cargo init --bin --name acme-epp "$POLICY_DIR/epp" # Add error handling and the asynchronous runtime. cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" anyhow cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" tokio@=1.48.0 --features macros,rt-multi-thread # Add the policy catalog and worker-selection API. cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$POLICY_DIR/catalog" acme-routing-catalog cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$DYNAMO_DIR/lib/kv-router" --features standalone-selection dynamo-kv-router # Add the standard EPP runner. cargo add --manifest-path "$POLICY_DIR/epp/Cargo.toml" --path "$DYNAMO_DIR/deploy/inference-gateway/ext-proc" dynamo-ext-proc ``` -------------------------------- ### Launch included custom encoder example Source: https://docs.nvidia.com/dynamo/multimodal/custom-vision-encoders Runs the aggregated path with the Qwen3.5-2B model and Qwen35VisionEncoder. The encoder loads the vision tower, preprocesses images, and returns projected features via the native TokensPrompt path. Favors readability over speed. ```bash bash examples/custom_encoder/launch/agg_qwen3_5_native.sh --gpu 0 ``` -------------------------------- ### Launch script: text-to-video-diffusion.sh Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang Starts a video-generation worker for Wan models. Default model is Wan-AI/Wan2.1-T2V-1.3B-Diffusers; --wan-size 1b uses one GPU, --wan-size 14b uses two GPUs. ```bash ./launch/text-to-video-diffusion.sh ``` -------------------------------- ### Define and register a Python route extension provider Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/frontend/python-route-extensions Defines a route provider that returns a FrontendRoute with a synchronous handler, registers it under the dynamo.frontend.routes entry-point group in pyproject.toml, installs the package, then starts the Frontend with the extension selected by name. Requires the package to be installable so the entry point is discoverable. ```python # hello_routes.py from dynamo.llm import FrontendExtensionContext, FrontendRoute def _hello(ctx: FrontendExtensionContext): return {"message": "hello world!"} def hello_world_routes(): return [FrontendRoute("GET", "/hello_world", _hello)] ``` ```toml # pyproject.toml [project.entry-points."dynamo.frontend.routes"] hello-world = "hello_routes:hello_world_routes" ``` ```bash pip install -e . ``` ```bash python -m dynamo.frontend --frontend-route-extension hello-world ``` -------------------------------- ### Quick Start: Create tilt-settings.yaml and Launch Tilt Source: https://docs.nvidia.com/dynamo/advanced-customizations/developing-with-tilt Commands to set up a personal Tilt settings file and start the development environment. The settings file is gitignored. Tilt opens a terminal UI and web dashboard at http://localhost:10350. Press Space to open the web UI, Ctrl-C to shut down (resources remain deployed), and run 'tilt down' to tear down. ```bash cd deploy/operator # Create your personal settings file (gitignored) cat > tilt-settings.yaml < EngineConfig: # ... load weights, build scheduler, warm up CUDA, etc. # Heavy: may take minutes. Emit logger.info checkpoints so # operators see progress (Worker logs around start() but not # inside it). self._engine = await heavy_init(self.model_name) return EngineConfig( model=self.model_name, served_model_name=self.model_name, context_length=8192, kv_cache_block_size=16, # None if no block-structured KV total_kv_blocks=1024, max_num_seqs=64, max_num_batched_tokens=8192, ) ``` -------------------------------- ### Launch single-worker text generation with ./launch/agg.sh Source: https://docs.nvidia.com/dynamo/recipes/cli-templates/sg-lang Starts the frontend and one SGLang worker handling prefill and decode. Default model is Qwen/Qwen3-0.6B, requires 1 GPU. Configure via OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, DYN_HTTP_PORT, DYN_SYSTEM_PORT, and flags --model-path and --enable-otel. ```bash ./launch/agg.sh ``` -------------------------------- ### Run router with policy-class queues config Source: https://docs.nvidia.com/dynamo/knowledge-base/modular-components/router/configuration-and-tuning Command to start the Dynamo frontend in KV router mode with a policy-class queues YAML configuration. The example uses the sample policy file from the repository. ```bash python -m dynamo.frontend \ --router-mode kv \ --router-policy-config examples/router/policy-class-queues.yaml ``` -------------------------------- ### Example 2: Two models, two pipelines Source: https://docs.nvidia.com/dynamo/advanced-customizations/writing-custom-backends/writing-python-workers Shows two separate pipelines with different namespaces and models, each with its own backend and generate endpoint. ```text Node 1: namespace: qwen3-32b, component: backend, endpoint: generate, model: /data/Qwen3-32B Node 2: namespace: llama3-1-8b, component: backend, endpoint: generat, model: /data/Llama-3.1-8B-Instruct/ ```