llm-d

https://llm-d.ai/
Admin
llm-d is a Kubernetes-native, high-performance distributed framework for scalable LLM inference,...
Tokens:13,117
Snippets:93
Trust Score:7.8
Update:1 year ago
Show doc for...
Tokens:
Raw
### Navigate to Deployment Guide Directory

Source: https://llm-d.ai/docs/guide/Installation/quickstart

This command navigates to a specific example directory within the llm-d quickstart guides. It's a prerequisite step before following the README instructions for deployment.

```shell
cd quickstarts/guides/inference-scheduling
```

--------------------------------

### List Helm Releases

Source: https://llm-d.ai/docs/guide/Installation/quickstart

Lists all Helm releases within a specified Kubernetes namespace. This command is used to view the charts installed by the guide. It requires the NAMESPACE environment variable to be set.

```bash
helm list -n ${NAMESPACE}
```

--------------------------------

### Clone llm-d-benchmark and Install Dependencies

Source: https://llm-d.ai/docs/architecture/Components/benchmark

This snippet clones the llm-d-benchmark repository and installs necessary dependencies using a setup script and pip. It sets up the environment for running benchmarks.

```bash
git clone https://github.com/llm-d/llm-d-benchmark.git
cd llm-d-benchmark
./setup/install_deps.sh
pip install ./config_explorer
```

--------------------------------

### Install llm-d Simulated Accelerators with Helmfile

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

This command installs the llm-d simulated accelerator stack using helmfile. It sets the namespace for deployment and applies the configuration. The release name can be customized using the RELEASE_NAME_POSTFIX environment variable. It defaults to using Istio as the provider.

```bash
export NAMESPACE=llm-d-sim # Or any namespace your heart desires
cd guides/simulated-accelerators
helmfile apply -n ${NAMESPACE}
```

--------------------------------

### Install llm-d Simulated Accelerators with kgateway Option

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

This command installs the llm-d simulated accelerator stack using helmfile, specifically selecting the 'kgateway' option. This allows for the deployment with a chosen gateway provider. The namespace is derived from the NAMESPACE environment variable.

```bash
export NAMESPACE=llm-d-sim
helmfile apply -e kgateway -n ${NAMESPACE}
```

--------------------------------

### Install HTTPRoute for P/D Disaggregation

Source: https://llm-d.ai/docs/guide/Installation/pd-disaggregation

Installs the HTTPRoute configuration for P/D disaggregation, with separate commands for 'kgateway'/'istio' and 'gke' providers.

```bash
kubectl apply -f httproute.yaml -n ${NAMESPACE}  
```

```bash
kubectl apply -f httproute.gke.yaml -n ${NAMESPACE}  
```

--------------------------------

### Install llm-d Simulated Accelerators with Helmfile and Custom Release Name

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

This demonstrates how to install the llm-d simulated accelerator stack with a custom release name postfix. This is useful for supporting concurrent installations of the simulator. The namespace is also explicitly set.

```bash
export NAMESPACE=llm-d-sim
export RELEASE_NAME_POSTFIX=sim-2
helmfile apply -n ${NAMESPACE}
```

--------------------------------

### Start vLLM Decoder Server (Podman)

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This command starts the vLLM decoder server using Podman. It configures GPU usage, model path, network settings, and enables the NIXLConnector for communication. Ensure the model path and image are correct for your setup. This server listens on port 8001.

```bash
$ podman run --network host --device nvidia.com/gpu=0 -v $HOME/models:/models \
    -e UCX_TLS="cuda_ipc,cuda_copy,tcp" \
    -e VLLM_NIXL_SIDE_CHANNEL_PORT=5555 \
    -e VLLM_NIXL_SIDE_CHANNEL_HOST=localhost \
    -e VLLM_LOGGING_LEVEL=DEBUG \
    -e HF_HOME=/models ghcr.io/llm-d/llm-d:0.0.8 --model Qwen/Qwen3-0.6B \
    --enforce-eager \
    --port 8001 \
    --kv-transfer-config='{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
```

--------------------------------

### Install HTTPRoute for kgateway or istio

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the HTTPRoute configuration for 'kgateway' or 'istio' providers to the specified namespace. Ensure the httproute.yaml file is present in the current directory.

```bash
kubectl apply -f httproute.yaml -n ${NAMESPACE}  
```

--------------------------------

### View Kubernetes Resources

Source: https://llm-d.ai/docs/guide/Installation/quickstart

Retrieves all resources within a specified Kubernetes namespace. This command is useful for verifying deployments and checking the status of various Kubernetes objects. It requires the NAMESPACE environment variable to be set.

```bash
kubectl get all -n ${NAMESPACE}
```

--------------------------------

### Install P/D Disaggregation Stack with Helmfile

Source: https://llm-d.ai/docs/guide/Installation/pd-disaggregation

Installs the P/D disaggregation stack using helmfile. Sets the NAMESPACE environment variable and applies the helmfile configuration. It also provides options for customizing release names and specifying gateway providers.

```bash
export NAMESPACE=llm-d-pd # Or any namespace your heart desires  
cd guides/pd-disaggregation  
helmfile apply -n ${NAMESPACE}  
```

```bash
helmfile apply -e kgateway -n ${NAMESPACE}  
```

--------------------------------

### Install HTTPRoute for gke

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the HTTPRoute configuration specific to the 'gke' provider to the specified namespace. Ensure the httproute.gke.yaml file is present in the current directory.

```bash
kubectl apply -f httproute.gke.yaml -n ${NAMESPACE}  
```

--------------------------------

### Install HTTPRoute for llm-d (kgateway or istio)

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

This command applies the Kubernetes HTTPRoute configuration for llm-d when using the 'kgateway' or 'istio' providers. It assumes the configuration is in a file named 'httproute.yaml' and applies it to the specified namespace.

```bash
kubectl apply -f httproute.yaml -n ${NAMESPACE}
```

--------------------------------

### Install HTTPRoute for llm-d (gke)

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

This command applies the Kubernetes HTTPRoute configuration for llm-d specifically for the 'gke' provider. It uses the 'httproute.gke.yaml' file and applies it to the namespace defined by the NAMESPACE environment variable.

```bash
kubectl apply -f httproute.gke.yaml -n ${NAMESPACE}
```

--------------------------------

### Install llm-d Inference Scheduler with Custom Release Name

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command demonstrates how to apply the llm-d inference scheduler stack with a custom release name postfix. This is useful for supporting concurrent installations. The target namespace is specified by the NAMESPACE environment variable.

```bash
RELEASE_NAME_POSTFIX=inference-scheduling-2 helmfile apply -n ${NAMESPACE}
```

--------------------------------

### Install InferencePool for GKE, Istio, or Kgateway

Source: https://llm-d.ai/docs/guide/Installation/wide-ep-lws

Installs the InferencePool using Helm, with configurations tailored for GKE, Istio, or Kgateway. It includes options for enabling monitoring.

```bash
# For GKE  
helm install deepseek-r1 \  
  -n ${NAMESPACE} \  
  -f inferencepool.values.yaml \  
  --set "provider.name=gke" \  
  --set "inferencePool.apiVersion=inference.networking.k8s.io/v1" \  
  --set "inferenceExtension.monitoring.gke.enable=true" \  
  oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool --version v1.0.1  
  
# For Istio  
helm install deepseek-r1 \  
  -n ${NAMESPACE} \  
  -f inferencepool.values.yaml \  
  --set "provider.name=istio" \  
  --set "inferenceExtension.monitoring.prometheus.enable=true" \  
  oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool --version v1.0.1  
  
# For Kgateway  
helm install deepseek-r1 \  
  -n ${NAMESPACE} \  
  -f inferencepool.values.yaml \  
  oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool --version v1.0.1  
```

--------------------------------

### Execute Standalone Benchmark Steps

Source: https://llm-d.ai/docs/architecture/Components/benchmark

This provides a granular approach to running benchmarks by explicitly calling the setup, run, and teardown scripts. This allows for more control over each stage of the benchmarking process.

```bash
./setup/standup.sh
./run.sh
./setup/teardown.sh
```

--------------------------------

### Apply llm-d Inference Scheduler for XPU Hardware

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the llm-d inference scheduler stack using helmfile, targeting XPU hardware and Istio as the gateway provider. The deployment occurs in the namespace specified by the NAMESPACE environment variable.

```bash
helmfile apply -e xpu  -n ${NAMESPACE} # targets istio as gateway provider with XPU hardware  
```

--------------------------------

### Install llm-d Inference Scheduler using Helmfile

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the llm-d inference scheduler stack using helmfile. It allows customization of the release name postfix and specifies the target namespace. The default namespace is 'llm-d-inference-scheduler'.

```bash
export NAMESPACE=llm-d-inference-scheduler # or any other namespace  
cd guides/inference-scheduling  
helmfile apply -n ${NAMESPACE}  
```

--------------------------------

### Start vLLM Prefiller Server (Podman)

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This command initiates the vLLM prefiller server using Podman, similar to the decoder but configured for the prefilling role. It utilizes a different GPU and side channel port (5556). This server listens on port 8002 and works in conjunction with the decoder via the NIXLConnector.

```bash
$ podman run --network host --device nvidia.com/gpu=1 -v $HOME/models:/models \
    -e UCX_TLS="cuda_ipc,cuda_copy,tcp" \
    -e VLLM_NIXL_SIDE_CHANNEL_PORT=5556 \
    -e VLLM_NIXL_SIDE_CHANNEL_HOST=localhost \
    -e VLLM_LOGGING_LEVEL=DEBUG \
    -e HF_HOME=/models ghcr.io/llm-d/llm-d:0.0.8 --model Qwen/Qwen3-0.6B \
    --enforce-eager \
    --port 8002 \
    --kv-transfer-config='{"kv_connector":"NixlConnector","kv_role":"kv_both"}'
```

--------------------------------

### Install llm-d Stack with Helmfile and Release Name Postfix

Source: https://llm-d.ai/docs/guide/Installation/precise-prefix-cache-aware

This command installs the llm-d stack using helmfile, allowing for concurrent installs by specifying a RELEASE_NAME_POSTFIX. The deployment namespace is derived from the NAMESPACE environment variable.

```bash
RELEASE_NAME_POSTFIX=kv-events-2 helmfile apply -n ${NAMESPACE}
```

--------------------------------

### Install llm-d Stack with Helmfile

Source: https://llm-d.ai/docs/guide/Installation/precise-prefix-cache-aware

This command installs the llm-d stack using helmfile. It derives the deployment namespace from the NAMESPACE environment variable, defaulting to 'llm-d-precise'. Ensure your client tools are installed and the Gateway control plane is configured.

```bash
export NAMESPACE=llm-d-precise
cd guides/precise-prefix-cache-aware
helmfile apply -n ${NAMESPACE}
```

--------------------------------

### Build Routing Proxy from Source

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This command builds the routing proxy from its source code. It is a prerequisite for running the proxy locally. Ensure you have the necessary build tools installed for your environment.

```bash
# Build the routing proxy from source:
$ make build  

```

--------------------------------

### Check Routing Proxy Build and Usage

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This command verifies that the routing proxy was built successfully and displays its usage information and available flags. This is useful for understanding the proxy's configuration options and starting it.

```bash
# Check the build was successful by running it locally:
$ ./bin/llm-d-routing-sidecar -help  

Usage of ./bin/llm-d-routing-sidecar:  
  -add_dir_header  
        If true, adds the file directory to the header of the log messages  
  -alsologtostderr  
        log to standard error as well as files (no effect when -logtostderr=true)  
  -cert-path string  
        The path to the certificate for secure proxy. The certificate and private key files are assumed to be named tls.crt and tls.key, respectively. If not set, and secureProxy is enabled, then a self-signed certificate is used (for testing).  
  -connector string  
        the P/D connector being used. Either nixl, nixlv2 or lmcache (default "nixl")  
  -decoder-tls-insecure-skip-verify  
        configures the proxy to skip TLS verification for requests to decoder  
  -decoder-use-tls  
        whether to use TLS when sending requests to the decoder  
  -enable-ssrf-protection  
        enable SSRF protection using InferencePool allowlisting  
  -inference-pool-name string  
        the specific InferencePool name to watch (defaults to INFERENCE_POOL_NAME env var)  
  -inference-pool-namespace string  
        the Kubernetes namespace to watch for InferencePool resources (defaults to INFERENCE_POOL_NAMESPACE env var)  
  -log_backtrace_at value  
        when logging hits line file:N, emit a stack trace  
  -log_dir string  
        If non-empty, write log files in this directory (no effect when -logtostderr=true)  
  -log_file string  
        If non-empty, use this log file (no effect when -logtostderr=true)  
  -log_file_max_size uint  
        Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)  
  -logtostderr  
        log to standard error instead of files (default true)  
  -one_output  
        If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)  
  -port string  
        the port the sidecar is listening on (default "8000")  
  -prefiller-tls-insecure-skip-verify  
        configures the proxy to skip TLS verification for requests to prefiller  
  -prefiller-use-tls  
        whether to use TLS when sending requests to prefillers  
  -secure-proxy  
        Enables secure proxy. Defaults to true. (default true)  
  -skip_headers  
        If true, avoid header prefixes in the log messages  
  -skip_log_headers  
        If true, avoid headers when opening log files (no effect when -logtostderr=true)  
  -stderrthreshold value  
        logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=true) (default 2)  
  -v value  
        number for the log level verbosity  
  -vllm-port string  
        the port vLLM is listening on (default "8001")  
  -vmodule value  
        comma-separated list of pattern=N settings for file-filtered logging  

> **Note:** lmcache and nixl connectors are deprecated. Use nixlv2

```

--------------------------------

### Install llm-d-modelservice Helm Chart with Custom Values

Source: https://llm-d.ai/docs/architecture/Components/modelservice

This command installs the llm-d-modelservice Helm chart using a custom values file and overrides specific configuration parameters using the `--set` flag. It's useful for deploying CPU-only inference and setting gateway parent references.

```bash
helm install cpu-only llm-d-modelservice -f examples/values-cpu.yaml --set prefill.replicas=0 --set "routing.parentRefs[0].name=MYGATEWAY"  
```

--------------------------------

### Apply llm-d Inference Scheduler for GKE TPU Hardware

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the llm-d inference scheduler stack using helmfile, targeting GKE externally managed TPU hardware and GKE externally managed as the gateway provider. The deployment occurs in the namespace specified by the NAMESPACE environment variable.

```bash
# or  
helmfile apply -e gke_tpu  -n ${NAMESPACE} # targets GKE externally managed as gateway provider with TPU hardware  
```

--------------------------------

### Add and Install llm-d Infrastructure Helm Chart

Source: https://llm-d.ai/docs/architecture/Components/infra

This snippet demonstrates how to add the llm-d-infra Helm repository, update it, and then install the infrastructure components into a Kubernetes cluster. Ensure prerequisites like Kubernetes, Helm, and Gateway API are met.

```bash
# Add the Helm repository  
helm repo add llm-d-infra https://llm-d-incubation.github.io/llm-d-infra/  
helm repo update  
  
# Install the infrastructure components  
helm install my-llm-d-infra llm-d-infra/llm-d-infra  
```

--------------------------------

### Apply llm-d Inference Scheduler with Specific Gateway Option

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the llm-d inference scheduler stack using helmfile with a specific gateway option, such as 'kgateway'. The target namespace is derived from the NAMESPACE environment variable.

```bash
helmfile apply -e kgateway -n ${NAMESPACE}  
```

--------------------------------

### Monitor Inference Metrics with Prometheus for llm-d

Source: https://context7.com/context7/llm-d_ai/llms.txt

Enables and queries inference metrics using Prometheus for llm-d components. Requires 'kubectl'. Demonstrates port-forwarding to Prometheus and example metric queries.

```bash
# Enable monitoring in Helm chart values
# Set in values.yaml or via --set flags:
# inferenceExtension.monitoring.prometheus.enable=true

# Verify PodMonitor resources
kubectl get podmonitor -n ${NAMESPACE}

# Query vLLM metrics via Prometheus
# Example metrics:
# - vllm:num_requests_running
# - vllm:num_requests_waiting
# - vllm:gpu_cache_usage_perc
# - vllm:time_to_first_token_seconds
# - vllm:time_per_output_token_seconds

# Port forward to Prometheus
kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090

# Access Prometheus UI at http://localhost:9090
# Query example: rate(vllm:request_success_total[5m])
```

--------------------------------

### Manual Cleanup of LLM-D AI Helm Releases

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

Uninstalls the individual Helm releases for the LLM-D AI project manually. This includes 'infra-inference-scheduling', 'gaie-inference-scheduling', and 'ms-inference-scheduling' in the specified namespace.

```bash
helm uninstall infra-inference-scheduling -n ${NAMESPACE}
helm uninstall gaie-inference-scheduling -n ${NAMESPACE}
helm uninstall ms-inference-scheduling -n ${NAMESPACE}
```

--------------------------------

### Destroy Helmfile Deployment

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

Destroys a deployment managed by Helmfile in the specified namespace. This is a convenient way to remove all resources associated with the deployment.

```shell
helmfile destroy -n ${NAMESPACE}
```

--------------------------------

### View PD Profile Handler Configuration

Source: https://llm-d.ai/docs/guide/Installation/pd-disaggregation

Displays the configuration for the 'pd-profile-handler' plugin within the GAIE values file using `cat`, `yq`, and `yq` again for nested selection. This is used to check and potentially modify the `threshold` parameter for Selective PD.

```shell
cat gaie-pd/values.yaml | yq '.inferenceExtension.pluginsCustomConfig."pd-config.yaml"' | yq '.plugins[] | select(.type == "pd-profile-handler")'
```

--------------------------------

### Cleanup HTTPRoute for kgateway or istio

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

Deletes the HTTPRoute configuration using kubectl for 'kgateway' or 'istio' environments. This command applies the specified 'httproute.yaml' file for removal.

```bash
kubectl delete -f httproute.yaml -n ${NAMESPACE}
```

--------------------------------

### Uninstall Helm Releases Manually

Source: https://llm-d.ai/docs/guide/Installation/simulated-accelerators

Manually uninstalls individual Helm releases (infra-sim, gaie-sim, ms-sim) from a specified namespace. This provides granular control over the cleanup process.

```shell
# From examples/sim
helm uninstall infra-sim -n ${NAMESPACE}
helm uninstall gaie-sim -n ${NAMESPACE}
helm uninstall ms-sim -n ${NAMESPACE}
```

--------------------------------

### Add and Update llm-d-modelservice Helm Repository

Source: https://llm-d.ai/docs/architecture/Components/modelservice

These commands add the llm-d-modelservice Helm repository to your Helm configuration and then update your local Helm repository cache. This is the first step to install or manage charts from this repository.

```bash
helm repo add llm-d-modelservice https://llm-d-incubation.github.io/llm-d-modelservice/  
helm repo update  
```

--------------------------------

### Cleanup HTTPRoute for gke

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

Deletes the HTTPRoute configuration using kubectl for 'gke' environments. This command applies the specified 'httproute.gke.yaml' file for removal.

```bash
kubectl delete -f httproute.gke.yaml -n ${NAMESPACE}
```

--------------------------------

### Apply llm-d Inference Scheduler for DigitalOcean Kubernetes Service (DOKS)

Source: https://llm-d.ai/docs/guide/Installation/inference-scheduling

This command applies the llm-d inference scheduler stack using helmfile, specifically configured for DigitalOcean Kubernetes Service (DOKS). It utilizes the 'digitalocean' gateway option and deploys to the namespace specified by the NAMESPACE environment variable.

```bash
helmfile apply -e digitalocean -n ${NAMESPACE}  
```

--------------------------------

### Deploy Wide Expert-Parallelism with Helm

Source: https://context7.com/context7/llm-d_ai/llms.txt

This snippet covers the deployment of large Mixture-of-Experts (MoE) models like DeepSeek-R1 using Wide Expert-Parallelism. It involves creating a namespace, applying model server configurations based on the infrastructure provider (GKE or CoreWeave), installing the InferencePool using Helm, and deploying the gateway and HTTPRoute. Verification of the LeaderWorkerSet deployment is also included.

```bash
# Deploy DeepSeek-R1 with DP=8 across 24 H200 GPUs
export NAMESPACE=llm-d-wide-ep

kubectl create namespace ${NAMESPACE}

# Deploy model servers (choose infrastructure provider)
kubectl apply -k ./manifests/modelserver/gke -n ${NAMESPACE}
# OR
kubectl apply -k ./manifests/modelserver/coreweave -n ${NAMESPACE}

# Deploy InferencePool with gateway provider configuration
helm install deepseek-r1 \
  -n ${NAMESPACE} \
  -f inferencepool.values.yaml \
  --set "provider.name=istio" \
  --set "inferenceExtension.monitoring.prometheus.enable=true" \
  oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool \
  --version v1.0.1

# Deploy gateway and HTTPRoute
kubectl apply -k ./manifests/gateway/istio -n ${NAMESPACE}

# Verify LeaderWorkerSet deployment (startup takes 7-10 minutes)
kubectl get all -n ${NAMESPACE}
# Expected: 1 prefill StatefulSet (DP=8), 2 decode StatefulSets (DP=8 each)
```

--------------------------------

### Cleanup Helm Deployments

Source: https://llm-d.ai/docs/guide/Installation/pd-disaggregation

Removes deployed resources using Helm commands. It first destroys all Helmfile deployments and then uninstalls individual Helm releases for model services, GAIE, and infrastructure.

```shell
# Remove the model services
helmfile destroy -n ${NAMESPACE}  

# Remove the infrastructure
helm uninstall ms-pd -n ${NAMESPACE}  
helm uninstall gaie-pd -n ${NAMESPACE}  
helm uninstall infra-pd -n ${NAMESPACE}
```

--------------------------------

### Send Inference Requests via Port Forwarded Gateway

Source: https://context7.com/context7/llm-d_ai/llms.txt

This code example shows how to send inference requests to a deployed model through the gateway endpoint. It involves port-forwarding to the gateway service and then using `curl` to send a completion request with specified parameters such as model, prompt, and temperature. The expected response structure from the API is also provided.

```bash
# Port forward to gateway service
export NAMESPACE=llm-d-inference-scheduler
kubectl port-forward -n ${NAMESPACE} \
  service/infra-inference-scheduling-inference-gateway-istio 8000:80

# Send completion request
curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "prompt": "Explain Kubernetes in simple terms:",
    "max_tokens": 100,
    "temperature": 0.7
  }' | jq

# Expected response structure:
# {
#   "id": "cmpl-",
#   "object": "text_completion",
#   "created": 1234567890,
#   "model": "Qwen/Qwen3-0.6B",
#   "choices": [{
#     "text": "Kubernetes is...",
#     "index": 0,
#
```

--------------------------------

### Send Chat Completions Request (curl)

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This curl command sends a chat completions request to the routing proxy running on port 8000. It specifies the prefiller to be used via the 'x-prefiller-host-port' header and provides a JSON payload with the model name and prompt. This tests the end-to-end functionality of the setup.

```bash
$ curl http://localhost:8000/v1/completions \
      -H "Content-Type: application/json" \
      -H "x-prefiller-host-port: http://localhost:8002" \
       -d '{
        "model": "Qwen/Qwen3-0.6B",
        "prompt": "Author-contribution statements and acknowledgements in research papers should state clearly and specifically whether, and to what extent, the authors used AI technologies such as ChatGPT in the preparation of their manuscript and analysis. They should also indicate which LLMs were used. This will alert editors and reviewers to scrutinize manuscripts more carefully for potential biases, inaccuracies and improper source crediting. Likewise, scientific journals should be transparent about their use of LLMs, for example when selecting submitted manuscripts. Mention the large language model based product mentioned in the paragraph above:"
      }'
```

--------------------------------

### Set Namespace and Navigate to Directory

Source: https://llm-d.ai/docs/guide/Installation/wide-ep-lws

Sets the Kubernetes namespace for deployment and navigates to the necessary directory. This is a prerequisite for deploying the model servers.

```bash
export NAMESPACE=llm-d-wide-ep # or any other namespace  
cd guides/wide-ep-lws/  
kubectl create namespace ${NAMESPACE}  
```

--------------------------------

### Deploy vLLM Simulator to Kubernetes

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Deploys the vLLM simulator to a Kubernetes cluster using the provided deployment manifest. Ensure the manifest is updated with the correct image tag.

```bash
kubectl apply -f manifests/deployment.yaml
```

--------------------------------

### Build vLLM Simulator Executable

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Builds the vLLM simulator as a standalone executable for local testing. This command compiles the simulator for local execution.

```bash
make build
```

--------------------------------

### Build and Run llm-d Routing Sidecar

Source: https://llm-d.ai/docs/architecture/Components/routing-sidecar

This sequence clones the llm-d routing sidecar repository, builds the executable using 'make', and then runs the sidecar proxy. The proxy routes requests to the vLLM servers, using port 8000 for incoming requests and 8001 for the vLLM server. The 'connector' is set to 'nixlv2'.

```bash
$ git clone https://github.com/llm-d/llm-d-routing-sidecar.git && \
  cd llm-d-routing-sidecar && \
  make build
```

```bash
$ ./bin/llm-d-routing-sidecar -port=8000 -vllm-port=8001 -connector=nixlv2
```

--------------------------------

### Run vLLM Simulator with Docker

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Runs the vLLM Simulator Docker image, exposing port 8000 and serving a specified model with optional LoRA modules. Replace 'dev' with the latest release tag for production use.

```bash
docker run --rm --publish 8000:8000 ghcr.io/llm-d/llm-d-inference-sim:dev --port 8000 --model "Qwen/Qwen2.5-1.5B-Instruct" --lora-modules ``{"name":"tweet-summary-0"}`` ``{"name":"tweet-summary-1"}``
```

--------------------------------

### Helmfile and Helm Uninstall for Cleanup

Source: https://llm-d.ai/docs/guide/Installation/precise-prefix-cache-aware

These commands are used to clean up deployed resources. `helmfile destroy` can remove all associated services from the 'examples/precise-prefix-cache-aware' directory. Alternatively, individual services can be uninstalled using `helm uninstall`. Note that release names might vary if `$RELEASE_NAME_POSTFIX` is set.

```bash
# Remove the model services
# From examples/precise-prefix-cache-aware
helmfile destroy -n ${NAMESPACE}

# Or uninstall manually
helm uninstall infra-kv-events -n ${NAMESPACE}
helm uninstall gaie-kv-events -n ${NAMESPACE}
helm uninstall ms-kv-events -n ${NAMESPACE}
```

--------------------------------

### OpenAI-Compatible API Endpoints

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

The simulator partially supports the OpenAI API, providing emulated responses for chat completions, text completions, and model listing.

```APIDOC
## POST /v1/chat/completions

### Description
Emulates the OpenAI Chat Completions API endpoint. Supports 'echo' and 'random' modes, streaming, and timing parameters.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **stream** (boolean) - Optional - Whether to stream the response.
- **model** (string) - Optional - The model to use for completion.
- **messages** (array) - Required - The chat messages to send to the model.
  - **role** (string) - Required - The role of the author of the message (e.g., 'user', 'assistant').
  - **content** (string) - Required - The content of the message.
- **time-to-first-token** (number) - Optional - Delay before the first token is returned.
- **inter-token-latency** (number) - Optional - Delay between subsequent tokens in a stream.
- **kv-cache-transfer-latency** (number) - Optional - Used instead of `time-to-first-token` if P/D is enabled.

### Request Example
```json
{
  "stream": true,
  "model": "some-model",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **created** (integer) - Timestamp of creation.
- **model** (string) - The model used for the completion.
- **choices** (array) - List of completion choices.
  - **index** (integer) - Index of the choice.
  - **finish_reason** (string) - The reason the model finished generating tokens (e.g., 'stop', 'length').
  - **message** (object) - The generated message.
    - **role** (string) - The role of the message sender (e.g., 'assistant').
    - **content** (string) - The content of the message.

#### Response Example
```json
{
  "id": "chatcmpl-123",
  "created": 1677652288,
  "model": "some-model",
  "choices": [
    {
      "index": 0,
      "finish_reason": "stop",
      "message": {
        "role": "assistant",
        "content": "Hello there!"
      }
    }
  ]
}
```

## POST /v1/completions

### Description
Emulates the OpenAI Completions API endpoint. Supports 'echo' and 'random' modes, streaming, and timing parameters.

### Method
POST

### Endpoint
/v1/completions

### Parameters
#### Query Parameters
- **stream** (boolean) - Optional - Whether to stream the response.
- **model** (string) - Optional - The model to use for completion.
- **prompt** (string) - Required - The prompt to send to the model.
- **max_tokens** (integer) - Optional - Maximum number of tokens to generate (for future usage).
- **time-to-first-token** (number) - Optional - Delay before the first token is returned.
- **inter-token-latency** (number) - Optional - Delay between subsequent tokens in a stream.
- **kv-cache-transfer-latency** (number) - Optional - Used instead of `time-to-first-token` if P/D is enabled.

### Request Example
```json
{
  "stream": false,
  "model": "some-model",
  "prompt": "Once upon a time"
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **created** (integer) - Timestamp of creation.
- **model** (string) - The model used for the completion.
- **choices** (array) - List of completion choices.
  - **text** (string) - The generated text.

#### Response Example
```json
{
  "id": "cmpl-123",
  "created": 1677652288,
  "model": "some-model",
  "choices": [
    {
      "text": " Once upon a time, in a land far, far away..."
    }
  ]
}
```

## GET /v1/models

### Description
Returns a list of simulated models available in the vLLM simulator.

### Method
GET

### Endpoint
/v1/models

### Response
#### Success Response (200)
- **object** (string) - The type of object, always 'list'.
- **data** (array) - List of model details.
  - **id** (string) - The ID of the model.
  - **object** (string) - The object type, always 'model'.
  - **created** (integer) - Timestamp of model creation.
  - **owned_by** (string) - The owner of the model.
  - **root** (string) - The root model ID.
  - **parent** (string) - The parent model ID.

#### Response Example
```json
{
  "object": "list",
  "data": [
    {
      "id": "model-xyz",
      "object": "model",
      "created": 1677652288,
      "owned_by": "owner",
      "root": "root-model",
      "parent": null
    }
  ]
}
```
```

--------------------------------

### Build Docker Image for vLLM Simulator

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Builds a Docker image for the vLLM Simulator. Environment variables can be used to customize the image tag. On macOS, specify TARGETOS=linux for the correct base image.

```bash
make image-build
```

--------------------------------

### Run End-to-End Benchmark

Source: https://llm-d.ai/docs/architecture/Components/benchmark

This script executes a complete end-to-end benchmark process. It includes standing up an llm-d stack, running an inference harness with a default load profile, and then tearing down the stack.

```bash
./e2e.sh
```

--------------------------------

### Run Throughput Benchmark with Vegeta

Source: https://context7.com/context7/llm-d_ai/llms.txt

This snippet demonstrates how to run a simple throughput benchmark using the `vegeta` tool. It involves port-forwarding to the gateway service and then using `vegeta attack` to send POST requests to the completions endpoint. The results are reported using `vegeta report`.

```bash
# Port forward to gateway
kubectl port-forward -n ${NAMESPACE} service/infra-inference-scheduling-inference-gateway-istio 8000:80

# Run simple throughput benchmark with vegeta
echo "POST http://localhost:8000/v1/completions
Content-Type: application/json

{
  \"model\": \"Qwen/Qwen3-0.6B\",
  \"prompt\": \"Explain the benefits of Kubernetes:\",
  \"max_tokens\": 100
}" | vegeta attack -rate=10 -duration=60s | vegeta report
```

--------------------------------

### Load Docker Image into Kind Cluster

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Loads the vLLM simulator Docker image into a local Kubernetes cluster managed by Kind. This step is necessary before deploying the image to the cluster.

```bash
kind load --name kind docker-image ghcr.io/llm-d/llm-d-inference-sim:dev
```

--------------------------------

### Verify Kubernetes Deployment

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Verifies the status of the vLLM simulator deployment and its associated service within the Kubernetes cluster. Use these commands to ensure the deployment was successful.

```bash
kubectl get deployment vllm-llama3-8b-instruct
kubectl get service vllm-llama3-8b-instruct-svc
```

--------------------------------

### Configure Gateway Providers for llm-d

Source: https://context7.com/context7/llm-d_ai/llms.txt

Configures different Gateway API implementations for llm-d. Requires 'helmfile' and 'kubectl'. Supports kgateway, GKE-managed, Istio, DigitalOcean, Intel XPU, and Google TPU.

```bash
# Deploy with kgateway provider
cd guides/inference-scheduling
helmfile apply -e kgateway -n ${NAMESPACE}

# Deploy with GKE-managed gateway
helmfile apply -e gke -n ${NAMESPACE}
kubectl apply -f httproute.gke.yaml -n ${NAMESPACE}

# Deploy with Istio (default)
helmfile apply -e istio -n ${NAMESPACE}

# Deploy on DigitalOcean Kubernetes Service
helmfile apply -e digitalocean -n ${NAMESPACE}

# Deploy with Intel XPU hardware backend
helmfile apply -e xpu -n ${NAMESPACE}

# Deploy with Google TPU hardware backend
helmfile apply -e gke_tpu -n ${NAMESPACE}
```

--------------------------------

### Deploy Prefill/Decode Disaggregation with Helmfile

Source: https://context7.com/context7/llm-d_ai/llms.txt

This section details the deployment of Prefill/Decode disaggregation for large models. It requires an RDMA-capable cluster and a minimum of 8 GPUs. The process includes navigating to the correct directory, applying the Helmfile configuration, setting up an HTTPRoute, and verifying the prefill and decode pods. Optional configuration for selective P/D routing is also shown.

```bash
# Deploy P/D disaggregation for Llama-70B with 4 prefill workers and 1 decode worker
export NAMESPACE=llm-d-pd

# Prerequisites: RDMA-capable cluster (InfiniBand or RoCE), 8 GPUs minimum
cd guides/pd-disaggregation
helmfile apply -n ${NAMESPACE}

kubectl apply -f httproute.yaml -n ${NAMESPACE}

# Verify prefill and decode pods are running
kubectl get pods -n ${NAMESPACE}
# Expected: 4 prefill pods (TP=1 each) and 1 decode pod (TP=4)

# Configure selective P/D routing threshold (optional)
cat gaie-pd/values.yaml | yq '.inferenceExtension.pluginsCustomConfig."pd-config.yaml"' \
  | yq '.plugins[] | select(.type == "pd-profile-handler")'

# Adjust threshold parameter for selective routing:
# threshold: 0 -> always use P/D split
# threshold: N -> route directly to decode if prompt length < N tokens
```

--------------------------------

### Deploy Intelligent Inference Scheduling with Helmfile

Source: https://context7.com/context7/llm-d_ai/llms.txt

This snippet demonstrates how to deploy the intelligent inference scheduling stack using Helmfile. It involves setting up a namespace, creating a Kubernetes secret for HuggingFace token, applying the Helmfile configuration, and setting up an HTTPRoute for routing. Finally, it includes verification steps to check the deployment status.

```bash
# Set up environment and deploy the inference scheduling stack
export NAMESPACE=llm-d-inference-scheduler

# Create HuggingFace token secret for model downloads
kubectl create secret generic llm-d-hf-token \
  --from-literal=HF_TOKEN=your_huggingface_token \
  -n ${NAMESPACE}

# Deploy the stack using helmfile
cd guides/inference-scheduling
helmfile apply -n ${NAMESPACE}

# Install HTTPRoute for routing configuration
kubectl apply -f httproute.yaml -n ${NAMESPACE}

# Verify deployment
helm list -n ${NAMESPACE}
kubectl get all -n ${NAMESPACE}

# Expected output includes:
# - gaie-inference-scheduling-epp (Endpoint Picker)
# - infra-inference-scheduling-inference-gateway-istio
# - ms-inference-scheduling-llm-d-modelservice-decode (2 replicas)
```

--------------------------------

### Deploy ModelService with Custom Configuration using Helm

Source: https://context7.com/context7/llm-d_ai/llms.txt

Deploys a ModelService with custom parallelism and routing settings using the llm-d-modelservice Helm chart. Requires 'helm', 'kubectl'. Supports custom values and CPU-only simulation.

```bash
# Add llm-d-modelservice Helm repository
helm repo add llm-d-modelservice \
  https://llm-d-incubation.github.io/llm-d-modelservice/
helm repo update

# Deploy with custom values
helm install my-model llm-d-modelservice \
  -n ${NAMESPACE} \
  --set modelArtifacts.name="meta-llama/Llama-3.1-8B-Instruct" \
  --set modelArtifacts.uri="hf://meta-llama/Llama-3.1-8B-Instruct" \
  --set modelArtifacts.authSecretName="llm-d-hf-token" \
  --set decode.replicas=4 \
  --set decode.parallelism.tensor=2 \
  --set decode.parallelism.data=2 \
  --set "routing.parentRefs[0].name=my-gateway" \
  --set "decode.containers[0].image=vllm/vllm-openai:v0.6.0" \
  --set "decode.containers[0].name=vllm"

# Deploy with CPU-only inference simulation
helm install cpu-only llm-d-modelservice \
  -f examples/values-cpu.yaml \
  --set prefill.replicas=0 \
  --set "routing.parentRefs[0].name=my-gateway" \
  -n ${NAMESPACE}

# Verify deployment
kubectl get pods -n ${NAMESPACE}
kubectl get inferencepool -n ${NAMESPACE}
```

--------------------------------

### Run Specific Workload with Custom Parameters

Source: https://llm-d.ai/docs/architecture/Components/benchmark

This command allows running a specific workload against a pre-deployed stack with custom parameters. It enables flexibility in testing different load profiles and inference services.

```bash
./run.sh --harness inference-perf --workload chatbot_synthetic --methods <a string that matches a inference service or pod>
```

--------------------------------

### Deploy Gateway and HTTPRoute for GKE, Istio, or Kgateway

Source: https://llm-d.ai/docs/guide/Installation/wide-ep-lws

Applies Kubernetes manifests to deploy Gateways and HTTPRoutes. Supports GKE, Istio, and Kgateway, including a specific configuration for OpenShift Container Platform (OCP).

```kubectl
# Deploy a gke-l7-regional-external-managed gateway.  
kubectl apply -k ./manifests/gateway/gke-l7-regional-external-managed -n ${NAMESPACE}  
  
# Deploy an Istio gateway.  
kubectl apply -k ./manifests/gateway/istio -n ${NAMESPACE}  
  
# Deploy a kgateway gateway.  
kubectl apply -k ./manifests/gateway/kgateway -n ${NAMESPACE}  
  
# Deploy a kgateway gateway on Openshift Container Platform (OCP).  
kubectl apply -k ./manifests/gateway/kgateway-openshift -n ${NAMESPACE}  
```

--------------------------------

### Configure Precise Prefix-Cache Aware Routing for llm-d

Source: https://context7.com/context7/llm-d_ai/llms.txt

Deploys llm-d with precise prefix-cache aware routing and tests its effectiveness. Requires 'kubectl', 'helmfile', and 'curl'. Verifies cache hits by analyzing scheduler logs.

```bash
# Deploy with precise prefix-cache aware routing
export NAMESPACE=llm-d-precise

cd guides/precise-prefix-cache-aware
helmfile apply -n ${NAMESPACE}

kubectl apply -f httproute.yaml -n ${NAMESPACE}

# Test cache-aware routing with repeated requests
export LONG_TEXT="Lorem ipsum dolor sit amet, consectetur adipiscing elit..."

# First request (cache miss)
curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"Qwen/Qwen3-0.6B\",
    \"prompt\": \"$LONG_TEXT\",
    \"max_tokens\": 50
  }" | jq

# Check scheduler logs for cache scoring
kubectl logs -l inferencepool=gaie-kv-events-epp -n ${NAMESPACE} --tail 100 \
  | grep "Calculated score" \
  | grep "precise-prefix-cache-scorer"

# Second identical request (cache hit - score increases from 0 to 1)
curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d "{
    \"model\": \"Qwen/Qwen3-0.6B\",
    \"prompt\": \"$LONG_TEXT\",
    \"max_tokens\": 50
  }" | jq

# Verify cache hit in logs (score: 1 indicates cached KV-blocks found)
kubectl logs -l inferencepool=gaie-kv-events-epp -n ${NAMESPACE} --tail 100 \
  | grep "precise-prefix-cache-scorer" | tail -2
```

--------------------------------

### Run vLLM Simulator Standalone

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Runs the vLLM simulator executable in a standalone test environment, specifying the model and port. This is useful for local development and testing.

```bash
./bin/llm-d-inference-sim --model my_model --port 8000
```

--------------------------------

### Test vLLM Simulator API with Curl

Source: https://llm-d.ai/docs/architecture/Components/inference-sim

Tests the vLLM simulator's chat completions API endpoint using curl. This command sends a POST request with a sample message to the running simulator.

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '`{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }`'
```
Tokens:
Raw