vLLM Production Stack

https://docs.vllm.ai/projects/production-stack/en/latest/
Admin
vLLM Production Stack is a Kubernetes-native reference architecture for deploying, scaling, and...
Tokens:70,990
Snippets:626
Trust Score:9.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### Example output from Minikube GPU install

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/prerequisite.html

Expected output when the installation succeeds, showing Minikube startup and gpu-operator deployment.

```bash
😄  minikube v1.35.0 on Ubuntu 22.04 (kvm/amd64)
❗  minikube skips various validations when --force is supplied; this may lead to unexpected behavior
✨  Using the docker driver based on user configuration
......
......
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
"nvidia" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
......
......
NAME: gpu-operator-1737507918
LAST DEPLOYED: Wed Jan 22 01:05:21 2025
NAMESPACE: gpu-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None

```

--------------------------------

### Install Minikube cluster with GPU support

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/prerequisite.html

Runs install-minikube-cluster.sh to install Minikube, enable NVIDIA Container Toolkit, start Minikube with GPU support, and install the NVIDIA gpu-operator chart.

```bash
bash install-minikube-cluster.sh

```

--------------------------------

### Install and verify kubectl

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/getting_started/prerequisite.rst

Clone the production-stack repository, navigate to the utils folder, run the install-kubectl.sh script to install kubectl, and verify the installation with kubectl version --client. The expected output shows the client version.

```bash
git clone https://github.com/vllm-project/production-stack.git
cd production-stack/utils
```

```bash
bash install-kubectl.sh
```

```bash
kubectl version --client
```

```text
Client Version: v1.32.1
```

--------------------------------

### Install pre-commit with uv sync and install

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/developer_guide/contributing.rst

Run these commands to set up pre-commit hooks in your development environment. The first command syncs all dependencies and groups, the second installs the pre-commit hooks.

```bash
uv sync --all-extras --all-groups
uv run pre-commit install
```

--------------------------------

### Install KEDA

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html

Create the keda namespace, add the KEDA Helm repository, update the repo, and install KEDA.

```bash
kubectl create namespace keda
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda

```

--------------------------------

### Expected output of kubectl get pods

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/getting_started/quickstart.rst

Example output showing the vLLM deployment pods are Ready and Running. It may take time for containers to download images and weights.

```text
NAME                                               READY   STATUS    RESTARTS   AGE
vllm-deployment-router-859d8fb668-2x2b7        1/1     Running   0          2m38s
vllm-opt125m-deployment-vllm-84dfc9bd7-vb9bs   1/1     Running   0          2m38s
```

--------------------------------

### Install kubectl via script

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/prerequisite.html

Runs install-kubectl.sh to download and install the latest kubectl client and place it in PATH.

```bash
bash install-kubectl.sh

```

--------------------------------

### Run vLLM Templates Setup Script

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/tool-enabled-installation.rst

Makes the setup script executable and runs it to download templates and create Kubernetes resources (PersistentVolume and PersistentVolumeClaim).

```bash
chmod +x scripts/setup_vllm_templates.sh

# Run the setup script
./scripts/setup_vllm_templates.sh
```

--------------------------------

### Run the example script to test tool calling

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/tool-enabled-installation.rst

Navigate to the examples directory and run the Python script tool_calling_example.py. The script connects to the vLLM service via the port-forwarded endpoint, sends a weather query, and demonstrates tool calling.

```bash
# Navigate to the examples directory
cd src/examples

# Run the example script
python tool_calling_example.py
```

--------------------------------

### Add Helm repo and deploy vLLM stack with tool calling

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/tool-enabled-installation.html

Adds the vLLM Helm repository and installs the vLLM stack using the example configuration file tutorials/assets/values-13-tool-enabled.yaml. The deployment uses the PersistentVolume from step 1 and mounts templates at /vllm/templates.

```bash
# Add the vLLM Helm repository if you haven't already
helm repo add vllm https://vllm-project.github.io/production-stack

# Deploy the vLLM stack with tool calling support using the example configuration
helm install vllm-tool vllm/vllm-stack -f tutorials/assets/values-13-tool-enabled.yaml

```

--------------------------------

### Deploy vLLM stack with Helm

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/pipeline-parallelism-kuberay.html

Adds the vLLM Helm repository and installs the vllm-stack chart with a pipeline-parallel example configuration file.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-15-a-minimal-pipeline-parallel-example-raycluster.yaml

```

--------------------------------

### Install and verify Helm

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/getting_started/prerequisite.rst

Run the install-helm.sh script to install Helm, then verify with helm version. The expected output shows the Helm build information.

```bash
bash install-helm.sh
```

```bash
helm version
```

```text
version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"}
```

--------------------------------

### Install Helm via script

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/prerequisite.html

Runs install-helm.sh to download and install the Helm package manager and place its binary in PATH.

```bash
bash install-helm.sh

```

--------------------------------

### Set up Docker, remove old Minikube, and install Minikube with GPU support

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/getting_started/prerequisite.rst

Add the current user to the docker group to run Docker without sudo, optionally uninstall an existing Minikube using the appropriate command for your OS, then run install-minikube-cluster.sh to install Minikube with GPU support and the NVIDIA gpu-operator chart.

```bash
sudo usermod -aG docker $USER && newgrp docker
```

```bash
# Ubuntu / Debian
sudo apt remove minikube

# RHEL / CentOS / Fedora
sudo yum remove minikube
# or
sudo dnf remove minikube

# macOS (installed via Homebrew)
brew uninstall minikube

# Arch Linux
sudo pacman -Rs minikube

# Windows (via Chocolatey)
choco uninstall minikube

# Windows (via Scoop)
scoop uninstall minikube
```

```bash
bash install-minikube-cluster.sh
```

--------------------------------

### Deploy disaggregated prefill with Helm install

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/disaggregated-prefill.rst

Installs the Helm chart with the configuration file, deploying prefill server, decode server, and router.

```bash
helm install pd helm/ -f tutorials/assets/values-16-disagg-prefill.yaml
```

--------------------------------

### Install KEDA

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/autoscaling-keda.rst

Commands to install KEDA in a dedicated namespace using the kedacore Helm chart.

```bash
kubectl create namespace keda
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda
```

--------------------------------

### Deploy vLLM router

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/crd.rst

Applies the VLLMRouter sample configuration to start the vLLM router.

```bash
kubectl apply -f operator/config/samples/production-stack_v1alpha1_vllmrouter.yaml
```

--------------------------------

### Clone repository and navigate to utils

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/prerequisite.html

Clone the production-stack repository and move to the utils/ folder before running installation scripts.

```bash
git clone https://github.com/vllm-project/production-stack.git
cd production-stack/utils

```

--------------------------------

### Deploy the Helm chart with predefined configuration

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/getting_started/quickstart.rst

Commands to add the vLLM Helm repository and install the vllm-stack chart using the predefined configuration file. The first command adds the repository, the second installs the release named 'vllm' with the specified values file.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml
```

--------------------------------

### Apply Jaeger/OTel manifests and install vLLM Helm chart

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/distributed-tracing.rst

Apply the Jaeger and OpenTelemetry collector manifests, then install the vLLM Helm chart with the OTel-enabled values file. After installation, check that the expected pods are running.

```bash
sudo kubectl apply -f otel-example/jaeger.yaml
sudo kubectl apply -f otel-example/jaeger-collector.yaml
sudo kubectl apply -f otel-example/jaeger-query.yaml
sudo kubectl apply -f otel-example/otel-collector-config.yaml
sudo kubectl apply -f otel-example/otel-collector.yaml
sudo helm install vllm ../../helm/ -f values-12-otel-vllm.yaml
```

```bash
sudo kubectl get pods
```

```text
NAME                                               READY   STATUS    RESTARTS   AGE
jaeger-744484b5bc-rdrgn                            1/1     Running   0          13m
otel-collector-859db69dd4-kj8x6                    1/1     Running   0          12m
vllm-deployment-router-6888598c6-m6gl9             1/1     Running   0          12m
vllm-opt125m-deployment-vllm-b489dfd8b-95gb5       1/1     Running   0          12m
```

--------------------------------

### Add Helm repo and install vLLM stack

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/helm.rst

Commands to add the vLLM Helm repository and install the vLLM stack using the prepared configuration file. This deploys the serving engine and router.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-02-basic-config.yaml
```

--------------------------------

### Expected output of Helm install

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/helm.rst

Expected output after running the Helm install command, indicating successful deployment of the chart.

```text
Release "vllm" has been deployed. Happy Helming!
NAME: vllm
LAST DEPLOYED: <timestamp>
NAMESPACE: default
STATUS: deployed
REVISION: 1
```

--------------------------------

### Install agentgateway with Inference Extension

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/gateway-inference-extension.rst

Installs agentgateway v1.4.1 with Inference Extension support enabled via Helm.

```bash
export AGENTGATEWAY_VERSION=v1.4.1

helm upgrade -i --create-namespace \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  agentgateway-crds oci://cr.agentgateway.dev/charts/agentgateway-crds

helm upgrade -i \
  --namespace agentgateway-system \
  --version "${AGENTGATEWAY_VERSION}" \
  --set inferenceExtension.enabled=true \
  agentgateway oci://cr.agentgateway.dev/charts/agentgateway
```

--------------------------------

### Apply Kubernetes resources and install Helm chart

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/distributed-tracing.html

Applies the Jaeger and OpenTelemetry collector configurations, then installs the vLLM Helm chart with the OpenTelemetry values file. Run from the 'tutorials/assets' directory.

```bash
sudo kubectl apply -f otel-example/jaeger.yaml
sudo kubectl apply -f otel-example/jaeger-collector.yaml
sudo kubectl apply -f otel-example/jaeger-query.yaml
sudo kubectl apply -f otel-example/otel-collector-config.yaml
sudo kubectl apply -f otel-example/otel-collector.yaml
sudo helm install vllm ../../helm/ -f values-12-otel-vllm.yaml
```

--------------------------------

### Verify KEDA installation

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/autoscaling-keda.rst

Command to check the KEDA pods in the keda namespace.

```bash
kubectl get pods -n keda
```

--------------------------------

### Expected output for engines list

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/sleep-wakeup-mode.rst

Example JSON response showing an engine with its ID, serving models, and creation timestamp.

```json
[
  {
    "engine_id": "b36921ab-6611-58c0-a941-16c51296446b",
    "serving_models": [
      "ibm-granite/granite-3.0-3b-a800m-instruct"
    ],
    "created": 1750035988
  }
]
```

--------------------------------

### Expected output of engine list

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/sleep-wakeup-mode.html

Example JSON response showing an engine with its ID and serving model.

```json
[
  {
    "engine_id": "b36921ab-6611-58c0-a941-16c51296446b",
    "serving_models": [
      "ibm-granite/granite-3.0-3b-a800m-instruct"
    ],
    "created": 1750035988
  }
]

```

--------------------------------

### Verify deployment with kubectl get

Source: https://docs.vllm.ai/projects/production-stack/en/latest/deployment/gateway-inference-extension.html

Check the status of the gateway, HTTPRoute, InferencePool, and deployment resources created by the extension.

```bash
kubectl get gateway inference-gateway
kubectl get httproute vllm-llama3-1b-instruct
kubectl get inferencepool vllm-llama3-1b-instruct
kubectl get deployment vllm-llama3-1b-instruct-epp

```

--------------------------------

### Expected ScaledObject output

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html

Example output showing the created ScaledObject for the vllm-llama3 deployment.

```text
NAME                        SCALETARGETKIND      SCALETARGETNAME                  MIN   MAX   TRIGGERS     AUTHENTICATION   READY   ACTIVE   FALLBACK   PAUSED    AGE
vllm-llama3-scaledobject   apps/v1.Deployment   vllm-llama3-deployment-vllm      1     3     prometheus                    True    False    Unknown    Unknown   30s

```

--------------------------------

### Install vLLM Helm chart

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/autoscaling-keda.rst

Command to deploy the vLLM Production Stack Helm chart with the provided values.yaml file.

```bash
helm install vllm vllm/vllm-stack -f values.yaml
```

--------------------------------

### Deploy Helm chart with sleep mode enabled

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/sleep-wakeup-mode.rst

Installs the vLLM Helm chart using the predefined configuration file values-19-sleep-mode-aware.yaml. Wait for the deployment to complete with kubectl get pods -w.

```bash
helm install vllm helm/ -f tutorials/assets/values-19-sleep-mode-aware.yaml
```

```bash
kubectl get pods -w
```

--------------------------------

### Deploy vLLM runtime

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/crd.rst

Applies the VLLMRuntime sample configuration to create a vLLM runtime instance in the cluster.

```bash
kubectl apply -f operator/config/samples/production-stack_v1alpha1_vllmruntime.yaml
```

--------------------------------

### Deploy vLLM with predefined configuration values-01-minimal-example.yaml

Source: https://docs.vllm.ai/projects/production-stack/en/latest/getting_started/quickstart.html

Defines a minimal vLLM serving engine configuration with the opt125m model, one replica, and resource requests. Optional CPU and memory limits are commented out. Use this file with the Helm chart to deploy a single vLLM pod.

```yaml
servingEngineSpec:
  runtimeClassName: ""
  modelSpec:
  - name: "opt125m"
    repository: "vllm/vllm-openai"
    tag: "latest"
    modelURL: "facebook/opt-125m"

    replicaCount: 1

    requestCPU: 6
    requestMemory: "16Gi"
    requestGPU: 1
    # Optional resource limits - if not specified, only GPU will have a limit
    # limitCPU: "8"
    # limitMemory: "32Gi"

```

--------------------------------

### Deploy vLLM Production Stack with Helm

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/semantic-router-integration.rst

Adds the vLLM Production Stack Helm repository and installs the stack using a sample values file that configures Qwen/Qwen3-8B with 2 replicas, round-robin routing, and 8 CPU/16Gi memory/1 GPU per instance.

```bash
helm repo add vllm-production-stack https://vllm-project.github.io/production-stack
helm install vllm-stack vllm-production-stack/vllm-stack -f https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-23-SR.yaml
```

--------------------------------

### Install Gateway API and Inference Extension CRDs

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/gateway-inference-extension.rst

Installs the pinned Gateway API v1.6.0 and Inference Extension v1.5.0 CRDs using server-side apply.

```bash
export GATEWAY_API_VERSION=v1.6.0
export INFERENCE_EXTENSION_VERSION=v1.5.0

kubectl apply --server-side -f \
  "https://github.com/kubernetes-sigs/gateway-api/releases/download/${GATEWAY_API_VERSION}/standard-install.yaml"

kubectl apply --server-side -f \
  "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${INFERENCE_EXTENSION_VERSION}/manifests.yaml"
```

--------------------------------

### Apply LLM demo backends and AI Gateway routes

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/semantic-router-integration.rst

Applies the base model and gateway API resources from the vLLM semantic-router repository. Run these commands to create the demo backends and routes for the semantic router.

```bash
# Apply LLM demo backends
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/base-model.yaml
```

```bash
# Apply AI Gateway routes
kubectl apply -f https://raw.githubusercontent.com/vllm-project/semantic-router/refs/heads/main/deploy/kubernetes/ai-gateway/aigw-resources/gwapi-resources.yaml
```

--------------------------------

### Troubleshoot with kubectl describe and logs

Source: https://docs.vllm.ai/projects/production-stack/en/latest/deployment/gateway-inference-extension.html

Inspect status conditions and component logs for troubleshooting the deployment.

```bash
kubectl describe gateway inference-gateway
kubectl describe httproute vllm-llama3-1b-instruct
kubectl logs deployment/vllm-llama3-1b-instruct-epp
kubectl logs -n agentgateway-system deployment/agentgateway

```

--------------------------------

### Install llm-d Router

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/gateway-inference-extension.rst

Installs the llm-d Router Gateway chart, which creates the InferencePool, EPP Deployment and Service, RBAC, and an HTTPRoute attached to inference-gateway. The values set provider.name=none and select vLLM pods via router.modelServers.matchLabels.

```bash
export LLM_D_ROUTER_VERSION=v0.9.0

helm upgrade -i vllm-llama3-1b-instruct \
  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \
  --version "${LLM_D_ROUTER_VERSION}" \
  -f src/gateway_inference_extension/configs/llm-d-router-values.yaml

kubectl rollout status deployment/vllm-llama3-1b-instruct-epp \
  --timeout=120s
```

--------------------------------

### Verify deployment resources

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/gateway-inference-extension.rst

Verifies the deployment by listing the Gateway, HTTPRoute, InferencePool, and EPP Deployment.

```bash
kubectl get gateway inference-gateway
kubectl get httproute vllm-llama3-1b-instruct
kubectl get inferencepool vllm-llama3-1b-instruct
kubectl get deployment vllm-llama3-1b-instruct-epp
```

--------------------------------

### View the created HPA

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html

Get the HorizontalPodAutoscaler created by KEDA.

```bash
kubectl get hpa

```

--------------------------------

### Expected HPA output

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/autoscaling-keda.html

Example output showing the HPA created by KEDA.

```text
NAME                            REFERENCE                                TARGETS     MINPODS   MAXPODS   REPLICAS
keda-hpa-vllm-llama3-scaledobject   Deployment/vllm-llama3-deployment-vllm   0/5 (avg)   1         3         1

```

--------------------------------

### Expected output from the example script

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/tool-enabled-installation.rst

The expected output shows the function call to get_weather with arguments and the result. This confirms the vLLM service, tool calling, and template system are working correctly.

```text
Function called: get_weather
Arguments: {"location": "San Francisco, CA", "unit": "celsius"}
Result: Getting the weather for San Francisco, CA in celsius...
```

--------------------------------

### Example configuration for tool calling (values-13-tool-enabled.yaml)

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/tool-enabled-installation.html

Helm values file enabling tool calling for Llama-3.1-8B-Instruct. Sets enableTool: true, toolCallParser: llama3_json, chatTemplate: tool_chat_template_llama3.1_json.jinja, mounts Hugging Face credentials, and requests 8 CPU, 32Gi memory, and 1 GPU.

```yaml
servingEngineSpec:
  runtimeClassName: ""
  modelSpec:
  - name: "llama3-8b"
    repository: "vllm/vllm-openai"
    tag: "latest"
    modelURL: "meta-llama/Llama-3.1-8B-Instruct"

    # Tool calling configuration
    enableTool: true
    toolCallParser: "llama3_json"  # Parser to use for tool calls (e.g., "llama3_json" for Llama models)
    chatTemplate: "tool_chat_template_llama3.1_json.jinja"  # Template file name (will be mounted at /vllm/templates)

    # Mount Hugging Face credentials
    env:
      - name: HUGGING_FACE_HUB_TOKEN
        valueFrom:
          secretKeyRef:
            name: huggingface-credentials
            key: HUGGING_FACE_HUB_TOKEN

    replicaCount: 1

    # Resource requirements for Llama-3.1-8B-Instruct
    requestCPU: 8
    requestMemory: "32Gi"
    requestGPU: 1

```

--------------------------------

### Router Decision Log

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/loadaware-routing.rst

Example of the log output generated by the router when making a routing decision.

```text
Routing request to http://... found by loadaware router
```

--------------------------------

### Create values-16-disagg-prefill.yaml configuration

Source: https://docs.vllm.ai/projects/production-stack/en/latest/use_cases/disaggregated-prefill.html

Create this file as values-16-disagg-prefill.yaml. It defines the prefill and decode engine specs with NIXL communication settings (kv_producer/sender and kv_consumer/receiver) and the router configuration with routingLogic 'disaggregated_prefill'. Replace <your-hf-token> with your Hugging Face token.

```yaml
# Unified configuration for disaggregated prefill setup
servingEngineSpec:
  enableEngine: true
  runtimeClassName: ""
  containerPort: 8000
  modelSpec:
    # Prefill node configuration
    - name: "llama-prefill"
      repository: "lmcache/vllm-openai"
      tag: "2025-05-17-v1"
      modelURL: "meta-llama/Llama-3.1-8B-Instruct"
      replicaCount: 1
      requestCPU: 8
      requestMemory: "30Gi"
      requestGPU: 1
      pvcStorage: "50Gi"
      vllmConfig:
        enableChunkedPrefill: false
        enablePrefixCaching: false
        maxModelLen: 32000
        v1: 1
      lmcacheConfig:
        enabled: true
        kvRole: "kv_producer"
        enableNixl: true
        nixlRole: "sender"
        nixlPeerHost: "pd-llama-decode-engine-service"
        nixlPeerPort: "55555"
        nixlBufferSize: "1073741824"  # 1GB
        nixlBufferDevice: "cuda"
        nixlEnableGc: true
        enablePD: true
      hf_token: <your-hf-token>
      labels:
        model: "llama-prefill"
    # Decode node configuration
    - name: "llama-decode"
      repository: "lmcache/vllm-openai"
      tag: "2025-05-17-v1"
      modelURL: "meta-llama/Llama-3.1-8B-Instruct"
      replicaCount: 1
      requestCPU: 8
      requestMemory: "30Gi"
      requestGPU: 1
      pvcStorage: "50Gi"
      vllmConfig:
        enableChunkedPrefill: false
        enablePrefixCaching: false
        maxModelLen: 32000
        v1: 1
      lmcacheConfig:
        enabled: true
        kvRole: "kv_consumer"
        enableNixl: true
        nixlRole: "receiver"
        nixlPeerHost: "0.0.0.0"
        nixlPeerPort: "55555"
        nixlBufferSize: "1073741824"  # 1GB
        nixlBufferDevice: "cuda"
        nixlEnableGc: true
        enablePD: true
      hf_token: <your-hf-token>
      labels:
        model: "llama-decode"
routerSpec:
  enableRouter: true
  repository: "lmcache/lmstack-router"
  tag: "pd-05-26"
  replicaCount: 1
  containerPort: 8000
  servicePort: 80
  routingLogic: "disaggregated_prefill"
  engineScrapeInterval: 15
  requestStatsWindow: 60
  enablePD: true
  resources:
    requests:
      cpu: "4"
      memory: "16G"
    limits:
      cpu: "4"
      memory: "32G"
  labels:
    environment: "router"
    release: "router"
  extraArgs:
    - "--prefill-model-labels"
    - "llama-prefill"
    - "--decode-model-labels"
    - "llama-decode"

```

--------------------------------

### Wait for deployment to complete

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/kv-cache-aware-routing.rst

Watches the Kubernetes pods until the deployment is ready. Use this after installing the Helm chart.

```bash
kubectl get pods -w
```

--------------------------------

### Get Kubernetes Services

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/use_cases/tool-enabled-installation.rst

Lists Kubernetes services to identify the router service name for port forwarding.

```bash
# Get the service name
kubectl get svc
```

--------------------------------

### Deploy vLLM model servers

Source: https://docs.vllm.ai/projects/production-stack/en/latest/_sources/deployment/gateway-inference-extension.rst

Deploys two replicas of meta-llama/Llama-3.2-1B-Instruct labeled with app: vllm-llama3-1b-instruct and waits for rollout. Adjust GPU resources, replica count, and model-server arguments for production.

```bash
kubectl apply -f \
  src/gateway_inference_extension/configs/vllm/gpu-deployment.yaml

kubectl rollout status deployment/vllm-llama3-1b-instruct \
  --timeout=15m
```
Tokens:
Raw