### Environment Variable Configuration Example

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/eks/README.md

Example of how to set environment variables for EKS cluster basics, NVIDIA setup, vLLM inference, and node-group sizing.

```bash
# Copy and customize
$ cp env-vars.template env-vars
$ vi env-vars
################################################################################
# EKS Cluster Configuration
################################################################################
# ☸️ EKS cluster basics
export TF_VAR_cluster_name="vllm-eks-prod" # default: "vllm-eks-prod"
export TF_VAR_cluster_version="1.30"       # default: "1.30" - Kubernetes cluster version
 ################################################################################
 # 🤖 NVIDIA setup selector
 #   • plugin           -> device-plugin only
 #   • operator_no_driver -> GPU Operator (driver disabled)
 #   • operator_custom  -> GPU Operator with your YAML
 ################################################################################
export TF_VAR_nvidia_setup="plugin" # default: "plugin"
################################################################################
# 🧠 LLM Inference Configuration
################################################################################
export TF_VAR_enable_vllm="true"         # default: "false" - Set to "true" to deploy vLLM
export TF_VAR_hf_token=""                # default: "" - Hugging Face token for model download (if needed)
export TF_VAR_inference_hardware="gpu"   # default: "cpu" - "cpu" or "gpu"
################################################################################
export TF_VAR_nvidia_setup="plugin" # default: ""
# Paths to Helm chart values templates for vLLM.
# These paths are relative to the root of your Terraform project.
export TF_VAR_gpu_vllm_helm_config="./modules/llm-stack/helm/gpu/gpu-tinyllama-light-ingress.tpl" # default: ""
export TF_VAR_cpu_vllm_helm_config="./modules/llm-stack/helm/cpu/cpu-tinyllama-light-ingress.tpl" # default: ""
################################################################################
# ⚙️ Node-group sizing
################################################################################
# CPU pool (always present)
export TF_VAR_cpu_node_min_size="1"     # default: 1
export TF_VAR_cpu_node_max_size="3"     # default: 3
export TF_VAR_cpu_node_desired_size="2" # default: 2
# GPU pool (ignored unless inference_hardware = "gpu")
export TF_VAR_gpu_node_min_size="1"     # default: 1
export TF_VAR_gpu_node_max_size="1"     # default: 1
export TF_VAR_gpu_node_desired_size="1" # default: 1
...snip
$ source env-vars
```

--------------------------------

### Example Router Usage

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/23-whisper-api-transcription.md

Execute the router script with the desired port and backend URL. This command starts the router, making it ready to receive requests.

```bash
./run-router.sh 8000 http://0.0.0.0:8002
```

--------------------------------

### Execute NVIDIA GPU Setup Script

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

Run the init-nvidia-gpu-setup-k8s.sh script to configure the system for GPU workloads and install the NVIDIA gpu-operator.

```bash
bash init-nvidia-gpu-setup-k8s.sh
```

--------------------------------

### Build and Install Helm Chart

Source: https://github.com/vllm-project/production-stack/blob/main/helm/README.md

Builds Helm chart dependencies and installs the 'llmstack' release using an example values file.

```bash
helm dependency build
helm install llmstack . -f values-example.yaml
```

--------------------------------

### Install Documentation Prerequisites

Source: https://github.com/vllm-project/production-stack/blob/main/docs/README.md

Installs necessary Python packages for building documentation. Ensure you have a virtual environment activated.

```bash
pip install -r requirements-docs.txt
```

--------------------------------

### Verify kind Installation

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-c-install-kind-kubernetes-env.md

Check the installed version of kind to confirm successful installation.

```bash
kind version
```

--------------------------------

### Install Observability Stack

Source: https://github.com/vllm-project/production-stack/blob/main/observability/README.md

Run this script to install the observability stack. Ensure you have a Kubernetes environment with GPUs set up beforehand.

```bash
bash install.sh
```

--------------------------------

### Make Setup Script Executable

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/tool-enabled-installation.rst

Ensures the vLLM setup script has execute permissions before running it.

```bash
chmod +x scripts/setup_vllm_templates.sh
```

--------------------------------

### Install kind using install-kind.sh

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-c-install-kind-kubernetes-env.md

Execute the install-kind.sh script to download and install the latest version of kind, making it available in your PATH.

```bash
bash install-kind.sh
```

--------------------------------

### Execute Kubeadm Installation Script

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

Run the install-kubeadm.sh script to download and install kubeadm, kubectl, and kubelet on the current node.

```bash
bash install-kubeadm.sh
```

--------------------------------

### Execute Minikube Installation Script

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

This script installs Minikube, configures GPU support, and installs the NVIDIA gpu-operator.

```bash
bash install-minikube-cluster.sh
```

--------------------------------

### Install vLLM with Audio Support

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/23-whisper-api-transcription.md

Install vLLM with the necessary audio dependencies. This is a prerequisite for serving audio models.

```bash
pip install vllm[audio]
```

--------------------------------

### Execute GKE vLLM Stack Setup Script

Source: https://github.com/vllm-project/production-stack/blob/main/deployment_on_cloud/gcp/README.md

Run the main script to configure the GKE LLM inference cluster. Ensure your GCP CLI is set up, logged in, and the region is configured. You also need kubectl and helm installed.

```bash
bash entry_point_basic.sh YAML_FILE_PATH
```

--------------------------------

### Deploy AI Inference Stack with Helm

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/cloud_deployments/01-AWS-EKS-deployment.md

Adds the vLLM Helm repository and installs the AI inference stack using a specified setup YAML file.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm ./vllm-stack -f $SETUP_YAML
```

--------------------------------

### Install kubectl

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

Execute the install-kubectl.sh script to download and install the latest version of kubectl.

```bash
bash install-kubectl.sh
```

--------------------------------

### Dynamic JSON Configuration Example

Source: https://github.com/vllm-project/production-stack/blob/main/src/vllm_router/README.md

Example of a dynamic configuration file in JSON format. This configuration mirrors the YAML example, using static service discovery and round-robin routing with comma-separated values for backends, models, and types.

```json
{
    "service_discovery": "static",
    "routing_logic": "roundrobin",
    "callbacks": "module.custom.callback_handler",
    "static_backends": "http://localhost:9001,http://localhost:9002,http://localhost:9003",
    "static_models": "facebook/opt-125m,meta-llama/Llama-3.1-8B-Instruct,facebook/opt-125m",
    "static_model_types": "completion,chat,completion",
    "static_aliases": "my-alias:meta-llama/Llama-3.1-8B-Instruct,my-other-alias:meta-llama/Llama-3.1-8B-Instruct"
}
```

--------------------------------

### Install vLLM Helm Chart

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/01-b-minimal-helm-installation-amd.md

Installs the vLLM stack using a specified configuration file. Ensure the Helm repository is added before installation.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-amd-example.yaml
```

--------------------------------

### Install Benchmark Dependencies

Source: https://github.com/vllm-project/production-stack/blob/main/benchmarks/multi-round-qa/README.md

Installs the necessary Python packages for running the benchmark. Ensure you have a `requirements.txt` file in your project.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install KEDA

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/autoscaling-keda.rst

Install KEDA into a dedicated namespace using Helm. This prepares the cluster for autoscaling.

```bash
kubectl create namespace keda
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --namespace keda
```

--------------------------------

### Example Kubeadm Version Output

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

This is an example of the expected output when verifying the kubeadm version.

```plaintext
kubeadm version: &version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.4", GitCommit:"59526cd4867447956156ae3a602fcbac10a2c335", GitTreeState:"clean", BuildDate:"2025-04-22T16:02:27Z", GoVersion:"go1.23.6", Compiler:"gc", Platform:"linux/amd64"}
```

--------------------------------

### Tool Calling Example Output

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/tool-enabled-installation.rst

Example output demonstrating a successful tool call and response from the vLLM model. This output confirms that tool calling is configured and working correctly.

```text
Function called: get_weather
Arguments: {"location": "San Francisco, CA", "unit": "celsius"}
Result: Getting the weather for San Francisco, CA in celsius...
```

--------------------------------

### Install Observability Stack

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/10-horizontal-autoscaling.md

Installs the Prometheus and Grafana monitoring tools necessary for exporting vLLM metrics.

```bash
# Navigate to the observability directory
cd production-stack/observability

# Install the observability stack
sudo bash install.sh
```

--------------------------------

### Install OCI CLI, kubectl, Helm, and jq

Source: https://github.com/vllm-project/production-stack/blob/main/deployment_on_cloud/oci/README.md

Installs essential command-line tools for OCI management, Kubernetes interaction, Helm package management, and JSON processing. Use 'brew' on macOS or 'pip' for OCI CLI installation.

```bash
# OCI CLI - https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm
brew install oci-cli  # macOS
# or
pip install oci-cli

# Kubernetes CLI
brew install kubectl  # macOS
# or see https://kubernetes.io/docs/tasks/tools/

# Helm
brew install helm  # macOS
# or see https://helm.sh/docs/intro/install/

# jq (JSON parsing)
brew install jq  # macOS

```

--------------------------------

### Install Helm

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

Execute the install-helm.sh script to download and install the Helm package manager for Kubernetes.

```bash
bash install-helm.sh
```

--------------------------------

### Remove Example Certificates

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/22-gateway-api-for-ingress.md

Delete the local directory containing example certificates.

```bash
rm -r example_certs/
```

--------------------------------

### Install kind Cluster with GPU Support

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-c-install-kind-kubernetes-env.md

Execute the install-kind-cluster.sh script to set up a kind cluster with GPU support, configuring NVIDIA container toolkit and installing the gpu-operator.

```bash
bash install-kind-cluster.sh
```

--------------------------------

### Example Operator Deployment Status Output

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/deployment/crd.rst

Example output showing the status of the operator deployment. A '1/1' ready status confirms the deployment is healthy.

```bash
NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
production-stack-production-stack-controller-manager   1/1     1            1           25s
```

--------------------------------

### Execute CRI-O Installation Script

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

Run the install-cri-o.sh script to install the CRI-O container runtime on the current node.

```bash
bash install-cri-o.sh
```

--------------------------------

### Install Naive Kubernetes Stack

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/07-benchmark-multi-round-qa-single-gpu.md

Install the vLLM Production Stack using Helm for a naive Kubernetes deployment. Ensure the Helm repository is added first.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f naive.yaml
```

--------------------------------

### Install Kgateway with Inference Extension

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/deployment/gateway-inference-extension.rst

Installs or upgrades Kgateway using Helm, enabling the inference extension.

```bash
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
```

--------------------------------

### Run vLLM Production Stack Setup

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/cloud_deployments/05-OCI-OKE-deployment.md

Execute the setup script for the vLLM production stack on OCI. Ensure you are in the correct directory and have set the OCI_COMPARTMENT_ID environment variable.

```bash
cd deployment_on_cloud/oci
export OCI_COMPARTMENT_ID="ocid1.compartment.oc1..xxxxx"
./entry_point.sh setup ./production_stack_specification.yaml
```

--------------------------------

### Install and Run Pre-commit Hooks

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/developer_guide/contributing.rst

Install pre-commit and its hooks to format code automatically before committing. Run pre-commit manually on all files to ensure code quality.

```bash
uv sync --all-extras --all-groups
uv run pre-commit install
```

```bash
uv run pre-commit run --all-files
```

--------------------------------

### Install Dependencies and Run Python Script

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/04-launch-multiple-model.md

Install the necessary OpenAI Python library and execute the provided script to query the deployed models. This is a prerequisite for running the Python code snippet.

```bash
pip install openai
python3 tutorials/assets/example-04-openai.py
```

--------------------------------

### Install KGateway with Inference Extension

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/21-gateway-inference-extension.md

Install KGateway with the inference extension enabled. This command deploys KGateway and configures it for inference tasks.

```bash
# Install KGateway with inference extension enabled
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
```

--------------------------------

### Example vLLM Runtime and Router Pod Status

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/deployment/crd.rst

Example output showing the status of vLLM router and runtime pods. A 'Running' status for both indicates a successful deployment.

```bash
NAME                                  READY   STATUS    RESTARTS   AGE
vllmrouter-sample-6fc78b7f85-lt5n7    1/1     Running   0          3m31s
vllmruntime-sample-7448f7547c-pdfml   1/1     Running   0          6m10s
```

--------------------------------

### Install Tools on Ubuntu/Debian

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/eks/README.md

Installs necessary tools including Terraform, AWS CLI v2, kubectl, and Helm. Ensure you have sudo privileges.

```bash
# Install tools
sudo apt update && sudo apt install -y jq curl unzip gpg
wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y terraform
curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip -q awscliv2.zip && sudo ./aws/install && rm -rf aws awscliv2.zip
curl -sLO "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" && sudo install kubectl /usr/local/bin/ && rm kubectl
curl -s https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg >/dev/null && echo "deb [signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm.list && sudo apt update && sudo apt install -y helm
```

--------------------------------

### Install vLLM Stack with Tool Calling

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/tool-enabled-installation.rst

Deploys the vLLM stack with tool calling support using a specified values file. This command installs the necessary components for tool-enabled inference.

```bash
helm install vllm-tool vllm/vllm-stack -f tutorials/assets/values-08-tool-enabled.yaml
```

--------------------------------

### Example Pod Status Output

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/01-b-minimal-helm-installation-amd.md

This is an example of the expected output when checking pod status, indicating that the vLLM deployment is running.

```plaintext
NAME                                               READY   STATUS    RESTARTS   AGE
vllm-deployment-router-74dd87b649-gh754        1/1     Running   0          45m
vllm-opt125m-deployment-vllm-7b5494fb4c-qbsxr  1/1     Running   0          42m
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/vllm-project/production-stack/blob/main/CONTRIBUTING.md

Installs pre-commit hooks to ensure code formatting and quality checks before committing. Run this after syncing dependencies.

```bash
uv sync --all-extras --all-groups
uv run pre-commit install
```

--------------------------------

### Add Helm Repository and Install vLLM Stack

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/01-minimal-helm-installation.md

Installs the vLLM Helm chart using a predefined configuration file. Ensure the Helm repository is added first.

```bash
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml

```

--------------------------------

### Install vLLM Production Stack

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/semantic-router-integration.rst

Installs the vLLM Production Stack using Helm, with a specified values file for configuration. The sample values file configures the model, router, and resources.

```bash
helm install vllm-stack vllm-production-stack/vllm-stack -f https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-23-SR.yaml
```

--------------------------------

### Install KGateway Controller

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/22-gateway-api-for-ingress.md

Installs the KGateway controller and its CRDs using Helm. This includes enabling the inference extension. Ensure the KGTW_VERSION is set correctly.

```bash
KGTW_VERSION=v2.0.3

# Install KGateway CRDs
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds

# Install KGateway with inference extension enabled
helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway
```

--------------------------------

### Observe Faster Startup with PV

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/03-load-model-from-pv.md

Demonstrates the performance benefit by uninstalling and reinstalling the deployment, showing a faster startup time due to model weights being pre-loaded on the Persistent Volume.

```bash
helm uninstall vllm
kubectl delete -f tutorials/assets/pv-03.yaml && kubectl apply -f tutorials/assets/pv-03.yaml
helm install vllm vllm/vllm-stack -f tutorials/assets/values-03-match-pv.yaml
```

--------------------------------

### Quick Start Deployment Script

Source: https://github.com/vllm-project/production-stack/blob/main/deployment_on_cloud/oci/README.md

Execute these commands sequentially to set up, establish a tunnel, and deploy vLLM after manual testing. Ensure environment variables are exported before running the script.

```bash
export OCI_PROFILE="your-profile"
export OCI_COMPARTMENT_ID="ocid1.compartment.oc1..xxx"
export OCI_REGION="us-ashburn-1"
export CLUSTER_NAME="vllm-production"
export GPU_AD_INDEX="1"  # Check GPU availability first!

# Step 1: Create infrastructure
./entry_point.sh setup

# Step 2: Start tunnel (in a SEPARATE terminal — auto-reconnects)
./entry_point.sh tunnel

# Step 3: Deploy vLLM (back in original terminal)
./entry_point.sh deploy-vllm
```

--------------------------------

### Clone Production Stack and Install Helm Chart

Source: https://github.com/vllm-project/production-stack/blob/main/README.md

Clone the production-stack repository and install the vLLM stack using Helm with a minimal configuration. This sets up the stack with an OpenAI API interface.

```bash
git clone https://github.com/vllm-project/production-stack.git
cd production-stack/
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/vllm-project/production-stack/blob/main/docs/README.md

Starts a local HTTP server to serve the built HTML documentation. Access it via your browser at http://localhost:8000.

```bash
python -m http.server 8000 -d build/html/
```

--------------------------------

### Example HPA Status Output

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/10-horizontal-autoscaling.md

Displays the typical output of `kubectl get hpa` command, showing the current state of the autoscaler, including replica counts and metric targets.

```plaintext
NAME       REFERENCE                                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
vllm-hpa   Deployment/vllm-llama3-deployment-vllm   0/1       1         2         1          34s
```

--------------------------------

### Get Current Dynamic Config via Health Endpoint

Source: https://github.com/vllm-project/production-stack/blob/main/src/vllm_router/README.md

Example of how to retrieve the current dynamic configuration of the vLLM router using a cURL command. The response includes the router's status and the current dynamic configuration object.

```bash
curl http://<router_host>:<router_port>/health
```

```json
{
    "status": "healthy",
    "dynamic_config": <current_dynamic_config (JSON object)>
}
```

--------------------------------

### Verify Kubeadm Installation

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

Check the installed version of kubeadm to confirm successful installation.

```bash
kubeadm version
```

--------------------------------

### Run Benchmarking for Naive Kubernetes

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/07-benchmark-multi-round-qa-single-gpu.md

Execute warmup and main benchmarking scripts for the naive Kubernetes setup. Specify the model and the local endpoint.

```bash
bash warmup_single.sh meta-llama/Llama-3.1-8B-Instruct http://localhost:30080/v1/
bash run_single.sh meta-llama/Llama-3.1-8B-Instruct http://localhost:30080/v1/ naive
```

--------------------------------

### Verify Helm Installation

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

Check the installed version of Helm to confirm successful installation.

```bash
helm version
```

--------------------------------

### Clone Repository and Navigate

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/eks/README.md

Clone the production-stack repository and navigate to the Terraform EKS tutorial directory.

```bash
git clone https://github.com/vllm-project/production-stack
cd production-stack/tutorials/terraform/eks/
```

--------------------------------

### Install KGateway CRDs

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/21-gateway-inference-extension.md

Install the Custom Resource Definitions for KGateway. This is a prerequisite for installing KGateway itself.

```bash
# Install KGateway CRDs
KGTW_VERSION=v2.0.2
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
```

--------------------------------

### Run vLLM Setup Script

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/tool-enabled-installation.rst

Executes the script to download templates and create necessary Kubernetes resources like PersistentVolumes.

```bash
./scripts/setup_vllm_templates.sh
```

--------------------------------

### Verify kubectl Installation

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

Check the installed client version of kubectl to confirm successful installation.

```bash
kubectl version --client
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/eks/README.md

Copy the template, customize it with your Hugging Face token and deployment options, and then source the file.

```bash
cp env-vars.template env-vars
vim env-vars  # Set HF token and customize deployment options
source env-vars
```

--------------------------------

### Initialize and Apply GKE Infrastructure with Terraform

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/gke/README.md

Navigate to the gke-infrastructure directory, initialize Terraform, and then apply the configuration to create the GKE infrastructure.

```bash
cd gke-infrastructure
terraform init
terraform apply
```

--------------------------------

### Deploy Infrastructure with Terraform

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/coreweave/README.md

Initialize Terraform, preview the deployment plan, and apply the changes to provision the infrastructure.

```bash
terraform init
terraform plan
terraform apply

```

--------------------------------

### Install vLLM Router

Source: https://github.com/vllm-project/production-stack/blob/main/src/vllm_router/README.md

Installs the vLLM router in editable mode. This is useful for development.

```bash
pip install -e .

```

--------------------------------

### Deploy Everything with Makefile

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/aks/README.md

Use this command to provision the Azure cluster, node pools, and deploy the vLLM stack in a single step.

```bash
make create
```

--------------------------------

### Execute GKE vLLM Stack Setup Script

Source: https://github.com/vllm-project/production-stack/blob/main/deployment_on_cloud/gcp/OPT125_CPU/README.md

Run the entrypoint script to configure the GKE LLM inference cluster. Ensure you have modified the production_stack_specification.yaml file before execution.

```bash
bash entrypoint_ql.sh YAML_FILE_PATH
```

--------------------------------

### LMCache Initialization Log Example (Text)

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/sharing-kv-cache.rst

Example log output indicating LMCache initialization and configuration for KV cache storage. This confirms that the LMCache connector is active and ready to manage KV caches.

```text
INFO 01-21 20:16:58 lmcache_connector.py:41] Initializing LMCacheConfig under kv_transfer_config kv_connector='LMCacheConnector' kv_buffer_device='cuda' kv_buffer_size=1000000000.0 kv_role='kv_both' kv_rank=None kv_parallel_size=1 kv_ip='127.0.0.1' kv_port=14579
INFO LMCache: Creating LMCacheEngine instance vllm-instance [2025-01-21 20:16:58,732] -- /usr/local/lib/python3.12/dist-packages/lmcache/experimental/cache_engine.py:237
```

--------------------------------

### Install Minikube Cluster

Source: https://github.com/vllm-project/production-stack/blob/main/README.md

Use this script to set up a local Kubernetes cluster with GPUs using Minikube. Ensure you are in the 'utils' directory before running.

```bash
cd utils && bash install-minikube-cluster.sh
```

--------------------------------

### Example Output of gpu-operator Deployment

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

This is an example of the expected output when the NVIDIA gpu-operator chart is successfully deployed.

```plaintext
...
NAME: gpu-operator-1737507918
LAST DEPLOYED: Wed Jan 22 01:05:21 2025
NAMESPACE: gpu-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None
```

--------------------------------

### Example CRI-O Service Status Output

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/00-a-install-multinode-kubernetes-env.md

This is an example of the expected output when checking the status of the crio service.

```plaintext
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/lib/systemd/system/crio.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2025-05-16 16:32:31 UTC; 20h ago
     Docs: https://github.com/cri-o/cri-o
    Main PID: 2332175 (crio)
       Tasks: 61
      Memory: 14.4G
         CPU: 17min 55.486s
      CGroup: /system.slice/crio.service
```

--------------------------------

### Install Nebius CLI Tool

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/nebius/README.md

Installs the Nebius CLI tool and configures bash auto-completion on Ubuntu/Debian systems.

```bash
# Install tools
sudo apt-get install jq
curl -sSL https://storage.eu-north1.nebius.cloud/cli/install.sh | bash
###### Auto completion
nebius completion bash > ~/.nebius/completion.bash.inc
echo 'if [ -f ~/.nebius/completion.bash.inc ]; then source ~/.nebius/completion.bash.inc; fi' >> ~/.bashrc
source ~/.bashrc

```

--------------------------------

### Initialize and Apply Terraform for Azure Infrastructure

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/terraform/aks/README.md

Initialize Terraform to download providers and then apply the configuration to create the Azure infrastructure.

```bash
cd azure-infrastructure
terraform init
terraform apply
```

--------------------------------

### Clone Repository and Navigate to Utils

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/getting_started/prerequisite.rst

Clone the production-stack repository and change the directory to the utils folder to access installation scripts.

```bash
git clone https://github.com/vllm-project/production-stack.git
cd production-stack/utils
```

--------------------------------

### Install vLLM Router with Semantic Cache

Source: https://github.com/vllm-project/production-stack/blob/main/src/vllm_router/README.md

Installs the vLLM router with support for the semantic cache. This requires additional dependencies.

```bash
pip install -e .[semantic_cache]

```

--------------------------------

### Install vLLM Helm Chart

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/autoscaling-keda.rst

Install the vLLM Production Stack Helm chart using a custom values.yaml file.

```bash
helm install vllm vllm/vllm-stack -f values.yaml
```

--------------------------------

### Install Kgateway CRDs

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/deployment/gateway-inference-extension.rst

Installs the Kgateway Custom Resource Definitions using Helm. Ensure you use the correct version.

```bash
# Install Kgateway CRDs
KGTW_VERSION=v2.0.2
helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
```

--------------------------------

### Launch Mock OpenAI Server

Source: https://github.com/vllm-project/production-stack/blob/main/src/tests/README.md

Use this script to start multiple instances of a mock OpenAI API server for performance testing. The arguments specify the number of servers and the port offset.

```bash
bash run-multi-server.sh 4 500
```

--------------------------------

### Install Envoy AI Gateway

Source: https://github.com/vllm-project/production-stack/blob/main/docs/source/use_cases/semantic-router-integration.rst

Installs or upgrades the Envoy AI Gateway using Helm from a specified OCI registry and version.

```bash
# Install Envoy AI Gateway
helm upgrade -i aieg oci://docker.io/envoyproxy/ai-gateway-helm \
  --version v0.0.0-latest \
  --namespace envoy-ai-gateway-system \
  --create-namespace
```

--------------------------------

### Run Benchmark with Naive Kubernetes

Source: https://github.com/vllm-project/production-stack/blob/main/tutorials/08-benchmark-multi-round-qa-multi-gpu.md

Bash script to initiate benchmarking against the naive Kubernetes deployment. Requires the model name and the service endpoint.

```bash
bash run.sh meta-llama/Llama-3.1-8B-Instruct http://localhost:30080/v1/ native
```