### Install dstack Client and Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_dstack.rst

Install the dstack client with all extras and start the dstack server. This is the initial setup step for using dstack.

```console
pip install "dstack[all]"
dstack server
```

--------------------------------

### Setup and Run vLLM API Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/run_on_sky.rst

Installs vLLM and its dependencies, then starts the OpenAI-compatible API server. This snippet is used for initial setup and launching the core serving component.

```bash
conda create -n vllm python=3.10 -y
conda activate vllm

pip install vllm==0.4.0.post1
# Install Gradio for web UI.
pip install gradio openai
pip install flash-attn==2.5.7
```

```bash
conda activate vllm
echo 'Starting vllm api server...'
python -u -m vllm.entrypoints.openai.api_server \
    --port 8081 \
    --model $MODEL_NAME \
    --trust-remote-code \
    --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
    2>&1 | tee api_server.log &

echo 'Waiting for vllm api server to start...'
while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done

echo 'Starting gradio server...'
git clone https://github.com/vllm-project/vllm.git || true
python vllm/examples/gradio_openai_chatbot_webserver.py \
    -m $MODEL_NAME \
    --port 8811 \
    --model-url http://localhost:8081/v1 \
    --stop-token-ids 128009,128001
```

--------------------------------

### Install and Start Ray for Distributed Serving

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/distributed_serving.rst

Install Ray and start a Ray runtime on the head node before launching vLLM for multi-machine serving. Ensure the model is accessible by all nodes.

```console
$ pip install ray

$ # On head node
$ ray start --head

$ # On worker nodes
$ ray start --address=<ray-head-address>
```

--------------------------------

### Build and Install vLLM CPU Backend from Source

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/cpu-installation.rst

Builds and installs the vLLM CPU backend by setting the target device and running the setup script.

```console
$ VLLM_TARGET_DEVICE=cpu python setup.py install
```

--------------------------------

### Build and Run vLLM OpenVINO Docker Image

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/openvino-installation.rst

Builds a Docker image with the vLLM OpenVINO environment and runs it interactively. This is a quick way to get started.

```console
$ docker build -f Dockerfile.openvino -t vllm-openvino-env .
$ docker run -it --rm vllm-openvino-env
```

--------------------------------

### Install Dependencies and Build Docs

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/README.md

Installs necessary dependencies for documentation and then builds the HTML documentation.

```bash
pip install -r requirements-docs.txt

make clean
make html
```

--------------------------------

### Build and Install vLLM XPU Backend

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/xpu-installation.rst

Builds and installs the vLLM XPU backend from source. Ensure OneAPI 2024.1 and drivers are installed prior to this step.

```console
$ VLLM_TARGET_DEVICE=xpu python setup.py install

```

--------------------------------

### Install Langchain and Langchain Community

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/serving_with_langchain.rst

Install the necessary Langchain packages to use vLLM integration. The -q flag ensures a quiet installation.

```console
pip install langchain langchain_community -q
```

--------------------------------

### Download Example Batch File

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/offline_inference_openai.md

Use wget to download the example batch file for offline inference.

```bash
wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/openai_example_batch.jsonl
```

--------------------------------

### View Example Batch File Content

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/offline_inference_openai.md

Display the content of the downloaded example batch file using cat.

```bash
$ cat openai_example_batch.jsonl
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
```

--------------------------------

### Install vLLM OpenVINO Prerequisites

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/openvino-installation.rst

Installs or upgrades pip and installs necessary prerequisites for the vLLM OpenVINO backend, including PyTorch CPU wheels.

```console
$ pip install --upgrade pip
$ pip install -r requirements-build.txt --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Multi-GPU Serving with API Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/distributed_serving.rst

Start the API server for multi-GPU serving by passing the `--tensor-parallel-size` argument. This example configures it for 4 GPUs.

```console
$ python -m vllm.entrypoints.openai.api_server \
$     --model facebook/opt-13b \
$     --tensor-parallel-size 4
```

--------------------------------

### Install AutoAWQ

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/quantization/auto_awq.rst

Install the AutoAWQ library using pip.

```console
$ pip install autoawq
```

--------------------------------

### Install vLLM with Pip

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/README.md

Install the vLLM library using pip. This is the recommended method for most users.

```bash
pip install vllm
```

--------------------------------

### Install SkyPilot and Check Environment

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/run_on_sky.rst

Install the SkyPilot nightly build and verify that your cloud or Kubernetes environment is correctly configured.

```console
pip install skypilot-nightly
sky check
```

--------------------------------

### Start OpenAI Compatible Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/openai_compatible_server.md

Start the vLLM OpenAI compatible server with a specified model and API key. Ensure the model is accessible.

```bash
python -m vllm.entrypoints.openai.api_server --model NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/README.md

Starts a local HTTP server to view the built documentation in a browser.

```bash
python -m http.server -d build/html/
```

--------------------------------

### Install and Login to Cerebrium Client

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_cerebrium.rst

Install the Cerebrium client and log in to your account to manage your deployments.

```console
pip install cerebrium
cerebrium login
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/CONTRIBUTING.md

Installs the necessary development dependencies, including linters, formatters, and testing tools.

```bash
pip install -r requirements-dev.txt
```

--------------------------------

### Install OpenTelemetry Packages

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/Otel.md

Installs necessary OpenTelemetry packages for SDK, API, OTLP exporter, and AI semantic conventions.

```bash
pip install \
    opentelemetry-sdk \
    opentelemetry-api \
    opentelemetry-exporter-otlp \
    opentelemetry-semantic-conventions-ai
```

--------------------------------

### Example: Benchmark vLLM with FP8 KV Cache

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/fp8/README.md

An example command to run the throughput benchmark with FP8 KV cache enabled. It specifies input/output lengths, tensor parallelism, FP8 KV cache dtype, the path to quantization parameters, and the model path.

```bash
python3 benchmarks/benchmark_throughput.py --input-len <INPUT_LEN> --output-len <OUTPUT_LEN> -tp <TENSOR_PARALLEL_SIZE> --kv-cache-dtype fp8 --quantization-param-path <path/to/kv_cache_scales.json> --model <path-to-llama2>
```

--------------------------------

### Clone vLLM Repository and Install from Source

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Clones the vLLM repository and installs it in editable mode, which may take several minutes due to compilation.

```console
git clone https://github.com/vllm-project/vllm.git
cd vllm
# export VLLM_INSTALL_PUNICA_KERNELS=1 # optionally build for multi-LoRA capability
pip install -e .
```

--------------------------------

### Install vLLM with Neuron Support

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/neuron-installation.rst

Clone the vLLM repository, install Neuron-specific requirements, and then install the vLLM package. This process ensures that vLLM is built with compatibility for AWS Inferentia hardware.

```console
$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm
$ pip install -U -r requirements-neuron.txt
$ pip install .
```

--------------------------------

### Install Python Packages for XPU

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/xpu-installation.rst

Installs or upgrades pip and then installs the required Python packages for building the vLLM XPU backend from source.

```console
$ pip install --upgrade pip
$ pip install -v -r requirements-xpu.txt 

```

--------------------------------

### Install FastAPI Instrumentation

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/Otel.md

Installs the OpenTelemetry instrumentation library specifically for FastAPI applications.

```bash
pip install opentelemetry-instrumentation-fastapi
```

--------------------------------

### Start OpenAI-Compatible API Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/quickstart.rst

Launch the vLLM OpenAI-compatible API server using the python -m vllm.entrypoints.openai.api_server command with a specified model.

```console
$ python -m vllm.entrypoints.openai.api_server \
$     --model facebook/opt-125m
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/hkunlp/chunkllama/blob/main/README.md

Installs the necessary Python packages for the project, including transformers and flash-attention2. Ensure FlashAttention is version 2.5.0 or higher.

```bash
pip install -r requirements.txt
pip install flash-attn --no-build-isolation (FlashAttention >= 2.5.0)
```

--------------------------------

### Install vLLM with OpenVINO Backend from Source

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/openvino-installation.rst

Installs vLLM with the OpenVINO backend from source, specifying custom index URLs for PyTorch CPU and OpenVINO nightly wheels, and targeting the OpenVINO device.

```console
$ PIP_PRE=1 PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu https://storage.openvinotoolkit.org/simple/wheels/nightly/" VLLM_TARGET_DEVICE=openvino python -m pip install -v .
```

--------------------------------

### Install AutoFP8 Library for Offline Quantization

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/quantization/fp8.rst

Clone the AutoFP8 repository and install it in editable mode to perform offline FP8 quantization.

```bash
git clone https://github.com/neuralmagic/AutoFP8.git
pip install -e AutoFP8
```

--------------------------------

### Install Recommended Compiler for vLLM CPU Backend

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/cpu-installation.rst

Installs GCC and G++ version 12.3.0 on Ubuntu 22.4, setting them as the default alternatives.

```console
$ sudo apt-get update  -y
$ sudo apt-get install -y gcc-12 g++-12
$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
```

--------------------------------

### Run AWQ Model via Command Line

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/quantization/auto_awq.rst

Execute an AWQ quantized model using vLLM's example script. This demonstrates how to specify the model and quantization type.

```console
$ python examples/llm_engine_example.py --model TheBloke/Llama-2-7b-Chat-AWQ --quantization awq
```

--------------------------------

### Inference Response Example

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_cerebrium.rst

Example of a successful response from the deployed vLLM model, showing the generated text for each prompt.

```python
{
    "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262",
    "result": {
        "result": [
            {
                "prompt": "Hello, my name is",
                "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of"
            },
            {
                "prompt": "The president of the United States is",
                "generated_text": " elected every four years. This is a democratic system.\n\n5. What"
            },
            {
                "prompt": "The capital of France is",
                "generated_text": " Paris.\n"
            },
            {
                "prompt": "The future of AI is",
                "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective."
            }
        ]
    },
    "run_time_ms": 152.53663063049316
}
```

--------------------------------

### Example: Extract KV Cache Scaling Factors

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/fp8/README.md

Provides an example command to run the `extract_scales.py` script, specifying the path to the quantized model, tensor parallel size, and output directory for the scaling factors.

```bash
python3 examples/fp8/extract_scales.py --quantized_model <QUANTIZED_MODEL_DIR> --tp_size <TENSOR_PARALLEL_SIZE> --output_dir <PATH_TO_OUTPUT_DIR>
```

--------------------------------

### Install vLLM from Source

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/CONTRIBUTING.md

Installs the vLLM package in editable mode from the local source code. This command is essential for development and testing.

```bash
pip install -e .
```

--------------------------------

### Install vLLM from Source with Limited Compilation Jobs

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Installs vLLM from source while limiting the number of simultaneous compilation jobs to prevent system overload.

```console
export MAX_JOBS=6
pip install -e .
```

--------------------------------

### Install Python on Ubuntu

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/openvino-installation.rst

Installs Python 3 on Ubuntu 22.04 using apt-get. This is a prerequisite for building vLLM from source.

```console
$ sudo apt-get update  -y
$ sudo apt-get install python3
```

--------------------------------

### Install Other Build Dependencies

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/tpu-installation.rst

Installs essential Python packages like 'packaging' and 'aiohttp' required for building vLLM from source.

```console
$ # Install other build dependencies.
$ pip install packaging aiohttp
```

--------------------------------

### Install transformers-neuronx and Dependencies

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/neuron-installation.rst

Sets up a Python virtual environment and installs the transformers-neuronx package along with its required dependencies, including Neuron Compiler and PyTorch Neuron.

```console
# Install Python venv
sudo apt-get install -y python3.10-venv g++

# Create Python venv
python3.10 -m venv aws_neuron_venv_pytorch

# Activate Python venv
source aws_neuron_venv_pytorch/bin/activate

# Install Jupyter notebook kernel
pip install ipykernel
python3.10 -m ipykernel install --user --name aws_neuron_venv_pytorch --display-name "Python (torch-neuronx)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install wget, awscli
python -m pip install wget
python -m pip install awscli

# Update Neuron Compiler and Framework
python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torchvision transformers-neuronx
```

--------------------------------

### Build vLLM from Source for TPU

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/tpu-installation.rst

Builds and installs vLLM from source with TPU target device specified. This command should be run after all dependencies are installed.

```console
$ VLLM_TARGET_DEVICE="tpu" python setup.py develop
```

--------------------------------

### Start API Server with Custom Chat Template

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/quickstart.rst

Start the vLLM API server and override the default chat template by specifying a custom Jinja template file using the --chat-template argument.

```console
$ python -m vllm.entrypoints.openai.api_server \
$     --model facebook/opt-125m \
$     --chat-template ./examples/template_chatml.jinja
```

--------------------------------

### Build Docker Image for XPU

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/xpu-installation.rst

Builds a Docker image with the vLLM XPU environment. Ensure you have Docker installed and the Dockerfile.xpu is present.

```console
$ docker build -f Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .

```

--------------------------------

### Run Docker Container for XPU

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/xpu-installation.rst

Launches an interactive Docker container with XPU support, mounting necessary devices. This is used for quick start.

```console
$ docker run -it \
             --rm \
             --network=host \
             --device /dev/dri \
             -v /dev/dri/by-path:/dev/dri/by-path \
             vllm-xpu-env

```

--------------------------------

### Install JAX and Pallas

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/tpu-installation.rst

Installs JAX with TPU support and Pallas, using specific index URLs for nightly releases. This step is crucial for leveraging JAX functionalities on TPUs.

```console
$ # Install JAX and Pallas.
$ pip install torch_xla[tpu] -f https://storage.googleapis.com/libtpu-releases/index.html
$ pip install torch_xla[pallas] -f https://storage.googleapis.com/jax-releases/jax_nightly_releases.html -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html
```

--------------------------------

### Example Batch File Content

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/offline_inference_openai.md

The content of an example batch file, showing multiple requests with custom IDs, methods, URLs, and bodies for chat completions.

```json
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
```

--------------------------------

### Install Python Packages for vLLM CPU Backend

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/cpu-installation.rst

Installs necessary Python packages including pip, wheel, packaging, ninja, setuptools, numpy, and vLLM CPU requirements with PyTorch CPU.

```console
$ pip install --upgrade pip
$ pip install wheel packaging ninja "setuptools>=49.4.0" numpy
$ pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Configure HTTP/protobuf Exporter Protocol

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/Otel.md

Sets the OpenTelemetry exporter protocol to HTTP/protobuf and configures the endpoint accordingly. It then starts the vLLM server using this configuration.

```bash
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://$JAEGER_IP:4318/v1/traces
python -m vllm.entrypoints.openai.api_server --model="facebook/opt-125m" --otlp-traces-endpoint="$OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
```

--------------------------------

### Launch vLLM Serving on a Single Instance

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/run_on_sky.rst

Launch the vLLM serving instance using the SkyPilot configuration. This command starts the API server and Gradio interface for the specified model.

```console
HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN
```

--------------------------------

### Install OpenBLAS for libopenblas.so.0 Error

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/tpu-installation.rst

Installs OpenBLAS, OpenMPI, and OpenMP development libraries using apt-get. This resolves the 'libopenblas.so.0: cannot open shared object file' error during PyTorch XLA import.

```console
$ sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
```

--------------------------------

### Setup and Run Gradio Chatbot GUI

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/run_on_sky.rst

Configures the environment and runs a Gradio-based chatbot web server that connects to a vLLM API endpoint. This allows for interactive use of the served model.

```yaml
envs:
    MODEL_NAME: meta-llama/Meta-Llama-3-70B-Instruct
    ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm. 

resources:
    cpus: 2

setup: |
    conda activate vllm
    if [ $? -ne 0 ]; then
        conda create -n vllm python=3.10 -y
        conda activate vllm
    fi

    # Install Gradio for web UI.
    pip install gradio openai

run: |
    conda activate vllm
    export PATH=$PATH:/sbin
    WORKER_IP=$(hostname -I | cut -d' ' -f1)
    CONTROLLER_PORT=21001
    WORKER_PORT=21002

    echo 'Starting gradio server...'
    git clone https://github.com/vllm-project/vllm.git || true
    python vllm/examples/gradio_openai_chatbot_webserver.py \
        -m $MODEL_NAME \
        --port 8811 \
        --model-url http://$ENDPOINT/v1 \
        --stop-token-ids 128009,128001 | tee ~/gradio.log
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Recommended step to create a new isolated environment for vLLM installation.

```console
conda create -n myenv python=3.10 -y
conda activate myenv
```

--------------------------------

### Install vLLM ROCm Dependencies

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/amd-installation.rst

Installs the necessary Python packages for vLLM ROCm installation. Ensure you are in the vLLM project directory.

```console
cd vllm
pip install -U -r requirements-rocm.txt
```

--------------------------------

### Provision VM Instance with dstack

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_dstack.rst

Run the dstack service configuration to provision a VM instance. The output shows available offers and prompts for confirmation.

```console
dstack run . -f serve.dstack.yml

⠸ Getting run plan...
 Configuration  serve.dstack.yml             
 Project        deep-diver-main              
 User           deep-diver                   
 Min resources  2..xCPU, 8GB.., 1xGPU (24GB) 
 Max price      -                            
 Max duration   -                            
 Spot policy    auto                         
 Retry policy   no                           

 #  BACKEND  REGION       INSTANCE       RESOURCES                               SPOT  PRICE       
 1  gcp   us-central1  g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804   
 2  gcp   us-east1     g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804   
 3  gcp   us-west1     g2-standard-4  4xCPU, 16GB, 1xL4 (24GB), 100GB (disk)  yes   $0.223804   
    ...                                                                                            
 Shown 3 of 193 offers, $5.876 max

Continue? [y/n]: y
⠙ Submitting run...
⠏ Launching spicy-treefrog-1 (pulling)
spicy-treefrog-1 provisioning completed (running)
Service is published at ...
```

--------------------------------

### Install python-json-logger

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/logging_configuration.md

Install the python-json-logger package using pip. This is required for JSON formatted logging.

```bash
pip install python-json-logger
```

--------------------------------

### Install Neuron Drivers and Tools

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/neuron-installation.rst

Installs necessary drivers, tools, and dependencies for AWS Neuron on Linux systems. This includes configuring the Neuron repository, updating OS packages, and installing the Neuron driver, runtime, and tools.

```console
# Configure Linux for Neuron repository updates
. /etc/os-release
sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <<EOF
deb https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main
EOF
wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -

# Update OS packages
sudo apt-get update -y

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install git
sudo apt-get install git -y

# install Neuron Driver
sudo apt-get install aws-neuronx-dkms=2.* -y

# Install Neuron Runtime
sudo apt-get install aws-neuronx-collectives=2.*
sudo apt-get install aws-neuronx-runtime-lib=2.*

# Install Neuron Tools
sudo apt-get install aws-neuronx-tools=2.*

# Add PATH
export PATH=/opt/aws/neuron/bin:$PATH
```

--------------------------------

### Install vLLM with CUDA 11.8

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Installs a specific vLLM version compiled with CUDA 11.8 from a wheel file.

```console
export VLLM_VERSION=0.4.0
export PYTHON_VERSION=310
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### Run ChunkLlama Demo

Source: https://github.com/hkunlp/chunkllama/blob/main/README.md

Launches a demo of ChunkLlama with specified parameters for context length, model scale, and PDF file. Adjust scale for 7b, 13b, or 70b models.

```python
python run_chunkllama_100k.py --max_length 16000 --scale 13b (7b/13b/70b) --pdf Popular_PDFs/longlora.pdf
```

--------------------------------

### Initialize dstack Project

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_dstack.rst

Create a new directory for your dstack project and navigate into it. This command initializes the project structure.

```console
mkdir -p vllm-dstack
cd vllm-dstack
dstack init
```

--------------------------------

### Install PyTorch for ROCm 5.7

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/amd-installation.rst

Installs PyTorch with ROCm 5.7 support using pip and a specific index URL.

```console
$ pip install torch --index-url https://download.pytorch.org/whl/rocm5.7
```

--------------------------------

### Install PyTorch for ROCm 6.0

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/amd-installation.rst

Installs PyTorch with ROCm 6.0 support using pip and a specific index URL.

```console
$ pip3 install torch --index-url https://download.pytorch.org/whl/rocm6.0
```

--------------------------------

### Serve LoRA Adapters with vLLM Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/models/lora.rst

Start the vLLM OpenAI-compatible server and specify LoRA modules using the `--lora-modules` flag. This allows serving multiple LoRA adapters alongside the base model.

```bash
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --enable-lora \
    --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
```

--------------------------------

### Configure Dummy Client for Tracing

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/Otel.md

Sets up environment variables for a client service to send traces to Jaeger, including its IP, trace endpoint, and service name. It then runs a dummy client script.

```bash
export JAEGER_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' jaeger)
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://$JAEGER_IP:4317
export OTEL_EXPORTER_OTLP_TRACES_INSECURE=true
export OTEL_SERVICE_NAME="client-service"
python dummy_client.py
```

--------------------------------

### vLLM Throughput Benchmark Script Help

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/fp8/README.md

Displays the help message for the `benchmark_throughput.py` script, outlining all available arguments and their descriptions. This is useful for understanding the script's capabilities and options.

```bash
python3 benchmarks/benchmark_throughput.py --help
```

--------------------------------

### Run NVIDIA PyTorch Docker Image

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Launches the recommended NVIDIA PyTorch Docker image, which is useful for troubleshooting build issues.

```console
docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
```

--------------------------------

### Query and Key Calculation Flow

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/dev/kernel/paged_attention.rst

Illustrates the high-level flow for calculating query-key (QK) dot products. It involves fetching query data, iterating through key tokens, preparing key vectors, and performing dot multiplications.

```cpp
q_vecs = ...
for ... {
   k_ptr = ...
   for ... {
      k_vecs[i] = ...
   }
   ...
   float qk = scale * Qk_dot<scalar_t, THREAD_GROUP_SIZE>::dot(q_vecs[thread_group_offset], k_vecs);
}
```

--------------------------------

### Install AMMO for FP8 Quantization

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/quantization/fp8_e4m3_kvcache.rst

Install the Algorithmic Model Optimization (AMMO) tool using pip. This is a prerequisite for generating FP8 quantization parameters.

```console
pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-ammo
```

--------------------------------

### Verify CUDA Toolkit Installation

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/installation.rst

Checks if the CUDA compiler (`nvcc`) is installed and accessible in the system's PATH and within the specified CUDA_HOME directory.

```console
nvcc --version
${CUDA_HOME}/bin/nvcc --version
```

--------------------------------

### Submit Sample Requests for Monitoring

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/README.md

Download a dataset and use a benchmark script to send sample requests to the vLLM server. This is necessary to generate metrics for Prometheus to scrape. Adjust the request rate as needed.

```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

python3 ../../benchmarks/benchmark_serving.py \
    --model mistralai/Mistral-7B-v0.1 \
    --tokenizer mistralai/Mistral-7B-v0.1 \
    --endpoint /v1/completions \
    --dataset-name sharegpt \
    --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \
    --request-rate 3.0
```

--------------------------------

### Install PyTorch and PyTorch XLA for TPU

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/tpu-installation.rst

Installs the nightly builds of PyTorch and PyTorch XLA compatible with TPU VMs. Ensure the DATE variable is set correctly.

```console
$ # Install PyTorch and PyTorch XLA.
$ export DATE="+20240601"
$ pip install https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch-nightly${DATE}-cp310-cp310-linux_x86_64.whl
$ pip install https://storage.googleapis.com/pytorch-xla-releases/wheels/tpuvm/torch_xla-nightly${DATE}-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Launch vLLM Serving with Larger Model and More GPUs

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/run_on_sky.rst

Optional command to serve a larger model (70B) and utilize more GPUs by overriding default environment variables and specifying GPU count.

```console
HF_TOKEN="your-huggingface-token" sky launch serving.yaml --gpus A100:8 --env HF_TOKEN --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
```

--------------------------------

### Install and Configure TCMalloc for vLLM CPU Performance

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/cpu-installation.rst

Installs the TCMalloc library, finds its path, and preloads it to enhance memory allocation and cache locality for vLLM CPU backend.

```console
$ sudo apt-get install libtcmalloc-minimal4 # install TCMalloc library
$ find / -name *libtcmalloc*
$ export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:$LD_PRELOAD # prepend the library to LD_PRELOAD
$ python examples/offline_inference.py # run vLLM
```

--------------------------------

### Launch vLLM OpenAI-Compatible Server

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/examples/production_monitoring/README.md

Start the vLLM server with Prometheus metric logging enabled by default. Ensure to specify the model and maximum model length. Requests logging can be disabled if not needed.

```bash
python3 -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-v0.1 \
    --max-model-len 2048 \
    --disable-log-requests
```

--------------------------------

### Run vLLM OpenAI Server from a Built Docker Image

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/deploying_with_docker.rst

Run the vLLM OpenAI server using a locally built Docker image. Mount necessary volumes and provide environment variables. Replace `<args...>` with your desired server arguments.

```console
$ docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    vllm/vllm-openai <args...>
```

--------------------------------

### Example Usage Stats Data Collected by vLLM

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/serving/usage_stats.md

This JSON object represents an example of the anonymous usage data collected by vLLM, including hardware, model, and configuration details. It is collected by default to aid development.

```json
{
  "uuid": "fbe880e9-084d-4cab-a395-8984c50f1109",
  "provider": "GCP",
  "num_cpu": 24,
  "cpu_type": "Intel(R) Xeon(R) CPU @ 2.20GHz",
  "cpu_family_model_stepping": "6,85,7",
  "total_memory": 101261135872,
  "architecture": "x86_64",
  "platform": "Linux-5.10.0-28-cloud-amd64-x86_64-with-glibc2.31",
  "gpu_count": 2,
  "gpu_type": "NVIDIA L4",
  "gpu_memory_per_device": 23580639232,
  "model_architecture": "OPTForCausalLM",
  "vllm_version": "0.3.2+cu123",
  "context": "LLM_CLASS",
  "log_time": 1711663373492490000,
  "source": "production",
  "dtype": "torch.float16",
  "tensor_parallel_size": 1,
  "block_size": 16,
  "gpu_memory_utilization": 0.9,
  "quantization": null,
  "kv_cache_dtype": "auto",
  "enable_lora": false,
  "enable_prefix_caching": false,
  "enforce_eager": false,
  "disable_custom_all_reduce": true
}
```

--------------------------------

### Define Prompts and Sampling Parameters

Source: https://github.com/hkunlp/chunkllama/blob/main/vllm/docs/source/getting_started/quickstart.rst

Define a list of input prompts and configure sampling parameters like temperature and top_p for text generation.

```python
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
```