### Start Aphrodite Server with Tool Calling

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/tool-call.md

Starts the Aphrodite server with tool calling enabled, using a specific model and chat template. This is the initial setup step for using tool calling.

```bash
aphrodite run meta-llama/Llama-3.1-8B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_json \
    --chat-template examples/tool_chat_template_llama3.1_json.jinja
```

--------------------------------

### Build and Run OpenVINO Docker Image

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Build a Docker image for the OpenVINO backend and run it interactively. This is a quick way to get started with the OpenVINO backend.

```sh
docker build -f Dockerfile.openvino -t aphrodite-openvino .
docker run -it --rm aphrodite-openvino
```

--------------------------------

### Download Example Batch File

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/openai_api/batched_inference.md

Use wget to download the example batch file for testing.

```bash
wget https://raw.githubusercontent.com/PygmalionAI/aphrodite-engine/rc_054/examples/openai_api/batched_example.jsonl
```

--------------------------------

### Set up Python Environment and Install transformers-neuronx

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-neuron.md

Creates and activates a Python virtual environment, installs Jupyter for notebook support, and configures pip to use the Neuron repository. It then installs the necessary Neuron-specific PyTorch and transformers-neuronx packages.

```shell
# Install Python venv
sudo apt-get install -y python3.10-venv g++

# Create Python venv
python3.10 -m venv aws_neuron_venv_pytorch

# Activate Python venv
source aws_neuron_venv_pytorch/bin/activate

# Install Jupyter notebook kernel
pip install ipykernel
python3.10 -m ipykernel install --user --name aws_neuron_venv_pytorch --display-name "Python (torch-neuronx)"
pip install jupyter notebook
pip install environment_kernels

# Set pip repository pointing to the Neuron repository
python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

# Install wget, awscli
python -m pip install wget
python -m pip install awscli

# Update Neuron Compiler and Framework
python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torchvision transformers-neuronx
```

--------------------------------

### Install DeepGEMM

Source: https://github.com/dphnai/aphrodite-engine/blob/main/benchmarks/kernels/deepgemm/README.md

Clone the DeepGEMM repository, navigate into it, and install it using setup.py. Finally, install Aphrodite in editable mode.

```bash
git clone --recursive https://github.com/deepseek-ai/DeepGEMM
cd DeepGEMM
python setup.py install
uv pip install -e .
```

--------------------------------

### Disaggregated Prefill Example (Aphrodite v0)

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/disaggregated_prefill/lmcache/README.md

Provides an example of how to run disaggregated prefill in Aphrodite version 0.

```python
disaggregated_prefill_lmcache_v0.py
```

--------------------------------

### Install System Dependencies for Building

Source: https://github.com/dphnai/aphrodite-engine/wiki/CPU-Support

Installs GCC and G++ version 12, and sets them as the default alternatives. Skip if you have a compatible version already.

```shell
$ sudo apt-get update -y
$ sudo apt-get install -y gcc-12 g++-12  # you can skip this if you already have a gcc/g++>=12.3.0 installed
$ sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
```

--------------------------------

### Install Dependencies for INT8 KV Cache Quantization

Source: https://github.com/dphnai/aphrodite-engine/wiki/8.-Quantization

Install the 'fire' library, which is required for the INT8 KV cache calibration process.

```sh
pip install fire
```

--------------------------------

### Install Transformers Library

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/quantization/quantization-methods.md

Install the Transformers library, which is a prerequisite for using AWQ quantization with Aphrodite. This command is typically run in your terminal.

```shell
pip install transformers
```

--------------------------------

### Example: Benchmark Throughput with FP8 KV Cache

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/fp8/README.md

An example command to run the throughput benchmark script. It specifies input/output lengths, tensor parallelism, FP8 KV cache data type, the path to KV cache scaling factors, and the model path.

```bash
python3 benchmarks/benchmark_throughput.py --input-len <INPUT_LEN> --output-len <OUTPUT_LEN> -tp <TENSOR_PARALLEL_SIZE> --kv-cache-dtype fp8 --quantization-param-path <path/to/kv_cache_scales.json> --model <path-to-llama2>
```

--------------------------------

### Install Transformers-Neuronx

Source: https://github.com/dphnai/aphrodite-engine/wiki/Neuron-Support

Sets up a Python virtual environment and installs the Transformers-Neuronx library along with its dependencies, including PyTorch and wget. This is necessary for using PyTorch models with the Neuron SDK.

```shell
sudo apt-get install -y python3.10-venv g++

python3.10 -m venv aws_neuron_venv_pytorch

source aws_neuron_venv_pytorch/bin/activate

pip install ipykernel
python3.10 -m ipykernel install --user --name aws_neuron_venv_pytorch --display-name "Python (torch-neuronx)"
pip install jupyter notebook
pip install environment_kernels

python -m pip config set global.extra-index-url https://pip.repos.neuron.amazonaws.com

python -m pip install wget awscli

python -m pip install --upgrade neuronx-cc==2.* --pre torch-neuronx==2.1.* torchvision transformers-neuronx
```

--------------------------------

### Install NVIDIA AMMO

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/quantization/kv-cache.md

Install the Algorithmic Model Optimization (AMMO) tool from NVIDIA's PyPI index. This is required for generating quantization parameters.

```sh
pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com nvidia-ammo
```

--------------------------------

### Install Compiler and Linker on Ubuntu

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Installs GCC/G++ version 12 and libnuma-dev on Ubuntu, and configures alternatives for gcc/g++ to use the installed version 12. This is a prerequisite for building the CPU backend from source.

```sh
sudo apt-get update
sudo apt-get install -y gcc-12 g++-12 libnuma-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12
```

--------------------------------

### Install Aphrodite Engine on Windows

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-windows.md

Install the Aphrodite Engine package using pip after all prerequisites and dependencies have been installed.

```bash
pip install aphrodite-engine
```

--------------------------------

### Example: Registering Custom Op for ShortConv

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/developer/adding-model.md

Example of registering a custom operation for a mamba-like layer (ShortConv) to support torch compile and CUDA graphs.

```python
# Found in aphrodite/modeling/layers/mamba/short_conv.py
# ... other imports ...
from aphrodite.utils.custom_op import direct_register_custom_op
from aphrodite.modeling.layers.mamba.short_conv import ShortConv
from aphrodite.v1.attention.backends.short_conv_attn import ShortConvAttentionMetadata

# ... layer definition ...

# Example call to register custom op
# direct_register_custom_op(
#     ShortConv(config),
#     ShortConvAttentionMetadata,
#     "short_conv_op"
# )

# Ensure "short_conv_op" is added to _attention_ops in aphrodite/config/compilation.py
```

--------------------------------

### Fuyu: `FuyuImageProcessor` Preprocessing Example

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/developer/adding-model.md

Shows a snippet from `FuyuImageProcessor.preprocess_with_tokenizer_info` where `image_input_ids` and `image_patches` are concatenated for each image in the batch.

```python
# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L673-L679
        image_input_ids.append(tensor_of_image_ids)
        image_patches.append(patches)
    else:
        image_input_ids.append(torch.tensor([], dtype=torch.int32, device=image_input.device))

batch_image_input_ids.append(image_input_ids)
batch_image_patches.append(image_patches)
```

--------------------------------

### Launch Docker Monitoring Containers

Source: https://github.com/dphnai/aphrodite-engine/wiki/7.-Logging-Metrics-for-Production

Navigate to the monitoring example directory and run this command to launch the Docker containers for Prometheus and Grafana.

```shell
cd examples/monitoring
docker compose up
```

--------------------------------

### Install Missing Libraries for OpenBLAS

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-tpu.md

Installs necessary libraries (OpenBLAS, OpenMPI, OpenMP) to resolve 'libopenblas.so.0' import errors. This command requires sudo privileges.

```sh
sudo apt-get install libopenblas-base libopenmpi-dev libomp-dev
```

--------------------------------

### Start Aphrodite Server for Benchmarking

Source: https://github.com/dphnai/aphrodite-engine/blob/main/benchmarks/multi_turn/README.md

Starts the aphrodite server with a specified model path and disables request logging. Ensure the MODEL_PATH variable points to your downloaded model files.

```bash
export MODEL_PATH=/models/meta-llama/Meta-Llama-3.1-8B-Instruct/

aphrodite run $MODEL_PATH --served-model-name Llama --disable-log-requests
```

--------------------------------

### Install Python on Ubuntu

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Installs Python 3 on Ubuntu 22.04 machines, a prerequisite for building Aphrodite from source.

```sh
sudo apt-get update
sudo apt-get install python3
```

--------------------------------

### Clone and Install Marlin Repository

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/marlin/README.md

Clone the Marlin repository and install it in editable mode. This is a prerequisite for converting GPTQ models to Marlin format.

```shell
git clone https://github.com/IST-DASLab/marlin && cd marlin

pip install -e .
```

--------------------------------

### KV Cache Sharing Example (Aphrodite v1)

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/disaggregated_prefill/lmcache/README.md

Demonstrates how to share KV caches between Aphrodite v1 instances.

```python
kv_cache_sharing_lmcache_v1.py
```

--------------------------------

### Install Aphrodite with OpenVINO Backend

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Installs Aphrodite from source with the OpenVINO target device, specifying PyTorch CPU wheels and setting the target device environment variable.

```sh
PIP_EXTRA_INDEX_URL="https://download.pytorch.org/whl/cpu" APHRODITE_TARGET_DEVICE=openvino python3 -m pip install -e .
```

--------------------------------

### Example: Registering Custom Op for MiniMaxText01

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/developer/adding-model.md

Example of registering a custom operation for a mamba-like layer (MiniMaxText01) to support torch compile and CUDA graphs.

```python
# Found in aphrodite/modeling/models/minimax_text_01.py
# ... other imports ...
from aphrodite.utils.custom_op import direct_register_custom_op
from aphrodite.modeling.layers.mamba.short_conv import ShortConv # Assuming ShortConv is used here for example

# ... model definition ...

# Example call to register custom op
# direct_register_custom_op(
#     ShortConv(config),
#     ShortConvAttentionMetadata,
#     "short_conv_op"
# )

# Ensure "short_conv_op" is added to _attention_ops in aphrodite/config/compilation.py
```

--------------------------------

### Install Xformers on Windows

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-windows.md

Install the Xformers library. Replace 'cp312-cp312' with your specific Python version (e.g., 'cp310-cp310' for Python 3.10) if necessary.

```bash
pip install https://downloads.pygmalion.chat/whl/windows/xformers/xformers-0.0.28-cp312-cp312-win_amd64.whl
```

--------------------------------

### CLI Example for MLPSpeculator Decoding

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/spec-decoding/mlpspeculator.md

This command-line interface example demonstrates how to enable MLPSpeculator decoding using Aphrodite. Specify the speculative model, draft tensor parallel size, and optionally enable the v2 block manager for enhanced performance.

```bash
aphrodite run meta-llama/Meta-Llama-3.1-70B-Instruct \
    --speculative-model ibm-fms/llama3-70b-accelerator \
    --speculative-draft-tensor-parallel-size 1 \
    --use-v2-block-manager  # [!code highlight]
```

--------------------------------

### Install Python Packages for XPU Backend

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-xpu.md

Installs required Python packages for the Aphrodite XPU backend after setting up OneAPI environment variables.

```bash
source /opt/intel/oneapi/setvars.sh
pip install -U pip
pip install -v -r requirements/xpu.txt
```

--------------------------------

### Example JSON Configuration for batch_auto_tune.sh

Source: https://github.com/dphnai/aphrodite-engine/blob/main/benchmarks/auto_tune/README.md

Define parameters for multiple auto_tune.sh runs in a JSON array. Each object represents a single experiment configuration.

```json
[
  {
    "base": "/home/user",
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "system": "TPU", # OR GPU
    "tp": 8,
    "input_len": 128,
    "output_len": 2048,
    "max_model_len": 2300,
    "num_seqs_list": "128 256",
    "num_batched_tokens_list": "8192 16384"
  },
  {
    "base": "/home/user",
    "model": "meta-llama/Llama-3.1-70B-Instruct",
    "system": "TPU", # OR GPU
    "tp": 8,
    "input_len": 4000,
    "output_len": 16,
    "max_model_len": 4096,
    "num_seqs_list": "64 128",
    "num_batched_tokens_list": "4096 8192",
    "max_latency_allowed_ms": 500
  }
]
```

--------------------------------

### Example Batch File Content

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/openai_api/batched_inference.md

This is an example of the OpenAI batch file format, with each line representing a JSON object for a separate request to the chat completions endpoint.

```json
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "NousResearch/Meta-Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
```

--------------------------------

### Start Aphrodite Server

Source: https://github.com/dphnai/aphrodite-engine/blob/main/tests/evals/gsm8k/README.md

Initiate the Aphrodite server on a specified port. This is a prerequisite for running the standalone evaluation script.

```bash
# Start Aphrodite server first
aphrodite run Qwen/Qwen2.5-1.5B-Instruct --port 8000
```

--------------------------------

### Install PyTorch with CUDA Support on Windows

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-windows.md

Install the PyTorch library with CUDA 12.4 support. Ensure your system meets the CUDA requirements before running this command.

```bash
pip install 'torch==2.4.1' --index-url https://download.pytorch.org/whl/cu124
```

--------------------------------

### Example: Extract FP8 KV Cache Scaling Factors

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/fp8/README.md

This example demonstrates how to run the `extract_scales.py` script to extract KV cache scaling factors. Ensure you replace placeholders with your actual model directory, tensor parallel size, and desired output directory.

```bash
python3 examples/fp8/extract_scales.py --quantized_model <QUANTIZED_MODEL_DIR> --tp_size <TENSOR_PARALLEL_SIZE> --output_dir <PATH_TO_OUTPUT_DIR>
```

--------------------------------

### Install Aphrodite CPU Backend Requirements

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Installs Python build tools, including wheel, packaging, ninja, setuptools, numpy, and Aphrodite's CPU-specific requirements, using PyTorch CPU wheels.

```sh
pip install -U pip
pip install wheel packaging ninja "setuptools>=49.4.0" numpy
pip install -v -r requirements/cpu.txt --extra-index-url http://download.pytorch.org/whl/cpu
```

--------------------------------

### Build Aphrodite from Source for XPU

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-xpu.md

Builds Aphrodite from source, targeting the XPU device. Ensure OneAPI and Python dependencies are installed first.

```bash
APHRODITE_TARGET_DEVICE=xpu python setup.py develop
```

--------------------------------

### Single Input Prompts for Encoder-Decoder Models

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/encoder-decoder.md

Examples of creating single input prompts for encoder-decoder models. The first example uses a raw string, the second uses TextPrompt with a string, and the third uses TokensPrompt with encoded token IDs. These are treated as encoder inputs with empty decoder inputs.

```python
single_text_prompt_raw = "Hello, my name is"
single_text_prompt = TextPrompt(prompt="The president of the United States is")
single_tokens_prompt = TokensPrompt(prompt_token_ids=tokenizer.encode("The capital of France is"))
```

--------------------------------

### KV Block Identification Example

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/prompt-caching/implementation.md

Illustrates how KV blocks are identified by their tokens and prefix tokens. This forms the basis for the hashing mechanism.

```console
                    Block 1                  Block 2                  Block 3
         [A gentle breeze stirred] [the leaves as children] [laughed in the distance]
Block 1: |<--- block tokens ---->|
Block 2: |<------- prefix ------>| |<--- block tokens --->|
Block 3: |<------------------ prefix -------------------->| |<--- block tokens ---->|
```

--------------------------------

### Run Disaggregated Prefill Example (NIXL)

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/disaggregated_prefill/lmcache/README.md

Execute the main script to run disaggregated prefill and benchmark performance using NIXL. Ensure you are in the correct directory.

```bash
bash disagg_example_nixl.sh
```

--------------------------------

### Clone AI Horde Worker Repository

Source: https://github.com/dphnai/aphrodite-engine/wiki/5.-Hosting-on-AI-Horde

Clone the AI Horde Worker repository from GitHub to get started.

```sh
git clone https://github.com/Haidra-Org/AI-Horde-Worker
```

--------------------------------

### Query Available Models via curl

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/1-getting-started.md

Use curl to query the /v1/models endpoint of the Aphrodite server to get a list of available models. Assumes jq is installed for pretty-printing.

```sh
curl http://localhost:2242/v1/models | jq .
```

--------------------------------

### CLI Example for Ngram Prompt Lookup Decoding

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/spec-decoding/ngram.md

Execute ngram prompt lookup decoding from the command line using the Aphrodite CLI. This command mirrors the Python configuration, enabling quick setup for speculative decoding.

```bash
aphrodite run facebook/opt-6.7b --speculative-model '[ngram]' --num-speculative-tokens 5 --ngram-prompt-lookup-max 4 --use-v2-block-manager
```

--------------------------------

### Example result.txt Content

Source: https://github.com/dphnai/aphrodite-engine/blob/main/benchmarks/auto_tune/README.md

Illustrates the typical content of the result.txt file, showing tested parameter combinations, their performance metrics, and the identified best parameters. It also shows how unmet latency requirements are reported.

```text
# Example result.txt content
hash:a1b2c3d4...
max_num_seqs: 128, max_num_batched_tokens: 2048, request_rate: 10.0, e2el: 450.5, throughput: 9.8, goodput: 9.8
max_num_seqs: 128, max_num_batched_tokens: 4096 does not meet latency requirement 500
...
best_max_num_seqs: 256, best_num_batched_tokens: 2048, best_throughput: 12.5, profile saved in: /home/user/aphrodite/auto-benchmark/2024_08_01_10_30/profile
```

--------------------------------

### Request Text Completion via curl

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/1-getting-started.md

Send a POST request to the /v1/completions endpoint using curl to get text completion tokens. Includes parameters like model, prompt, max_tokens, temperature, and min_p. Assumes jq is installed.

```sh
curl http://localhost:2242/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
    "prompt": "Once upon a time",
    "max_tokens": 128,
    "temperature": 1.1,
    "min_p": 0.1
  }' | jq .
```

--------------------------------

### Install Aphrodite Engine

Source: https://github.com/dphnai/aphrodite-engine/blob/main/README.md

Install the Aphrodite Engine using pip. This command installs the latest version of the engine.

```sh
pip install -U aphrodite-engine
```

--------------------------------

### Build Aphrodite from Source with ROCm

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-rocm.md

Installs Aphrodite Engine from source after cloning the repository. This process involves patching AMD specific code, installing ROCm-compatible requirements, and performing a development installation. Ensure ROCm, PyTorch, and hipBLAS are installed beforehand.

```sh
git clone https://github.com/PygmalionAI/aphrodite-engine.git
cd aphrodite-engine

chmod +x ./amdpatch.sh
./amdpatch.sh 
pip install -U -r requirements/rocm.txt
python setup.py develop  #  pip install -e . won't work for now
```

--------------------------------

### Install AWS Neuron Drivers

Source: https://github.com/dphnai/aphrodite-engine/wiki/Neuron-Support

Installs necessary AWS Neuron drivers and tools on a Linux system. Ensure your system has the correct release version and headers installed.

```shell
. /etc/os-release
sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <<EOF
deb https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main
EOF
wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -


sudo apt-get update -y

sudo apt-get install linux-headers-$(uname -r) -y

sudo apt-get install git -y

sudo apt-get install aws-neuronx-dkms=2.* aws-neuronx-collectives=2.* aws-neuronx-runtime-lib=2.* aws-neuronx-tools=2.* -y

export PATH=/opt/aws/neuron/bin:$PATH
```

--------------------------------

### Build and Run Docker Image for XPU

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-xpu.md

Builds a Docker image for XPU and runs it with necessary host configurations for device access.

```bash
docker build -f Dockerfile.xpu -t aphrodite-xpu --shm-size=4g .
docker run -it \
  --rm \
  --network=host \
  --ipc=host \
  --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path \
  aphrodite-xpu
```

--------------------------------

### Install Aphrodite from Source for Neuron

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-neuron.md

Clones the Aphrodite Engine repository, installs its dependencies specifically for Neuron, and then installs the engine in editable mode with the target device set to 'neuron'.

```shell
git clone https://github.com/PygmalionAI/aphrodite-engine.git
cd aphrodite-engine
pip install -U -r requirements/neuron.txt
APHRODITE_TARGET_DEVICE="neuron" pip install -e .
```

--------------------------------

### Install Python and Dependencies on WSL2

Source: https://github.com/dphnai/aphrodite-engine/wiki/1.-Installation

Install essential packages including Python, pip, git, and other utilities on a WSL2 environment. This is a prerequisite for installing Aphrodite Engine on Windows via WSL2.

```sh
sudo apt update && sudo apt upgrade -y && sudo apt install python3 python3-pip git wget curl bzip2 tar
```

--------------------------------

### Install Neuron Drivers and Tools

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-neuron.md

Installs the Neuron driver, runtime, and tools on Ubuntu 22.04 LTS. Ensure your Linux system is configured for Neuron repository updates and has the necessary OS headers and git installed.

```shell
# Configure Linux for Neuron repository updates
. /etc/os-release
sudo tee /etc/apt/sources.list.d/neuron.list > /dev/null <<EOF
deb https://apt.repos.neuron.amazonaws.com ${VERSION_CODENAME} main
EOF
wget -qO - https://apt.repos.neuron.amazonaws.com/GPG-PUB-KEY-AMAZON-AWS-NEURON.PUB | sudo apt-key add -

# Update OS packages
sudo apt-get update -y

# Install OS headers
sudo apt-get install linux-headers-$(uname -r) -y

# Install git
sudo apt-get install git -y

# install Neuron Driver
sudo apt-get install aws-neuronx-dkms=2.* -y

# Install Neuron Runtime
sudo apt-get install aws-neuronx-collectives=2.*

```

```shell
sudo apt-get install aws-neuronx-runtime-lib=2.*

# Install Neuron Tools
sudo apt-get install aws-neuronx-tools=2.*

# Add PATH
export PATH=/opt/aws/neuron/bin:$PATH
```

--------------------------------

### Prevent Dependency Installation

Source: https://github.com/dphnai/aphrodite-engine/blob/main/CMakeLists.txt

Configures CMake to prevent the installation of dependencies like cutlass by default.

```cmake
install(CODE "set(CMAKE_INSTALL_LOCAL_ONLY TRUE)" ALL_COMPONENTS)
```

--------------------------------

### Start Head Node for Ray Cluster

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/distributed.md

Run this command on the designated head node to initiate the Ray cluster. Ensure the image, head node IP, and model path are correctly specified.

```bash
bash run_cluster.sh \
  alpindale/aphrodite-openai \
  ip_of_head_node \
  --head \
  /path/to/the/huggingface/home/in/this/node
```

--------------------------------

### Install DeepSpeed for Quantization

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/quantization/quantization-methods.md

Install the DeepSpeed library to enable DeepspeedFP quantization. Ensure you have version 0.14.2 or higher.

```sh
pip install deepspeed>=0.14.2
```

--------------------------------

### Quantized Model Output Directory Structure

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/fp8/quantizer/README.md

Example output showing the file structure within the specified output directory after quantization. This includes JSON configuration, NPZ (for reference), and Safetensors model files.

```shell
# ll ./ll2_7b_fp8/
total 19998244
drwxr-xr-x 2 root root        4096 Feb  7 01:08 ./
drwxrwxr-x 8 1060 1061        4096 Feb  7 01:08 ../
-rw-r--r-- 1 root root      176411 Feb  7 01:08 llama_tp1.json
-rw-r--r-- 1 root root 13477087480 Feb  7 01:09 llama_tp1_rank0.npz
-rw-r--r-- 1 root root  7000893272 Feb  7 01:08 rank0.safetensors
#
```

--------------------------------

### Install Aphrodite Build Requirements

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Installs the necessary Python packages for building Aphrodite, including PyTorch CPU wheels.

```sh
python3 -m pip install -U pip
python3 -m pip install -r requirements/build.txt --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Aphrodite Throughput Benchmark Script Help

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/fp8/README.md

Displays the help message and available arguments for the `benchmark_throughput.py` script. This is useful for understanding all configuration options for benchmarking.

```bash
python3 benchmarks/benchmark_throughput.py --help
```

--------------------------------

### Install Python Dependencies for CPU

Source: https://github.com/dphnai/aphrodite-engine/wiki/CPU-Support

Installs necessary Python packages including PyTorch with CPU-specific wheels. Ensure pip is up-to-date.

```shell
$ pip install -U pip
$ pip install wheel packaging ninja setuptools>=49.4.0 numpy
$ pip install -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/dphnai/aphrodite-engine/blob/main/CONTRIBUTING.md

Install the Aphrodite Engine in editable mode and its development dependencies. Ensure you build in editable mode with '-e'.

```sh
pip install -e .
pip install -r requirements/dev.txt
```

--------------------------------

### Configure and Run Auto-Tuning Script

Source: https://github.com/dphnai/aphrodite-engine/blob/main/benchmarks/auto_tune/README.md

Set essential variables like model, system, and parallelism, then execute the auto-tuning script. Environment variables can override default values.

```bash
MODEL=meta-llama/Llama-3.3-70B-Instruct SYSTEM=TPU TP=8 DOWNLOAD_DIR='' INPUT_LEN=128 OUTPUT_LEN=2048 MAX_MODEL_LEN=2300 MIN_CACHE_HIT_PCT=0 MAX_LATENCY_ALLOWED_MS=100000000000 NUM_SEQS_LIST="128 256" NUM_BATCHED_TOKENS_LIST="1024 2048 4096" APHRODITE_LOGGING_LEVEL=DEBUG bash auto_tune.sh
```

--------------------------------

### Build and Run CPU Backend Docker Image

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/installation/installation-cpu.md

Builds a Docker image for the CPU backend and runs it with host networking and IPC enabled. Includes optional arguments for CPU and memory binding.

```sh
docker build -f Dockerfile.cpu -t aphrodite-cpu --shm-size=4g .
docker run -it \
  --rm \
  --network=host \
  --ipc=host \
  -p 2242:2242 \
  #--cpuset-cpus=<cpu-id-list, optional> \
  #--cpuset-mems=<memory-node, optional> \
  aphrodite-cpu
```

--------------------------------

### Run Aphrodite Server with Custom Chat Template via CLI

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/usage/1-getting-started.md

Start the Aphrodite server with a custom chat template file specified via the --chat-template argument.

```sh
aphrodite run meta-llama/Meta-Llama-3.1-8B-Instruct -tp 2 --chat-template ./examples/chat_templates/chatml.jinja
```

--------------------------------

### View Example Batch File Content

Source: https://github.com/dphnai/aphrodite-engine/blob/main/examples/openai_api/batched_inference.md

Display the content of the downloaded batch file using the cat command. The file should contain JSONL objects, each representing a request.

```bash
cat batched_example.jsonl
```

--------------------------------

### Enable and Use Prompt Caching

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/prompt-caching/introduction.md

Enable prompt caching by setting `enable_prefix_caching=True` during LLM initialization. Subsequent queries with the same prompt prefix will automatically benefit from the cached KV cache, as shown in the second `get_generation_time` call.

```python
from fastchat.model import LLM, SamplingParams
from fastchat.utils import get_generation_time

# Assume LONG_PROMPT is defined elsewhere
LONG_PROMPT = """
This is a very long prompt that will be repeated across queries.
It contains a lot of context and information.
"""

# Enable prompt caching
llm = LLM(
    model='lmsys/longchat-13b-16k',
    enable_prefix_caching=True
)

sampling_params = SamplingParams(temperature=0, max_tokens=100)

# First query: Computes and caches the KV cache for LONG_PROMPT
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of John Doe? Your answer: The age of John Doe is ",
)

# Second query: Reuses the cached KV cache for LONG_PROMPT, resulting in faster generation
get_generation_time(
    llm,
    sampling_params,
    LONG_PROMPT + "Question: what is the age of Zack Blue? Your answer: The age of Zack Blue is ",
)
```

--------------------------------

### Initialize LLM and Generate Outputs

Source: https://github.com/dphnai/aphrodite-engine/wiki/2.-Usage

Initializes the LLM with a specified model and tensor parallel size, then generates outputs for the given prompts and sampling parameters. Ensure the model path is correct.

```python
llm = LLM(model="mistralai/Mistral-7B-v0.1", tensor_parallel_size=2)
outputs = llm.generate(prompts, sampling_params)
```

--------------------------------

### Example `adjust_request` Implementation

Source: https://github.com/dphnai/aphrodite-engine/blob/main/docs/src/content/docs/developer/tool-plugin.md

Demonstrates how to use the optional `adjust_request` method to modify the `ChatCompletionRequest` before it's processed. This example disables special token skipping.

```python
def adjust_request(
    self, 
    request: ChatCompletionRequest
) -> ChatCompletionRequest:
    # Example: Disable special token skipping
    request.skip_special_tokens = False
    return request

```