### Start Dynasor Server with vLLM Backend

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

One-click server setup for Dynasor reasoning optimization using vLLM. Ensure vLLM is installed and configured.

```bash
# Option 1: One-click server setup with vLLM backend
VLLM_USE_V1=1 python -m arctic_inference.dynasor.vllm_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    -tp 1 \
    --enable-chunked-prefill \
    --enforce-eager \
    --port 8080
```

--------------------------------

### Run Arctic Speculator Example

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/spec_dec/README.md

Execute an example conversion using specified base and draft models for Arctic Speculator. Ensure Python is installed and the script is available.

```console
$ python offline_inference_spec_dec.py
```

--------------------------------

### Install ArcticInference

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Clone the ArcticInference repository, checkout a specific commit, and install it.

```bash
git clone https://github.com/snowflakedb/ArcticInference.git
cd ArcticInference
git checkout d096fdf
pip install .
```

--------------------------------

### Start vLLM Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/dynasor/README.md

This command starts a standalone vLLM server. It serves as the backend for the Dynasor proxy server.

```bash
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B -tp 1 --enable-chunked-prefill --enforce-eager
```

--------------------------------

### Start Dynasor Server (vLLM Backend)

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/dynasor/README.md

Use this command to start the Dynasor server with a vLLM backend and OpenAI proxy server. Ensure the model path is correct.

```bash
VLLM_USE_V1=1 python -m arctic_inference.dynasor.vllm_server \
--model deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B  \
-tp 1 --enable-chunked-prefill --enforce-eager \
--port 8080
```

--------------------------------

### Run SwiftKV Example Conversation

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/swiftkv/README.md

Execute an example conversation using the Snowflake/Llama-3.1-SwiftKV-8B-Instruct model via the provided Python script. This demonstrates prompt processing with SwiftKV.

```console
$ python offline_inference_swiftkv.py
```

--------------------------------

### Start vLLM Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Launches a vLLM server with a specified model. This is a prerequisite for running the HTTP benchmark against vLLM.

```bash
vllm serve Snowflake/snowflake-arctic-embed-m-v1.5
```

--------------------------------

### Install Arctic Inference from Source

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/index.rst

Clone the Arctic Inference repository and build/install it from source if you prefer not to use PyPI.

```bash
git clone https://github.com/snowflakedb/ArcticInference.git && pip install ./ArcticInference
```

--------------------------------

### Install vLLM

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Install a specific version of vLLM using pip.

```bash
pip install vllm==v0.10.1
```

--------------------------------

### Install Matplotlib

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Install the matplotlib library for plotting results.

```bash
pip install matplotlib
```

--------------------------------

### Run Arctic Ulysses Example Conversation

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/ulysses/README.md

Execute an example conversation using Arctic Ulysses with TP=2 and SP=2. This command initiates the inference process and will output the conversation, including the model's responses.

```console
$ python offline_inference_ulysses.py
```

--------------------------------

### Start Dynasor Client

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/dynasor/README.md

Initiate the Dynasor client to interact with the running server. Adjust probe-interval and certainty-window as needed for performance tuning.

```bash
cd projects/dynasor/
python openai_client.py \
--base-url http://localhost:8080/v1 \
--probe-interval 32 \
--certainty-window 2 \
--prompt "2 + 2 ="
```

--------------------------------

### Start ArcticInference gRPC Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Starts the ArcticInference embedding replica manager, which serves models via gRPC. This is necessary for running the gRPC benchmark.

```bash
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --port 50050
```

--------------------------------

### Install gRPC Dependencies

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Install the necessary Python packages for gRPC communication, including grpcio, grpcio-tools, protobuf, and vllm.

```bash
pip install grpcio grpcio-tools protobuf vllm
```

--------------------------------

### Start Arctic Inference gRPC Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Use this command to start the Arctic Inference gRPC server. Specify the model, number of replicas, and load balancing strategy.

```bash
python -m arctic_inference.embedding.replica_manager \
        --model Snowflake/snowflake-arctic-embed-m-v1.5 \
        --num-replicas 4 \
        --load-balancing round_robin
```

--------------------------------

### Start Replica Manager for Embedding Benchmark

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Initiates the replica manager for embedding benchmarks. Ensure this is run before starting the benchmark. It specifies the model, number of replicas, and load balancing strategy.

```bash
python -m arctic_inference.embedding.replica_manager \
    --model Snowflake/snowflake-arctic-embed-m-v1.5 \
    --num-replicas 4 \
    --load-balancing round_robin
```

--------------------------------

### Install Arctic Inference from PyPI

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/index.rst

Use this command to install the latest Arctic Inference and a compatible vLLM version from PyPI.

```bash
pip install arctic-inference[vllm]
```

--------------------------------

### Start Replica Manager

Source: https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/embedding/README.md

Launches the replica manager for horizontal scaling of vLLM inference. Specify the model name and desired number of replicas. The load balancing policy can also be set.

```bash
python -m arctic_inference.embedding.replica_manager [options]
```

```bash
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 4
```

```bash
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 4 --load-balancing least_loaded
```

```bash
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 32 --load-balancing round_robin
```

--------------------------------

### Configure Full Speculative Decoding with Arctic

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Configure speculative decoding using the `--speculative-config` parameter with the 'arctic' method. This example includes all available parameters for fine-tuning.

```bash
# Full speculative config with all parameters
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-config '{
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true,
        "disable_by_batch_size": 64,
        "suffix_cache_max_depth": 64,
        "suffix_max_spec_factor": 1.0,
        "suffix_max_spec_offset": 0.0,
        "suffix_min_token_prob": 0.1,
        "suffix_cache_max_requests": 100000
    }'
```

--------------------------------

### JSONL Dataset Format Example

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/trace/README.md

Each line in the JSONL file represents a request with a timestamp, input length, and output length. The timestamp indicates when the request should be sent relative to the start time.

```jsonl
{"timestamp": 15, "input_length": 1000, "output_length": 128}
...
```

--------------------------------

### Run Mooncake Conversation Trace Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Start the vLLM server for the Mooncake conversation trace with ArcticInference and Shift Parallelism enabled. Requires specific model and configurations.

```bash
ARCTIC_INFERENCE_ENABLED=1 VLLM_DISABLE_COMPILE_CACHE=1 vllm serve Qwen/Qwen3-32B-FP8 \
  --disable-log-requests \
  --no-enable-prefix-caching \
  --ulysses-sequence-parallel-size 8 \
  --enable-shift-parallel \
  --max-num-batched-tokens 131072
```

--------------------------------

### Compare ArcticInference and vLLM Performance via HTTP Benchmark

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Runs an HTTP benchmark comparing ArcticInference and vLLM performance. This involves starting both servers and then executing the benchmark with specific configurations for batch sizes, requests, prompt length, concurrency, and distribution.

```bash
python benchmark_http.py --server http://localhost:8000 --batch-sizes 1,8,64 --requests 1280 --concurrency 256 --prompt-length 508 --distribution normal
```

--------------------------------

### Run Azure LLM Trace Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Start the vLLM server for the Azure LLM trace with ArcticInference and Shift Parallelism enabled. Requires specific model and configurations.

```bash
ARCTIC_INFERENCE_ENABLED=1 VLLM_DISABLE_COMPILE_CACHE=1 vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
  --disable-log-requests \
  --no-enable-prefix-caching \
  --ulysses-sequence-parallel-size 8 \
  --enable-shift-parallel \
  --max-num-batched-tokens 131072
```

--------------------------------

### Start Separate Dynasor Proxy Server

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Starts the Dynasor proxy server after a vLLM server has been initiated. Requires specifying the target vLLM server URL.

```bash
# Option 2: Separate proxy server (requires running vLLM server first)
# Start vLLM server
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    -tp 1 --enable-chunked-prefill --enforce-eager

# Start Dynasor proxy server
python -m arctic_inference.dynasor.openai_server \
    --target-base-url http://localhost:8000 \
    --port 8080
```

--------------------------------

### Install Arctic Inference with Embedding Support

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Install the arctic-inference package with the embedding extra to enable optimized embedding functionalities. This command installs the necessary dependencies for embedding.

```bash
pip install arctic-inference[embedding]
```

--------------------------------

### Install Arctic Inference with Embedding Support

Source: https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/embedding/README.md

Installs the arctic_inference package with embedding capabilities and its dependencies. This command also compiles the proto file into Python code.

```bash
pip install arctic_inference[embedding]
```

--------------------------------

### Start Dynasor Proxy Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/projects/dynasor/README.md

Launch the Dynasor proxy server, which forwards requests to the vLLM backend. Specify the target vLLM base URL and the desired port.

```bash
python -m arctic_inference.dynasor.openai_server \
--target-base-url http://localhost:8000 \
--port 8080
```

--------------------------------

### Run Arctic Inference Replica Server Manually

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Manually start a replica server for debugging purposes. The replica manager typically starts these automatically. Specify the model name and optionally host, port, and workers.

```bash
python -m arctic_inference.embedding.replica --model <MODEL_NAME> [--host <HOST>] [--port <PORT>] [--workers <WORKERS>] [--disable-log-request]
# e.g., 
python -m arctic_inference.embedding.replica --model Snowflake/snowflake-arctic-embed-m-v1.5 --host 127.0.0.1 --port 50000
```

--------------------------------

### Create Python 3.10 Virtual Environment

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Use this command to create a virtual environment with Python 3.10 if it's already installed.

```bash
python3.10 -m venv myvenv
source myvenv/bin/activate
```

--------------------------------

### Benchmark Script for H200 (Short Sequences)

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Uses a shell script for benchmarking on H200 GPUs with configurations for short sequences. This example specifies a large number of replicas for high concurrency.

```bash
bash benchmark/embedding/run_benchmark.sh \
    Snowflake/snowflake-arctic-embed-m-v1.5 \
    10240 50 1024 fixed 1,16,64 32
```

--------------------------------

### Run Replica Server Manually

Source: https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/embedding/README.md

Manually starts a replica server for debugging purposes. Requires specifying the model name and optionally host, port, workers, and request logging behavior.

```bash
python -m arctic_inference.embedding.replica --model <MODEL_NAME> [--host <HOST>] [--port <PORT>] [--workers <WORKERS>] [--disable-log-request]
```

```bash
python -m arctic_inference.embedding.replica --model Snowflake/snowflake-arctic-embed-m-v1.5 --host 127.0.0.1 --port 50000
```

--------------------------------

### Run Embedding Benchmark

Source: https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/embedding/README.md

Executes the embedding benchmark. This involves starting the replica manager first, then running the benchmark script with specified server and performance parameters.

```bash
python -m arctic_inference.embedding.replica_manager \
    --model Snowflake/snowflake-arctic-embed-m-v1.5 \
    --num-replicas 4 \
    --load-balancing round_robin
```

```bash
python -m benchmark/embedding/benchmark.py \
    --model "Snowflake/snowflake-arctic-embed-m-v1.5" \
    --server localhost:50050 \
    --batch-sizes 1,16,64 \
    --requests 1024 \
    --concurrency 64 \
    --prompt-length 512
```

--------------------------------

### Shift Parallelism Configuration Example

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Configure Shift Parallelism for Llama 70B on 8 GPUs, demonstrating dynamic switching between tensor and sequence parallelism based on batch size. The threshold is set to 256.

```bash
# Shift Parallelism with Llama 70B on 8 GPUs (TP=4, SP=2)
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --enable-shift-parallel \
    --tensor-parallel-size 4 \
    --ulysses-sequence-parallel-size 2 \
    --shift-parallel-threshold 256

# When batch_size <= 256: uses TP=8 (SP*TP combined) for low latency
# When batch_size > 256: uses SP=2, TP=4 for high throughput
```

--------------------------------

### Load SwiftKV Model with Arctic Inference

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/swiftkv.rst

To use SwiftKV, load a model fine-tuned with SwiftKV. Ensure ARCTIC_INFERENCE_ENABLED is set to 1. This example loads a Llama-3.3 SwiftKV model using vLLM's OpenAI API server.

```bash
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model Snowflake/Llama-3.3-SwiftKV-70B-Instruct \
    --tensor-parallel-size 8
```

--------------------------------

### Create Conda Environment with Python 3.10

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Use this command to create a Conda environment with Python 3.10 if it's not installed.

```bash
conda create -n myenv python=3.10
conda activate myenv
```

--------------------------------

### Start Arctic Inference Replica Manager

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Launch the replica manager to horizontally scale vLLM inference. This command manages multiple replica instances and load balances requests between them. Specify the model name and desired number of replicas.

```bash
python -m arctic_inference.embedding.replica_manager [options]
```

```bash
# Start a manager with 4 replicas using a round-robin load balancing policy
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 4
```

```bash
# Use a least-loaded policy for better handling of varying request complexities
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 4 --load-balancing least_loaded
```

```bash
# this is the command we use in benchmark H200 using short sequence
python -m arctic_inference.embedding.replica_manager --model Snowflake/snowflake-arctic-embed-m-v1.5 --num-replicas 32 --load-balancing round_robin
```

--------------------------------

### Deploy High-Throughput Embedding Inference

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Scale embedding inference with parallel tokenization, load balancing, and multiple GPU replicas using the replica manager. Install embedding dependencies first.

```bash
# Install embedding dependencies
pip install arctic-inference[embedding]
```

```bash
# Start replica manager with 4 replicas and round-robin load balancing
python -m arctic_inference.embedding.replica_manager \
    --model Snowflake/snowflake-arctic-embed-m-v1.5 \
    --num-replicas 4 \
    --load-balancing round_robin \
    --port 50050
```

```bash
# Start with least-loaded policy for variable request complexity
python -m arctic_inference.embedding.replica_manager \
    --model Snowflake/snowflake-arctic-embed-m-v1.5 \
    --num-replicas 4 \
    --load-balancing least_loaded
```

```bash
# High-throughput configuration for H200 GPUs (short sequences)
python -m arctic_inference.embedding.replica_manager \
    --model Snowflake/snowflake-arctic-embed-m-v1.5 \
    --num-replicas 32 \
    --load-balancing round_robin
```

--------------------------------

### Run gRPC Benchmark with Default Settings

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Executes the gRPC benchmark tool using all default parameters for server address, model, batch sizes, requests, concurrency, and prompt length.

```bash
python benchmark.py
```

--------------------------------

### Serve with vLLM CLI using Arctic Inference

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Launch an optimized vLLM server with Shift Parallelism, Arctic Speculator, Suffix Decoding, and SwiftKV enabled. Set the ARCTIC_INFERENCE_ENABLED environment variable to activate optimizations.

```bash
# Start vLLM server with all Arctic Inference optimizations
ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
    --quantization "fp8" \
    --tensor-parallel-size 1 \
    --ulysses-sequence-parallel-size 2 \
    --enable-shift-parallel \
    --speculative-config '{
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true,
        "disable_by_batch_size": 64
    }'
```

```bash
# Query the server using curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Snowflake/Llama-3.1-SwiftKV-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Write a short poem about AI."}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'
```

--------------------------------

### Run HTTP Benchmark with Default Settings

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Executes the HTTP benchmark tool using default settings for server address, model, endpoint, batch sizes, requests, concurrency, and prompt length.

```bash
python benchmark_http.py
```

--------------------------------

### Serve Model with Arctic Inference Optimizations

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/index.rst

This command serves a model using vLLM with Arctic Inference optimizations, including Shift Parallelism, Ulysses, and Speculative Decoding.

```bash
vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
   --quantization "fp8" \
   --tensor-parallel-size 1 \
   --ulysses-sequence-parallel-size 2 \
   --enable-shift-parallel \
   --speculative-config "{
      \"method\": \"arctic\",
      \"model\":\"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct\",
      \"num_speculative_tokens\": 3,
      \"enable_suffix_decoding\": true,
      \"disable_by_batch_size\": 64
   }"
```

--------------------------------

### Download Trace Datasets

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Download JSONL files containing trace data for benchmarking.

```bash
wget https://zenodo.org/records/18240909/files/AzureLLMInferenceTrace_code_15mins.jsonl
wget https://zenodo.org/records/18240909/files/conversation_trace_15mins.jsonl
```

--------------------------------

### Run vLLM Benchmark Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/trace/README.md

Execute the vLLM benchmark server with a specified model, trace dataset, and options to ignore the end-of-sequence token. The `--trace-dataset-path` flag points to the custom JSONL dataset.

```bash
vllm bench serve --model $model --trace-dataset-path example_trace.jsonl --ignore-eos
```

--------------------------------

### Run Azure LLM Trace Client

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Execute the vLLM benchmark client for the Azure LLM trace dataset.

```bash
vllm bench serve --model RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic --trace-dataset-path AzureLLMInferenceTrace_code_15mins.jsonl --ignore-eos --trace-output-path code_output.csv
```

--------------------------------

### Set Prompt Length Distribution for HTTP Benchmark

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Runs the HTTP benchmark with a fixed prompt length distribution. Other options include 'uniform' and 'normal'.

```bash
python benchmark_http.py --distribution fixed
```

--------------------------------

### Download Hugging Face Models

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Download specified models from Hugging Face to local directories.

```bash
huggingface-cli download RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic --local-dir RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic
huggingface-cli download Qwen/Qwen3-32B-FP8 --local-dir Qwen/Qwen3-32B-FP8
```

--------------------------------

### Run Mooncake Conversation Trace Client

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Execute the vLLM benchmark client for the Mooncake conversation trace dataset.

```bash
vllm bench serve --model Qwen/Qwen3-32B-FP8 --trace-dataset-path conversation_trace_15mins.jsonl --ignore-eos --trace-output-path conversation_output.csv
```

--------------------------------

### Minimal Arctic Speculator Configuration

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/arctic-speculator.rst

Enables Arctic Speculator with default settings. Ensure ARCTIC_INFERENCE_ENABLED is set to 1.

```bash
export ARCTIC_INFERENCE_ENABLED=1

vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{
    "method": "arctic",
    "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
    "num_speculative_tokens": 3
}'
```

--------------------------------

### Serve Models with Arctic Inference

Source: https://github.com/snowflakedb/arcticinference/blob/main/README.md

Serve a vLLM model with Arctic Inference enabled, utilizing optimizations like Shift Parallelism and Speculative Decoding. Ensure ARCTIC_INFERENCE_ENABLED is set to 1.

```bash
ARCTIC_INFERENCE_ENABLED=1 vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
    --quantization "fp8" \
    --tensor-parallel-size 1 \
    --ulysses-sequence-parallel-size 2 \
    --enable-shift-parallel \
    --speculative-config '{
        "method": "arctic",
        "model":"Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true,
        "disable_by_batch_size": 64
    }'
```

--------------------------------

### Configure gRPC Benchmark for Specific Server and Load

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Tests the gRPC server with a specified address, custom batch sizes, and increased concurrency. Adjust these parameters to match your testing environment.

```bash
python benchmark.py --server 0.0.0.0:50050 --batch-sizes 1,4,16,64 --concurrency 64
```

--------------------------------

### Enable Arctic Ulysses with vLLM

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/arctic-ulysses.rst

Specify both tensor-parallel-size and ulysses-sequence-parallel-size when launching vLLM to enable Arctic Ulysses optimization. Ensure ARCTIC_INFERENCE_ENABLED is set to 1.

```bash
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4 \
    --ulysses-sequence-parallel-size 2
```

--------------------------------

### Run Benchmark Script with A10g GPU

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Benchmark script for A10g GPU with long and short sequences. Parameters include model, requests, sequence length, concurrency, fixed, batch sizes, and replicas.

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 1024 512 16 fixed 1,16,64 2
```

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 4096 50 256 fixed 1,16,64 8
```

--------------------------------

### Run Embedding Benchmark

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Execute the embedding benchmark script. Configure the model, server address, batch sizes, number of requests, concurrency, and prompt length.

```bash
python -m benchmark/embedding/benchmark.py \
        --model "Snowflake/snowflake-arctic-embed-m-v1.5" \
        --server localhost:50050 \
        --batch-sizes 1,16,64 \
        --requests 1024 \
        --concurrency 64 \
        --prompt-length 512
```

--------------------------------

### Run Benchmark Script with H200 GPU

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Benchmark script for H200 GPU with long and short sequences. Parameters include model, requests, sequence length, concurrency, fixed, batch sizes, and replicas.

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 1024 512 16 fixed 1,16,64 4
```

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 10240 50 1024 fixed 1,16,64 32
```

--------------------------------

### Run gRPC Benchmark for Arctic Inference

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Execute the benchmark script with specified server, batch sizes, request counts, concurrency, and prompt length. Adjust parameters to identify performance bottlenecks.

```bash
python benchmark.py --server localhost:50050 --batch-sizes 1,8,64 --requests 1280 --concurrency 256 --prompt-length 508 --distribution normal
```

--------------------------------

### Vibe Test Script

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Python script to test vLLM with ArcticInference and Shift Parallelism enabled. Requires vLLM and specific model.

```python
import vllm
from vllm import LLM, SamplingParams

vllm.plugins.load_general_plugins()

llm = LLM(
    model="RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic",
    ulysses_sequence_parallel_size=8,
    enable_shift_parallel=True,
    shift_parallel_threshold=8,
)

conversation = [
    {
        "role": "user",
        "content": "Write an essay about the importance of higher education.",
    },
]

sampling_params = SamplingParams(temperature=0.0, max_tokens=800)

outputs = llm.chat(conversation, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)
```

--------------------------------

### Utilize SwiftKV Model Optimization

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Reduce prefill compute by up to 50% by using SwiftKV-compatible models that enable KV cache reuse across transformer layers. This is automatically enabled when using a SwiftKV model.

```bash
# SwiftKV with Llama 70B (automatically enabled with SwiftKV model)
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model Snowflake/Llama-3.3-SwiftKV-70B-Instruct \
    --tensor-parallel-size 8
```

```bash
# SwiftKV with 8B model and additional optimizations
vllm serve Snowflake/Llama-3.1-SwiftKV-8B-Instruct \
    --quantization "fp8" \
    --tensor-parallel-size 1 \
    --ulysses-sequence-parallel-size 2 \
    --enable-shift-parallel
```

--------------------------------

### Configure HTTP Benchmark for Specific Server

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Tests the HTTP endpoint of a vLLM server at a specified address. Ensure the server is running and accessible.

```bash
python benchmark_http.py --server http://localhost:8000
```

--------------------------------

### Use Normal Distribution for HTTP Prompt Lengths

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Configures the HTTP benchmark to use a normal distribution for prompt lengths. This can simulate more varied real-world input.

```bash
python benchmark_http.py --distribution normal
```

--------------------------------

### High-Performance gRPC Benchmark Configuration

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

This command configures the gRPC benchmark for high-throughput testing on specific hardware (H200), using a particular model, server, batch sizes, request count, concurrency, and prompt length.

```bash
python benchmark.py --model "Snowflake/snowflake-arctic-embed-m-v1.5" \
    --server localhost:50050 \
    --batch-sizes 1,16,64 \
    --requests 10240 \
    --concurrency 1024 \
    --prompt-length 50
```

--------------------------------

### Enable Arctic Ulysses Sequence Parallelism

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Use sequence parallelism to partition input sequences across GPUs, reducing time-to-first-token for long-context tasks. Requires setting the ARCTIC_INFERENCE_ENABLED environment variable.

```bash
# Arctic Ulysses with Llama 70B across 8 GPUs (TP=4, SP=2)
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 4 \
    --ulysses-sequence-parallel-size 2

# Total parallelism = TP * SP = 4 * 2 = 8 GPUs
# Ideal for RAG, summarization, and code generation with long contexts
```

--------------------------------

### Generate gRPC Code

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Run this Python script to generate gRPC client and server code from the proto file. This creates inference_pb2.py and inference_pb2_grpc.py.

```bash
python arctic_inference/embedding/generate_proto.py
```

--------------------------------

### Benchmark Script for H200 (Long Sequences)

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Utilizes a shell script to run benchmarks on H200 GPUs with configurations optimized for long sequences. It takes the model name, sequence length, prompt length, batching strategy, batch sizes, and number of replicas as arguments.

```bash
bash benchmark/embedding/run_benchmark.sh \
    Snowflake/snowflake-arctic-embed-m-v1.5 \
    1024 512 16 fixed 1,16,64 4
```

--------------------------------

### Enable Shift Parallelism in Arctic Inference

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/shift-parallel.rst

Use this command to launch Arctic Inference with Shift Parallelism enabled. Set the model, enable the feature, and configure tensor and sequence parallel sizes. The threshold determines when the switch between TP and SP occurs.

```bash
export ARCTIC_INFERENCE_ENABLED=1

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.3-70B-Instruct \
    --enable-shift-parallel \
    --tensor-parallel-size 4 \
    --ulysses-sequence-parallel-size 2 \
    --shift-parallel-threshold 256
```

--------------------------------

### Run Vibe Test

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/reproducibility/README.md

Execute the vibe test script with specific environment variables to disable compile cache.

```bash
ARCTIC_INFERENCE_ENABLED=1 VLLM_DISABLE_COMPILE_CACHE=1 python vibe_test.py
```

--------------------------------

### Configure Arctic Speculator for Speculative Decoding

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Utilize lightweight draft models (MLP or LSTM) for fast speculative decoding, enabling up to 4x faster task completion. Ensure ARCTIC_INFERENCE_ENABLED is set. Pre-trained speculators are available on Hugging Face.

```bash
# Arctic Speculator with LSTM draft model
export ARCTIC_INFERENCE_ENABLED=1

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-config '{
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3
    }'
```

```bash
# Combined with Suffix Decoding for additional speedup on repetitive workloads
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --speculative-config '{
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": true
    }'
```

--------------------------------

### Use Arctic Inference Client for Embeddings

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/embeddings.rst

Interact with the Arctic Inference service using the client. Provide a prompt and optionally configure sampling parameters like temperature, top-p, top-k, and max tokens. Streaming is also supported.

```bash
python -m arctic_inference.embedding.client --prompt "Your prompt here" [--host <HOST>] [--port <PORT>] [--temperature <TEMP>] [--top-p <TOP_P>] [--top-k <TOP_K>] [--max-tokens <MAX_TOKENS>] [--stream] [--lora-name <LORA_NAME>]
```

--------------------------------

### Run Benchmark Script for Arctic Embeddings

Source: https://github.com/snowflakedb/arcticinference/blob/main/arctic_inference/embedding/README.md

Use this script to benchmark the Snowflake Arctic embedding model. Adjust parameters for sequence length, request count, concurrency, and replicas based on your hardware and testing needs.

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 1024 512 16 fixed 1,16,64 4
```

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 10240 50 1024 fixed 1,16,64 32
```

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 1024 512 16 fixed 1,16,64 2
```

```bash
bash benchmark/embedding/run_benchmark.sh Snowflake/snowflake-arctic-embed-m-v1.5 4096 50 256 fixed 1,16,64 8
```

--------------------------------

### Configure Custom Ops Build with CMake

Source: https://github.com/snowflakedb/arcticinference/blob/main/csrc/custom_ops/CMakeLists.txt

This CMake script sets up the build environment for custom C++/CUDA operations. It requires C++17, specifies CUDA architectures, and finds necessary packages like Torch, pybind11, and CUDA.

```cmake
cmake_minimum_required(VERSION 3.14)

project(CustomOps CXX CUDA)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

set(TORCH_CUDA_ARCH_LIST "7.0 7.5 8.0 8.6 8.9 9.0+PTX")

message(STATUS "TORCH_CMAKE_PREFIX_PATH: ${TORCH_CMAKE_PREFIX_PATH}")

list(APPEND CMAKE_PREFIX_PATH ${TORCH_CMAKE_PREFIX_PATH})

find_package(CUDA REQUIRED)
find_package(pybind11 REQUIRED)
find_package(Torch REQUIRED)

pybind11_add_module(custom_ops
  reshape_and_cache_flash_fp4.cu
  reshape_and_cache_flash_bulk.cu
  speculator_ln.cu
  sum_lstm.cu
  torch_bindings.cpp
)

target_compile_definitions(custom_ops PRIVATE
  -DTORCH_API_INCLUDE_EXTENSION_H
  -DTORCH_EXTENSION_NAME=custom_ops
  -DTORCH_API_INCLUDE_TYPEDEFS
)

target_include_directories(custom_ops PRIVATE ${TORCH_INCLUDE_DIRS})

target_link_libraries(custom_ops PRIVATE
  torch
  ${GPU_LIBRARIES}
  CUDA::cudart
  CUDA::cuda_driver
)
```

--------------------------------

### Hybrid Suffix Decoding with Arctic Speculator Configuration

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/suffix-decoding.rst

Configure suffix decoding to work with Arctic Speculator for a hybrid approach. This combines model-based speculation with suffix-based prediction. Ensure ARCTIC_INFERENCE_ENABLED is set to 1.

```bash
export ARCTIC_INFERENCE_ENABLED=1

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "arctic",
    "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
    "num_speculative_tokens": 3,
    "enable_suffix_decoding": true
  }'
```

--------------------------------

### Specify Model for gRPC Benchmark

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/embedding/README.md

Runs the gRPC benchmark using a specific embedding model. Ensure the model name is valid and accessible by the server.

```bash
python benchmark.py --model "Snowflake/snowflake-arctic-embed-m-v1.5"
```

--------------------------------

### Arctic Speculator with Suffix Decoding

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/arctic-speculator.rst

Combines Arctic Speculator with Suffix Decoding for repetitive patterns. Requires ARCTIC_INFERENCE_ENABLED to be set to 1.

```bash
export ARCTIC_INFERENCE_ENABLED=1

vllm serve meta-llama/Llama-3.1-8B-Instruct \
--speculative-config '{
    "method": "arctic",
    "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
    "num_speculative_tokens": 3,
    "enable_suffix_decoding": true
}'
```

--------------------------------

### Run Embedding Benchmark with Various Batch Sizes

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Executes the embedding benchmark script. This command allows customization of batch sizes, number of requests, concurrency, and prompt length to measure performance.

```bash
python benchmark/embedding/benchmark.py \
    --model "Snowflake/snowflake-arctic-embed-m-v1.5" \
    --server localhost:50050 \
    --batch-sizes 1,16,64 \
    --requests 1024 \
    --concurrency 64 \
    --prompt-length 512
```

--------------------------------

### Apply vLLM Benchmarking Patch

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/trace/README.md

This bash command applies a specific patch to the vLLM benchmarking scripts. Ensure that the vLLM version matches the patch requirements.

```bash
bash apply_vllm_bench_patch_v10p1.sh
```

--------------------------------

### Offline Inference with Arctic Inference Optimizations

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/index.rst

This Python code demonstrates how to perform offline inference using vLLM with Arctic Inference optimizations, including Shift Parallelism, Ulysses, and Speculative Decoding.

```python
import vllm
from vllm import LLM, SamplingParams

vllm.plugins.load_general_plugins()

llm = LLM(
   model="Snowflake/Llama-3.1-SwiftKV-8B-Instruct",
   quantization="fp8",
   tensor_parallel_size=1,
   ulysses_sequence_parallel_size=2,
   enable_shift_parallel=True,
   speculative_config={
      "method": "arctic",
      "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
      "num_speculative_tokens": 3,
      "enable_suffix_decoding": True,
      "disable_by_batch_size": 64,
   },
)

conversation = [
   {
       "role": "user",
       "content": "Write an essay about the importance of higher education.",
   },
]

sampling_params = SamplingParams(temperature=0.0, max_tokens=800)

outputs = llm.chat(conversation, sampling_params=sampling_params)
```

--------------------------------

### Offline Batch Inference with Python API

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Use the vLLM Python API with Arctic Inference for offline batch processing. Load the general plugins explicitly to enable optimizations before initializing the LLM.

```python
import vllm
from vllm import LLM, SamplingParams

# Load Arctic Inference plugin to enable optimizations
vllm.plugins.load_general_plugins()

# Initialize LLM with Arctic optimizations
llm = LLM(
    model="Snowflake/Llama-3.1-SwiftKV-8B-Instruct",
    quantization="fp8",
    tensor_parallel_size=1,
    ulysses_sequence_parallel_size=2,
    enable_shift_parallel=True,
    speculative_config={
        "method": "arctic",
        "model": "Snowflake/Arctic-LSTM-Speculator-Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 3,
        "enable_suffix_decoding": True,
        "disable_by_batch_size": 64,
    },
)

# Define conversation and sampling parameters
conversation = [
    {
        "role": "user",
        "content": "Write an essay about the importance of higher education.",
    },
]

sampling_params = SamplingParams(temperature=0.0, max_tokens=800)

# Generate response
outputs = llm.chat(conversation, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
```

--------------------------------

### Integrate Dynasor with OpenAI Python Client

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Use the standard OpenAI Python client to interact with the Dynasor proxy. Custom parameters like `dynasor` can be passed in `extra_body`.

```python
from openai import OpenAI

# Initialize client pointing to Dynasor proxy
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="EMPTY"
)

# Generate with Dynasor optimization
response = client.chat.completions.create(
    messages=[
        {"role": "system", "content": "You are a helpful math assistant."},
        {"role": "user", "content": "Solve: What is 15% of 240?"}
    ],
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B",
    max_tokens=2048,
    extra_body={
        # Dynasor parameters for dynamic execution
        "dynasor": {
            "probe_interval": 32,      # Check certainty every N tokens
            "certainty_window": 2       # Window size for certainty evaluation
        }
    },
    stream=True
)

# Process streaming response
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
```

--------------------------------

### Generate Sequences with Specific Lengths

Source: https://github.com/snowflakedb/arcticinference/blob/main/benchmark/rollout/README.md

Use `max_tokens_n` as a list and set `ignore_eos=True` to ensure each sequence generates a specific number of tokens. This is useful when precise output lengths are required.

```python
# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    # "The future of AI is",
]

sampling_params = [SamplingParams(n=2,
                                 temperature=0.8,
                                 top_p=1.0,
                                 max_tokens_n=[25, 50],
                                 ignore_eos=True,
                                ),
                   SamplingParams(n=3,
                                 temperature=0.8,
                                 top_p=1.0,
                                 max_tokens_n=[5, 10, 15],
                                 ignore_eos=True,
                                ),
                   SamplingParams(n=1,
                                 temperature=0.8,
                                 top_p=1.0,
                                 max_tokens=100,
                                 ignore_eos=True,
                                 # max_tokens_n=[100], this will be ineffective since n = 1
                                ),
                   ]

outputs = llm.generate(prompts, sampling_params=sampling_params)
```

--------------------------------

### Generate Embeddings with InferenceClient

Source: https://context7.com/snowflakedb/arcticinference/llms.txt

Use the gRPC client to generate embeddings from text prompts. Requires server health check and replica info retrieval.

```python
import asyncio
from arctic_inference.embedding.client import InferenceClient

async def generate_embeddings():
    # Initialize client
    client = InferenceClient(host="localhost", port=50050)

    try:
        # Check server health
        health = await client.health_check()
        if not health.healthy:
            print(f"Server unhealthy: {health.message}")
            return

        # Get replica information
        replica_info = await client.get_replica_info()
        print(f"Replicas: {replica_info.n_healthy_replicas}/{replica_info.n_replicas}")

        # Generate embeddings for batch of texts
        texts = [
            "What is machine learning?",
            "How does neural network training work?",
            "Explain transformer architecture.",
        ]

        embeddings = await client.embed(
            prompts=texts,
            model_name="Snowflake/snowflake-arctic-embed-m-v1.5"
        )

        for i, emb in enumerate(embeddings):
            print(f"Text {i}: embedding shape = {emb.shape}")

    finally:
        await client.close()

# Run the async function
asyncio.run(generate_embeddings())
```

--------------------------------

### Minimal Suffix-Only Decoding Configuration

Source: https://github.com/snowflakedb/arcticinference/blob/main/docs/suffix-decoding.rst

Use this configuration for suffix-only decoding with Llama-3.1-8B-Instruct. Ensure ARCTIC_INFERENCE_ENABLED is set to 1.

```bash
export ARCTIC_INFERENCE_ENABLED=1

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "suffix"
  }'
```