### Build ExLlamaV3 from Source

Source: https://github.com/turboderp-org/exllamav3/blob/master/README.md

Clone the repository and install dependencies. Ensure Torch is installed separately.

```sh
# Clone the repo
git clone https://github.com/turboderp-org/exllamav3
cd exllamav3

# (Optional) switch to dev branch for latest in-progress features
git checkout dev

# Install requirements (make sure you install Torch separately)
pip install -r requirements.txt
```

```sh
pip install .
```

--------------------------------

### Install ExLlamaV3 via prebuilt wheel

Source: https://github.com/turboderp-org/exllamav3/blob/master/README.md

Install the library using a specific prebuilt wheel from the releases page. Ensure PyTorch with CUDA 12.4 or later is installed beforehand.

```sh
pip install https://github.com/turboderp-org/exllamav3/releases/download/v0.0.6/exllamav3-0.0.6+cu128.torch2.8.0-cp313-cp313-linux_x86_64.whl
```

--------------------------------

### Install ExLlamaV3 via PyPi

Source: https://github.com/turboderp-org/exllamav3/blob/master/README.md

Install the package using pip. Requires CUDA toolkit and build prerequisites.

```sh
pip install exllamav3
```

--------------------------------

### Initialize Models with model_init

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Simplify script setup by using the model_init module for argument parsing and standardized model loading.

```python
import argparse
from exllamav3 import model_init, Generator

# Create argument parser with model loading args
parser = argparse.ArgumentParser()
model_init.add_args(
    parser,
    cache=True,
    default_cache_size=8192,
    add_sampling_args=True
)
parser.add_argument("-prompt", type=str, default="Hello!")
args = parser.parse_args()

# Load model using helper (returns model, config, cache, tokenizer)
model, config, cache, tokenizer = model_init.init(args)

# Get sampler from args
sampler = model_init.get_arg_sampler(args)

# Create generator and generate
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)
response = generator.generate(
    prompt=args.prompt,
    sampler=sampler,
    max_new_tokens=200,
    add_bos=True,
    completion_only=True
)
print(response)

# Run with: python script.py -m /path/to/model -prompt "Your prompt here"
```

--------------------------------

### Run CLI Chatbot Example

Source: https://github.com/turboderp-org/exllamav3/blob/master/README.md

Execute the provided chatbot script with a specified model path and prompt mode.

```sh
python examples/chat.py -m <input_dir> -mode <prompt_mode> 

# E.g.:
python examples/chat.py -m /mnt/models/llama3.1-8b-instruct-exl3 -mode llama3

# Wealth of options
python examples/chat.py -h
```

--------------------------------

### Resume Interrupted EXL3 Conversion Job

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md

If a conversion job is interrupted, use this command to resume it from the last checkpoint. Ensure you specify the working directory where the job was originally started.

```sh
python convert.py -w /mnt/temp/exl3 -r
```

--------------------------------

### Configure samplers for generation

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Initialize the generator and apply different sampling strategies including greedy, Top-P, and complex combinations.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator
from exllamav3.generator.sampler import (
    DefaultSampler,      # Min-P 0.08, temperature 0.8
    ArgmaxSampler,       # Greedy decoding (top-1)
    TopKSampler,         # Top-K truncation
    TopPSampler,         # Nucleus sampling
    ComboSampler,        # Configurable multi-step sampler
)

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

prompt = "The meaning of life is"

# Greedy decoding (deterministic)
response = generator.generate(
    prompt=prompt,
    sampler=ArgmaxSampler(),
    max_new_tokens=50,
    add_bos=True,
    completion_only=True
)
print(f"Greedy: {response}")

# Top-P (nucleus) sampling with temperature
response = generator.generate(
    prompt=prompt,
    sampler=TopPSampler(top_p=0.9, temperature=0.7),
    max_new_tokens=50,
    add_bos=True,
    completion_only=True
)
print(f"Top-P: {response}")

# Full control with ComboSampler
response = generator.generate(
    prompt=prompt,
    sampler=ComboSampler(
        rep_p=1.1,              # Repetition penalty
        freq_p=0.1,             # Frequency penalty
        temperature=0.8,        # Temperature scaling
        min_p=0.05,             # Min-P truncation
        top_k=50,               # Top-K truncation
        top_p=0.95,             # Top-P truncation
        temp_last=True,         # Apply temperature after truncation
    ),
    max_new_tokens=50,
    add_bos=True,
    completion_only=True
)
print(f"Combo: {response}")
```

--------------------------------

### Load ExLlamaV3 Model and Tokenizer

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Load model configuration, tokenizer, and inference engine from a local directory. Initializes a KV cache and loads the model to the GPU. Demonstrates basic text encoding and decoding.

```python
from exllamav3 import Config, Model, Cache, Tokenizer

# Load configuration from model directory
config = Config.from_directory("/path/to/model/")

# Create tokenizer from config
tokenizer = Tokenizer.from_config(config)

# Create model instance (not loaded yet)
model = Model.from_config(config)

# Create a KV cache with 8192 token capacity
cache = Cache(model, max_num_tokens=8192)

# Load model to GPU with progress bar
model.load(progressbar=True)

# Encode and decode text
input_ids = tokenizer.encode("Hello, world!", add_bos=True)
text = tokenizer.decode(input_ids)
print(f"Encoded shape: {input_ids.shape}, Decoded: {text}")
```

--------------------------------

### Convert Models to EXL3 Format

Source: https://github.com/turboderp-org/exllamav3/blob/master/README.md

Use the conversion script to quantize models. Requires an input directory, output directory, working directory, and bitrate.

```sh
# Convert model
python convert.py -i <input_dir> -o <output_dir> -w <working_dir> -b <bitrate>

# Resume an interrupted quant job
python convert.py -w <working_dir> -r

# More options
python convert.py -h
```

--------------------------------

### Convert Models to EXL3 Format

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Utilize the `convert.py` script to transform HuggingFace models into the optimized EXL3 quantization format. Options include specifying bits per weight, working directory, and using multiple GPUs for faster conversion.

```bash
# Basic conversion to 4 bits per weight
python convert.py \
    -i /path/to/source/model \
    -o /path/to/output/model \
    -w /path/to/working/directory \
    -b 4.0

# Resume interrupted conversion
python convert.py -w /path/to/working/directory -r

# Multi-GPU conversion for faster processing
python convert.py \
    -i /path/to/source/model \
    -o /path/to/output/model \
    -w /path/to/working/directory \
    -b 3.75 \
    -d 0,1,2 \
    -hb 6 \
    -hq

# Options:
# -b / --bits: Target bits per weight (e.g., 2.0, 3.0, 4.0, 6.0)
# -hb / --head_bits: Bits for output layer (1-8, default 6)
# -hq / --hq: Higher quality for attention/shared layers
# -d / --devices: GPUs to use (e.g., "0,1,2")
# -r / --resume: Resume interrupted conversion
```

--------------------------------

### Configure and Use Loop Detection in ExLlamaV3

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

This snippet shows how to initialize ExLlamaV3 components and set up a generation job with loop detection. The `stop_on_loop` parameter takes a tuple specifying the window size and minimum repetitions to trigger detection. Ensure the model, cache, and tokenizer are properly loaded before creating the generator.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator, Job

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

prompt = "Count from 1 to infinity:"

job = Job(
    input_ids=tokenizer.encode(prompt, add_bos=True),
    max_new_tokens=1000,
    stop_on_loop=(300, 3),  # (window_size, min_reps) - detect 3+ reps in 300 tokens
)
generator.enqueue(job)

response = ""
eos_reason = None
while generator.num_remaining_jobs():
    for result in generator.iterate():
        if result["stage"] == "streaming":
            response += result.get("text", "")
        if result["eos"]:
            eos_reason = result["eos_reason"]

print(f"Response: {response[:500]}...")  # Truncate for display
print(f"Stopped due to: {eos_reason}")  # "loop_detected" if loop found
```

--------------------------------

### Initialize ExLlamaV3 Generator

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Sets up the Generator class for managing inference jobs with continuous batching and paged attention. Requires model, cache, and tokenizer instances. Configurable with max batch and chunk sizes.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator

# Initialize model components
config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=16384)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)

# Create generator with model, cache, and tokenizer
generator = Generator(
    model=model,
    cache=cache,
    tokenizer=tokenizer,
    max_batch_size=256,      # Max concurrent sequences
    max_chunk_size=2048,     # Max tokens per prefill chunk
)

# Simple generation
response = generator.generate(
    prompt="Write a haiku about programming:",
    max_new_tokens=100,
    completion_only=True,
    add_bos=True
)
print(response)
```

--------------------------------

### Convert Model to EXL3 Format

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md

Use this command to convert a source model in Hugging Face format to the EXL3 format. Specify input, output, and working directories, along with the target bits per weight.

```sh
python convert.py -i /mnt/models/llama3.1-70b-instruct \
                  -o /mnt/models/llama3.1-70b-instruct-exl3-3.75bpw \
                  -w /mnt/temp/exl3 \
                  -b 3.75
```

--------------------------------

### EXL3 Quantizer Implementation

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md

This snippet refers to the Python code for the EXL3 quantizer. It is used for implementing the EXL3 quantization format, a variant of QTIP.

```python
from exllamav3.modules.quant.exl3_lib.quantize import quantize
```

--------------------------------

### Compare Quantization Formats Script

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md

This Python script is used to perform an apples-to-apples comparison between different quantization formats by measuring perplexity on the wiki2 test set across various bitrates.

```python
from eval.compare_q import compare_quantizers
```

--------------------------------

### Create and Enqueue a Streaming Generation Job

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Demonstrates creating a Job object for fine-grained control over generation, including streaming output, multiple stop conditions, and custom identifiers. The job is then enqueued for processing by the generator.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator, Job

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# Create a streaming job
prompt = "Write a short story about a robot learning to paint:"
input_ids = tokenizer.encode(prompt, add_bos=True)

job = Job(
    input_ids=input_ids,
    max_new_tokens=300,
    stop_conditions=[tokenizer.eos_token_id, "THE END"],
    identifier="story_job",  # Custom identifier for tracking
)

generator.enqueue(job)
```

--------------------------------

### Implement Quantized KV Cache

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Reduce VRAM usage by applying 2-8 bit quantization to keys and values in the cache layer.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator
from exllamav3.cache import CacheLayer_quant

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)

# Create quantized cache (4-bit keys and values)
cache = Cache(
    model,
    max_num_tokens=32768,
    layer_type=CacheLayer_quant,
    k_bits=4,  # 4-bit quantized keys
    v_bits=4,  # 4-bit quantized values
)

model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# Use normally - quantization is transparent
response = generator.generate(
    prompt="Explain the benefits of cache quantization:",
    max_new_tokens=200,
    add_bos=True,
    completion_only=True
)
print(response)
```

--------------------------------

### Configure Multi-GPU Loading

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Distribute model weights across multiple GPUs using automatic splitting, per-device VRAM limits, or tensor parallelism.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/large-model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)

# Option 1: Automatic split based on available VRAM
model.load(progressbar=True)

# Option 2: Specify VRAM limits per GPU (in GB)
model.load(
    use_per_device=[20, 20, 20],  # Use up to 20GB on GPUs 0, 1, 2
    progressbar=True
)

# Option 3: Tensor parallelism for maximum throughput
model.load(
    tensor_p=True,  # Enable tensor parallelism
    tp_backend="native",  # or "nccl" for multi-node
    progressbar=True
)

tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

response = generator.generate(
    prompt="Hello from multiple GPUs!",
    max_new_tokens=100,
    add_bos=True,
    completion_only=True
)
print(response)
```

--------------------------------

### Speculative Decoding with Draft Model

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Accelerate text generation by using a smaller, faster draft model to propose tokens that the main model then verifies. Configure the `draft_model`, `draft_cache`, and `num_draft_tokens` for this feature.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator

# Load main model
config = Config.from_directory("/path/to/large-model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)

# Load draft model (smaller, faster version)
draft_config = Config.from_directory("/path/to/small-model/")
draft_model = Model.from_config(draft_config)
draft_cache = Cache(draft_model, max_num_tokens=8192)
draft_model.load(progressbar=True)

tokenizer = Tokenizer.from_config(config)

# Create generator with draft model
generator = Generator(
    model=model,
    cache=cache,
    tokenizer=tokenizer,
    draft_model=draft_model,
    draft_cache=draft_cache,
    num_draft_tokens=4,  # Number of tokens to draft ahead
)

response, metrics = generator.generate(
    prompt="Explain the theory of relativity:",
    max_new_tokens=200,
    add_bos=True,
    completion_only=True,
    return_last_results=True
)

print(f"Response: {response}")
if metrics:
    accepted = metrics.get("accepted_draft_tokens", 0)
    rejected = metrics.get("rejected_draft_tokens", 0)
    print(f"Draft tokens - Accepted: {accepted}, Rejected: {rejected}")
```

--------------------------------

### HumanEval Benchmark Script

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md

This Python script is used to evaluate models on the HumanEval benchmark. It helps in comparing the performance of different quantization methods.

```python
from eval.humaneval import evaluate_humaneval
```

--------------------------------

### Perform Simple Text Generation

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Uses the generator.generate() method for single or batched text completion. Supports specifying stop conditions and whether to return only the completion.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# Single prompt generation
response = generator.generate(
    prompt="Explain quantum computing in simple terms:",
    max_new_tokens=200,
    stop_conditions=[tokenizer.eos_token_id],
    completion_only=True,
    add_bos=True
)
print("Response:", response)

# Batched generation with multiple prompts
prompts = [
    "What is the capital of France?",
    "Write a limerick about cats.",
    "Explain the theory of relativity."
]
responses = generator.generate(
    prompt=prompts,
    max_new_tokens=150,
    completion_only=True,
    add_bos=True
)
for i, resp in enumerate(responses):
    print(f"Response {i+1}: {resp}\n")
```

--------------------------------

### Integrate HuggingFace Chat Templates

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Format conversational messages using HuggingFace chat templates with the `hf_chat_template` method of the tokenizer. This ensures proper message structure for models trained with these templates.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# Format messages using HF chat template
messages = [
    {"role": "system", "content": "You are a helpful assistant.",},
    {"role": "user", "content": "What is the weather like today?"},
]

# Get tokenized input using chat template
input_ids = tokenizer.hf_chat_template(
    messages,
    add_generation_prompt=True
)

response = generator.generate(
    prompt=input_ids,  # Pass pre-tokenized input
    max_new_tokens=200,
    stop_conditions=[tokenizer.eos_token_id],
    completion_only=True
)
print(response)
```

--------------------------------

### Execute batched streaming jobs

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Process multiple concurrent prompts by enqueuing jobs and tracking them via unique identifiers.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator, Job

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=16384)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)

# Multiple prompts to process concurrently
prompts = [
    "Translate to French: Hello, how are you?",
    "Summarize: The quick brown fox jumps over the lazy dog.",
    "Complete: Once upon a time in a land far away,"
]

# Enqueue all jobs with identifiers
responses = [""] * len(prompts)
for idx, prompt in enumerate(prompts):
    job = Job(
        input_ids=tokenizer.encode(prompt, add_bos=True),
        max_new_tokens=100,
        identifier=idx,
    )
    generator.enqueue(job)

# Process all jobs concurrently
while generator.num_remaining_jobs():
    results = generator.iterate()
    for result in results:
        idx = result["identifier"]
        if result["stage"] == "streaming":
            responses[idx] += result.get("text", "")
        if result["eos"]:
            print(f"Job {idx} complete: {result['eos_reason']}")

# Print all responses
for i, response in enumerate(responses):
    print(f"\nPrompt {i}: {prompts[i]}")
    print(f"Response: {response}")
```

--------------------------------

### Quantize model across multiple GPUs

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/convert.md

Distributes the quantization workload across specified devices using the -d flag.

```sh
python convert.py -i /mnt/models/llama3.1-70b-instruct \
                  -o /mnt/models/llama3.1-70b-instruct-exl3-3.75bpw \
                  -w /mnt/temp/exl3 \
                  -b 3.75 \
                  -d 0,1,2
```

```sh
python convert.py -i /mnt/models/qwen3.5-35b-a3b \
                  -o /mnt/models/qwen3.5-35b-a3b-4.00bpw-plus \
                  -w /mnt/temp/exl3 \
                  -b 4.00 \
                  -eb \
                  -d 2,0,1 \
                  -dr 3,4,5
```

--------------------------------

### Apply Token Healing

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Use token healing to mitigate artifacts at prompt boundaries by regenerating the final input token.

```python
from exllamav3 import Config, Model, Cache, Tokenizer, Generator, Job

config = Config.from_directory("/path/to/model/")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load(progressbar=True)
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)
```

--------------------------------

### Perform Asynchronous Generation

Source: https://context7.com/turboderp-org/exllamav3/llms.txt

Utilize AsyncGenerator and AsyncJob for non-blocking inference in async-compatible frameworks.

```python
import asyncio
from exllamav3 import Config, Model, Cache, Tokenizer
from exllamav3.generator import AsyncGenerator, AsyncJob

async def main():
    # Setup model
    config = Config.from_directory("/path/to/model/")
    model = Model.from_config(config)
    cache = Cache(model, max_num_tokens=8192)
    model.load(progressbar=True)
    tokenizer = Tokenizer.from_config(config)

    # Create async generator
    generator = AsyncGenerator(
        model=model,
        cache=cache,
        tokenizer=tokenizer
    )

    # Create and enqueue async job
    prompt = "Write a poem about the ocean:"
    input_ids = tokenizer.encode(prompt, add_bos=True)

    job = AsyncJob(
        generator,
        input_ids=input_ids,
        max_new_tokens=150,
        stop_conditions=[tokenizer.eos_token_id],
    )

    # Async iteration over results
    print("Generated: ", end="", flush=True)
    async for result in job:
        if result["stage"] == "streaming":
            print(result.get("text", ""), end="", flush=True)
        if result["eos"]:
            print(f"\nDone: {result['new_tokens']} tokens")

    # Clean up
    await generator.close()

asyncio.run(main())
```

--------------------------------

### EXL3 Quantization Kernels

Source: https://github.com/turboderp-org/exllamav3/blob/master/doc/exl3.md

This snippet refers to the C++/CUDA kernels used in EXL3 quantization. These kernels are part of the extended functionality for EXL3.

```c++
#include <exllamav3/exllamav3_ext/quant/kernels.h>
```