### Install from source

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/getting-started/install.md

Install the latest development version directly from the GitHub repository.

```bash
pip install git+https://github.com/vllm-project/llm-compressor.git
```

--------------------------------

### Install Transformers

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/qwen3.5/index.md

Install the required version of the transformers library to run the quantization examples.

```bash
uv pip install --upgrade transformers
```

--------------------------------

### Run QuIP-style transformation example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/transform/README.md

Executes the provided example script for QuIP-style transforms.

```bash
python3 quip_example.py
```

--------------------------------

### Install from local clone

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/getting-started/install.md

Install the library from a local directory, optionally in editable mode for development.

```bash
pip install .
```

```bash
pip install -e .[dev]
```

--------------------------------

### Install Transformers v5

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/gemma4/index.md

Install the required version of the transformers library. This is necessary for the quantization examples.

```bash
uv pip install --upgrade transformers==5.8.1
```

--------------------------------

### Install llm-compressor

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/autoround/README.md

Clone the repository and install the llm-compressor package in editable mode.

```bash
git clone https://github.com/vllm-project/llm-compressor.git
cd llm-compressor
pip install -e .
```

--------------------------------

### Run FP8 Dynamic Llama Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/awq/RESULTS.md

Execute the example script for FP8_DYNAMIC quantization. Navigate to the 'examples/awq' directory before running.

```bash
cd examples/awq
python fp8_dynamic_llama_example.py
```

--------------------------------

### Install vLLM and lm-evaluation-harness

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/README.md

Install the necessary libraries for running models with vLLM and evaluating their accuracy.

```bash
pip install vllm lm_eval==0.4.3
```

--------------------------------

### Run AutoRound Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/autoround/quantization_w4a16/README.md

Execute the provided example script to apply AutoRound quantization to a model.

```bash
python3 llama3_example.py
```

--------------------------------

### Install from PyPI

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/getting-started/install.md

Install the latest stable release of LLM Compressor from the Python Package Index.

```bash
pip install llmcompressor
```

--------------------------------

### Run iMatrix Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/imatrix/README.md

Execute the llama3_imatrix_example.py script to demonstrate iMatrix functionality.

```bash
python3 llama3_imatrix_example.py
```

--------------------------------

### Install llmcompressor from source

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_kv_cache/README.md

Install the llmcompressor library from its GitHub repository. This is necessary as the FP8 KV cache quantization feature is new.

```bash
pip install git+https://github.com/vllm-project/llm-compressor.git@cb98f34d4ec9dd175e6995d12fb02dec39c6f27a
```

--------------------------------

### Run FP8 Block Llama Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/awq/RESULTS.md

Execute the example script for FP8_BLOCK quantization. Navigate to the 'examples/awq' directory before running.

```bash
python fp8_block_llama_example.py
```

--------------------------------

### Run Qwen3-VL AutoRound Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/autoround/quantization_w4a4_fp4/README.md

Execute the provided script to apply AutoRound quantization to the Qwen3-VL model.

```bash
python3 qwen3_vl_example.py
```

--------------------------------

### Run DeepSeekV4 Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/deepseek-v4/fp8-nvfp4-example.md

Execute the example script to quantize the DeepSeekV4 model. Replace MODEL_ID with your desired model.

```bash
python3 examples/quantizing_moe/deepseek_v4_example.py
```

--------------------------------

### Run end-to-end FP8 KV cache quantization example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_kv_cache/README.md

Execute the provided example script to apply FP8 KV cache quantization to a model. The resulting model is optimized for vLLM.

```bash
python3 llama3_fp8_kv_example.py
```

--------------------------------

### Install vLLM and LM Evaluation Harness

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/README_granite4.md

Install the required libraries for model evaluation. Ensure you have Python and pip available.

```bash
pip install vllm lm_eval
```

--------------------------------

### Install vLLM for Inference

Source: https://github.com/vllm-project/llm-compressor/blob/main/README.md

Install the vLLM library, which is used for efficient inference with LLM models.

```bash
pip install vllm
```

--------------------------------

### Install vLLM with Experimental Branch

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/deepseek-v4/fp8-nvfp4-example.md

Clone the vLLM repository, checkout the experimental branch for DeepSeekV4 support, and install vLLM with transformers.

```bash
git clone https://github.com/neuralmagic/vllm
cd vllm
git checkout kylesayrs/deepseek-ct
uv pip install -U transformers==5.8.0
# optional:
# git rebase main
VLLM_USE_PRECOMPILED=1 uv pip install -e .
```

--------------------------------

### Install LLM Compressor

Source: https://github.com/vllm-project/llm-compressor/blob/main/README.md

Install the llmcompressor library using pip. This is the first step to using the library's features.

```bash
pip install llmcompressor
```

--------------------------------

### Install specific version from PyPI

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/getting-started/install.md

Install a specific version of the library by appending the version number.

```bash
pip install llmcompressor==0.5.1
```

--------------------------------

### Install LLM Compressor and Dependencies

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/deepseek-v4/fp8-nvfp4-example.md

Clone the LLM Compressor repository, checkout the experimental branch, and install necessary packages including transformers.

```bash
git clone https://github.com/vllm-project/llm-compressor
cd llm-compressor
git checkout deepseekv4-experimental
uv pip install -e .
uv pip install -U transformers==5.8.0
```

--------------------------------

### Install LLM Compressor from Source

Source: https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

Installs the LLM Compressor package in editable mode with development dependencies.

```bash
pip install -e ./[dev]
```

--------------------------------

### Install Compressed Tensors from Source

Source: https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

Clones the Compressed Tensors repository and installs it in editable mode. This is recommended for development.

```bash
git clone https://github.com/vllm-project/compressed-tensors.git
pip install -e ./compressed-tensors
```

--------------------------------

### Distributed GPTQ Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/index.md

Example demonstrating Distributed Data Parallel (DDP) functionality with GPTQ for improved calibration runtime.

```python
https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16/llama3_ddp_example.py
```

--------------------------------

### Run Granite4 Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/README_granite4.md

Execute the `granite4_example.py` script to demonstrate the quantization process for mamba, mlp, and moe layers. This will generate a quantized model ready for vLLM.

```bash
python3 granite4_example.py
```

--------------------------------

### Start vLLM HTTP Server

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/steps/deploy.md

Launch the vLLM server to serve the model via a RESTful API.

```bash
vllm serve "Qwen3-30B-A3B-FP8-BLOCK"
```

--------------------------------

### Run Llama 3.1 AutoRound Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/autoround/quantization_w4a4_fp4/README.md

Execute the provided script to apply AutoRound quantization to the Llama 3.1 model.

```bash
python3 llama3.1_example.py
```

--------------------------------

### NVFP4 GPTQ Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/index.md

Example for NVFP4 quantization scheme, a type of FP4 quantization using GPTQ.

```python
https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a4_fp4/llama3_gptq_example.py
```

--------------------------------

### vLLM Serving Benchmark Result (InternVL3-8B-hf-FP8-GPTQ)

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/README_internvl3.md

Example benchmark results for InternVL3-8B-hf-FP8-GPTQ, highlighting improved performance metrics compared to the non-FP8 version.

```text
============ Serving Benchmark Result ============
Successful requests:                     500       
Maximum request concurrency:             100       
Benchmark duration (s):                  163.36    
Total input tokens:                      6193      
Total generated tokens:                  34831     
Request throughput (req/s):              3.06      
Output token throughput (tok/s):         213.22    
Peak output token throughput (tok/s):    1787.00   
Peak concurrent requests:                109.00    
Total Token throughput (tok/s):          251.13    
---------------Time to First Token----------------
Mean TTFT (ms):                          14510.84  
Median TTFT (ms):                        14371.25  
P99 TTFT (ms):                           28978.21  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          257.52    
Median TPOT (ms):                        270.19    
P99 TPOT (ms):                           330.58    
---------------Inter-token Latency----------------
Mean ITL (ms):                           247.34    
Median ITL (ms):                         268.16    
P99 ITL (ms):                            386.99    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          31725.93  
Median E2EL (ms):                        32227.14  
P99 E2EL (ms):                           64293.40  
==================================================
```

--------------------------------

### vLLM Serving Benchmark Result (InternVL3-8B-hf)

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/README_internvl3.md

Example benchmark results for InternVL3-8B-hf, showing key performance metrics like throughput, latency, and concurrency.

```text
============ Serving Benchmark Result ============
Successful requests:                     500       
Maximum request concurrency:             100       
Benchmark duration (s):                  251.15    
Total input tokens:                      6193      
Total generated tokens:                  30487     
Request throughput (req/s):              1.99      
Output token throughput (tok/s):         121.39    
Peak output token throughput (tok/s):    1055.00   
Peak concurrent requests:                107.00    
Total Token throughput (tok/s):          146.05    
---------------Time to First Token----------------
Mean TTFT (ms):                          25349.32  
Median TTFT (ms):                        25498.75  
P99 TTFT (ms):                           45969.02  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          415.96    
Median TPOT (ms):                        414.47    
P99 TPOT (ms):                           857.11    
---------------Inter-token Latency----------------
Mean ITL (ms):                           379.15    
Median ITL (ms):                         410.25    
P99 ITL (ms):                            524.16    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          48444.73  
Median E2EL (ms):                        50202.88  
P99 E2EL (ms):                           91705.52  
==================================================
```

--------------------------------

### Run End-to-End Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantizing_moe/README.md

Execute the provided Python script to perform end-to-end quantization of the GLM-4.7 MoE model.

```bash
python3 glm4_7_example.py
```

--------------------------------

### AWQ Mappings for Llama Models

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/awq/README.md

Provides example AWQ mappings for the Llama family of models. These mappings specify which layers are targeted for weight and activation scaling.

```python
[
    AWQMapping(
        "re:.*input_layernorm",
        ["re:.*q_proj", "re:.*k_proj", "re:.*v_proj"],
    ),
    AWQMapping("re:.*v_proj", ["re:.*o_proj"]),
    AWQMapping(
        "re:.*post_attention_layernorm",
        ["re:.*gate_proj", "re:.*up_proj"],
    ),
    AWQMapping(
        "re:.*up_proj",
        ["re:.*down_proj"],
    ),
]
```

--------------------------------

### Apply FP8 Quantization with Round-to-Nearest

Source: https://github.com/vllm-project/llm-compressor/blob/main/README.md

Applies FP8 quantization to model weights and activations using the Round-to-Nearest algorithm. This example demonstrates loading a model, configuring the quantization recipe, applying it using the oneshot API, and saving the quantized model.

```python
from compressed_tensors.offload import dispatch_model
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "Qwen/Qwen3-30B-A3B"

# Load model.
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Configure the quantization algorithm and scheme.
# In this case, we:
#   * quantize the weights to FP8 using RTN with block_size 128
#   * quantize the activations dynamically to FP8 during inference
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",
    ignore=["lm_head", "re:.*mlp.gate$"],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Confirm generations of the quantized model look sane.
print("========== SAMPLE GENERATION ==============")
dispatch_model(model)
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to(
    model.device
)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
print("==========================================")

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-BLOCK"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
```

--------------------------------

### Experimental Scripts for Attention Quantization

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/index.md

Scripts to get started with extended KV cache and attention quantization support.

```python
https://github.com/vllm-project/llm-compressor/tree/main/experimental
```

--------------------------------

### Upgrade pip

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/getting-started/install.md

Ensure the latest version of pip is installed before proceeding with library installation.

```bash
python -m pip install --upgrade pip
```

--------------------------------

### MXFP4 Quantization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/index.md

Example for MXFP4 quantization scheme, a type of FP4 quantization.

```python
https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16_fp4/mxfp4/llama3_example.py
```

--------------------------------

### Dispatch Model and Generate Sample Output

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/big_models_with_sequential_onloading/README.md

After quantization, dispatch the model to available devices. Then, prepare sample input, generate output, and decode it using the tokenizer.

```python
dispatch_model(model)
sample = tokenizer("Hello my name is", return_tensors="pt")
sample = {key: value.to(model.device) for key, value in sample.items()}
output = model.generate(**sample, max_new_tokens=100)
print(tokenizer.decode(output[0]))
```

--------------------------------

### Load Qwen3.5 Model

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/qwen3.5/nvfp4-vl-example.md

Initialize the model and processor using the Hugging Face Transformers library.

```python
from compressed_tensors.offload import dispatch_model
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Load model.
MODEL_ID = "Qwen/Qwen3.5-27B"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
    MODEL_ID, dtype="auto", trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
```

--------------------------------

### Evaluation Results

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16/README.md

Example output from the accuracy evaluation.

```bash
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
|-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.728|±  |0.0282|
|     |       |strict-match    |     5|exact_match|↑  |0.720|±  |0.0285|
```

--------------------------------

### Load and Use Observer for Quantization

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/guides/observers.md

Demonstrates loading an observer from the registry, accumulating statistics from input tensors, and computing quantization parameters. Requires torch and llmcompressor.observers.

```python
import torch
from llmcompressor.observers import Observer
from compressed_tensors.quantization import QuantizationArgs

args = QuantizationArgs(num_bits=4, strategy="group", group_size=128)
observer = Observer.load_from_registry(
    "minmax",
    base_name="weight",
    args=args,
)

# Phase 1: accumulate statistics
x = torch.randn(64, 512)
observer(x)

# Phase 2: compute quantization parameters
qparams = observer.get_qparams()
scale = qparams["scale"]
zero_point = qparams["zero_point"]
```

--------------------------------

### COCO Dataset Citation

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/README.md

BibTeX citation for the Microsoft COCO dataset used in the examples.

```bibtex
@article{cocodataset,
  author    = {Tsung{-}Yi Lin and Michael Maire and Serge J. Belongie and Lubomir D. Bourdev and Ross B. Girshick and James Hays and Pietro Perona and Deva Ramanan and Piotr Doll{'{a} }r and C. Lawrence Zitnick},
  title     = {Microsoft {COCO:} Common Objects in Context},
  journal   = {CoRR},
  volume    = {abs/1405.0312},
  year      = {2014},
  url       = {http://arxiv.org/abs/1405.0312},
  archivePrefix = {arXiv},
  eprint    = {1405.0312},
  timestamp = {Mon, 13 Aug 2018 16:48:13 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/LinMBHPRDZ14},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```

--------------------------------

### Serve Model with vLLM

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/README_internvl3.md

Use this command to serve a model with vLLM. Configure GPU memory, logging, and specific model parameters for multimodal capabilities.

```bash
vllm serve OpenGVLab/InternVL3-8B-hf-FP8-GPTQ \
        --served-model-name InternVL3-8B-hf-FP8-GPTQ \
        --gpu-memory-utilization 0.9 \
        --uvicorn_log_level error \
        --disable-log-stats \
        --trust-remote-code \
        --allowed-local-media-path /path/to/sharegpt4v/images \
        --limit-mm-per-prompt '{"image": 20}'  \
        --mm-processor-kwargs '{"max_dynamic_patch": 1}' \
        --no-enable-prefix-caching \
        --disable-mm-preprocessor-cache  \
        --max-model-len 6144
```

--------------------------------

### Defining Ignore Patterns

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/guides/entrypoints/model-free-ptq.md

Example configuration for the ignore argument using both exact strings and regex patterns.

```python
ignore=[
    "lm_head",            # exact name match
    "re:.*gate$",         # regex: any module ending in "gate"
    "model.embed_tokens", # exact name match
]
```

--------------------------------

### Run Code Styling and Formatting Checks

Source: https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

Executes the make commands to check code styling and quality.

```bash
make style
```

```bash
make quality
```

--------------------------------

### Load Qwen3.6 MoE Model and Processor

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/qwen3.6/nvfp4-moe-example.md

Loads the Qwen3.6-35B-A3B model and its corresponding processor. Ensure transformers v5 is installed.

```python
import torch
from compressed_tensors.utils import save_mtp_tensors_to_checkpoint
from datasets import load_dataset
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# NOTE: This example requires transformers >= v5

MODEL_ID = "Qwen/Qwen3.6-35B-A3B"

# Load model.
model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)
```

--------------------------------

### Execute oneshot quantization

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/transform/README.md

Applies the recipe to the model and saves the compressed output to disk.

```python
from llmcompressor import oneshot

# Apply algorithms.
oneshot(model=model, recipe=recipe, pipeline="datafree")

# Save to disk compressed.
SAVE_DIR = MODEL_ID.split("/")[1] + "-quip-w4a16"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
```

--------------------------------

### Default SmoothQuant Mapping Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/src/llmcompressor/modifiers/transform/smoothquant/README.md

This is the default mapping used by SmoothQuant, specifying that inputs to projection layers should be smoothed based on the output of post_attention_layernorm.

```python
[["re:.*gate_proj", "re:.*up_proj"], "re:.*post_attention_layernorm"]
```

--------------------------------

### Load and Run Model with vLLM

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_fp8/README.md

Load the quantized model into vLLM and perform a basic generation task.

```python
from vllm import LLM
model = LLM("./Meta-Llama-3-8B-Instruct-FP8-Dynamic")
model.generate("Hello my name is")
```

--------------------------------

### Prepare Calibration Dataset

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/autoround/quantization_w4a16/README.md

Fetch a calibration dataset using `get_dataset` from `auto_round.calib_dataset`. Configure the number of samples, sequence length, and tuning steps as recommended.

```python
# Select calibration dataset.
from auto_round.calib_dataset import get_dataset

NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 2048

# Get aligned calibration dataset.
ds = get_dataset(
    tokenizer=tokenizer,
    seqlen=MAX_SEQUENCE_LENGTH,
    nsamples=NUM_CALIBRATION_SAMPLES,
)
```

--------------------------------

### Basic oneshot Usage

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/guides/entrypoints/oneshot.md

Demonstrates loading a model and applying a FP8 dynamic quantization modifier using the oneshot entrypoint.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", dtype="auto")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

oneshot(
    model=model,
    recipe=QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"]),
    output_dir="Meta-Llama-3-8B-Instruct-FP8",
)
```

--------------------------------

### DDP Synchronization Example

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/guides/observers.md

Illustrates how observer statistics are synchronized across distributed ranks using `sync_activation_stats` and `dist.all_reduce`. This method is called at each layer boundary.

```python
pending_comms = observer.sync_activation_stats()
# ... wait_for_comms(pending_comms) once all observers in the subgraph are queued
```

--------------------------------

### Run Sample Generation

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/qwen3.5/nvfp4-vl-example.md

Perform inference to verify the quantized model's output.

```python
print("\n\n========== SAMPLE GENERATION ==============")
dispatch_model(model)
messages = [{"role": "user", "content": "Hello my name is"}]
prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(text=prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=100)
print(processor.decode(output[0], skip_special_tokens=True))
print("==========================================\n\n")
```

--------------------------------

### Compose iMatrix with GPTQ

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/imatrix/README.md

Integrate iMatrix's importance-weighted ranges with GPTQ for enhanced Hessian-based rounding. This setup is beneficial for advanced quantization scenarios.

```python
from llmcompressor.modifiers.gptq import GPTQModifier

scheme = preset_name_to_scheme("W4A16", ["Linear"])
scheme.weights.observer = "imatrix_mse"

recipe = [
    IMatrixGatherer(ignore=["lm_head"]),
    GPTQModifier(
        config_groups={"group_0": scheme},
        ignore=["lm_head"],
    ),
]
```

--------------------------------

### Prepare Calibration Data

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a16/README.md

Load and preprocess the calibration dataset to match the model's training format.

```python
from datasets import load_dataset

NUM_CALIBRATION_SAMPLES=512
MAX_SEQUENCE_LENGTH=2048

# Load dataset.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)

# Preprocess the data into the format the model is trained with.
def preprocess(example):
    return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False,)}
ds = ds.map(preprocess)

# Tokenize the data (be careful with bos tokens - we need add_special_tokens=False since the chat_template already added it).
def tokenize(sample):
    return tokenizer(sample["text"], padding=False, max_length=MAX_SEQUENCE_LENGTH, truncation=True, add_special_tokens=False)
ds = ds.map(tokenize, remove_columns=ds.column_names)
```

--------------------------------

### Run Tests

Source: https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

Executes the test suite using the make test command. Note that running all tests can be time-consuming and may require multiple GPUs.

```bash
make test
```

--------------------------------

### Configure and Apply Quantization Recipe

Source: https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w8a8_int8/README.md

Define a quantization recipe using `SmoothQuantModifier` and `GPTQModifier` for W8A8 quantization. Apply the recipe to the model using the `oneshot` function and save the quantized model.

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier

# Configure the quantization algorithms to run.
recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),
    GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"]),
]

# Apply quantization.
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

# Save to disk compressed.
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-W8A8-Dynamic-Per-Token"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
```

--------------------------------

### Ignore lm_head Layer During Quantization

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/faq/faq.md

Example demonstrating how to exclude the lm_head layer from quantization to maintain model accuracy. This is a common practice for the final layer of a model.

```python
ignore_modules=['lm_head']
```

--------------------------------

### Prepare Calibration Dataset

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/gemma4/nvfp4-example.md

Loads and preprocesses a dataset for calibration. It shuffles, maps text to input IDs, filters by sequence length, and selects a specified number of samples. Requires the 'datasets' library.

```python
from datasets import load_dataset

BATCH_SIZE = 1
NUM_CALIBRATION_SAMPLES = 32
MAX_SEQUENCE_LENGTH = 2048

DATASET_ID = "mit-han-lab/pile-val-backup"
DATASET_SPLIT = "validation"


def get_calib_dataset(processor):
    ds = load_dataset(
        DATASET_ID,
        split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES*10}]",
    )

    def preprocess(example):
        return {
            "input_ids": processor.tokenizer.encode(example["text"].strip())[
                :MAX_SEQUENCE_LENGTH
            ]
        }

    ds = (
        ds.shuffle(seed=42)
        .map(preprocess, remove_columns=ds.column_names)
        .filter(lambda example: len(example["input_ids"]) >= MAX_SEQUENCE_LENGTH)
        .select(range(NUM_CALIBRATION_SAMPLES))
    )
    return ds
```

--------------------------------

### Load Gemma 4 Model and Processor

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/key-models/gemma4/nvfp4-example.md

Loads the specified Gemma 4 multimodal model and its corresponding processor from Hugging Face. Ensure transformers and llmcompressor are installed.

```python
from transformers import AutoProcessor, Gemma4ForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "google/gemma-4-31B-it"
model = Gemma4ForConditionalGeneration.from_pretrained(model_id, dtype="auto")
processor = AutoProcessor.from_pretrained(model_id)
```

--------------------------------

### Import Calibration Module and Run Oneshot Quantization

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/developer-tutorials/add-moe-support.md

Import the custom MoE calibration module to enable the `@MoECalibrationModule.register` decorator. Then, use the `oneshot` function with a `QuantizationModifier` to quantize the MoE model.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modeling.your_model import SequentialYourModelMoE  # noqa: F401
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "your-org/your-moe-model"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

oneshot(
    model=model,
    dataset=ds,
    recipe=[QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"])],
    num_calibration_samples=512,
    max_seq_length=2048,
)

model.save_pretrained("your-moe-model-FP8", save_compressed=True)
tokenizer.save_pretrained("your-moe-model-FP8")

```

--------------------------------

### Python: Reference Observer in QuantizationArgs

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/developer-tutorials/add-observer.md

Use the 'observer' field in QuantizationArgs to specify the registered name of the observer, such as 'percentile'. This example shows how to set up a QuantizationModifier with a percentile observer.

```python
from llmcompressor.modifiers.quantization import QuantizationModifier
from compressed_tensors.quantization import QuantizationArgs

recipe = QuantizationModifier(
    targets="Linear",
    scheme={
        "weights": QuantizationArgs(
            num_bits=8,
            type="int",
            symmetric=True,
            strategy="channel",
            observer="percentile",
            observer_kwargs={"percentile": 99.5},
        )
    },
    ignore=["lm_head"],
)
```

--------------------------------

### YAML Configuration for Observer in Quantization Stage

Source: https://github.com/vllm-project/llm-compressor/blob/main/docs/guides/observers.md

Shows how to configure an 'mse' observer with specific keyword arguments within a YAML file for a quantization stage, targeting Linear layers.

```yaml
quantization_stage:
  quantization_modifiers:
    GPTQModifier:
      weights:
        observer: mse
        observer_kwargs:
          maxshrink: 0.1
          patience: 10
          averaging_constant: 0.05
          grid: 128.0
          norm: 2.0
        num_bits: 4
        type: int
        symmetric: true
        strategy: channel
      targets:
        - Linear
```