LLM Compressor

https://docs.vllm.ai/projects/llm-compressor/en/latest
Admin
LLM Compressor provides tools and techniques for compressing Large Language Models using various...
Tokens:531,454
Snippets:2,957
Trust Score:9.5
Update:1 week ago
Show doc for...
Tokens:
Raw
### Run the iMatrix example script

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/imatrix

Executes the example script for iMatrix quantization. This is the simplest way to run the example.

```bash
python3 llama3_imatrix_example.py

```

--------------------------------

### Run QuIP-style transform example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/transform

Execute the provided example script for applying quip-style transforms.

```bash
python3 quip_example.py
```

--------------------------------

### Run Llama 3.3 70B quantization example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/autoround

Run the quantization script from the repository root. Requires LLM Compressor installed, sufficient memory/disk for the 70B model, and access to the gated Hugging Face model or a local checkpoint via model_id.

```bash
python examples/autoround/quantization_attention/llama3_3_quantize_example.py

```

--------------------------------

### Install vllm and lm-evaluation-harness

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_kv_cache

Install the required packages for evaluating the quantized model.

```bash
pip install "vllm>=0.5.5" lm_eval==0.4.3

```

--------------------------------

### Install transformers >= v5

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/key-models/gemma4

Required to run the Gemma 4 quantization examples. Install with uv pip.

```bash
uv pip install --upgrade transformers==5.8.1

```

--------------------------------

### Install vllm and lm_eval

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w8a8_fp8

Install vllm and lm-evaluation-harness for evaluation.

```bash
pip install vllm lm_eval==0.4.3

```

--------------------------------

### Run the end-to-end quantization example script

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w4a16

Run the provided example script to apply the quantization algorithm. The resulting model is saved as Meta-Llama-3-8B-Instruct-W4A16-G128 and is ready to load into vLLM.

```bash
python3 llama3_example.py

```

--------------------------------

### Run Quickstart Script

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w4a4_fp4

Execute the end-to-end quantization script.

```bash
python3 llama3_example.py
```

--------------------------------

### Install AutoRound from source

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/autoround

Clone the llm-compressor repository and install it in editable mode to use AutoRound.

```bash
git clone https://github.com/vllm-project/llm-compressor.git
cd llm-compressor
pip install -e .

```

--------------------------------

### Install LLM Compressor from a local clone

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/getting-started/install

Install LLM Compressor from a local clone of the repository.

```bash
pip install .
```

--------------------------------

### Install LLM Compressor from source

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/getting-started/install

Install the latest development version of LLM Compressor from the main branch on GitHub.

```bash
pip install git+https://github.com/vllm-project/llm-compressor.git
```

--------------------------------

### Install LLM Compressor from PyPI

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/getting-started/install

Install the latest stable release of LLM Compressor from PyPI.

```bash
pip install llmcompressor
```

--------------------------------

### Oneshot Usage Example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/entrypoints/oneshot

Example of initializing and executing the Oneshot calibration process, followed by accessing the processed components.

```python
oneshot = Oneshot(model=model, recipe=recipe, dataset=dataset)
oneshot()

# Access the processed components
model = oneshot.model
processor = oneshot.processor
recipe = oneshot.recipe
```

--------------------------------

### Install llmcompressor from source

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_kv_cache

Install the llmcompressor library from a specific commit to access the new KV cache quantization feature.

```bash
pip install git+https://github.com/vllm-project/llm-compressor.git@cb98f34d4ec9dd175e6995d12fb02dec39c6f27a

```

--------------------------------

### Install transformers >= v5

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/key-models/qwen3.6

Install the required transformers version (>= v5) using uv pip. This is a prerequisite for running the quantization examples end-to-end.

```bash
uv pip install --upgrade transformers
```

--------------------------------

### Run llama3_example.py with ModelOpt data mix

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/qad

Run the example from the current directory, selecting the sample count and epoch budget. The first command uses 512 samples and 3 epochs; the second uses 1024 samples and 7 epochs. These commands demonstrate loading and training, not the full configuration for the long-context epoch-ablation benchmark.

```bash
python llama3_example.py --dataset modelopt_qad_mix --quantizer gptq --samples 512 --epochs 3 --max-seq-length 2048 --output ./llama-mix-512-e3
python llama3_example.py --dataset modelopt_qad_mix --quantizer gptq --samples 1024 --epochs 7 --max-seq-length 2048 --output ./llama-mix-1024-e7

```

--------------------------------

### Install llmcompressor

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_w8a8_fp8

Install the llmcompressor package.

```bash
pip install llmcompressor

```

--------------------------------

### on_initialize method

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/quantization/quantization/base

Prepares the model for activation and weight calibration by attaching quantization schemes and hooks.

```python
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Prepare to calibrate activations and weights

    According to the quantization config, a quantization scheme is attached to each
    targeted module. The module's forward call is also overwritten to perform
    quantization to inputs, weights, and outputs.

    Then, according to the module's quantization scheme, observers and calibration
    hooks are added. These hooks are disabled until the modifier starts.
    """
    if not QuantizationMixin.has_config(self):
        raise ValueError(
            "QuantizationModifier requires that quantization fields be specified"
        )
    QuantizationMixin.initialize_quantization(self, state.model)

    return True
```

--------------------------------

### on_calibration_start method

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/quantization/quantization/base

Begins the activation calibration process.

```python
def on_calibration_start(self, state: State, event: Event, **kwargs):
    """
    Begin calibrating activations.
    """
    QuantizationMixin.start_calibration(self, state.model)
```

--------------------------------

### NVFP4ExpandedIMatrixObserver usage example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/observers/imatrix

Example of configuring the observer with the 'nvfp4_expanded_imatrix' string in QuantizationArgs.

```python
QuantizationArgs(
    ...
    observer="nvfp4_expanded_imatrix",
)

```

--------------------------------

### SmoothQuantModifier Configuration Example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/logarithmic_equalization/base

Example configuration for using SmoothQuantModifier as a replacement for LogarithmicEqualizationModifier.

```yaml
    SmoothQuantModifier:
      algorithm: log_equalization
      mappings: [...]
```

--------------------------------

### Run llama3_example.py with QAD

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/qad

Commands to run the complete NVFP4A16 example with QAD for RTN and GPTQ quantizers, including a variant with targets-per-subgraph set to 2.

```bash
python llama3_example.py --quantizer rtn --output ./llama-rtn-qad
python llama3_example.py --quantizer gptq --output ./llama-gptq-qad
python llama3_example.py --quantizer gptq --targets-per-subgraph 2 --output ./llama-gptq-qad-2

```

--------------------------------

### start_calibration(model: Module)

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/quantization/quantization/mixin

Attaches observers, registers activation calibration hooks (including kv_cache quantization) and enables quantization as we calibrate. This method prepares the model for calibration by initializing observers and hooks on target modules.

```APIDOC
## start_calibration(model: Module)

### Description
Attach observers, register activation calibration hooks (including kv_cache quantization) and enable quantization as we calibrate.

### Method
start_calibration

### Parameters
- **model** (Module) - Required - model to prepare for calibration

### Source Code
```python
def start_calibration(self, model: torch.nn.Module):
    """
    Attach observers, register activation calibration hooks (including
    kv_cache quantization) and enable quantization as we calibrate

    :param model: model to prepare for calibration
    """
    targets = match_named_modules(model, self.resolved_targets, self.ignore)
    if targets_embeddings(model, targets):
        untie_word_embeddings(model)
    warn_inference_mode_forwards(model)

    for _, module in match_named_modules(model, self.resolved_targets, self.ignore):
        self._initialize_observers(module)
        self._calibration_hooks |= self._initialize_hooks(module)
        apply_calibration_status(module)

    # Link weight observers in fused groups (Q/K/V, gate/up) for shared global_scale
    fuse_weight_observers(model)
```
```

--------------------------------

### Run GLM-4.7 quantization example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantizing_moe

Executes the complete GLM-4.7 quantization example script.

```bash
python3 glm4_7_example.py

```

--------------------------------

### on_calibration_start method

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/transform/awq

Starts AWQ on the given state. Runs after quantization mixin initialization. Resolves mappings, validates mappings and quant scheme, and sets up activation cache hooks. Raises ValueError for unsupported token masking with MoE up_proj->down_proj mappings and for duo_scaling with tensor quantization.

```python
def on_calibration_start(self, state: State, event: Event, **kwargs):
    """
    Start AWQ on the given state. This runs after quantization mixin has been
    initialized (i.e. after quantization config has been applied)

    - resolve mappings
    - validate mappings and quant scheme
    - setup activation cache hooks

    :param state: state to run AWQ on
    :return: True on a successful run, False otherwise
    """

    self._set_resolved_mappings(state.model)

    # Check for unsupported token masking with MoE up_proj -> down_proj mappings
    if state.loss_masks is not None and self._has_moe_up_down_proj_mapping():
        raise ValueError(
            "Token masking (use_loss_mask=True) is not supported with "
            "up_proj -> down_proj mappings in MoE models. The MoE routing "
            "mechanism dispatches tokens to different experts, and the loss mask "
            "cannot be properly aligned with this dispatch. Please either "
            "disable token masking or exclude the up_proj -> down_proj mapping "
            "for MoE layers from the AWQ configuration."
        )

    # Validate that duo_scaling is only used with per-channel quantization
    if self.duo_scaling is not False:

        def _validate_balance_layer(name, module):
            if (
                hasattr(module, "quantization_scheme")
                and hasattr(module.quantization_scheme, "weights")
                and module.quantization_scheme.weights.strategy
                == QuantizationStrategy.TENSOR
            ):
                raise ValueError(
                    "duo_scaling is only supported with per-channel quantization "
                    "strategies (group or channel), but found TENSOR strategy on "
                    f"layer {name}. Please set duo_scaling=False or use a "
                    "per-channel quantization strategy."
                )

        for mapping in self._resolved_mappings:
            for balance_name, balance_layer in zip(
                mapping.balance_names, mapping.balance_layers
            ):
                _validate_balance_layer(balance_name, balance_layer)

    self._setup_activation_cache_hooks()

```

--------------------------------

### Install llm-compressor dependencies

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/transform

Commands to clone the repository and install the package in editable mode.

```bash
git clone https://github.com/vllm-project/llm-compressor.git
cd llm-compressor
pip install -e .
```

--------------------------------

### start_calibration implementation

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/quantization/quantization/mixin

Implementation of the start_calibration method in QuantizationMixin.

```python
def start_calibration(self, model: torch.nn.Module):
    """
    Attach observers, register activation calibration hooks (including
    kv_cache quantization) and enable quantization as we calibrate

    :param model: model to prepare for calibration
    """
    targets = match_named_modules(model, self.resolved_targets, self.ignore)
    if targets_embeddings(model, targets):
        untie_word_embeddings(model)
    warn_inference_mode_forwards(model)

    for _, module in match_named_modules(model, self.resolved_targets, self.ignore):
        self._initialize_observers(module)
        self._calibration_hooks |= self._initialize_hooks(module)
        apply_calibration_status(module)

    # Link weight observers in fused groups (Q/K/V, gate/up) for shared global_scale
    fuse_weight_observers(model)
```

--------------------------------

### on_initialize Method Implementation

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/smoothquant

The on_initialize method prepares the state and runs the SmoothQuant initialization, requiring a calibration dataset.

```python
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Initialize and run SmoothQuant on the given state

    :param state: state to run SmoothQuant on
    :return: True on a successful run, False otherwise
    """
    if self.end and self.end != -1:
        raise ValueError(
            f"{self.__class__.__name__} can only be applied during one-shot. "
            f" Expected end to be None or -1, got {self.end}"
        )
    if self.start and self.start != -1:
        raise ValueError(
            f"{self.__class__.__name__} can only be applied during one-shot. "
            f"Expected start to be None or -1, got {self.end}"
        )

    if not hasattr(state, "data") or state.data.calib is None:
        raise ValueError(
            f"{self.__class__.__name__} requires a calibration dataset to be "
            "provided"
        )
    self.ignore = [] if not self.ignore else self.ignore
    self.mappings = self._infer_mappings_from_model(state.model)
    self.resolved_mappings_ = self._resolve_mappings(state.model)
    self.scales_ = {}

    return True
```

--------------------------------

### Install vLLM via pip

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/steps/deploy

Install the vLLM package to enable model deployment and inference.

```bash
pip install vllm
```

--------------------------------

### on_initialize

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/transform/awq

Method to start AWQ on the given state before quantization config is applied.

```APIDOC
## on_initialize(state: State, **kwargs) -> bool

### Description
Start AWQ on the given state. This runs before quantization config has been applied. It also infers unresolved mappings based on model architecture, if not set manually.

### Parameters
- **state** (State) - state to run AWQ on

### Returns
- **bool** - True on a successful run, False otherwise

```

--------------------------------

### on_initialize method implementation

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/transform/awq

Starts AWQ on the given state before quantization config is applied. Infers unresolved mappings from the model if not set manually, and sets default offload_device based on MoE model detection.

```python
def on_initialize(self, state: State, **kwargs) -> bool:
    """
    Start AWQ on the given state. This runs before quantization config has been
    applied

    - infer unresolved mappings based on model architecture, if not set manually

    :param state: state to run AWQ on
    :return: True on a successful run, False otherwise
    """

    if self.mappings is None:
        logger.info("No AWQModifier.mappings provided, inferring from model...")
        self.mappings = get_layer_mappings_from_model(state.model)

    # Set default offload_device
    if self.offload_device == Sentinel("not_provided"):
        # Check if we have a MoE model
        if is_moe_model(state.model):
            self.offload_device = torch.device("cpu")
            logger.info(
                "MoE model detected: setting offload_device to 'cpu' by default "
                "to reduce memory usage. You can override this by explicitly "
                "setting offload_device in your recipe."
            )
        else:
            # For non-MoE models, convert sentinel to None
            # (no offloading by default)
            self.offload_device = None

    return True
```

--------------------------------

### symbolic_trace usage example

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/pipelines/sequential/transformers_helpers

Example of using symbolic_trace from transformers.utils.fx to trace a model with specified input names.

```python
from transformers.utils.fx import symbolic_trace

traced_model = symbolic_trace(model, input_names=["input_ids", "attention_mask", "token_type_ids"])
```

--------------------------------

### SmoothQuantModifier example recipe

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/smoothquant/base

Example YAML recipe for configuring SmoothQuantModifier with smoothing_strength, mappings, and ignore parameters.

```yaml
SmoothQuantModifier:
  smoothing_strength: 0.5
  mappings: [
    [["re:.*q_proj", "re:.*k_proj", "re:.*v_proj"], "re:.*self_attn_layer_norm"],
    [["re:.*fc1"], "re:.*final_layer_norm"]
  ]
  ignore: ["model.decoder.final_layer_norm"]

```

--------------------------------

### Sample usage of QADModifier with oneshot

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/qad/base

Shows how to configure a recipe with QuantizationModifier and QADModifier and run oneshot with the sequential pipeline. Place QADModifier after a weight quantization modifier and use the sequential pipeline.

```python
from llmcompressor import oneshot
from llmcompressor.modifiers.qad import QADModifier
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = [
    QuantizationModifier(
        targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]
    ),
    QADModifier(num_epochs=1, lr=2e-6),
]
oneshot(model=model, dataset=ds, recipe=recipe, pipeline="sequential")
```

--------------------------------

### calibration_start

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/core/session_functions

Invokes a start event for the active session during calibration. Should be called before calibration starts for one epoch.

```APIDOC
## calibration_start

### Description
Invoke a start event for the active session during calibration. This event should be called before calibration starts for one epoch.

### Method
classmethod

### Signature
calibration_start(**kwargs) -> ModifiedState

### Parameters
#### Keyword Arguments
- **kwargs** - Additional keyword arguments to pass to the current session's event method.

### Returns
- **ModifiedState** - The modified state of the active session after invoking the event.

### Usage Example
See `src/llmcompressor/pipelines/basic/pipeline.py` for usage example.
```

--------------------------------

### should_start method in src/llmcompressor/modifiers/modifier.py

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/transform/spinquant

Determines whether the modifier should start based on the current event index and the configured start/end bounds. Returns False if start is None, otherwise checks if the current index falls within the start and end range.

```python
def should_start(self, event: Event) -> bool:
    """
    :param event: The event to check if the modifier should start
    :return: True if the modifier should start based on the given event
    """
    if self.start is None:
        return False

    current = event.current_index

    return self.start <= current and (self.end is None or current < self.end)

```

--------------------------------

### KimiK3VisionProcessor.__init__

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modeling/kimi_k3/kimi_k3_vision_processing

Initializes the KimiK3VisionProcessor with media processing configuration.

```APIDOC
## KimiK3VisionProcessor.__init__

### Description
Initializes the KimiK3VisionProcessor instance with the provided media processing configuration.

### Method
Constructor

### Signature
```python
KimiK3VisionProcessor(media_proc_cfg: dict, **kwargs)
```

### Parameters
- **media_proc_cfg** (dict) - Required - Configuration for media processing, including image mean, std, and patch size.
- **kwargs** - Optional - Additional keyword arguments passed to the base class.

### Source Code
```python
def __init__(
    self,
    media_proc_cfg: dict,
    **kwargs,
):
    super().__init__(**kwargs)
    self.media_proc_cfg = media_proc_cfg
```
```

--------------------------------

### REAPPruningModifier Example Recipe

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/api/llmcompressor/modifiers/pruning/reap/base

An example configuration for the REAPPruningModifier, specifying the sparsity fraction of experts to remove per layer.

```yaml
REAPPruningModifier:
  sparsity: 0.25
```

--------------------------------

### Run the end-to-end quantization script

Source: https://docs.vllm.ai/projects/llm-compressor/en/latest/examples/quantization_kv_cache

Execute the provided script to apply the quantization algorithm. The resulting model is ready to be loaded into vLLM.

```bash
python3 llama3_fp8_kv_example.py

```
Tokens:
Raw