Fujitsu One Compression

https://FujitsuResearch.github.io/OneCompression/
Admin
Fujitsu One Compression is an open-source Python library for post-training quantization of Large...
Tokens:85,046
Snippets:788
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Set up Open WebUI Environment with uv

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/vllm-inference

Create a dedicated Python virtual environment for Open WebUI using `uv`, ensuring compatibility with Python 3.12. Install Open WebUI and start its server.

```bash
# Create a dedicated venv (uv auto-downloads Python 3.12 if needed)
uv venv ~/open-webui-env --python 3.12
source ~/open-webui-env/bin/activate

# Install and launch
uv pip install open-webui
open-webui serve --port 3000

```

--------------------------------

### Quantizer Setup Method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

The setup method initializes the quantizer with a given model. It handles specific logic for VLMs and MoE models.

```python
setup(model)
```

--------------------------------

### setup Method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

The `setup` method prepares the quantizer by associating it with a model. It handles specific model architectures like VLMs and MoE models.

```APIDOC
### `setup`
```
setup(model)
```

Setup the quantizer with the model
For VLMs (e.g. Gemma3, Qwen3-VL), only language-model submodule layers are considered for quantization. Vision / audio encoder layers are automatically excluded.
For MoE models with fused 3D expert parameters (e.g. Gemma4,), expert tensors are automatically unfused into per-expert nn.Linear layers before the module scan.

### Parameters:
Name | Type | Description | Default
---|---|---|---
`model` |  | The model to be quantized | _required_
```

--------------------------------

### Setup Logger

Source: https://fujitsuresearch.github.io/OneCompression/api/utilities

Configures the logging system for the 'onecomp' logger and the root logger. Requires a logfile name.

```python
setup_logger(level: str = 'INFO', basic: str = 'WARNING', filemode: str = 'w', **kwargs)
```

--------------------------------

### DBF Quick Start with TinyLlama

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/dbf

Demonstrates a quick start for DBF quantization using a small model (TinyLlama) and a lightweight calibration configuration. This is recommended for verifying the pipeline end-to-end.

```python
from onecomp import CalibrationConfig, ModelConfig, Runner
from onecomp.quantizer.dbf import DBF

model_config = ModelConfig(
    model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
    device="cuda:0",
)
calib_config = CalibrationConfig(
    max_length=512,
    num_calibration_samples=128,
)
dbf = DBF(target_bits=1.5)
runner = Runner(
    model_config=model_config,
    quantizer=dbf,
    calibration_config=calib_config,
)
runner.run()

```

--------------------------------

### Install OneComp

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/cli

Install the OneComp package using pip. Ensure PyTorch is installed first.

```bash
pip install onecomp
```

--------------------------------

### Build and Serve Documentation

Source: https://fujitsuresearch.github.io/OneCompression/contributing

Install documentation dependencies with `uv sync --extra docs` and then serve the documentation locally using `mkdocs serve` for previewing.

```bash
uv sync --extra docs
uv run mkdocs serve

```

--------------------------------

### Install OneComp with Visualization Extras

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install OneComp along with matplotlib for visualization features.

```bash
pip install onecomp[visualize]

```

--------------------------------

### Clarify CalibrationConfig Usage in Examples

Source: https://fujitsuresearch.github.io/OneCompression/changelog

Adds NOTE comments to example scripts clarifying that compact `CalibrationConfig` settings are for demonstration only. Recommends default settings for real quantization and suggests `batch_size` for chunked calibration with large datasets.

```python
example/example_gptq.py
```

```python
example/example_qep_gptq.py
```

```python
example/example_jointq.py
```

```python
example/example_autobit.py
```

```python
example/example_save_load.py
```

```python
example/example_lpcd_gptq.py
```

```python
example/example_custom_calibration.py
```

```python
post_process/example_blockwise_ptq.py
```

```python
post_process/example_lora_sft.py
```

```python
post_process/example_lora_sft_knowledge.py
```

```python
pre_process/example_preprocess_save_load.py
```

```python
vllm_inference/example_gptq_vllm_inference.py
```

--------------------------------

### Basic LPCD with GPTQ + QEP

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/lpcd

This snippet demonstrates the basic setup for LPCD using GPTQ quantization and QEP for error compensation. Ensure 'cuda:0' is available for device placement.

```python
from onecomp import (
    CalibrationConfig,
    GPTQ,
    LPCDConfig,
    ModelConfig,
    Runner,
    setup_logger,
)

setup_logger()

model_config = ModelConfig(
    model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
    device="cuda:0",
)
gptq = GPTQ(wbits=3, groupsize=128)

lpcd_config = LPCDConfig(
    enable_residual=True,
    perccorr=0.5,
    percdamp=0.01,
    use_closed_form=True,
    device="cuda:0",
)

runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    calibration_config=CalibrationConfig(max_length=512, num_calibration_samples=128),
    qep=True,
    lpcd=True,
    lpcd_config=lpcd_config,
)
runner.run()

```

--------------------------------

### QEPConfig Initialization with Custom Parameters

Source: https://fujitsuresearch.github.io/OneCompression/api/qep_config

Initialize QEPConfig with custom parameters for damping and correction percentages. This example shows setting specific values for percdamp and perccorr.

```python
>>> # Generic implementation
>>> config = QEPConfig(general=True, percdamp=0.05, perccorr=0.3)
```

--------------------------------

### Runner Usage with Post-Processes

Source: https://fujitsuresearch.github.io/OneCompression/api/post_process

Example demonstrating how to use the Runner with a specified model configuration, quantizer, and a list of post-processing steps like BlockWisePTQ.

```python
>>> from onecomp import Runner, ModelConfig, GPTQ, BlockWisePTQ
>>> model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf")
>>> quantizer = GPTQ(wbits=4, groupsize=128)
>>> runner = Runner(
...     model_config=model_config,
...     quantizer=quantizer,
...     post_processes=[BlockWisePTQ()],
... )
>>> runner.run()
```

--------------------------------

### Install PyTorch with CPU Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Use this command to install PyTorch for systems without a CUDA-enabled GPU.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

```

--------------------------------

### Direct Execution Example

Source: https://fujitsuresearch.github.io/OneCompression/api/post_process

Shows how to directly run PostProcessLoraSFT on a pre-loaded quantized model. This is useful when you have already loaded a quantized model and want to apply LoRA SFT.

```python
import torch
from onecomp import ModelConfig, PostProcessLoraSFT
model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf")
quantized_model = torch.load(
    "quantized_model.pt",
    map_location="cpu",
    weights_only=False,
)
post_process = PostProcessLoraSFT(data_files="train.jsonl")
post_process.run(quantized_model, model_config)
```

--------------------------------

### Install OneComp via pip

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install the OneComp package using pip.

```bash
pip install onecomp

```

--------------------------------

### BlockWisePTQ Runner Configuration

Source: https://fujitsuresearch.github.io/OneCompression/api/post_process

Example showing how to instantiate and use the Runner with BlockWisePTQ as a post-processing step, including custom parameters for BlockWisePTQ.

```python
>>> from onecomp import Runner, ModelConfig, GPTQ, BlockWisePTQ
>>> model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf")
>>> quantizer = GPTQ(wbits=4, groupsize=128)
>>> runner = Runner(
...     model_config=model_config,
...     quantizer=quantizer,
...     post_processes=[BlockWisePTQ(lr=1e-4, epochs=10, cbq_enable=True)],
... )
>>> runner.run()
```

--------------------------------

### Install Development Dependencies

Source: https://fujitsuresearch.github.io/OneCompression/contributing

Install the necessary development dependencies using `uv`. The `--extra cu128` and `--extra dev` flags specify additional dependency groups.

```bash
uv sync --extra cu128 --extra dev

```

--------------------------------

### Example 3: AutoBit Quantization Usage

Source: https://fujitsuresearch.github.io/OneCompression/changelog

Provides a usage example for the AutoBit quantization functionality, demonstrating how to apply it in a practical scenario.

```python
example/example3.py
```

--------------------------------

### Install PyTorch with CUDA 12.1 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 12.1 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

```

--------------------------------

### Install PyTorch with CUDA 12.8 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 12.8 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

```

--------------------------------

### Install PyTorch with CUDA 13.0 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 13.0 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

```

--------------------------------

### Instantiate GPTQ with Included Keywords

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

Example of creating a GPTQ quantizer instance to quantize layers whose names contain specific keywords.

```python
quantizer = GPTQ( include_layer_keywords=["q_proj", "k_proj", "v_proj"] )
```

--------------------------------

### Install PyTorch with CUDA 11.8 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 11.8 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

```

--------------------------------

### Install PyTorch with CUDA 12.6 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 12.6 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

```

--------------------------------

### Basic QEP Usage with GPTQ

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/qep

Demonstrates basic QEP integration with GPTQ quantization. Ensure 'onecomp' is installed and the model ID is valid.

```python
from onecomp import ModelConfig, Runner, GPTQ

model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf", device="cuda:0")
gptq = GPTQ(wbits=3)

runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
)
runner.run()

```

--------------------------------

### Load Model and Tokenizer

Source: https://fujitsuresearch.github.io/OneCompression/api/model_config

Example of initializing ModelConfig and then loading the model and tokenizer. Ensure the model_id is valid for Hugging Face Hub.

```python
model_config = ModelConfig(model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T")
model = model_config.load_model()
tokenizer = model_config.load_tokenizer()
```

--------------------------------

### Install PyTorch with CUDA 12.4 Support

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Install PyTorch with CUDA 12.4 support for GPU acceleration.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124

```

--------------------------------

### Example: End-to-End LPCD GPTQ Quantization

Source: https://fujitsuresearch.github.io/OneCompression/changelog

An end-to-end example for TinyLlama GPTQ 3-bit quantization with groupsize 128, QEP, and LPCD. Reports original, dequantized, and quantized perplexity.

```python
example/example_lpcd_gptq.py
```

--------------------------------

### Basic 4-bit Quantization Setup

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/jointq

Initializes and runs the JointQ quantizer for 4-bit quantization with a specified model and device. Requires the Runner class for execution.

```python
from onecomp import JointQ, ModelConfig, Runner

model_config = ModelConfig(
    model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
    device="cuda:0",
)
jointq = JointQ(bits=4, group_size=128)

runner = Runner(model_config=model_config, quantizer=jointq, qep=False)
runner.run()

```

--------------------------------

### Install uv Package Manager

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Installs the `uv` package manager on macOS or Linux using a curl script.

```bash
# Install uv (macOS or Linux)
curl -LsSf https://astral.sh/uv/install.sh | sh

```

--------------------------------

### Runner Integration Example

Source: https://fujitsuresearch.github.io/OneCompression/api/post_process

Demonstrates how to integrate PostProcessLoraSFT with the Runner class for model quantization and post-processing. Use this when performing a full pipeline including quantization.

```python
from onecomp import Runner, ModelConfig, GPTQ, PostProcessLoraSFT
model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf")
quantizer = GPTQ(wbits=4, groupsize=128)
runner = Runner(
    model_config=model_config,
    quantizer=quantizer,
    post_processes=[
        PostProcessLoraSFT(data_files="train.jsonl")
    ],
)
runner.run()
```

--------------------------------

### Set up Development Environment with uv and vLLM

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Configures the development environment using `uv`, including vLLM support which requires the `cu130` PyTorch index. This command installs OneComp, development tools, visualization libraries, and vLLM.

```bash
uv sync --extra cu130 --extra dev --extra visualize --extra vllm

```

--------------------------------

### ResultLoader Example Usage

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

Demonstrates how to instantiate and use the ResultLoader to load quantization results from a specified file. It shows how to access the keys of the loaded results.

```python
from onecomp.quantizer import ResultLoader
loader = ResultLoader(results_file="quantization_results.pt")
loader.results.keys()
```

--------------------------------

### Quantizer Post-Initialization Method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

The __post_init__ method is called after the object is initialized to perform any necessary setup or validation.

```python
__post_init__()
```

--------------------------------

### Runner Initialization with Calibration Configuration

Source: https://fujitsuresearch.github.io/OneCompression/api/runner

Instantiate the Runner with a custom CalibrationConfig for large-scale calibration data. This example uses GPTQ and specifies batch size and number of calibration samples.

```python
>>> from onecomp import Runner, ModelConfig, CalibrationConfig
>>> from onecomp.quantizer.gptq import GPTQ
>>> model_config = ModelConfig(
...     model_id_or_path="meta-llama/Llama-2-7b-hf"
... )
>>> quantizer = GPTQ(wbits=4, groupsize=128)
>>> calib_config = CalibrationConfig(
...     max_length=2048,
...     num_calibration_samples=1024,
...     batch_size=128,
... )
>>> runner = Runner(
...     model_config=model_config,
...     quantizer=quantizer,
...     calibration_config=calib_config,
... )
>>> runner.run()
```

--------------------------------

### Launch Open WebUI with Docker

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/vllm-inference

Start Open WebUI using Docker. The `--add-host` flag is necessary on Linux to allow the container to reach the vLLM server running on the host machine.

```bash
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  --name open-webui \
  ghcr.io/open-webui/open-webui:latest

```

--------------------------------

### Example: Custom Calibration Dataset for Quantization

Source: https://fujitsuresearch.github.io/OneCompression/changelog

Demonstrates using `CalibrationConfig` with a custom calibration dataset. Quantizes TinyLlama with GPTQ 3-bit and compares inference outputs using default C4 and custom Python-code calibration.

```python
example/example_custom_calibration.py
```

--------------------------------

### Example: Block-wise PTQ with GPTQ 4-bit

Source: https://fujitsuresearch.github.io/OneCompression/changelog

Showcases GPTQ 4-bit quantization combined with BlockWisePTQ (Phase 1 greedy + Phase 2 CBQ). Includes a Perplexity (PPL) comparison.

```python
example/post_process/example_blockwise_ptq.py
```

--------------------------------

### Initialize GPTQ for 4-bit Quantization

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/gptq

Instantiate the GPTQ quantizer for basic 4-bit quantization with a group size of 128. Ensure 'onecomp' is installed.

```python
from onecomp import GPTQ

gptq = GPTQ(wbits=4, groupsize=128)
```

--------------------------------

### Quick Start: Rotation Preprocessing and Quantization

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/pre-process

Performs rotation preprocessing and then quantizes the model using GPTQ. Ensure wbits, groupsize, and sym parameters match between preprocessing and quantization steps for optimal quality.

```python
from onecomp import ModelConfig, Runner, GPTQ, prepare_rotated_model, setup_logger

setup_logger()

# Step 1: Rotation preprocessing
model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf", device="cuda:0")

rotated_config = prepare_rotated_model(
    model_config=model_config,
    save_directory="./rotated_model",
    wbits=4,
    groupsize=128,
)

# Step 2: Quantize (wbits/groupsize/sym must match Step 1)
gptq = GPTQ(wbits=4, groupsize=128)
runner = Runner(model_config=rotated_config, quantizer=gptq)
runner.run()

```

--------------------------------

### Start vLLM Server for OpenAI-Compatible API

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/vllm-inference

Launch a vLLM server to expose an OpenAI-compatible API endpoint, typically used for chat interfaces like Open WebUI. The server listens on http://localhost:8000 by default.

```bash
vllm serve ./Llama-3.1-8B-Instruct-gptq-4bit

```

--------------------------------

### Install OneComp for Developers with pip

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Installs OneComp in editable mode with development dependencies, after cloning the repository and installing PyTorch with CUDA support.

```bash
git clone https://github.com/FujitsuResearch/OneCompression.git
cd OneCompression

# Install PyTorch with CUDA support
pip install torch --index-url https://download.pytorch.org/whl/cu128

# Install onecomp with development dependencies
pip install -e ".[dev]"

```

--------------------------------

### Install vLLM with uv

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/vllm-inference

Install vLLM using uv, ensuring compatibility with CUDA 13.0. Avoid mixing with older CUDA versions or CPU-only installations.

```bash
uv sync --extra cu130 --extra vllm
```

--------------------------------

### Initialize Runner with GPTQ, QEP, and LPCD

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/overview

Configure the Runner to use GPTQ quantization, enable QEP, and activate Layer-Projected Coordinate Descent (LPCD) with custom configuration.

```python
from onecomp import GPTQ, LPCDConfig, Runner

runner = Runner(
    model_config=model_config,
    quantizer=GPTQ(wbits=3, groupsize=128),
    qep=True,
    lpcd=True,
    lpcd_config=LPCDConfig(),
)
runner.run()

```

--------------------------------

### setup_logger

Source: https://fujitsuresearch.github.io/OneCompression/api/utilities

Sets up a logger for the 'onecomp' library. Allows configuration of logging levels and file output.

```APIDOC
## setup_logger

### Description
Sets up the logger for the 'onecomp' library.

### Parameters
#### Keyword Arguments
- **level** (str) - Optional - Logging level for the 'onecomp' logger. Defaults to 'INFO'.
- **basic** (str) - Optional - Logging level for the root logger used in `basicConfig`. Defaults to 'WARNING'.
- **filemode** (str) - Optional - File mode for the log file. Defaults to 'w'.
- **logfile** - Required - Name of the log file to write logs to.
- **kwargs** - Optional - Additional keyword arguments.
```

--------------------------------

### setup_logger Function

Source: https://fujitsuresearch.github.io/OneCompression/api

Configures the logging output for the One Compression library.

```APIDOC
## setup_logger Function

### Description
Configure logging output.

### Usage
```python
from onecomp import setup_logger

# Set up logging
setup_logger(log_level='INFO')
```
```

--------------------------------

### Set up Development Environment with uv

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Clones the repository and synchronizes dependencies using `uv`, including optional extras like development tools, visualization, and vLLM support. Replace `cu128` with your specific CUDA variant.

```bash
# Clone and set up
git clone https://github.com/FujitsuResearch/OneCompression.git
cd OneCompression
uv sync --extra cu128 --extra dev --extra visualize

```

--------------------------------

### Verify OneComp Installation

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/cli

Check the installed version of OneComp using the command-line interface or by running it as a Python module.

```bash
onecomp --version
```

```bash
python -m onecomp --version
```

--------------------------------

### JointQ with All Initialization Strategies

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/jointq

Shows how to enable all available initialization strategies for JointQ.

```APIDOC
## JointQ with All Initialization Strategies

### Description
Initializes the JointQ quantizer with all initialization strategies enabled.

### Parameters
- **bits** (int) - Required - The number of bits for quantization.
- **symmetric** (bool) - Required - Whether to use symmetric quantization.
- **group_size** (int) - Required - The group size for quantization.
- **enable_clip_optimize** (bool) - Optional - Whether to use Clip-Optimize initialization. Default is True.
- **enable_clip_optimize_ep** (bool) - Optional - Whether to use Clip-Optimize with Error Propagation initialization. Default is False.
- **enable_gptq** (bool) - Optional - Whether to use GPTQ initialization. Default is True.

### Request Example
```python
quantizer = JointQ(
    bits=4,
    symmetric=False,
    group_size=128,
    enable_clip_optimize=True,
    enable_clip_optimize_ep=True,
    enable_gptq=True,
)
```
```

--------------------------------

### Install vLLM with pip

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/vllm-inference

Install the vLLM package using pip. Note that vLLM requires CUDA and a compatible GPU.

```bash
pip install vllm
```

--------------------------------

### Initialize Runner and Run Quantization

Source: https://fujitsuresearch.github.io/OneCompression/user-guide/basic-usage

Instantiate the `Runner` with the `model_config` and `quantizer`, then call `run()` to perform the quantization. Optional logging can be enabled using `setup_logger()`.

```python
from onecomp import Runner, setup_logger

setup_logger()  # Optional: enable logging output

runner = Runner(model_config=model_config, quantizer=gptq)
runner.run()

```

--------------------------------

### Runner Initialization and Usage

Source: https://fujitsuresearch.github.io/OneCompression/api/runner

Demonstrates how to initialize and use the Runner class for single-GPU and multi-GPU quantization.

```APIDOC
## Runner Class

### Description
Runner class for model quantization. Supports quantization using calibration data and parallel quantization on multiple GPUs.

### Parameters

- **model_config** (`ModelConfig`) - Required - Model configuration.
- **quantizer** (`Quantizer`) - Optional - The quantizer to use. Specify either `quantizer` or `quantizers`, not both. At least one must be given.
- **quantizers** (`list[Quantizer]`) - Optional - Specify multiple quantizers. When used with `calibration_config.batch_size`, the X^T X accumulation is shared, reducing the forward pass to a single execution. Specify either `quantizer` or `quantizers`, not both. Currently, this is only available when `calibration_config.batch_size` is set and `qep=False`.
- **calibration_config** (`CalibrationConfig or None`) - Optional - Calibration data configuration. When `None` (default), a :class:`CalibrationConfig` with default values is created automatically. See :class:`CalibrationConfig` for available fields.
- **qep** (`bool`) - Optional - Whether to use QEP. Default is `False`.
- **qep_config** (`QEPConfig or None`) - Optional - Configuration for QEP. If None and `qep=True`, a default `QEPConfig()` is used.
- **lpcd** (`bool`) - Optional - Whether to use LPCD. Default is `False`.
- **lpcd_config** (`LPCDConfig or None`) - Optional - Configuration for LPCD. If None and `lpcd=True`, a default `LPCDConfig()` is used.
- **multi_gpu** (`bool`) - Optional - Whether to use multi-GPU for layer-wise parallel quantization. Default is `False`.
- **gpu_ids** (`list[int]`) - Optional - List of GPU IDs to use for multi-GPU quantization. If None and `multi_gpu` is True, all available GPUs will be used.
- **post_processes** (`list[PostQuantizationProcess] or None`) - Optional - Optional list of post-quantization processes to execute after the main quantization step. Each process receives a quantized model on CPU (built via `create_quantized_model`) and may modify it in-place. Processes are executed in order. Default is None.

### Method
`Runner(model_config, quantizer=None, quantizers=None, calibration_config=None, qep=False, qep_config=None, lpcd=False, lpcd_config=None, multi_gpu=False, gpu_ids=None, post_processes=None)`

### Example
```python
# Single GPU quantization (default)
from onecomp import Runner, ModelConfig
from onecomp.quantizer.gptq import GPTQ
model_config = ModelConfig(model_id_or_path="meta-llama/Llama-2-7b-hf")
quantizer = GPTQ(wbits=4, groupsize=128)
runner = Runner(
    model_config=model_config,
    quantizer=quantizer,
)
runner.run()

# Multi-GPU quantization (layer-wise parallel)
from onecomp.quantizer.jointq import JointQ
quantizer = JointQ(bits=4, group_size=128)
# Use all available GPUs
runner = Runner(
    model_config=model_config,
    quantizer=quantizer,
    multi_gpu=True,
)
runner.run()

# Use specific GPUs (e.g., GPU 0, 2, 3)
runner = Runner(
    model_config=model_config,
    quantizer=quantizer,
    multi_gpu=True,
    gpu_ids=[0, 2, 3],
)
runner.run()
```
```

--------------------------------

### validate_params method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/gptq

Validates the GPTQ parameters during the setup phase.

```APIDOC
## validate_params method

### Signature
```python
validate_params()
```

### Description
Validate GPTQ parameters once in setup().

### Validated Ranges
- `blocksize`: int >= 1
- `percdamp`: float >= 3.95e-4
- `wbits`: int, 1 <= wbits <= 63
- `groupsize`: int, -1 or >= 1
- `q_grid`: int >= 1 (when mse=True)
- `q_norm`: float > 0 (when mse=True)
```

--------------------------------

### Check CUDA Version

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Verify your installed CUDA version using `nvcc` or `nvidia-smi`.

```bash
nvcc --version

```

```bash
nvidia-smi

```

--------------------------------

### Get Assignment Function from Strategy

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/autobit

Retrieve the specific assignment function associated with an AssignmentStrategy enum member.

```python
fn
```

--------------------------------

### Get Model ID or Path

Source: https://fujitsuresearch.github.io/OneCompression/api/model_config

Retrieve the model identifier or the local path configured for the ModelConfig instance.

```python
get_model_id_or_path()
```

--------------------------------

### validate_params method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/dbf

Validates the DBF parameters. This method should be called once during setup to ensure all parameters are within their acceptable ranges.

```APIDOC
### validate_params

```python
validate_params()
```

Validate DBF parameters once in setup().

Validated ranges:
- `target_bits`: float > 0
- `iters`: int >= 1
- `reg`: float >= 0
- `balance_iters`: int >= 1 (when `use_balancing`=True)
- `balance_alpha`: float > 0 (when `use_balancing`=True)
- `balance_mode`: str in {"l1", "l2"} (when `use_balancing`=True)
```

--------------------------------

### Run Quantization with Custom QEP Configuration

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/quickstart

For fine-grained control over QEP, create a `QEPConfig` object with specific parameters like `percdamp` and `perccorr`, and pass it to the `Runner` constructor.

```python
from onecomp import QEPConfig

qep_config = QEPConfig(
    percdamp=0.01,
    perccorr=0.5,
)
runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
    qep_config=qep_config,
)
runner.run()

```

--------------------------------

### Instantiate GPTQ with Excluded Layers

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

Example of creating a GPTQ quantizer instance, excluding the 'lm_head' layer by default.

```python
quantizer = GPTQ(exclude_layer_names=["lm_head"])
```

--------------------------------

### QEPConfig Initialization for Architecture-Aware Implementation

Source: https://fujitsuresearch.github.io/OneCompression/api/qep_config

Initialize QEPConfig for the architecture-aware implementation by setting general to False. This is the default behavior.

```python
>>> # Architecture-aware implementation (default)
>>> config = QEPConfig(general=False)
```

--------------------------------

### Run Commands with Activated Virtualenv

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Activate the virtual environment and run OneComp commands, tests, or examples directly.

```bash
source .venv/bin/activate
onecomp --version
onecomp TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
pytest tests/ -v
python example/example_gptq.py

```

--------------------------------

### Initialize Runner with GPTQ and QEP

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/overview

Instantiate the Runner with a specific quantizer (GPTQ) and enable Quantization Error Propagation (QEP).

```python
runner = Runner(
    model_config=model_config,
    quantizer=GPTQ(wbits=3),
    qep=True,
)

```

--------------------------------

### validate_params Method

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/onebit

Validates the parameters of the OneBit quantizer. This method should be called once during setup to ensure parameters are within acceptable ranges.

```APIDOC
### `validate_params`
```python
validate_params()
```

Validate OneBit parameters once in setup().
Validated ranges
`iters`: int >= 0
`balance_iters`: int >= 1 (when `use_balancing=True`)
`balance_alpha`: float > 0 (when `use_balancing=True`)
```

--------------------------------

### Instantiate GPTQ for Specific Layers

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

Example of creating a GPTQ quantizer instance to quantize only specified layers by their exact names.

```python
quantizer = GPTQ( include_layer_names=["model.layers.0.self_attn.q_proj"] )
```

--------------------------------

### JointQ Initialization

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/jointq

Demonstrates basic initialization of the JointQ quantizer with essential parameters.

```APIDOC
## JointQ Initialization

### Description
Initializes the JointQ quantizer with basic configuration.

### Parameters
- **bits** (int) - Required - The number of bits for quantization.
- **symmetric** (bool) - Required - Whether to use symmetric quantization.
- **group_size** (int) - Required - The group size for quantization.

### Request Example
```python
from onecomp.quantizer.jointq import JointQ

quantizer = JointQ(
    bits=4,
    symmetric=False,
    group_size=128,
)
```
```

--------------------------------

### run

Source: https://fujitsuresearch.github.io/OneCompression/api/runner

Executes the quantization and related processing steps.

```APIDOC
## run

### Description
Execute quantization (and related) processing

### Method
`run()`
```

--------------------------------

### Run Commands with uv

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/installation

Execute OneComp commands, tests, or examples using `uv run` without activating a virtual environment.

```bash
uv run onecomp --version
uv run onecomp TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
uv run pytest tests/ -v
uv run python example/example_gptq.py

```

--------------------------------

### Step-by-step Quantization Workflow

Source: https://fujitsuresearch.github.io/OneCompression/getting-started/quickstart

Orchestrate the quantization pipeline manually using `ModelConfig`, `GPTQ`, and `Runner`. This provides fine-grained control over each component. Ensure the logger is set up.

```python
from onecomp import ModelConfig, Runner, GPTQ, setup_logger

setup_logger()

model_config = ModelConfig(
    model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
    device="cuda:0",
)
gptq = GPTQ(wbits=4, groupsize=128)

runner = Runner(model_config=model_config, quantizer=gptq)
runner.run()

```

--------------------------------

### Training Data for Knowledge Injection

Source: https://fujitsuresearch.github.io/OneCompression/changelog

Provides training data in JSON Lines format for the knowledge injection example, defining the 'OneCompression' concept.

```json
example/post_process/onecomp_knowledge.jsonl
```

--------------------------------

### auto_run

Source: https://fujitsuresearch.github.io/OneCompression/api/runner

Performs one-liner quantization with sensible defaults. It sets up the model configuration, auto bit quantizer, and QEP, then runs the quantization process. The target bitwidth can be estimated automatically from available VRAM if not specified. Optionally evaluates perplexity and accuracy, and saves the quantized model.

```APIDOC
## auto_run `classmethod`

### Description
One-liner quantization with sensible defaults. Sets up ModelConfig, AutoBitQuantizer (ILP-based mixed-precision), and QEP, then runs quantization. When `wbits` is `None`, the target bitwidth is estimated automatically from available VRAM. Optionally evaluates perplexity and accuracy, and saves the quantized model.

### Parameters
#### Path Parameters
- **model_id** (str) - Required - Hugging Face model ID or local path.

#### Query Parameters
- **wbits** (float or None) - Optional - Target quantization bitwidth. When `None` (default), estimated from VRAM via `estimate_wbits_from_vram`.
- **total_vram_gb** (float or None) - Optional - Total VRAM budget in GB for bitwidth estimation. Only used when `wbits` is `None`. When `None`, the installed GPU VRAM is detected automatically.
- **groupsize** (int) - Optional - GPTQ group size (default: 128). Use -1 to disable grouping.
- **device** (str) - Optional - Device to place the model on (default: "cuda:0").
- **qep** (bool) - Optional - Whether to use QEP (default: True).
- **evaluate** (bool) - Optional - Whether to calculate perplexity and accuracy after quantization (default: True).
- **eval_original_model** (bool) - Optional - Whether to also evaluate the original (unquantized) model (default: False).
- **save_dir** (str or None) - Optional - Directory to save the quantized model. "auto" (default) derives the path from model_id (e.g., "TinyLlama-1.1B-...-autobit-3.5bit"). Set to `None` to skip saving.

#### Request Body
- **kwargs** - Additional keyword arguments forwarded to the `GPTQ` constructor (e.g., `actorder`, `sym`).

### Response
#### Success Response (200)
- **Runner** - The configured Runner instance (with quantization results accessible via `runner.quantizer.results`).

### Request Example
Minimal usage (QEP + GPTQ 4-bit, groupsize=128, auto-save):
```python
>>> from onecomp import Runner
>>> runner = Runner.auto_run(
...     model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
... )
```

Custom save directory:
```python
>>> runner = Runner.auto_run(
...     model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
...     save_dir="./my_quantized_model",
... )
```

Skip saving:
```python
>>> runner = Runner.auto_run(
...     model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
...     save_dir=None,
... )
```

Evaluate both original and quantized models:
```python
>>> runner = Runner.auto_run(
...     model_id="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
...     eval_original_model=True,
... )
```
```

--------------------------------

### Run Quantization with QEP for Improved Quality

Source: https://fujitsuresearch.github.io/OneCompression/algorithms/gptq

Utilize the Runner with GPTQ and QEP enabled for enhanced quantization quality. This requires specifying model configuration and the quantizer.

```python
from onecomp import Runner, ModelConfig

model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf", device="cuda:0")
gptq = GPTQ(wbits=3, groupsize=128)

runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
)
runner.run()
```

--------------------------------

### Get Quantization Configuration

Source: https://fujitsuresearch.github.io/OneCompression/api/quantizers/base

Returns the quantization configuration dictionary, which is used for saving quantized models. This method should be overridden by quantizers that support saving.

```python
get_quant_config() -> dict


```
Tokens:
Raw