### Colab Notebook for Serving

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-serve.md

Example code for a Colab notebook to serve models, including setting up ngrok for public API access. Installs necessary libraries and starts the server.

```python
!pip install pyngrok fastapi uvicorn
from pyngrok import ngrok
# ... start server + ngrok tunnel ...
print(f'API endpoint: {ngrok_url}/v1/chat/completions')

```

--------------------------------

### Install vLLM Fork and Transformers

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/nemotron-offload.md

Installs the vLLM fork with expert offloading capabilities and the transformers library. Ensure vllm-expert-offload is cloned or available for installation.

```python
# Install vLLM fork with expert offloading
import subprocess, sys, os
os.environ["VLLM_USE_PRECOMPILED"] = "1"
subprocess.run([sys.executable, "-m", "pip", "install",
    "git+https://github.com/caiovicentino/vllm-expert-offload.git@nemotron-expert-offload",
    "-q"], check=True)
!pip install transformers -q
print("Installed!")
```

--------------------------------

### Install Dependencies

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-multi.md

Installs necessary Python packages for model quantization. Use this at the beginning of your notebook.

```python
!pip install safetensors huggingface_hub scipy -q
```

--------------------------------

### Install Dependencies for Fine-tuning

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-finetune.md

Installs necessary libraries for fine-tuning, including transformers, accelerate, peft, trl, bitsandbytes, and torchao. Use this at the beginning of your notebook.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy torchao
!pip install -q peft trl datasets bitsandbytes
```

--------------------------------

### Install Dependencies for Transformers and TorchAO

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant.md

Installs necessary libraries including the latest transformers from GitHub, datasets, accelerate, safetensors, sentencepiece, tiktoken, scipy, and torchao. Use this cell to set up the environment for quantization and benchmarking. Avoid installing flash-attn as SDPA offers comparable performance.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece tiktoken scipy torchao

```

--------------------------------

### Install PolarEngine-vLLM

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Install the PolarEngine-vLLM package from PyPI or build from source for optional CUDA kernel support.

```bash
pip install polarengine-vllm
```

```bash
git clone https://github.com/caiovicentino/polarengine-vllm
cd polarengine-vllm
pip install -e ".[vllm,triton]"
```

--------------------------------

### OpenAI Client Example

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-serve.md

Python client example using the `openai` library to interact with the served PolarQuant model. Demonstrates making a chat completion request and streaming the response.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="{MODEL}",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content, end="")

```

--------------------------------

### Install Dependencies and Clone llama.cpp

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-gguf.md

Installs necessary Python libraries and clones the llama.cpp repository for conversion tools. Use this cell to set up the environment in a Colab notebook.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy

# Clone llama.cpp for conversion tools
!git clone https://github.com/ggerganov/llama.cpp.git /content/llama.cpp
!cd /content/llama.cpp && pip install -r requirements/requirements-convert_hf_to_gguf.txt -q

```

--------------------------------

### Install Dependencies for Benchmark Notebook

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-bench.md

Installs necessary libraries including transformers, datasets, accelerate, safetensors, sentencepiece, scipy, torchao, and bitsandbytes. Use this cell to ensure all required packages are available.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q datasets accelerate safetensors sentencepiece scipy torchao bitsandbytes

```

--------------------------------

### Install PolarEngine vLLM

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/README.md

Install the PolarEngine vLLM package using pip. For CUDA kernel support, install with the optional extras.

```bash
pip install polarengine-vllm
```

```bash
git clone https://github.com/caiovicentino/polarengine-vllm
cd polarengine-vllm
pip install -e .
```

```bash
pip install -e ".[cuda]"
```

--------------------------------

### Configure and Run SFTTrainer

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-finetune.md

Sets up the training arguments for SFTTrainer, including output directory, epochs, batch size, learning rate, and sequence length. BF16 is enabled for training. The trainer is then initialized and the training process is started.

```python
training_args = SFTConfig(
    output_dir='/content/lora_output',
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-4,
    warmup_ratio=0.1,
    logging_steps=10,
    save_steps=100,
    bf16=True,
    max_seq_length=2048,
    dataset_text_field='text',
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()
```

--------------------------------

### Import and verify package versions

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/notebooks/full_pipeline_test.ipynb

Imports the transformers and polarengine-vllm libraries and prints their versions to ensure correct installation. Requires runtime restart after installation.

```python
# REINICIAR RUNTIME!
import transformers; print(f'transformers: {transformers.__version__}')
import polarengine_vllm; print(f'polarengine-vllm: {polarengine_vllm.__version__}')
```

--------------------------------

### Install polarengine-vllm and dependencies

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/notebooks/full_pipeline_test.ipynb

Installs the polarengine-vllm package from GitHub, including a fix for H128 GPU cache. Also installs necessary libraries like transformers, datasets, and triton.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall
```

```python
!pip install -q datasets accelerate safetensors sentencepiece tiktoken scipy triton
```

```python
!pip install -q git+https://github.com/caiovicentino/polarengine-vllm.git
```

--------------------------------

### Install Dependencies for Video Quantization

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-video.md

Installs necessary libraries for video model quantization, including transformers, diffusers, and torchao. Use --force-reinstall for transformers to ensure the latest version.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q diffusers accelerate safetensors scipy torchao imageio[ffmpeg] opencv-python pillow
```

--------------------------------

### Dockerfile for PolarQuant Serving

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-serve.md

Docker image definition for serving PolarQuant models. Installs necessary Python packages and copies the serving script. The CMD starts the server with a specified model and port.

```dockerfile
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

RUN pip install torch transformers torchao safetensors scipy sentencepiece \
    fastapi uvicorn polarengine-vllm

COPY serve.py /app/serve.py
WORKDIR /app

EXPOSE 8000
CMD ["python", "serve.py", "--model", "{MODEL}", "--port", "8000"]

```

--------------------------------

### Lloyd-Max Centroids Precomputation Example

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-vllm-kv.md

Illustrates the use of `scipy.stats.norm` for precomputing Lloyd-Max centroids, which are essential for weight quantization. This example shows the import statement and a comment indicating precomputation for specific bit values.

```python
# scipy.stats.norm, 100 iterations, EXACT same as weight quantization
from scipy.stats import norm as sp_norm
# Precompute for bits [2, 3, 4]
```

--------------------------------

### Configure and Initialize PolarKVCache

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Configure and initialize `PolarKVCache` for a specific model (e.g., Qwen3.5 9B) with a desired number of bits for compression. This setup enables efficient KV cache management.

```python
from polarengine_vllm.kv_cache import PolarKVConfig, PolarKVCache
import torch

# Pre-configured for Qwen3.5 9B (head_dim=128, 48 layers, 8 KV heads)
config = PolarKVConfig.for_qwen35(nbits=3, size="9b")
print(f"Compression: {config.compression_ratio:.1f}x")          # 5.3x
print(f"Max context @ 4 GB: {config.max_context(4.0)} tokens")  # ~22K

cache = PolarKVCache(config, device="cuda")
```

--------------------------------

### Check Installed Library Versions

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant.md

Verifies the installed versions of the transformers and torchao libraries. This is a crucial step to ensure compatibility and proper functioning of the quantization pipeline.

```python
import transformers; print(f'transformers: {transformers.__version__}')
import torchao; print(f'torchao: {torchao.__version__}')

```

--------------------------------

### Usage of PolarQuant Q3 KV Cache

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/ggml-integration/polar-quant.md

Example command to load a model using PolarQuant Q3 for both key and value caches with an extended context length.

```bash
./llama-cli -m model.gguf \
    --cache-type-k pq3_0 \
    --cache-type-v pq3_0 \
    -c 131072
```

--------------------------------

### Install Dependencies for Transformers

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/eoq.md

Installs core HuggingFace libraries and essential tools for model quantization and processing. Add extra packages like 'gguf' or 'mamba-ssm' based on model requirements.

```python
!pip install -q git+https://github.com/huggingface/transformers.git datasets accelerate safetensors sentencepiece tiktoken huggingface_hub
```

--------------------------------

### Install Dependencies for PolarQuant Arena

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-arena.md

Installs necessary libraries including transformers, accelerate, safetensors, sentencepiece, scipy, torchao, and the lm-eval harness. Use --force-reinstall for transformers to ensure the latest version is used.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q safetensors sentencepiece scipy torchao
!pip install -q lm-eval  # EleutherAI evaluation harness
```

--------------------------------

### Initialize Environment and Load Model (Python)

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-mlx.md

Sets up necessary environment variables for HuggingFace cache and temporary directories, and loads the model in bfloat16 precision onto the CPU.

```python
#!/usr/bin/env python3
"""PolarQuant MLX Converter — {MODEL_NAME}"""

import os
os.environ['HF_HOME'] = '/Volumes/SSD Major/fish/.hf_cache'
os.environ['TMPDIR'] = '/Volumes/SSD Major/fish/.tmp'
os.makedirs(os.environ['HF_HOME'], exist_ok=True)
os.makedirs(os.environ['TMPDIR'], exist_ok=True)

import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np, math, gc, time, json, shutil
from transformers import AutoModelForCausalLM, AutoTokenizer
from scipy.stats import norm
from huggingface_hub import HfApi, login

MODEL = '{owner/model-name}'
HF_TOKEN = 'YOUR_HF_TOKEN'
WORK_DIR = '/Volumes/SSD Major/fish/polar_mlx_{short_name}'
DEQUANT_DIR = os.path.join(WORK_DIR, 'dequanted_bf16')
MLX_DIR = os.path.join(WORK_DIR, 'mlx_4bit')
REPO_ID = 'caiovicentino1/{Model-Name}-PolarQuant-MLX-4bit'
BS = 128

os.makedirs(DEQUANT_DIR, exist_ok=True)
os.makedirs(MLX_DIR, exist_ok=True)
login(token=HF_TOKEN)

model = AutoModelForCausalLM.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map='cpu', trust_remote_code=True
)
```

--------------------------------

### Initialize model, tokenizer, and kernels

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/notebooks/full_pipeline_test.ipynb

Sets up the environment by importing necessary PyTorch and Hugging Face modules, defining model parameters, and initializing the Hadamard matrix and tokenizer. Displays GPU information.

```python
import torch, torch.nn as nn, torch.nn.functional as F
import numpy as np, time, math, gc, os, json
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from polarengine_vllm.utils import get_centroids, get_bits_for_layer
from polarengine_vllm.kernels.fwht import build_hadamard
# Import kernel JIT functions directly (skip wrapper to avoid per-call allocation)
from polarengine_vllm.kernels.polar_gemv import (
    polar_gemv_kernel, polar_gemv_packed_kernel,  # raw Triton JIT
    polar_gemv, polar_gemv_packed, pack_codes_int4,  # wrappers (for reference)
)

DEVICE = 'cuda'
MODEL = 'Qwen/Qwen3.5-9B'
BS = 128
Half_K = BS // 2

print(f'GPU: {torch.cuda.get_device_name()}')
print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')

H128 = build_hadamard(BS, DEVICE)
print(f'H128 cached on {H128.device}')

tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)
if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token
print('Ready!')
```

--------------------------------

### ExpertOffloadStore for MoE Expert CPU Offloading

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Shows how to initialize and use the ExpertOffloadStore for managing Mixture-of-Experts (MoE) expert weights on CPU-pinned memory. Supports synchronous and asynchronous transfers to GPU.

```python
import torch
from polarengine_vllm.expert_offload import ExpertOffloadStore
from polarengine_vllm.loader import PolarWeightLoader  # example usage

# Initialise store (called once at model load time)
polar_config = {"layers": {...}, "block_size": 128}
weight_loader = PolarWeightLoader("./Nemotron-30B-PolarQuant/")

store = ExpertOffloadStore(polar_config, weight_loader)
store.load_all_experts()  # ~23 GB pinned CPU memory for Nemotron 30B
print(store)
# ExpertOffloadStore(experts=2944, moe_layers=23, device='cpu')

print(store.moe_layer_indices)  # [0, 1, 2, ..., 22]
print(store.experts_for_layer(0)[:5])  # [0, 1, 2, 3, 4]

# Synchronous transfer (blocking)
gpu_expert = store.transfer_to_gpu(layer_idx=0, expert_id=42, device="cuda:0")
gate_codes = gpu_expert["gate_proj"]["codes"]   # int8/uint8 on GPU

# Asynchronous transfer with CUDA stream (overlaps with compute)
stream = torch.cuda.Stream()
gpu_expert_async = store.async_transfer(layer_idx=1, expert_id=7, stream=stream)
stream.synchronize()  # wait before using tensors
up_norms = gpu_expert_async["up_proj"]["norms"]  # fp16 on GPU

# Check membership
print(store.has_expert(layer_idx=5, expert_id=100))  # True
print(store.num_experts_loaded)  # 2944
```

--------------------------------

### Initialize and Update PolarKV Cache

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/polarengine_vllm/kv_cache/README.md

Demonstrates how to initialize the PolarKVConfig for a specific model and number of bits, then create a PolarKVCache instance. It also shows how to update the cache for a given layer and prints the cache statistics.

```python
from polarengine_vllm.kv_cache import PolarKVConfig, PolarKVCache

# For Gemma 4 31B-it (head_dim=256)
config = PolarKVConfig.for_gemma4_31b(nbits=3)
cache = PolarKVCache(config)

# Update cache per layer
full_k, full_v = cache.update(key_states, value_states, layer_idx=0)

# Check stats
print(cache.stats())
# {'seq_length': 1024, 'memory_mb': 45.2, 'fp16_memory_mb': 240.0, 'compression_ratio': 5.3}
```

--------------------------------

### Install Dependencies for PolarQuant Inference

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-inference.md

Installs necessary libraries including transformers, accelerate, safetensors, sentencepiece, scipy, torchao, and gradio. It explicitly avoids flash-attn.

```python
!pip install git+https://github.com/huggingface/transformers.git --force-reinstall -q
!pip install -q accelerate safetensors sentencepiece scipy torchao gradio
```

--------------------------------

### Serve Model with vLLM using PolarEngine

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/README.md

Serve a quantized model using vLLM, specifying the quantization method as 'polarengine'.

```bash
vllm serve ./Qwen3.5-9B-PolarEngine/ --quantization polarengine
```

--------------------------------

### Pack and Unpack Codes with Half Block

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Demonstrates packing and unpacking of codes using a half-block strategy. Ensure the input codes are compatible with the packing function.

```python
from polarengine_vllm.packing import pack_codes_half_block, unpack_codes_half_block
import torch

codes = torch.randint(0, 16, (4096, 4096), dtype=torch.int8)
print(f"Original: {codes.shape}, {codes.numel() * 1 / 1e6:.1f} MB")

packed = pack_codes_half_block(codes, block_size=128)
print(f"Packed:   {packed.shape}, {packed.numel() * 1 / 1e6:.1f} MB")

unpacked = unpack_codes_half_block(packed, block_size=128)
assert torch.all((unpacked & 0x0F) == (codes & 0x0F))
print("Round-trip: OK")
```

--------------------------------

### Create HF Repository

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-mlx.md

Initializes a new model repository on Hugging Face using the HfApi. The `exist_ok=True` argument prevents errors if the repository already exists.

```python
api = HfApi()
api.create_repo(REPO_ID, repo_type='model', exist_ok=True)
```

--------------------------------

### Get PolarKVCache Statistics

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Retrieve statistics from the `PolarKVCache` object to monitor its usage, including sequence length, memory consumption, and compression ratio.

```python
stats = cache.stats()
print(stats)
# {'seq_length': 64, 'memory_mb': 1.2, 'fp16_memory_mb': 6.0,
#  'compression_ratio': 5.0, 'seen_tokens': 64}
```

--------------------------------

### Pack and Unpack Codes with Q5 Quantization

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Illustrates packing integer codes using 5-bit quantization (Q5). The resulting packed tensor has a reduced shape compared to the original.

```python
from polarengine_vllm.packing import pack_codes_q5
import torch

# 5-bit packing (Q5 codes, 0–31)
codes_q5 = torch.randint(0, 32, (4096, 4096), dtype=torch.int8)
packed_q5 = pack_codes_q5(codes_q5, block_size=128)
print(f"Q5 packed: {packed_q5.shape}")  # (4096, 2560) → 5/8 of original
```

--------------------------------

### Define Model Presets with PolarKVConfig

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Use predefined configurations for common models like Gemma, Llama, and Qwen. Adjust `nbits` for quantization level. The `max_context` method estimates token capacity for a given VRAM size.

```python
configs = {
    "gemma4-31b": PolarKVConfig.for_gemma4_31b(nbits=3),
    "llama3-8b":  PolarKVConfig.for_llama3(nbits=3, size="8b"),
    "llama3-70b": PolarKVConfig.for_llama3(nbits=3, size="70b"),
    "qwen35-9b":  PolarKVConfig.for_qwen35(nbits=3, size="9b"),
    "qwen35-27b": PolarKVConfig.for_qwen35(nbits=3, size="27b"),
}

for name, cfg in configs.items():
    print(f"{name}: {cfg.compression_ratio:.1f}x compression, "
          f"max_context(4GB)={cfg.max_context(4.0):,} tokens")
```

--------------------------------

### PolarQuant CLI Usage

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/blog/polarquant-blog.md

Command-line interface commands for interacting with PolarQuant. Includes options for checking model information, starting a chat session, and quantizing/uploading models.

```bash
pip install polarquant[all]

# Info about any model
polarquant info google/gemma-4-31B-it

# Start chatting immediately
polarquant chat google/gemma-4-31B-it --vision

# Quantize and upload
polarquant quantize google/gemma-4-31B-it --upload

```

--------------------------------

### Get Model Stats via Hugging Face API

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-monitor.md

Fetches download and like counts for models by a specific author, sorted by downloads. Requires the `huggingface_hub` library.

```python
from huggingface_hub import HfApi, ModelInfo
api = HfApi()

# Get download stats
models = api.list_models(author='caiovicentino1', sort='downloads', direction=-1)
for m in models:
    info = api.model_info(m.id)
    print(f'{m.id}: {info.downloads} downloads, {info.likes} likes')
```

--------------------------------

### Build and Test llama.cpp with PolarQuant

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-llamacpp.md

Build llama.cpp with CPU-only support and then test the integration using the command-line interface with PQ3_0 KV cache types.

```bash
cmake -B build -DGGML_CUDA=OFF
cmake --build build -j$(nproc)
./build/bin/llama-cli -m model.gguf --cache-type-k pq3_0 --cache-type-v pq3_0 -c 65536
```

--------------------------------

### Display Full Pipeline Results

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/notebooks/full_pipeline_test.ipynb

Prints a formatted table summarizing the performance metrics (tokens/sec, VRAM, PPL) for different model configurations, including baselines and the packed Polarengine.

```python
# ===============================================================
# RESULTS
# ===============================================================
final_vram = torch.cuda.memory_allocated() / 1e9
print(f'
{"="*60}')
print(f'  POLARENGINE PACKAGE — FULL PIPELINE RESULTS')
print(f'  Qwen3.5-9B | {torch.cuda.get_device_name()}')
print(f'{"="*60}')
print(f'  {"Method":<30} {"tok/s":>8} {"VRAM":>10} {"PPL":>8}')
print(f'  {"
```

```python
"-"*60}')
print(f'  {"FP16 baseline":<30} {"45.7":>8} {fp16_vram:>8.1f} GB {"6.37":>8}')
print(f'  {"Standalone v4 (ref)":<30} {"34.2":>8} {"7.9 GB":>10} {"6.89":>8}')
print(f'  {"Package unpacked":<30} {unpacked_tps:>8.1f} {unpacked_vram:>8.1f} GB')
print(f'  {"Package + INT4 pack":<30} {packed_tps:>8.1f} {packed_vram:>8.1f} GB {ppl:>8.4f}')
print(f'  {"
```

```python
"-"*60}')
print(f'  Package vs standalone: {packed_tps/34.2:.2f}x speed')
print(f'  Package vs FP16: {packed_tps/45.7*100:.0f}% speed, {fp16_vram/packed_vram:.1f}x less VRAM')
print(f'{"="*60}')

```

--------------------------------

### Build Standalone C Library

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-llamacpp.md

Compile the standalone C library for PolarQuant. Ensure you have a C compiler like GCC installed. This command builds the test executable for the library.

```bash
gcc -O2 -Iinclude -o test src/polar_quants.c src/test_polar.c -lm && ./test
```

--------------------------------

### Standard 3-bit Quantization Process

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/ggml-integration/polar-quant.md

Illustrates the simplified process of standard 3-bit quantization, involving finding a scale, rounding, and storing.

```text
weight → find scale → round to nearest 3-bit value → store
```

--------------------------------

### Load Model and Run Benchmarks with lm-eval

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-arena.md

Wraps the PolarQuant model for use with lm-eval and runs standard chat and code benchmarks. Specify the number of few-shot examples for tasks like MMLU.

```python
import lm_eval
from lm_eval.models.huggingface import HFLM

# Wrap our model for lm-eval
lm = HFLM(pretrained=model, tokenizer=tokenizer, batch_size=4)

# Select benchmarks based on model type
benchmarks_chat = ['mmlu', 'gsm8k', 'arc_challenge', 'hellaswag']
benchmarks_code = ['humaneval', 'mbpp']

results = lm_eval.simple_evaluate(
    model=lm,
    tasks=benchmarks_chat,
    num_fewshot=5,  # standard for MMLU
    batch_size=4,
)
```

--------------------------------

### Python Serving Script (`serve.py`)

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-serve.md

Main script for serving PolarQuant models with a vLLM-compatible API. Includes streaming loader, PolarQuant KV cache, and an OpenAI-compatible endpoint. Use with `python serve.py --model <model_name> --port <port>`.

```python
"""Serve PolarQuant model with vLLM-compatible API.

Usage:
    python serve.py --model google/gemma-4-31B-it --port 8000
    curl http://localhost:8000/v1/chat/completions -d '{...}'
"""

import argparse
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from polarengine_vllm.kv_cache import PolarKVConfig, PolarKVCache

# ... streaming loader code ...
# ... FastAPI/uvicorn OpenAI-compatible endpoint ...

```

--------------------------------

### polarquant-convert CLI Tool

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

A command-line tool to convert PolarQuant models from HuggingFace to BF16 safetensors format, making them compatible with standard vLLM and transformers setups without requiring the PolarEngine plugin.

```APIDOC
## `polarquant-convert` CLI — BF16 conversion for vLLM

Command-line tool that downloads a PolarQuant model from HuggingFace, dequantizes all coded layers to BF16 safetensors, and writes a standard model directory loadable by any vLLM or transformers setup without the plugin. Handles multi-shard models, recovers missing buffers from the base model repo, and fixes tokenizer configs.

```bash
# Convert a HuggingFace PolarQuant model to plain BF16
polarquant-convert caiovicentino1/Qwen3.5-9B-PolarQuant-Q5 /tmp/qwen35-bf16

# Output:
#   Downloading PolarQuant codes...  (downloads safetensors)
#   342 quantized layers, block_size=128
#   Converting...
#     model-00001-of-00003.safetensors: 156 tensors
#     model-00002-of-00003.safetensors: 148 tensors
#     model-00003-of-00003.safetensors: 38 tensors
#   Done in 312s!
#   Dequanted: 342 layers
#   Passthrough: 48 tensors
#   Load with vLLM: vllm serve /tmp/qwen35-bf16 --trust-remote-code --dtype bfloat16

# Then serve without the PolarEngine plugin
vllm serve /tmp/qwen35-bf16 --trust-remote-code --dtype bfloat16
```
```

--------------------------------

### Interact with HuggingFace Hub API

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-collection.md

Provides basic functions to interact with the HuggingFace Hub, such as listing models, retrieving collection details, listing repository files, and downloading files. Ensure `huggingface_hub` is installed.

```python
from huggingface_hub import HfApi
api = HfApi()

# List models
models = list(api.list_models(author='caiovicentino1'))

# Get collection
col = api.get_collection('caiovicentino1/polarquant-models-69cbc96292c5174df2088b08')

# List repo files
files = api.list_repo_files('caiovicentino1/Model-Name')

# Read file
content = api.hf_hub_download('repo', 'file.json')
```

--------------------------------

### Explore Video Model Structure and Config

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-video.md

Examines the files and configuration of a video model repository. Crucial for understanding model architecture and parameters before quantization. It checks for class name, number of layers, dimensions, and attention heads.

```python
# Check repo files
api = HfApi()
files = list(api.list_repo_files(MODEL))

# Check config
config = json.load(open(hf_hub_download(MODEL, 'config.json')))
print(f'Class: {config.get("_class_name")}')
print(f'Layers: {config.get("num_layers")}')
print(f'Dim: {config.get("dim")}')
print(f'Heads: {config.get("num_heads")}')
# head_dim = dim / num_heads — must be power of 2 for Hadamard
```

--------------------------------

### Convert PolarQuant Model to BF16 using CLI

Source: https://context7.com/caiovicentino/polarengine-vllm/llms.txt

Demonstrates the usage of the `polarquant-convert` CLI tool to dequantize a PolarQuant model from HuggingFace to BF16 safetensors format, making it compatible with standard vLLM and transformers setups.

```bash
# Convert a HuggingFace PolarQuant model to plain BF16
polarquant-convert caiovicentino1/Qwen3.5-9B-PolarQuant-Q5 /tmp/qwen35-bf16

# Output:
#   Downloading PolarQuant codes...  (downloads safetensors)
#   342 quantized layers, block_size=128
#   Converting...
#     model-00001-of-00003.safetensors: 156 tensors
#     model-00002-of-00003.safetensors: 148 tensors
#     model-00003-of-00003.safetensors: 38 tensors
#   Done in 312s!
#   Dequanted: 342 layers
#   Passthrough: 48 tensors
#   Load with vLLM: vllm serve /tmp/qwen35-bf16 --trust-remote-code --dtype bfloat16

# Then serve without the PolarEngine plugin
vllm serve /tmp/qwen35-bf16 --trust-remote-code --dtype bfloat16
```

--------------------------------

### Run KV Cache Benchmark

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/polarengine_vllm/kv_cache/README.md

Provides commands to run the KV cache benchmark. The first command uses a preset configuration for Gemma 4 31B, while the second allows manual specification of head dimension and number of bits.

```bash
python -m polarengine_vllm.kv_cache.benchmark --preset gemma4-31b
```

```bash
python -m polarengine_vllm.kv_cache.benchmark --head-dim 128 --nbits 3
```

--------------------------------

### Upload README to Repository

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-gguf.md

Use this to upload a README file to a Hugging Face model repository. Ensure the path and repository ID are correctly specified.

```python
api.upload_file(path_or_fileobj=card.encode(), path_in_repo='README.md',
                repo_id=REPO, repo_type='model')
```

--------------------------------

### CPU Backend Quantization Implementation: ggml-cpu/quants.c

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-llamacpp.md

Implement the CPU-specific quantization and dequantization wrappers in `ggml/src/ggml-cpu/quants.c`. This file provides the concrete implementations for CPU-based operations.

```c
void quantize_row_pq3_0(void * dst, const float * src, int n, int k, const float * imatrix);
void ggml_vec_dot_pq3_0_q8_0(int n, float * s, size_t bs, const void * vx, size_t bx, const void * vy, size_t by, int nrc);
```

--------------------------------

### Quantize Model with PolarEngine CLI

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/README.md

Use the PolarEngine command-line interface to quantize a specified model and save the output to a directory.

```bash
python -m polarengine_vllm.quantize \
    --model Qwen/Qwen3.5-9B \
    --output ./Qwen3.5-9B-PolarEngine/
```

--------------------------------

### Model Quantization Workflow Outline

Source: https://github.com/caiovicentino/polarengine-vllm/blob/main/skills/polarquant-multi.md

Outlines the process for loading a safetensors model, quantizing all its 2D weights for each supported PolarQuant variant (PQ2 to PQ8), verifying cosine similarity on the first 5 layers, and saving the quantized models.

```python
# Load safetensors
# For each variant (PQ2 to PQ8):
#   1. Quantize all 2D weights
#   2. Verify cos_sim on first 5 layers
#   3. Save to /content/pq{bits}/

```