### Quantization-Aware Training (QAT) Setup

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Applies quantizers using `replace_layers` before training to enable gradients to flow through fake-quantization operations. Assumes a standard PyTorch training loop.

```python
import torch
import torchvision
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import INT8, FP16

model = torchvision.models.resnet50(pretrained=True)

# Quantize before training — gradients flow through fake-quant ops
replace_layers(model, input_quantizer=INT8, weight_quantizer=INT8, output_quantizer=FP16)

# Standard PyTorch training loop
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4, momentum=0.9, weight_decay=1e-4)
criterion = torch.nn.CrossEntropyLoss()

model.train()
for epoch in range(5):
    for images, labels in train_loader:          # your DataLoader
        images, labels = images.cuda(), labels.cuda()
        optimizer.zero_grad()
        loss = criterion(model(images), labels)
        loss.backward()                          # backward through FPQuantFunction / fake_tensor_quant
        optimizer.step()

# Save and evaluate
torch.save(model.state_dict(), "resnet50_qat_int8.pth")
```

--------------------------------

### Install PyTorch with CUDA 11.6

Source: https://github.com/lightmatter-ai/int-fp-qsim/blob/main/README.md

Installs PyTorch version 1.13.0 with CUDA 11.6 and torchvision version 0.14.0 with CUDA 11.6. Use the --index-url option for specific CUDA or CPU versions.

```bash
pip install torch==1.13.0+cu116 torchvision==0.14.0+cu116 -f https://download.pytorch.org/whl/torch_stable.html
```

--------------------------------

### Quickstart: ResNet50 E4M3 Quantization

Source: https://github.com/lightmatter-ai/int-fp-qsim/blob/main/README.md

Quantizes inputs and weights to E4M3 format and outputs to BF16 for a ResNet50 model. The replace_layers function modifies the model in-place. Ensure PyTorch and torchvision are installed.

```python
import torch
import torchvision
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import E4M3, BF16

model = torchvision.models.resnet50()
# Quantize inputs and weights to E4M3 and outputs in BF16
# Following function replaces layers of the model to support quantization
# Note that replace_layers is an in-place function
replace_layers(
    model,
    input_quantizer=E4M3,
    weight_quantizer=E4M3,
    output_quantizer=BF16
)
print(model)  # You can see the quantizer objects attached to layers

# Continue to evaluation
inputs = torch.randn(1, 3, 225, 225)
model.eval()
with torch.no_grad():
    model(inputs)
```

--------------------------------

### Full Static Calibration Workflow

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Demonstrates a three-step pipeline: calibrate input activations, calibrate weights, and then replace layers. This workflow is recommended for static inference.

```python
import torch
import torchvision
from torchvision.models import resnet50
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
import torchvision.datasets as datasets

from int_fp_qsim.formats import INT8, INT4, FP16
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.static_calib import calibrate_inputs, calibrate_weights

# ── 1. Load FP32 model ────────────────────────────────────────────────────────
model = resnet50(pretrained=True)

# ── 2. Build a small calibration dataloader ──────────────────────────────────
transform = transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224),
                                transforms.ToTensor()])
calib_dataset = datasets.ImageFolder("/data/imagenet/val", transform=transform)
calib_loader  = DataLoader(calib_dataset, batch_size=32, num_workers=4, shuffle=True)

# ── 3. Calibrate inputs (5 batches, percentile method) ────────────────────────
def batch_fn(model, data):
    images, _ = data
    model(images.cuda())

calibrate_inputs(
    model.cuda(), num_bits=8, dataloader=calib_loader,
    batch_process_func=batch_fn,
    method="percentile", percentile=99.99, num_samples=5
)

# ── 4. Calibrate weights (per-channel max) ───────────────────────────────────
calibrate_weights(model, num_bits=8, unsigned=False, method="max", perchannel=True)

# ── 5. Replace layers — static scales are carried over ───────────────────────
replace_layers(model, input_quantizer=INT8, weight_quantizer=INT4, output_quantizer=FP16)

# ── 6. Evaluate ──────────────────────────────────────────────────────────────
val_loader = DataLoader(calib_dataset, batch_size=64, num_workers=4)
model.eval()
correct = total = 0
with torch.no_grad():
    for images, labels in val_loader:
        preds = model(images.cuda()).argmax(1).cpu()
        correct += (preds == labels).sum().item()
        total   += labels.size(0)
print(f"Top-1 accuracy: {100 * correct / total:.2f}%")
```

--------------------------------

### Create Custom Quantized Linear Layer with QuantMixin

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Demonstrates how to create a custom quantized linear layer by inheriting from torch.nn.Linear and QuantMixin. This allows for direct control over input, weight, and output quantization.

```python
from int_fp_qsim.utils import QuantMixin
import torch
from functools import partial
from int_fp_qsim.quantizer import quantize_to_int

# Build a custom quantized layer using QuantMixin directly
class MyQuantLinear(torch.nn.Linear, QuantMixin):
    def __init__(self, in_features, out_features, input_q, weight_q, output_q):
        super().__init__(in_features, out_features)
        self.init_quantizer(input_q, weight_q, output_q,
                            amax_input=None, amax_weight=None)

    def forward(self, x):
        qx = self._quantize_input(x)
        qw = self._quantize_weight(self.weight)
        out = torch.nn.functional.linear(qx, qw, self.bias)
        return self._quantize_output(out)

INT8 = partial(quantize_to_int, num_bits=8)
INT4 = partial(quantize_to_int, num_bits=4)

layer = MyQuantLinear(128, 64, input_q=INT8, weight_q=INT4, output_q=INT8)
x = torch.randn(4, 128)
print(layer(x).shape)   # torch.Size([4, 64])
```

```python
# Inject a static scale at runtime
layer.amax_input  = torch.tensor([2.5])
layer.amax_weight = torch.tensor([1.8])
print(layer(x).shape)   # now uses static scales internally
```

--------------------------------

### Apply Quantizers for Quantization-Aware Training (QAT)

Source: https://github.com/lightmatter-ai/int-fp-qsim/blob/main/int_fp_qsim/README.md

This snippet demonstrates how to apply quantizers to a model for Quantization-Aware Training (QAT). It uses `replace_layers` to set input and weight quantizers to INT8 and the output quantizer to FP16. Ensure the quantizer's backward pass is well-defined for correct training.

```python
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import INT8, FP16

model = torchvision.models.resnet50()
# Quantize inputs and weights using INT8 and outputs in FP16
replace_layers(
    model,
    input_quantizer=INT8,
    weight_quantizer=INT8,
    output_quantizer=FP16 
)

# Proceed with training...
```

--------------------------------

### Floating-Point Quantization with `quantize_to_fp`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Performs floating-point quantization with configurable exponent and mantissa bit widths, inspired by AIMET. Supports max or percentile calibration, per-element or per-channel scales, and clipping. Use `functools.partial` for integration with layer replacement.

```python
from int_fp_qsim.quantizer import quantize_to_fp
import torch

x = torch.randn(8, 32)

# E4M3 equivalent using quantize_to_fp directly
x_e4m3 = quantize_to_fp(x, mantissa_bits=3, exponent_bits=4)

# E2M1 with percentile calibration
x_e2m1_pct = quantize_to_fp(
    x, mantissa_bits=1, exponent_bits=2,
    calibration="percentile", percentile=99.5
)

# Static scale (float or tensor)
x_fp_static = quantize_to_fp(
    x, mantissa_bits=3, exponent_bits=4,
    static_scale=torch.tensor([2.0])
)

# Per-channel (axis=0 for first dimension)
weight = torch.randn(64, 32)
w_perchan = quantize_to_fp(
    weight, mantissa_bits=3, exponent_bits=4,
    per_channel_axis=0
)

# Use as a partial for replace_layers
from functools import partial
E3M2 = partial(quantize_to_fp, mantissa_bits=2, exponent_bits=3)
```

--------------------------------

### Quantized Attention Layers Integration

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Shows how to import and use quantized attention variants from `int_fp_qsim`. These are automatically applied by `replace_layers` via `DEFAULT_MAPPING` and respect static calibration scales.

```python
# Supported attention types (registered in DEFAULT_MAPPING):
#   MultiheadAttention, OPTAttention, BertSelfAttention,
#   DetrAttention, MaskFormerSwinSelfAttention,
#   CodeGenAttention, Attention (Stable Diffusion cross-attn),
#   GraphormerMultiheadAttention

from transformers import AutoModelForCausalLM
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import E4M3, FP16
```

--------------------------------

### Static Activation Calibration with `calibrate_inputs`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Calibrates input activations by attaching forward hooks, passing data through the model, and computing per-layer scales. Supports custom `batch_process_func` for non-standard data inputs like HuggingFace models.

```python
from int_fp_qsim.static_calib import calibrate_inputs
from torchvision.models import resnet50
from torch.utils.data import DataLoader, TensorDataset
import torch

model = resnet50(pretrained=True).cuda()

# Synthetic calibration dataloader (replace with real ImageNet loader)
dummy_data = TensorDataset(torch.randn(50, 3, 224, 224))
calib_loader = DataLoader(dummy_data, batch_size=10)

# Max calibration — fastest, uses observed maximum activation
calibrate_inputs(
    model.cuda(), num_bits=8, dataloader=calib_loader,
    method="max", num_samples=5
)

# Percentile calibration — clips top 0.01% outliers
calibrate_inputs(
    model.cuda(), num_bits=8, dataloader=calib_loader,
    method="percentile", percentile=99.99, num_samples=5
)

# Custom batch_process_func for HuggingFace-style dict inputs (e.g. OPT)
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

opt_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").cuda()
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")

def batch_process_func(model, batch):
    inputs = {k: v.cuda() for k, v in batch.items()}
    model(**inputs)

# calib_loader yields dicts with input_ids, attention_mask, etc.
calibrate_inputs(
    opt_model, num_bits=8, dataloader=calib_loader,
    batch_process_func=batch_process_func,
    method="percentile", percentile=99.99, num_samples=5
)
```

--------------------------------

### Built-in Quantizer Callables from `formats.py`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Provides ready-to-use quantizer callables for common formats like E4M3, FP16, BF16, INT8, and INT4. These can be directly passed to `replace_layers`. Some formats support additional arguments for rounding and calibration.

```python
from int_fp_qsim.formats import E4M3, FP16, BF16, E1M2, E2M1, INT8, INT4
import torch

x = torch.randn(4, 8)

# FP8 E4M3 (QPyTorch) — common for LLM weights
x_e4m3 = E4M3(x)

# BF16 (native PyTorch cast)
x_bf16 = BF16(x)           # equivalent to x.to(torch.bfloat16).float()

# FP16 (5 exponent, 10 mantissa bits via QPyTorch)
x_fp16 = FP16(x)

# AIMET-style FP formats with configurable exponent/mantissa
x_e1m2 = E1M2(x)           # 1 exp bit, 2 mantissa bits
x_e2m1 = E2M1(x)           # 2 exp bit, 1 mantissa bit

# TensorRT-style integer quantization
x_int8 = INT8(x)            # 8-bit integer
x_int4 = INT4(x)            # 4-bit integer

# ── Custom QPyTorch format ────────────────────────────────────────────────────
from qtorch import FloatingPoint
from qtorch.quant import quantizer

format_e3m4 = FloatingPoint(exp=3, man=4)
format_e5m2 = FloatingPoint(exp=5, man=2)

# Different precision in forward vs backward (useful for QAT)
E3M4_E5M2 = quantizer(
    forward_number=format_e3m4,
    forward_rounding="nearest",
    backward_number=format_e5m2,
    backward_rounding="nearest",
)
x_custom = E3M4_E5M2(x)
```

--------------------------------

### quantize_to_fp

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Performs floating-point quantization with configurable exponent and mantissa bit widths. Supports max or percentile calibration, and per-channel scaling.

```APIDOC
## `quantize_to_fp` — Floating-point quantization with configurable exponent/mantissa

AIMET-derived FP quantizer that supports arbitrary exponent and mantissa bit widths. Computes a per-element or per-channel scale from the input tensor using max or percentile calibration, clips to the representable range, and rounds to the FP grid. Also implements a QAT-compatible backward pass via `FPQuantFunction`.

### Usage Examples

```python
from int_fp_qsim.quantizer import quantize_to_fp
import torch

x = torch.randn(8, 32)

# E4M3 equivalent using quantize_to_fp directly
x_e4m3 = quantize_to_fp(x, mantissa_bits=3, exponent_bits=4)

# E2M1 with percentile calibration
x_e2m1_pct = quantize_to_fp(
    x, mantissa_bits=1, exponent_bits=2,
    calibration="percentile", percentile=99.5
)

# Static scale (float or tensor)
x_fp_static = quantize_to_fp(
    x, mantissa_bits=3, exponent_bits=4,
    static_scale=torch.tensor([2.0])
)

# Per-channel (axis=0 for first dimension)
weight = torch.randn(64, 32)
w_perchan = quantize_to_fp(
    weight, mantissa_bits=3, exponent_bits=4,
    per_channel_axis=0
)

# Use as a partial for replace_layers
from functools import partial
E3M2 = partial(quantize_to_fp, mantissa_bits=2, exponent_bits=3)
```
```

--------------------------------

### Perform Static Calibration for ResNet50

Source: https://github.com/lightmatter-ai/int-fp-qsim/blob/main/int_fp_qsim/README.md

This function calibrates model inputs and weights for static quantization. It requires a model, dataloader, number of bits, calibration method, and percentile. A custom batch process function can be provided to handle model input processing.

```python
from torchvision.models import resnet50
from int_fp_qsim.formats import INT8, INT4, FP16
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.static_calib import calibrate_inputs, calibrate_weights

def do_static_calibration(
    model, dataloader, num_bits, method, percentile, num_samples=5
):
    # [OPTIONAL] Function that tells the model how to process the inputs.
    def batch_process_func(model, inputs):
        # Most commonly used for BERT / OPT
        model(inputs[0].cuda())

    # This function will pass `num_samples` batches of data
    # from `dataloader` and calibrate the model for `num_bits`
    # based on the specified `method`.
    calibrate_inputs(
        model.cuda(),
        num_bits,
        dataloader,
        method=method,
        batch_process_func=batch_process_func,
        percentile=percentile,
        num_samples=num_samples
    )

    # This following function performs the same for the model.
    # No data batches are required here since this is done on weights.
    calibrate_weights(model, num_bits, False, method="max", perchannel=True)

model = resnet50(pretrained=True)
data_dir = "/data/datasets/imagenet"
data_loaders = get_dataloaders(
    data_dir,
    128,
    32,
    4,
)

# The following function will compute and attach static
# scales as attributes to the FP32 model layers.
do_static_calibration(
    model,
    data_loaders.train_loader,
    num_bits=8,
    method="percentile",
    percentile=99.99,
    num_samples=1
)

# Replace layers will copy the static scale attributes
# from the FP32 layers to the replaced quant layers.
replace_layers(
    model,
    input_quantizer=INT8,
    weight_quantizer=INT4,
    output_quantizer=FP16
)

# If you print the model at this stage, you should
# be able to see the computed scales attached as
# `amax_input` and `amax_weight`
print(model)

# During inference, each layer's forward call will
# check if scales are attached as layer attributes.
# If they are, then the model will use the attached
# scales to perform the quantization.
eval(model, data_loaders.val_loader, "cuda")
```

--------------------------------

### Integer Quantization with `quantize_to_int`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Simulates integer quantization using TensorRT's `fake_tensor_quant`. Supports dynamic (max, percentile) and static scales, signed/unsigned modes, and per-channel quantization. Use `functools.partial` for custom quantizer callables.

```python
from int_fp_qsim.quantizer import quantize_to_int
import torch

x = torch.randn(16, 64)

# Dynamic max calibration (default)
x_int8_dynamic = quantize_to_int(x, num_bits=8)

# Percentile calibration (clips outliers)
x_int8_pct = quantize_to_int(x, num_bits=8, calibration="percentile", percentile=99.9)

# Static scale (from prior calibration run)
static_amax = torch.tensor([3.5])
x_int8_static = quantize_to_int(x, num_bits=8, static_scale=static_amax)

# 4-bit unsigned per-channel (axis=0 for output channels)
x_int4_perchan = quantize_to_int(
    x, num_bits=4, unsigned=True, per_channel_axis=0
)

# Use as a custom quantizer callable with functools.partial
from functools import partial
INT6_unsigned = partial(quantize_to_int, num_bits=6, unsigned=True)
x_int6 = INT6_unsigned(x)
```

--------------------------------

### Static Weight Calibration with `calibrate_weights`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Computes per-tensor or per-channel weight scales for supported layers using max, MSE, or percentile criteria. The results are attached as `module.amax_weight` and must be called before `replace_layers`.

```python
from int_fp_qsim.static_calib import calibrate_weights
from torchvision.models import resnet50

model = resnet50(pretrained=True)

# Per-channel max calibration (recommended for Conv/Linear)
calibrate_weights(
    model, num_bits=8, unsigned=False,
    method="max", perchannel=True
)

# Per-tensor percentile calibration
calibrate_weights(
    model, num_bits=8, unsigned=False,
    method="percentile", perchannel=False, percentile=99.99
)
```

--------------------------------

### In-place Model Conversion with `replace_layers`

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Recursively replaces supported layers in a `torch.nn.Module` with their quantized counterparts. The function is in-place and propagates pre-computed static calibration scales. Use this to convert models for mixed precision inference.

```python
import torch
import torchvision
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import E4M3, BF16, INT8, INT4, FP16

# ── Example 1: FP8 (E4M3) inputs/weights, BF16 outputs ──────────────────────
model = torchvision.models.resnet50(pretrained=False)

replace_layers(
    model,
    input_quantizer=E4M3,   # callable: tensor -> simulated-quantized tensor
    weight_quantizer=E4M3,
    output_quantizer=BF16,
)

print(model)  # Each Linear/Conv now shows quantizer info in repr

inputs = torch.randn(1, 3, 224, 224)
model.eval()
with torch.no_grad():
    out = model(inputs)       # forward uses E4M3 for inputs/weights, BF16 for outputs
print(out.shape)              # torch.Size([1, 1000])

# ── Example 2: Mixed INT8 inputs, INT4 weights, FP16 outputs ─────────────────
model2 = torchvision.models.resnet50(pretrained=False)
replace_layers(model2, input_quantizer=INT8, weight_quantizer=INT4, output_quantizer=FP16)

# ── Example 3: Custom layer mapping ──────────────────────────────────────────
from int_fp_qsim.layers import Linear as QLinear
custom_mapping = {torch.nn.Linear: QLinear}   # only replace Linear layers
model3 = torchvision.models.resnet50(pretrained=False)
replace_layers(model3, input_quantizer=E4M3, weight_quantizer=E4M3,
               output_quantizer=FP16, mapping=custom_mapping)
```

--------------------------------

### Replace Layers with Quantizers in OPT Model

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Replaces Linear and OPTAttention layers in a HuggingFace OPT model with specified quantizers. This is useful for exploring different quantization formats.

```python
from transformers import AutoModelForCausalLM
from int_fp_qsim.quantizer import E4M3, FP16
from int_fp_qsim.layers import replace_layers

model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m")

replace_layers(model, input_quantizer=E4M3, weight_quantizer=E4M3, output_quantizer=FP16)
print(model)  # decoder layers now show QuantMixin attributes
```

```python
import torch
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
inputs = tokenizer("The meaning of life is", return_tensors="pt")
model.eval()
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

--------------------------------

### calibrate_inputs

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Performs static calibration of input activations for a model. Collects activation histograms over a specified number of samples and computes optimal per-layer scales.

```APIDOC
## `calibrate_inputs` — Static calibration of input activations

Attaches forward hooks to every supported layer, passes `num_samples` batches through the model, collects activation histograms, computes the optimal per-layer scale (amax), and attaches it as `module.amax_input`. A custom `batch_process_func` can override how data is fed to the model (required for dict-based inputs such as BERT/OPT tokenizer outputs).

### Usage Examples

```python
from int_fp_qsim.static_calib import calibrate_inputs
from torchvision.models import resnet50
from torch.utils.data import DataLoader, TensorDataset
import torch

model = resnet50(pretrained=True).cuda()

# Synthetic calibration dataloader (replace with real ImageNet loader)
dummy_data = TensorDataset(torch.randn(50, 3, 224, 224))
calib_loader = DataLoader(dummy_data, batch_size=10)

# Max calibration — fastest, uses observed maximum activation
calibrate_inputs(
    model.cuda(), num_bits=8, dataloader=calib_loader,
    method="max", num_samples=5
)

# Percentile calibration — clips top 0.01% outliers
calibrate_inputs(
    model.cuda(), num_bits=8, dataloader=calib_loader,
    method="percentile", percentile=99.99, num_samples=5
)

# Custom batch_process_func for HuggingFace-style dict inputs (e.g. OPT)
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

opt_model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m").cuda()
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")

def batch_process_func(model, batch):
    inputs = {k: v.cuda() for k, v in batch.items()}
    model(**inputs)

# calib_loader yields dicts with input_ids, attention_mask, etc.
calibrate_inputs(
    opt_model, num_bits=8, dataloader=calib_loader,
    batch_process_func=batch_process_func,
    method="percentile", percentile=99.99, num_samples=5
)
```
```

--------------------------------

### replace_layers — In-place model conversion to quantized layers

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Recursively replaces supported layers in a `torch.nn.Module` with their quantized counterparts. This function operates in-place and can propagate pre-computed static calibration scales.

```APIDOC
## `replace_layers` — In-place model conversion to quantized layers

`replace_layers` recursively walks a `torch.nn.Module` and replaces every supported layer with its quantized counterpart. The three quantizer callables (input, weight, output) are attached to every replaced layer and are invoked on each forward pass. The function is in-place and also propagates any pre-computed static calibration scales (`amax_input`, `amax_weight`) when present.

```python
import torch
import torchvision
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import E4M3, BF16, INT8, INT4, FP16

# ── Example 1: FP8 (E4M3) inputs/weights, BF16 outputs ──────────────────────
model = torchvision.models.resnet50(pretrained=False)

replace_layers(
    model,
    input_quantizer=E4M3,   # callable: tensor -> simulated-quantized tensor
    weight_quantizer=E4M3,
    output_quantizer=BF16,
)

print(model)  # Each Linear/Conv now shows quantizer info in repr

inputs = torch.randn(1, 3, 224, 224)
model.eval()
with torch.no_grad():
    out = model(inputs)       # forward uses E4M3 for inputs/weights, BF16 for outputs
print(out.shape)              # torch.Size([1, 1000])

# ── Example 2: Mixed INT8 inputs, INT4 weights, FP16 outputs ─────────────────
model2 = torchvision.models.resnet50(pretrained=False)
replace_layers(model2, input_quantizer=INT8, weight_quantizer=INT4, output_quantizer=FP16)

# ── Example 3: Custom layer mapping ──────────────────────────────────────────
from int_fp_qsim.layers import Linear as QLinear
custom_mapping = {torch.nn.Linear: QLinear}   # only replace Linear layers
model3 = torchvision.models.resnet50(pretrained=False)
replace_layers(model3, input_quantizer=E4M3, weight_quantizer=E4M3,
               output_quantizer=FP16, mapping=custom_mapping)
```
```

--------------------------------

### Replace Layers with Quantizers

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Replaces model layers with specified quantizers after calibration. Scales are automatically transferred from the calibration step.

```python
from int_fp_qsim.replace import replace_layers
from int_fp_qsim.formats import INT8, INT4, FP16

replace_layers(model, input_quantizer=INT8, weight_quantizer=INT4, output_quantizer=FP16)
```

--------------------------------

### Built-in format constants

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Exposes ready-to-use quantizer callables from `formats.py`. Each callable `f(tensor) -> tensor` can be directly passed as an argument to `replace_layers`.

```APIDOC
## Built-in format constants — Pre-configured quantizer callables (`formats.py`)

`formats.py` exposes ready-to-use quantizer callables. Each is a plain Python callable `f(tensor) -> tensor` that can be passed directly as `input_quantizer`, `weight_quantizer`, or `output_quantizer`. QPyTorch-based formats support a `forward_rounding` argument; AIMET-based FP formats and TensorRT-based INT formats accept optional calibration keyword arguments.

```python
from int_fp_qsim.formats import E4M3, FP16, BF16, E1M2, E2M1, INT8, INT4
import torch

x = torch.randn(4, 8)

# FP8 E4M3 (QPyTorch) — common for LLM weights
x_e4m3 = E4M3(x)

# BF16 (native PyTorch cast)
x_bf16 = BF16(x)           # equivalent to x.to(torch.bfloat16).float()

# FP16 (5 exponent, 10 mantissa bits via QPyTorch)
x_fp16 = FP16(x)

# AIMET-style FP formats with configurable exponent/mantissa
x_e1m2 = E1M2(x)           # 1 exp bit, 2 mantissa bits
x_e2m1 = E2M1(x)           # 2 exp bit, 1 mantissa bit

# TensorRT-style integer quantization
x_int8 = INT8(x)            # 8-bit integer
x_int4 = INT4(x)            # 4-bit integer

# ── Custom QPyTorch format ────────────────────────────────────────────────────
from qtorch import FloatingPoint
from qtorch.quant import quantizer

format_e3m4 = FloatingPoint(exp=3, man=4)
format_e5m2 = FloatingPoint(exp=5, man=2)

# Different precision in forward vs backward (useful for QAT)
E3M4_E5M2 = quantizer(
    forward_number=format_e3m4,
    forward_rounding="nearest",
    backward_number=format_e5m2,
    backward_rounding="nearest",
)
x_custom = E3M4_E5M2(x)
```
```

--------------------------------

### Per-channel MSE Calibration

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Calibrates model weights using the Mean Squared Error (MSE) method on a per-channel basis. This is useful for fine-tuning quantization scales for weights.

```python
calibrate_weights(
    model, num_bits=4,
    unsigned=False,
    method="mse", perchannel=True, num_bins=2048
)
```

--------------------------------

### quantize_to_int

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Performs integer quantization on tensors. Supports dynamic (max, percentile) or static calibration, signed/unsigned modes, and per-channel quantization.

```APIDOC
## `quantize_to_int` — Integer quantization with dynamic or static scale

Wraps TensorRT's `fake_tensor_quant` to produce simulated integer-quantized tensors. Supports dynamic (max or percentile) and static (pre-computed `amax`) calibration, signed/unsigned modes, and optional per-channel quantization.

### Usage Examples

```python
from int_fp_qsim.quantizer import quantize_to_int
import torch

x = torch.randn(16, 64)

# Dynamic max calibration (default)
x_int8_dynamic = quantize_to_int(x, num_bits=8)

# Percentile calibration (clips outliers)
x_int8_pct = quantize_to_int(x, num_bits=8, calibration="percentile", percentile=99.9)

# Static scale (from prior calibration run)
static_amax = torch.tensor([3.5])
x_int8_static = quantize_to_int(x, num_bits=8, static_scale=static_amax)

# 4-bit unsigned per-channel (axis=0 for output channels)
x_int4_perchan = quantize_to_int(
    x, num_bits=4, unsigned=True, per_channel_axis=0
)

# Use as a custom quantizer callable with functools.partial
from functools import partial
INT6_unsigned = partial(quantize_to_int, num_bits=6, unsigned=True)
x_int6 = INT6_unsigned(x)
```
```

--------------------------------

### Custom Floating Point Format Specification

Source: https://github.com/lightmatter-ai/int-fp-qsim/blob/main/int_fp_qsim/README.md

Define custom floating-point formats with specific exponent and mantissa bits for forward and backward passes using QPytorch. This quantizer function can then be applied to model layers.

```python
from qtorch import FloatingPoint
from qtorch.quant import quantizer

format_e3m4 = FloatingPointNumber(exp=3, man=4)
format_e5m2 = FloatingPointNumber(exp=5, man=2)

# A custom FP format that uses E3M4 in forward pass and E5M2 in backward
quant_func = quantizer(
    forward_num=format_e3m4,
    forward_rounding='nearest,
    backward_num=format_e5m2,
    backward_rounding='nearest'
)

replace_layers(
    model,
    input_quantizer=quant_func,
    weight_quantizer=quant_func,
    output_quantizer=quant_func 
)
```

--------------------------------

### calibrate_weights

Source: https://context7.com/lightmatter-ai/int-fp-qsim/llms.txt

Performs static calibration of model weights. Computes per-tensor or per-channel weight scales using max, MSE, or percentile criteria.

```APIDOC
## `calibrate_weights` — Static calibration of model weights

Iterates over all supported layers and computes per-tensor or per-channel weight scales using max, MSE, or percentile criteria. Attaches the result as `module.amax_weight` (numpy array). Must be called on the original FP32 model before `replace_layers`.

### Usage Examples

```python
from int_fp_qsim.static_calib import calibrate_weights
from torchvision.models import resnet50

model = resnet50(pretrained=True)

# Per-channel max calibration (recommended for Conv/Linear)
calibrate_weights(
    model, num_bits=8, unsigned=False,
    method="max", perchannel=True
)

# Per-tensor percentile calibration
calibrate_weights(
    model, num_bits=8, unsigned=False,
    method="percentile", perchannel=False, percentile=99.99
)
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.