### Get and Verify Quantization Backend

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/torch/generated/aimet_torch.quantization.get_backend.html

This example demonstrates how to set a specific backend (e.g., 'triton') and then retrieve and verify the active backend using `get_backend()`.

```python
>>> aimet_torch.quantization.set_backend("triton")
>>> aimet_torch.quantization.get_backend().__name__
'aimet_torch.quantization.affine.backends.triton'
```

--------------------------------

### AIMET Installation Verification: Imports and Setup

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/index.html

This snippet handles necessary imports and initial setup for verifying AIMET installation, including numpy and AIMET's libpymo.

```python
import numpy as np
from aimet_common import libpymo

x = np.random.randn(100)

quant_scheme = libpymo.QuantizationMode.QUANTIZATION_TF
analyzer = libpymo.EncodingAnalyzerForPython(quant_scheme)

```

--------------------------------

### Compile and Install Dependencies with uv

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/build_from_source.html

Navigates to the AIMET root directory, compiles project requirements into a temporary file, and installs them using uv.

```shell
# cd to AIMET root directory
cd aimet/

# Compile requirements from pyproject.toml with constraints
uv pip compile pyproject.toml --extra=dev --extra=test --output-file=/tmp/requirements.txt

# Install the compiled dependencies
uv pip install -r /tmp/requirements.txt

```

--------------------------------

### Setup for Manual Mixed Precision with PyTorch

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/mixed_precision/mmp.html

Initializes a PyTorch model and sets up the QuantizationSimModel and MixedPrecisionConfigurator for manual mixed precision operations. Ensure PyTorch is installed and the model is in evaluation mode.

```python
import torch
from torchvision.models import mobilenet_v2
from aimet_torch.common.quantsim_config.utils import get_path_for_per_channel_config
from aimet_torch import QuantizationSimModel
from aimet_torch.v2.mixed_precision import MixedPrecisionConfigurator

input_shape = (1, 3, 224, 224)
dummy_input = torch.randn(input_shape).cuda()

device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = mobilenet_v2(pretrained=True).eval().to(device)

# create the sim object. Feel free to change the default settings as you wish
quant_sim = QuantizationSimModel(model,
                                 dummy_input=dummy_input,
                                 default_param_bw=8,
                                 default_output_bw=8,
                                 config_file=get_path_for_per_channel_config())

# create the MMP configurator object
mp_configurator = MixedPrecisionConfigurator(quant_sim)

```

--------------------------------

### Build and Install AIMET Wheel

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/build_from_source.html

Builds the AIMET wheel package from the current directory and then installs it using pip.

```shell
# Build AIMET wheel
python3 -m build --wheel --no-isolation .

# Install the built wheel
pip install dist/aimet*.whl

```

--------------------------------

### Install Bokeh for Visualization

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/compression/visualization_compression.html

Installs the Bokeh library, a prerequisite for running AIMET visualization features.

```python
pip install bokeh
```

--------------------------------

### Start Bokeh Server for AIMET Visualization

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/compression/visualization_compression.html

Command to start a Bokeh server session, allowing for visualization of AIMET features. Specify allowed origins and the port to listen on.

```bash
bokeh serve --allow-websocket-origin=<host name>:<port number> --port=<port number>
```

--------------------------------

### Install AIMET PyTorch

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Install the AIMET PyTorch package using pip.

```bash
pip install aimet-torch
```

--------------------------------

### Install AIMET ONNX

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Install the AIMET ONNX package using pip. Optionally, install onnxruntime-gpu for CUDA acceleration.

```bash
pip install aimet-onnx
# Optional: To accelerate quantization with CUDA
pip install onnxruntime-gpu
```

--------------------------------

### Install Jupyter Metapackage

Source: https://quic.github.io/aimet-pages/releases/latest/tutorials/notebooks.html

Installs the Jupyter metapackage required to run the notebooks. Prepend with `sudo -H` if administrator privileges are needed.

```bash
python3 -m pip install jupyter

```

--------------------------------

### Install AIMET ONNX with CPU only from Wheel

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/index.html

Install the ONNX version of AIMET with CPU-only support using a specific wheel file from the GitHub releases.

```bash
python3 -m pip install https://github.com/quic/aimet/releases/download/2.31.0/aimet_onnx-2.31.0+cpu-cp310-abi3-manylinux_2_34_x86_64.whl
```

--------------------------------

### Verify AIMET ONNX Installation

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Verify the installation of AIMET ONNX by importing the library and printing its version.

```python
import aimet_onnx
print(aimet_onnx.__version__)
```

--------------------------------

### QuantizeDequantize Affine Quantizer Example

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/torch/quantization.html

An example demonstrating the creation and usage of the `QuantizeDequantize` affine quantizer for 8-bit asymmetric quantization. It shows how to initialize the quantizer, compute encodings from input data, and then use the quantizer to transform tensors.

```APIDOC
## Affine Quantizers: QuantizeDequantize

### Description

Demonstrates the instantiation and usage of an 8-bit asymmetric affine quantizer.

### Method

```python
import aimet_torch.quantization as Q
import torch

# Create an 8-bit asymmetric affine quantizer
qtzr = Q.affine.QuantizeDequantize(shape=(), bitwidth=8, symmetric=False)

# Initialize quantization parameters by computing encodings
print(f"Before compute_encodings:")
print(f"  * is_initialized: {qtzr.is_initialized()}")
print(f"  * scale: {qtzr.get_scale()}")
print(f"  * offset: {qtzr.get_offset()}")

input_tensor = torch.arange(256) / 256 # Example input tensor

with qtzr.compute_encodings():
    _ = qtzr(input_tensor)

print(f"After compute_encodings:")
print(f"  * is_initialized: {qtzr.is_initialized()}")
print(f"  * scale: {qtzr.get_scale()}")
print(f"  * offset: {qtzr.get_offset()}")

# Use the quantizer to transform the input tensor
input_qdq = qtzr(input_tensor)

print("Output (dequantized representation):")
print(input_qdq)
print(f"  * scale: {input_qdq.encoding.scale}")
print(f"  * offset: {input_qdq.encoding.offset}")
print(f"  * bitwidth: {input_qdq.encoding.bitwidth}")
print(f"  * signed: {input_qdq.encoding.signed}")

# Obtain the integer representation
input_q = input_qdq.quantize()
print("Output (quantized representation):")
print(input_q)
print(f"  * scale: {input_q.encoding.scale}")
print(f"  * offset: {input_q.encoding.offset}")
print(f"  * bitwidth: {input_q.encoding.bitwidth}")
print(f"  * signed: {input_q.encoding.signed}")
```
```

--------------------------------

### Verify AIMET PyTorch Installation

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Verify the installation of AIMET PyTorch by importing the library and printing its version.

```python
import aimet_torch
print(aimet_torch.__version__)
```

--------------------------------

### Install AIMET PyTorch with CPU only from Wheel

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/index.html

Install the PyTorch version of AIMET with CPU-only support using a specific wheel file from the GitHub releases.

```bash
python3 -m pip install https://github.com/quic/aimet/releases/download/2.31.0/aimet_torch-2.31.0+cpu-py310-none-any.whl
```

--------------------------------

### Example of Computing Encodings with a Callback

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_onnx/quantsim.html

This example demonstrates how to compute encodings for a QuantizationSimModel using a user-defined callback function that runs forward passes on a representative dataset. Ensure the dataset and callback are correctly defined before calling.

```python
>>> sim = QuantizationSimModel(...)
>>> def run_forward_pass(session: ort.InferenceSession):
...     for input in dataset:
...         _ = sess.run(None, {"input": input})
...
>>> sim.compute_encodings(run_forward_pass)
```

--------------------------------

### Install AIMET ONNX with CUDA 12.x from Wheel

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/index.html

Install the ONNX version of AIMET with CUDA 12.x support using a specific wheel file from the GitHub releases.

```bash
python3 -m pip install https://github.com/quic/aimet/releases/download/2.31.0/aimet_onnx-2.31.0+cu126-cp310-abi3-manylinux_2_34_x86_64.whl
```

--------------------------------

### Get Node Attribute Value

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/model_preparer.html

Retrieves the attribute value associated with a given FX graph node. This function is adapted from PyTorch FX interpreter examples.

```python
def get_node_attr(node: torch.fx.node):
    """
    Codes modified from https://pytorch.org/docs/stable/fx.html#the-interpreter-pattern

```

--------------------------------

### Step 1: Setup for PyTorch Model

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Loads a pre-trained MobileNetV2 model and prepares a dummy input tensor, selecting the appropriate device (CUDA or CPU).

```python
import torch
from torchvision.models import mobilenet_v2

device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = mobilenet_v2(weights='DEFAULT').eval().to(device)
dummy_input = torch.randn((10, 3, 224, 224), device=device)
```

--------------------------------

### Initialize and Use Quantize Module

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/quantization/affine/quantizer.html

Demonstrates how to initialize the `Quantize` module with specific parameters and then use it to quantize an input tensor after computing encodings.

```python
import aimet_torch.quantization as Q
import torch

input = torch.randn(5, 10)
q = Q.affine.Quantize(shape=(5, 1), bitwidth=8, symmetric=False, block_size=(1, 5))
q.is_initialized()
with q.compute_encodings():
    _ = q(input)

q.is_initialized()
q(input)
```

--------------------------------

### PyTorch Model and Data Setup

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/qat.html

Sets up a PyTorch model (MobileNetV2), data loader for ImageNet, and necessary callback functions for QAT. Includes dummy input generation and batch normalization folding.

```python
import itertools
import torch
import torchvision
from tqdm import tqdm
from aimet_torch.batch_norm_fold import fold_all_batch_norms

# General setup that can be changed as needed
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = torchvision.models.mobilenet_v2(pretrained=True).eval().to(device)

PATH_TO_IMAGENET = ...
data = torchvision.datasets.ImageNet(PATH_TO_IMAGENET, split="train")
data_loader = torch.utils.data.DataLoader(data, batch_size=64)

dummy_input = torch.randn(1, 3, 224, 224, device=device)
fold_all_batch_norms(model, dummy_input.shape)

@torch.no_grad()
def pass_calibration_data(model: torch.nn.Module):
    # Pass N batches of calibration data through the model
    for images, _ in itertools.islice(data_loader, 10):
        _ = model(images.to(device))

@torch.no_grad()
def evaluate(model, data_loader):
    # Basic ImageNet evaluation function
    correct = 0
    for data, labels in tqdm(data_loader):
        data, labels = data.to(device), labels.to(device)
        logits = model(data)
        correct += (logits.argmax(1) == labels).sum().item()
    return correct / len(data_loader.dataset)

```

--------------------------------

### Get Ordered List of Convolutional Modules

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_onnx/cross_layer_equalization.html

Retrieves an ordered list of convolutional modules from a given list of starting operations in an ONNX graph. This is useful for identifying sequential convolutional layers for cross-layer scaling.

```python
# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
# SPDX-License-Identifier: BSD-3-Clause


"""Cross Layer Equalization

Some terminology for this code.
CLS set: Set of layers (2 or 3) that can be used for cross-layer scaling
Layer groups: Groups of layers that are immediately connected and can be decomposed further into CLS sets
"""

from typing import List, Optional, Tuple, Union, Dict
import numpy as np
import onnx
from onnx import numpy_helper
from onnxruntime.quantization.onnx_quantizer import ONNXModel
from packaging import version

from aimet_onnx.common.utils import AimetLogger
from aimet_onnx.common.connected_graph.connectedgraph import get_ordered_ops
from aimet_onnx.common.cross_layer_equalization import (
    GraphSearchUtils,
    CrossLayerScaling as CLS,
    ClsImpl,
    ClsSetInfo,
    HbfImpl,
)

from aimet_onnx.meta.connectedgraph import ConnectedGraph
from aimet_onnx.meta.operations import Op
from aimet_onnx.utils import (
    ParamUtils,
    replace_relu6_with_relu,
    find_shared_param_names,
)
from aimet_onnx.batch_norm_fold import BNLayer, fold_all_batch_norms_to_weight

# pylint: disable=no-name-in-module, ungrouped-imports
if version.parse(onnx.__version__) >= version.parse("1.14.0"):
    from onnx import ModelProto
else:
    from onnx.onnx_pb import ModelProto

logger = AimetLogger.get_area_logger(AimetLogger.LogAreas.Quant)

ClsSet = Union[Tuple[Op, Op], Tuple[Op, Op, Op]]
ScaleFactor = Union[np.ndarray, Tuple[np.ndarray]]
cls_supported_layer_types = ["Conv", "ConvTranspose"]
cls_supported_activation_types = ["Relu", "PRelu"]


def get_ordered_list_of_conv_modules(list_of_starting_ops: List) -> List:
    """
    Finds order of nodes in graph
    :param list_of_starting_ops: list of starting ops for the model
    :return: List of names in graph in order
    """
    module_list = get_ordered_ops(list_of_starting_ops)
    module_list = [
        [module.dotted_name, module]
        for module in module_list
        if module.type in cls_supported_layer_types
    ]
    return module_list

```

--------------------------------

### ImageNet Data Pipeline Setup

Source: https://quic.github.io/aimet-pages/releases/latest/tutorials/notebooks/torch/compression/channel_pruning.html

This class provides static methods to manage the ImageNet dataset pipeline for evaluation and fine-tuning. It includes functions to get the validation dataloader, evaluate model accuracy, and fine-tune the model.

```python
import os
import torch
from typing import List
from Examples.common import image_net_config
from Examples.torch.utils.image_net_evaluator import ImageNetEvaluator
from Examples.torch.utils.image_net_trainer import ImageNetTrainer
from Examples.torch.utils.image_net_data_loader import ImageNetDataLoader

# Assuming DATASET_DIR is defined elsewhere
# For example: DATASET_DIR = "/path/to/imagenet"

class ImageNetDataPipeline:

    @staticmethod
    def get_val_dataloader() -> torch.utils.data.DataLoader:
        """
        Instantiates a validation dataloader for ImageNet dataset and returns it
        """
        data_loader = ImageNetDataLoader(DATASET_DIR,
                                         image_size=image_net_config.dataset['image_size'],
                                         batch_size=image_net_config.evaluation['batch_size'],
                                         is_training=False,
                                         num_workers=image_net_config.evaluation['num_workers']).data_loader
        return data_loader

    @staticmethod
    def evaluate(model: torch.nn.Module, iterations: int, use_cuda: bool) -> float:
        """
        Given a torch model, evaluates its Top-1 accuracy on the dataset
        :param model: the model to evaluate
        :param iterations: the number of batches to be used to evaluate the model. A value of 'None' means the model will be
                           evaluated on the entire dataset once.
        :param use_cuda: whether or not the GPU should be used.
        """
        evaluator = ImageNetEvaluator(DATASET_DIR, image_size=image_net_config.dataset['image_size'],
                                      batch_size=image_net_config.evaluation['batch_size'],
                                      num_workers=image_net_config.evaluation['num_workers'])

        return evaluator.evaluate(model, iterations=iterations, use_cuda=use_cuda)

    @staticmethod
    def finetune(model: torch.nn.Module, epochs: int, learning_rate: float, learning_rate_schedule: List, use_cuda: bool):
        """
        Given a torch model, finetunes the model to improve its accuracy
        :param model: the model to finetune
        :param epochs: The number of epochs used during the finetuning step.
        :param learning_rate: The learning rate used during the finetuning step.
        :param learning_rate_schedule: The learning rate schedule used during the finetuning step.
        :param use_cuda: whether or not the GPU should be used.
        """
        trainer = ImageNetTrainer(DATASET_DIR, image_size=image_net_config.dataset['image_size'],
                                  batch_size=image_net_config.train['batch_size'],
                                  num_workers=image_net_config.train['num_workers'])

        trainer.train(model, max_epochs=epochs, learning_rate=learning_rate,
                      learning_rate_schedule=learning_rate_schedule, use_cuda=use_cuda)

```

--------------------------------

### Initialize and Use Affine Quantizer

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/quantization/affine/quantizer.html

Demonstrates the initialization and usage of the `Quantize` class. It shows how to set quantization parameters (min, max) and then apply the quantizer to an input tensor.

```python
import aimet_torch.quantization as Q
import torch

input_tensor = torch.randn(5, 10)
q = Q.affine.Quantize(shape=(5, 1), bitwidth=8, symmetric=False, block_size=(1, 5))

print(f"Is initialized before setting params: {q.is_initialized()}")

q.min = torch.nn.Parameter(-torch.ones_like(q.min))
q.max = torch.nn.Parameter(torch.ones_like(q.max))

print(f"Is initialized after setting params: {q.is_initialized()}")

quantized_output = q(input_tensor)
print(quantized_output)
```

--------------------------------

### Get Block Start and End Names for AdaScale

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_onnx/experimental/adascale/adascale_optimizer.html

This static method retrieves the input and output tensor names for a specific block within the model's endpoint structure. It's used to isolate and process individual blocks during AdaScale optimization.

```python
    @staticmethod
    def get_block_start_end_name(
        blocks_end_points: List[Tuple], block_idx: int, input_list_names: List[str]
    ) -> Tuple[List[str], List[str]]:
        block_inputs = [blocks_end_points[block_idx][0].inputs[0].name]
        block_input_names = block_inputs + input_list_names

        block_output_names = [blocks_end_points[block_idx][1].inputs[0].name]

        return block_input_names, block_output_names
```

--------------------------------

### Setup PyTorch Model and Data Loader for AdaRound

Source: https://quic.github.io/aimet-pages/releases/latest/ptq_techniques/adaround.html

Sets up a PyTorch MobileNetV2 model and a streaming ImageNet dataloader for AdaRound. This includes defining the device, loading the model, and creating a dummy input for forward passes. Ensure the 'datasets' and 'evaluate' libraries are installed.

```python
import torch
from torchvision.models import mobilenet_v2
from torch.utils.data import DataLoader
from datasets import load_dataset
from evaluate import evaluator

# General setup that can be changed as needed
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = mobilenet_v2(pretrained=True).eval().to(device)
num_batches = 32
data = load_dataset('imagenet-1k', streaming=True, split="train")
data_loader = DataLoader(data, batch_size=num_batches, num_workers = 4)
dummy_input = torch.randn(1, 3, 224, 224).to(device)

def forward_pass(model: torch.nn.Module):
    with torch.no_grad():
        for images, _ in data_loader:
            model(images)

path = './'
filename = 'mobilenet'


```

--------------------------------

### Initialize and Print QuantizedLinear Module

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/nn/true_quant.html

Demonstrates how to instantiate a QuantizedLinear layer and print its configuration, showing the structure for weight, bias, and input/output quantizers.

```python
>>> qlinear = QuantizedLinear(in_features=10, out_features=10)
>>> print(qlinear)
QuantizedLinear(
  in_features=10, out_features=10, bias=True
  (param_quantizers): ModuleDict(
    (weight): None
    (bias): None
  )
  (input_quantizers): ModuleList(
    (0): None
  )
  (output_quantizers): ModuleList(
    (0): None
  )
)
```

--------------------------------

### QuantizationSimModel Example

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/quantsim/quantsim.html

Demonstrates how to initialize and use QuantizationSimModel with a PyTorch model and dummy input. Shows the transformation from a standard ResNet18 model to its quantized version.

```python
>>> model = torchvision.models.resnet18()
>>> dummy_input = torch.randn(1, 3, 224, 224)
>>> sim = QuantizationSimModel(model, dummy_input)
>>> print(model)
ResNet(
  (conv1): Conv2d(
    3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
  )
  ...
)
>>> print(sim.model)
ResNet(
  (conv1): QuantizedConv2d(
    3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
    (param_quantizers): ModuleDict(
      (weight): QuantizeDequantize(shape=(), qmin=-128, qmax=127, symmetric=True)
    )
    (input_quantizers): ModuleList(
      (0): QuantizeDequantize(shape=(), qmin=0, qmax=255, symmetric=False)
    )
    (output_quantizers): ModuleList(
      (0): None
    )
  )
  ...
)
```

--------------------------------

### Install uv

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/build_from_source.html

Installs the uv package manager on Linux/MacOS systems using a curl command.

```shell
curl -LsSf https://astral.sh/uv/install.sh | sh

```

--------------------------------

### Initialize Quantization Encodings

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/torch/quantsim.html

Demonstrates how to initialize quantization parameters for a QuantizationSimModel. This is necessary before running the model forward pass.

```python
>>> sim = QuantizationSimModel(...)
>>> _ = sim.model(input) # Can't run forward until quantizer encodings are initialized
RuntimeError: Failed to run QuantizeDequantize since quantization parameters are not initialized.
Please initialize the quantization parameters using `compute_encodings()`.
>>> def run_forward_pass(quantized_model: torch.nn.Module):
...     for input in train_dataloader:
...         with torch.no_grad():
...             _ = quantized_model(input) \
>>> sim.compute_encodings(run_forward_pass)
>>> _ = sim.model(input) # Now runs successfully!
```

--------------------------------

### Initialize QuantizeDequantize Manually

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/quantization/affine/quantizer.html

Shows how to manually initialize quantization parameters by assigning values to `qdq.min` and `qdq.max`. This is useful when the quantization range is known beforehand or needs to be set explicitly.

```python
import aimet_torch.quantization as Q
import torch

input = torch.randn(5, 10)
qdq = Q.affine.QuantizeDequantize(shape=(5, 2), bitwidth=8, symmetric=False, block_size=(1, 5))
print(qdq.is_initialized())
qdq.min = torch.nn.Parameter(-torch.ones_like(qdq.min))
qdq.max = torch.nn.Parameter(torch.ones_like(qdq.max))
print(qdq.is_initialized())
print(qdq(input))
```

--------------------------------

### Step 2: Create QuantizationSimModel for PyTorch

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Initializes a QuantizationSimModel for PyTorch, setting default parameter and output bit widths for quantization.

```python
from aimet_torch.common.defs import QuantScheme
from aimet_torch.common.quantsim_config.utils import get_path_for_per_channel_config
from aimet_torch import QuantizationSimModel

sim = QuantizationSimModel(model, 
                           dummy_input,
                           default_param_bw=8,
                           default_output_bw=16)
print(sim)
```

--------------------------------

### Execute Model on Target

Source: https://quic.github.io/aimet-pages/releases/latest/tutorials/on_target_inference.html

Use the `qnn-net-run` tool to execute the model (represented as serialized context binary) on the target device. Provide the backend library, context binary, input list, and output directory.

```bash
qnn-net-run --backend <libQnnHtp.so> --retrieve_context <binary_file_name>
            --input_list <input_list>.txt --output_dir <output_path>
```

--------------------------------

### Per-Tensor Encoding Example

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/encoding_spec.html

Example of a per-tensor quantization encoding. This is the simplest form, applying a single scale and zero-point to the entire tensor.

```json
{
    "name": "tensor_name",
    "y_scale": 0.01,
    "y_zero_point": 41,
    "output_dtype": "uint8"
}
```

--------------------------------

### MixedPrecisionAlgo Initialization

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/_base/amp/mixed_precision_algo.html

Initializes the MixedPrecisionAlgo with simulation model, dummy input, quantization candidates, evaluation callbacks, and configuration flags. It sets up internal structures for supported candidates and baseline options based on supported kernels and user preferences.

```python
def __init__(
    self,
    sim: QuantizedSimModel,
    dummy_input: Any,
    candidates: List[Tuple[int, QuantizationDataType]],
    eval_callback_for_phase1: Callable,
    eval_callback_for_phase2: Callable,
    results_dir: str,
    clean_start: bool,
    forward_pass_callback: Callable = None,
    use_all_amp_candidates: bool = False,
    phase2_reverse: bool = False,
    phase1_optimize: bool = True,
):
    """
    :param sim: Quantized sim model
    :param dummy_input: Dummy input to the model. If the model has more than one input, pass a tuple.
                        User is expected to place the tensors on the appropriate device.
    :param candidates: List of Tuple of all possible [bitwidth, QuantizationDataType] values to quantize to
    :param eval_callback_for_phase1: Callable object used to measure sensitivity of each
                                 quantizer group during phase 1. The phase 1 involves finding accuracy list/sensitivity of each
                                 module. Therefore, a user might want to run the phase 1 with a smaller dataset
    :param eval_callback_for_phase2: Callale object used to get accuracy of quantized model
                                 for phase 2 calculations. The phase 2 involves finding pareto front curve
    :param results_dir: Path to save results and cache intermediate results
    :param clean_start: If true, any cached information from previous runs will be deleted prior to starting the
                        mixed-precision analysis. If false, prior cached information will be used if applicable. Note
                        it is the user's responsibility to set this flag to true if anything in the model or
                        quantization parameters changes compared to the previous run.
    :param forward_pass_callback: Callable object used to compute quantization encodings
                              function parameters. Forward pass callback used to compute quantization encodings
    :param use_all_amp_candidates: Using the “supported_kernels” field in the config file (under defaults
                and op_type sections), a list of supported candidates can be specified. All the AMP candidates
                which are passed through the “candidates” field may not be supported based on the data passed
                through “supported_kernels”. When the field “use_all_amp_candidates” is set to True, the AMP algo
                will ignore the "supported_kernels" in the config file and will continue to use all the candidates.
    :param phase2_reverse: If user will set this parameter to True, then phase1 of amp algo, that is calculating accuracy list will not be changed,
                        whereas the phase2 algo of amp, which generate the pareto list will be changed. In phase2, algo will start, model with all quantizer groups in least candidate, and
                        one by one, it will put nodes in higher candidate till target accuracy does not meet.
    :phase1_optimize: If user set this parameter to True then phase1 optimized logic will be executed else common code will be executed
    """
    mac_dict = mixed_precision_utils.create_mac_dict(sim.model, dummy_input)
    self.phase1_optimize = phase1_optimize
    self.dummy_input = dummy_input

    super().__init__(
        sim,
        candidates,
        eval_callback_for_phase1,
        eval_callback_for_phase2,
        forward_pass_callback,
        mac_dict,
        results_dir,
        clean_start,
        phase2_reverse,
    )

    supported_kernels = reformat_supported_kernels(sim.get_supported_kernels())

    # Find 1. candidates for each of the quantizers by using supported_kernels, candidates and
    # use_all_amp_candidates (flag)
    # 2. max_candidate_options based on the candidates which are present in all the quantizers
    (
        self._supported_candidates_per_quantizer_group,
        self._baseline_candidate_options,
    ) = find_supported_candidates(
        self.quantizer_groups,
        candidates,
        supported_kernels,
        get_module_name_to_module_dict(sim),
        use_all_amp_candidates,
    )

```

--------------------------------

### PyTorch Mixed Precision Quantization Setup

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/mixed_precision/amp.html

Configures quantization parameters, defines candidate bitwidths, and sets up callbacks for mixed precision quantization.

```python
default_bitwidth = 16

# ((activation bitwidth, activation data type), (param bitwidth, param data type))
candidates = [((16, QuantizationDataType.int), (16, QuantizationDataType.int)),
             ((16, QuantizationDataType.int), (8, QuantizationDataType.int)),
             ((8, QuantizationDataType.int), (16, QuantizationDataType.int))]
# Allowed accuracy drop in absolute value
allowed_accuracy_drop = 0.5 # Implies 50% drop

eval_callback_for_phase_1 = CallbackFunc(evaluate, func_callback_args=data_loader)
eval_callback_for_phase_2 = CallbackFunc(evaluate, func_callback_args=data_loader)

calibration_batches = 10
forward_pass_call_back = CallbackFunc(forward_pass, func_callback_args=calibration_batches)

# Create quant sim
sim = QuantizationSimModel(model,
                           default_param_bw=default_bitwidth,
                           default_output_bw=default_bitwidth,
                           dummy_input=dummy_input)
```

--------------------------------

### Per-Block Encoding Example

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/encoding_spec.html

Example of a per-block quantization encoding. Features nested lists for scales and zero-points, along with 'axis' and 'block_size' fields.

```json
{
    "name": "tensor_name",
    "y_scale": [
        [0.01, 0.02],
        [0.03, 0.04],
        [0.05, 0.06]
    ],
    "y_zero_point": [
        [0, 0],
        [0, 0],
        [0, 0]
    ],
    "axis": 1,
    "block_size": 32,
    "output_dtype": "int4"
}
```

--------------------------------

### Per-Channel Encoding Example

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/encoding_spec.html

Example of a per-channel quantization encoding. Uses a list of scales and zero-points, with an 'axis' field specifying the channel dimension.

```json
{
    "name": "tensor_name",
    "y_scale": [0.01, 0.02, 0.03],
    "y_zero_point": [0, 0, 0],
    "axis": 0,
    "output_dtype": "int8"
}
```

--------------------------------

### Configure Per-Channel Weight Quantization for QuantizedLinear

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/torch/nn.html

Demonstrates how to configure a QuantizedLinear layer to perform per-channel weight quantization by assigning a specific quantizer to the 'weight' parameter.

```python
import aimet_torch
import aimet_torch.quantization as Q
qlinear = aimet_torch.nn.QuantizedLinear(out_features=10, in_features=5)
# Per-channel weight quantization is performed over the `out_features` dimension, so encodings are shape (10, 1)
per_channel_quantizer = Q.affine.QuantizeDequantize(shape=(10, 1), bitwidth=8, symmetric=True)
qlinear.param_quantizers["weight"] = per_channel_quantizer
```

--------------------------------

### Initialize QuantAnalyzer

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/onnx/quant_analyzer.html

Instantiate the QuantAnalyzer with the FP32 model, dummy input, and necessary callbacks for calibration and evaluation.

```python
analyzer = QuantAnalyzer(model, dummy_input, forward_pass_callback, eval_callback)
```

--------------------------------

### Instantiating and Printing a QuantizedLinear Module

Source: https://quic.github.io/aimet-pages/releases/latest/apiref/torch/generated/aimet_torch.nn.QuantizationMixin.html

Demonstrates how to create an instance of QuantizedLinear and print its structure, showing the default None quantizers.

```python
>>> qlinear = QuantizedLinear(in_features=10, out_features=10)
>>> print(qlinear)
QuantizedLinear(
  in_features=10, out_features=10, bias=True
  (param_quantizers): ModuleDict(
    (weight): None
    (bias): None
  )
  (input_quantizers): ModuleList(
    (0): None
  )
  (output_quantizers): ModuleList(
    (0): None
  )
)

```

--------------------------------

### Install AIMET PyTorch with CUDA 12.x from Wheel

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/index.html

Install the PyTorch version of AIMET with CUDA 12.x support using a specific wheel file from the GitHub releases.

```bash
python3 -m pip install https://github.com/quic/aimet/releases/download/2.31.0/aimet_torch-2.31.0+cu126-py310-none-any.whl
```

--------------------------------

### PyTorch General Setup for BN Re-estimation

Source: https://quic.github.io/aimet-pages/releases/latest/ptq_techniques/bn.html

Sets up the PyTorch model, dataloader, and device for BN re-estimation. Ensure your model is in evaluation mode and moved to the appropriate device.

```python
import torch
from torchvision.models import mobilenet_v2
from torch.utils.data import DataLoader
from datasets import load_dataset

# General setup that can be changed as needed
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = mobilenet_v2(pretrained=True).eval().to(device)
num_batches = 32
data = load_dataset('imagenet-1k', streaming=True, split="train")
data_loader = DataLoader(data, batch_size=num_batches, num_workers = 4)
dummy_input = torch.randn(1, 3, 224, 224).to(device)


```

--------------------------------

### Step 2: Create QuantizationSimModel for ONNX

Source: https://quic.github.io/aimet-pages/releases/latest/overview/install/quick-start.html

Initializes a QuantizationSimModel for ONNX, specifying parameter and activation bit widths for quantization.

```python
from aimet_onnx.quantsim import QuantizationSimModel
from aimet_onnx import int8, int16
from aimet_onnx.utils import make_dummy_input

sim = QuantizationSimModel(onnx_model,
                           param_type=int8,
                           activation_type=int16)
```

--------------------------------

### ONNX Calibration Callback Example

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/analysis_tools/quant_analyzer.html

Example of a user-defined model calibration function for ONNX. This callback is used by AIMET to compute encoding (delta/offset) using representative data.

```python
# NOTE: In the actual use cases, the users should implement this part to serve
#       their own goals if necessary.
def forward_pass_callback(session: InferenceSession, _: Any = None) -> None:
    """
    NOTE: This is intended to be the user-defined model calibration function.
    AIMET requires the above signature. So if the user's calibration function does not
    match this signature, please create a simple wrapper around this callback function.

    A callback function for model calibration that simply runs forward passes on the model to
    compute encoding (delta/offset). This callback function should use representative data and should
    be subset of entire train/validation dataset (~1000 images/samples).

    :param session: OnnxRuntime Inference Session.
    :param _: Argument(s) of this callback function. Up to the user to determine the type of this parameter.
    E.g. could be simply an integer representing the number of data samples to use. Or could be a tuple of
    parameters or an object representing something more complex.
    """
    # User action required
    # User should create data loader/iterable using representative dataset and simply run
    # forward passes on the model.


```

--------------------------------

### PyTorch Model Setup and Initial State

Source: https://quic.github.io/aimet-pages/releases/latest/ptq_techniques/cle.html

Sets up a PyTorch MobileNetV2 model and prints its initial state before cross-layer equalization. This includes inspecting specific convolutional layers and their weights.

```python
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model = mobilenet_v2(pretrained=True).eval().to(device)
print(model)
input_shape = (1, 3, 224, 224)

print('*** Before cross-layer equalization ***')

print('\nmodel.features[1].conv[0][0]')
print(model.features[1].conv[0][0])

print('\nmodel.features[1].conv[1]')
print(model.features[1].conv[1])

print('\nmodel.features[1].conv[0][0]')
print(model.features[1].conv[0][0].weight)

print('\nmodel.features[1].conv[1]')
print(model.features[1].conv[1].weight)
```

--------------------------------

### PyTorch Calibration Callback Example

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/analysis_tools/quant_analyzer.html

Example of a user-defined model calibration function for PyTorch. This callback is used by AIMET to compute encoding (delta/offset) using representative data.

```python
# NOTE: In the actual use cases, the users should implement this part to serve
#       their own goals if necessary.
def forward_pass_callback(model: torch.nn.Module) -> None:
    """
    NOTE: This is intended to be the user-defined model calibration function.
    AIMET requires the above signature. So if the user's calibration function does not
    match this signature, please create a simple wrapper around this callback function.

    A callback function for model calibration that simply runs forward passes on the model to
    compute encoding (delta/offset). This callback function should use representative data and should
    be subset of entire train/validation dataset (~1000 images/samples).

    :param model: PyTorch model.
    """
    # User action required
    # User should create data loader/iterable using representative dataset and simply run
    # forward passes on the model.


```

--------------------------------

### Create PyTorch QuantizationSimModel

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/ptq.html

Initializes a QuantizationSimModel for PyTorch. Requires model, dummy input, quantization scheme, and configuration file.

```python
from aimet_torch.common.defs import QuantScheme
from aimet_torch.common.quantsim_config.utils import get_path_for_per_channel_config
from aimet_torch import QuantizationSimModel

input_shape = (1, 3, 224, 224)
dummy_input = torch.randn(input_shape).to(device)
sim = QuantizationSimModel(model,
                           dummy_input=dummy_input,
                           quant_scheme=QuantScheme.training_range_learning_with_tf_init,
                           default_param_bw=8,
                           default_output_bw=16,
                           config_file=get_path_for_per_channel_config())

```

--------------------------------

### Create and Navigate to Workspace

Source: https://quic.github.io/aimet-pages/releases/latest/tutorials/notebooks.html

Creates the workspace directory and changes the current directory to it.

```bash
mkdir $WORKSPACE && cd $WORKSPACE

```

--------------------------------

### Start Jupyter Notebook Server

Source: https://quic.github.io/aimet-pages/releases/latest/tutorials/notebooks.html

Starts the Jupyter notebook server, making it accessible from your browser. The `--ip=*` flag allows remote access, and `--no-browser` prevents it from opening automatically.

```bash
jupyter notebook --ip=* --no-browser &

```

--------------------------------

### AffineQuantizerBase Initialization

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/quantization/affine/quantizer.html

Initializes the AffineQuantizerBase class. Supports initialization with either explicit qmin/qmax and symmetric flag, or with bitwidth and symmetric flag to derive qmin/qmax. Optional parameters include encoding_analyzer, block_size, and zero_point_shift.

```python
# Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries.
# SPDX-License-Identifier: BSD-3-Clause


# pylint: disable=redefined-builtin, too-many-lines
"""Affine quantizers"""

from itertools import chain, repeat
from typing import (
    Any,
    Dict,
    List,
    Optional,
    overload,
    Protocol,
    runtime_checkable,
    Tuple,
)
from contextlib import contextmanager, nullcontext
import functools

import torch
from torch import nn

from aimet_torch.common.utils import docstring
from aimet_torch.utils import (
    patch_attr,
    _is_expandable,
    StatisticsNotFoundError,
    _torch_compiler_is_exporting,
    _is_qtensor_casting_enabled,
)
from aimet_torch.quantization.encoding_analyzer import (
    EncodingAnalyzer,
    MinMaxEncodingAnalyzer,
    _flag_extreme_min_max,
)
from aimet_torch.quantization.affine import AffineEncoding, GroupedBlockEncoding
from aimet_torch.quantization.tensor import QuantizedTensor, DequantizedTensor
from aimet_torch.quantization.base import QuantizerBase
from aimet_torch.quantization.affine.backends import (
    quantize,
    quantize_dequantize,
    dequantize,
    _derive_qmin_qmax,
)
from aimet_torch.utils import ste_round
from aimet_torch.deepspeed_utils import SafeGatheredParameters
from aimet_torch.common.quantsim import _get_minimum_scale
from ._utils import _GridMixin, _register_signature
from aimet_torch.quantization._utils import (
    interleave,
    concretize_block_size,
)


__all__ = [
    "AffineQuantizerBase",
    "Dequantize",
    "GroupedBlockQuantizeDequantize",
    "MinMaxQuantizer",
    "Quantize",
    "QuantizeDequantize",
    "ScaleOffsetQuantizer",
]


class AffineQuantizerBase(QuantizerBase, _GridMixin):  # pylint: disable=too-many-instance-attributes
    """
    Base class for linear quantization modules.

    Args:
        shape (tuple): Shape of the quantization parameters
        bitwidth (int): Quantization bitwidth
        symmetric (bool): If True, performs symmetric quantization;
                          otherwise, performs asymmetric quantization
        encoding_analyzer (EncodingAnalyzer, optional): Encoding analyzer for calibrating quantization encodings
                                                        (default: min-max encoding analyzer)

    """

    _init_signatures = []

    @overload
    @_register_signature(_init_signatures)
    def __init__(
        self,
        shape,
        qmin: int,
        qmax: int,
        symmetric: bool,
        encoding_analyzer: EncodingAnalyzer = None,
        block_size: Optional[Tuple[int, ...]] = None,
        zero_point_shift: Optional[float] = None,
    ): ...

    @overload
    @_register_signature(_init_signatures)
    def __init__(
        self,
        shape,
        bitwidth: int,
        symmetric: bool,
        encoding_analyzer: EncodingAnalyzer = None,
        block_size: Optional[Tuple[int, ...]] = None,
        zero_point_shift: Optional[float] = None,
    ): ...

    def __init__(self, shape, *args, **kwargs):
        super().__init__()
        if isinstance(shape, int):
            shape = (shape,)
        self.shape = tuple(shape)
        full_args = (shape, *args)

        # Pad positional args with None's such that len(args) == 6
        args = tuple(chain(args, repeat(None, 6 - len(args))))
        arg0 = kwargs.pop("qmin", kwargs.pop("bitwidth", args[0]))
        arg1 = kwargs.pop("qmax", args[1])

        if arg1 is not None and not isinstance(arg1, bool):
            # (arg0, arg1, arg2) == (qmin, qmax, symmetric)
            qmin, qmax = arg0, arg1
            symmetric = kwargs.pop("symmetric", args[2])

            if (qmin is None) or (qmax is None) or (symmetric is None):
                raise self._arg_parsing_error(full_args, kwargs)

            encoding_analyzer = kwargs.pop("encoding_analyzer", args[3])
            block_size = kwargs.pop("block_size", args[4])
            zero_point_shift = kwargs.pop("zero_point_shift", args[5])
        else:
            # (arg0, arg1) == (bitwidth, symmetric)
            bitwidth = arg0
            symmetric = kwargs.pop("symmetric", args[1])

            if (bitwidth is None) or (symmetric is None):
                raise self._arg_parsing_error(full_args, kwargs)

            # We support two quantization modes: (unsigned) asymmetric and signed-symmetric
            qmin, qmax = _derive_qmin_qmax(bitwidth=bitwidth, signed=symmetric)
            encoding_analyzer = kwargs.pop("encoding_analyzer", args[2])
            block_size = kwargs.pop("block_size", args[3])
            zero_point_shift = kwargs.pop("zero_point_shift", args[4])

        assert qmin is not None
        assert qmax is not None

```

--------------------------------

### Run Mixed Precision Algorithm (Clean Start)

Source: https://quic.github.io/aimet-pages/releases/latest/techniques/mixed_precision/amp.html

Executes the mixed precision algorithm with a clean start, generating new accuracy and Pareto lists. Exports the model based on allowed accuracy drop.

```python
# Call the mixed precision algo with clean start = True i.e. new accuracy list and pareto list will be generated
# If set to False then pareto front list and accuracy list will be loaded from the provided directory path
# A allowed_accuracy_drop can be specified to export the final model with reference to the pareto list
pareto_front_list = choose_mixed_precision(sim, dummy_input, candidates, eval_callback_for_phase_1,
                                           eval_callback_for_phase_2, allowed_accuracy_drop, results_dir='./data',
                                           clean_start=True, forward_pass_callback=forward_pass_call_back)
print(pareto_front_list)
```

--------------------------------

### Initialize Quant Stats Visualization

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/visualization_tools/quant_stats_visualization.html

Initializes the visualization tool with statistics dictionary and a list of percentiles. Sets up plot and boxplot figures with labels and tools.

```python
def __init__(self, stats_dict: dict, percentile_list: list):
        self.stats_dict = stats_dict
        self.plot = figure(
            title="Min Max Activations/Weights of quantized modules for given model",
            x_axis_label="Layer index",
            y_axis_label="Activation/Weight",
            tools="pan,wheel_zoom,box_zoom",
        )
        self.boxplot = figure(
            x_range=FactorRange(),
            title="Boxplots of selected layers",
            x_axis_label="Layer index",
            y_axis_label="Activation/Weight",
            tools="pan,wheel_zoom,box_zoom",
        )
        self.default_values = {}
        self.percentiles = []
        for percentile in percentile_list:
            if percentile not in [25, 50, 75]:
                self.percentiles.append(percentile)
```

--------------------------------

### Get Original Module

Source: https://quic.github.io/aimet-pages/releases/latest/_modules/aimet_torch/seq_mse.html

Returns the original, unquantized module.

```python
    @classmethod
    def _get_original_module(cls, quant_module: BaseQuantizationMixin):
        return quant_module
```