### Install the TinyNeuralNetwork framework

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/README.md

Instructions for installing the TinyNeuralNetwork framework using git clone and setup.py.

```shell
# Install the TinyNeuralNetwork framework
git clone https://github.com/alibaba/TinyNeuralNetwork.git
cd TinyNeuralNetwork
python setup.py install

# Alternatively, you may try the one-liner
pip install git+https://github.com/alibaba/TinyNeuralNetwork.git
```

--------------------------------

### Prepare Example Input Image

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/model_conversion/basic.ipynb

Downloads an example image and preprocesses it to be used as input for the TFLite model.

```python
import os
from torch.hub import download_url_to_file
from PIL import Image
from torchvision import transforms
import numpy as np

cwd = os.path.abspath(os.getcwd())
img_path = os.path.join(cwd, 'dog.jpg')

img_urls = [
    'https://github.com/pytorch/hub/raw/master/images/dog.jpg',
    'https://raw.fastgit.org/pytorch/hub/master/images/dog.jpg',
]

# If you have diffculties accessing Github, then you may try out the second link
download_url_to_file(img_urls[0], img_path)

img = Image.open(img_path)

mean = np.array([0.485, 0.456, 0.406], dtype='float32')
std = np.array([0.229, 0.224, 0.225], dtype='float32')

preprocess = transforms.Compose(
    [
        transforms.Resize(256),
        transforms.CenterCrop(224),
    ]
)

# Image preprocessing
processed_img = preprocess(img)
arr = np.asarray(processed_img).astype('float32') / 255
normalized = (arr - mean) / std
input_arr = np.expand_dims(normalized, 0)
```

--------------------------------

### dynamic.py

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/converter/README.md

Example script for dynamic quantization.

```python
# Example script for dynamic quantization.
```

--------------------------------

### Run TFLite Model with Example Input

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/model_conversion/basic.ipynb

Loads the converted TFLite model and runs inference with the prepared example input.

```python
import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path=model_path)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]['index'], input_arr)
interpreter.invoke()
output_arr = interpreter.get_tensor(output_details[0]['index'])

print('TFLite out:', np.argmax(output_arr))
```

--------------------------------

### Post-Quantization Algorithm Example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example demonstrating how to use the KL divergence-based post-quantization algorithm in TinyNeuralNetwork.

```python
with model_tracer():
    model = Mobilenet()
    model.load_state_dict(torch.load(DEFAULT_STATE_DICT))
    dummy_input = torch.rand((1, 3, 224, 224))

    # Set the algorithm options you need, the default option is l2.
    quantizer = PostQuantizer(model, dummy_input, work_dir='out', config={'algorithm':'kl'})
    ptq_model = quantizer.quantize()
```

--------------------------------

### convert.py

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/converter/README.md

Example script for performing conversion directly from models.

```python
# Example script for performing conversion directly from models.
```

--------------------------------

### convert_from_json.py

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/converter/README.md

Example script for performing conversion from JSON configuration files.

```python
# Example script for performing conversion from JSON configuration files.
```

--------------------------------

### Pruning configuration example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example configuration for setting pruning sparsity.

```yaml
# example config
sparsity: 0.25
metrics: l2_norm # The available metrics are listed in `tinynn/graph/modifier.py`
```

--------------------------------

### Install and use ruff and black for code formatting

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/CONTRIBUTING.md

Commands to install and run ruff and black for code formatting.

```bash
pip install black flake8

python3 -m black .
python3 -m ruff .
```

--------------------------------

### OneShotChannelPruner Example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/pruning/basic.ipynb

This code snippet demonstrates the basic usage of the OneShotChannelPruner to prune a MobileNetV2 model.

```python
import sys
sys.path.append('../..')

import torch
import torchvision

from tinynn.prune.oneshot_pruner import OneShotChannelPruner

model = torchvision.models.mobilenet_v2(pretrained=True)
model.train()

dummy_input = torch.randn(1, 3, 224, 224)

pruner = OneShotChannelPruner(model, dummy_input, config={'sparsity': 0.25, 'metrics': 'l2_norm'})

st_flops = pruner.calc_flops()
pruner.prune()

ed_flops = pruner.calc_flops()
print(f"Pruning over, reduced FLOPS {100 * (st_flops - ed_flops) / st_flops:.2f}%  ({st_flops} -> {ed_flops})")

```

--------------------------------

### Install Additional Dependencies

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/quantization/specific/vit/readme.md

Installs the transformers library required for the ViT model.

```shell
pip install transformers==4.26.0
```

--------------------------------

### PyTorch Quantization Process Example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/quantization/README.md

Illustrates the manual steps required for quantization in PyTorch, including modifying forward functions, replacing operations, fusing modules, and preparing/converting the model.

```python
x = a + b
x = self.add0.add(a, b)
```

--------------------------------

### Image Normalization Example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example of image preprocessing using torchvision transforms, including Resize, ToTensor, and Normalize.

```python
transforms = transforms.Compose(
    [
        Resize(img_size),
        ToTensor(),
        Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
    ]
)
```

--------------------------------

### Function annotation and docstring example

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/CONTRIBUTING.md

Sample Python code demonstrating function annotations using typing and Google-style docstrings.

```python
def test_func(param1: int, param2: str) -> bool:
    """Documentation for the function

    Args:
        param1 (int): The first parameter.
        param2 (str): The second parameter.

    Returns:
        bool: The return value. True for success, False otherwise.

    """
```

--------------------------------

### Custom Qconfig with MinMaxObserver

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example of how to set different quantization observers for different layers using override_qconfig_func.

```python
import torch
from torch.quantization import FakeQuantize, MinMaxObserver
form torch.ao.nn.intrinsic import ConvBnReLU2d
def set_MinMaxObserver(name, module):
   # Set the corresponding weight and activation observers to MinMaxObserver based on model_name and module_type.
   if name in ['model_0_0', 'model_0_1'] or isinstance(module, ConvBnReLU2d):
        weight_fq = FakeQuantize.with_args(
            observer=MinMaxObserver,
            quant_min=-128,
            quant_max=127,
            dtype=torch.qint8,
            qscheme=torch.per_tensor_symmetric,
            reduce_range=False,
        )
        act_fq = FakeQuantize.with_args(
            observer=MinMaxObserver,
            quant_min=0,
            quant_max=255,
            dtype=torch.quint8,
            reduce_range=False,
        )
        qconfig_new = torch.quantization.QConfig(act_fq, weight_fq)
        return qconfig_new
```

```python
with model_tracer():
    quantizer = QATQuantizer(model, dummy_input, work_dir='out', config={'override_qconfig_func': set_MinMaxObserver})
    qat_model = quantizer.quantize()
```

--------------------------------

### Using pre-commit for automatic code checks

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/CONTRIBUTING.md

Instructions for installing and using pre-commit to automatically check code before committing.

```python
# Install pre-commit and pre-configured git hooks
pip install pre-commit
pre-commit install

# The code check will be done automatically before committing the code
git commit -m "test"

# Or you can also trigger the code check manually
pre-commit run
```

--------------------------------

### Applying Quantization with QATQuantizer

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

This code snippet shows how to initialize and use the QATQuantizer to quantize a PyTorch model, followed by converting the quantized model and printing its structure.

```python
import sys
sys.path.append('../..')

from tinynn.graph.quantization.quantizer import QATQuantizer

quantizer = QATQuantizer(model, dummy_input, work_dir='out')
q_model = quantizer.quantize()

for _ in range(10):
    dummy_input = torch.randn(1, 3, 224, 224)
    q_model(dummy_input)

q_model = torch.quantization.convert(q_model)

print(q_model)
```

--------------------------------

### Training loop with dummy data

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

This code snippet demonstrates a simple training loop where the model is fed with randomly generated data to observe the effect of FakeQuantize nodes.

```python
for _ in range(10):
    dummy_input = torch.randn(1, 3, 224, 224)
    m(dummy_input)

print(m)
```

--------------------------------

### Model after training with FakeQuantize nodes

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

The output shows the model's structure after the training loop, with updated statistics in the FakeQuantize nodes.

```python
Model(
  (conv): ConvBnReLU2d(
    3, 3, kernel_size=(1, 1), stride=(1, 1)
    (bn): BatchNorm2d(3, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (weight_fake_quant): FakeQuantize(
      fake_quant_enabled=tensor([1], dtype=torch.uint8), observer_enabled=tensor([1], dtype=torch.uint8),            quant_min=-128, quant_max=127, dtype=torch.qint8, qscheme=torch.per_tensor_symmetric, ch_axis=-1,         scale=tensor([0.0045]), zero_point=tensor([0])
      (activation_post_process): MovingAverageMinMaxObserver(min_val=-0.5689650177955627, max_val=0.4857633411884308)
    )
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=tensor([1], dtype=torch.uint8), observer_enabled=tensor([1], dtype=torch.uint8),            quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine, ch_axis=-1,         scale=tensor([0.0171]), zero_point=tensor([0])
      (activation_post_process): MovingAverageMinMaxObserver(min_val=0.0, max_val=4.370081901550293)
    )
  )
  (bn): Identity()
  (relu): Identity()
  (fake_quant): QuantStub(
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=tensor([1], dtype=torch.uint8), observer_enabled=tensor([1], dtype=torch.uint8),            quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine, ch_axis=-1,         scale=tensor([0.0340]), zero_point=tensor([122])
      (activation_post_process): MovingAverageMinMaxObserver(min_val=-4.156678199768066, max_val=4.5101141929626465)
    )
  )
  (fake_dequant): DeQuantStub()
  (float_functional): FloatFunctional(
    (activation_post_process): FakeQuantize(
      fake_quant_enabled=tensor([1], dtype=torch.uint8), observer_enabled=tensor([1], dtype=torch.uint8),            quant_min=0, quant_max=255, dtype=torch.quint8, qscheme=torch.per_tensor_affine, ch_axis=-1,         scale=tensor([0.0460]), zero_point=tensor([87])
      (activation_post_process): MovingAverageMinMaxObserver(min_val=-4.007132530212402, max_val=7.722269058227539)
    )
  )
)

```

--------------------------------

### Load Pretrained MobileNetV2 Model

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/model_conversion/basic.ipynb

Loads a pretrained MobileNetV2 model from torchvision.

```python
import torch
import torchvision

model = torchvision.models.mobilenet_v2(pretrained=True)
```

--------------------------------

### Tracer Example Script

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/examples/tracer/README.md

This script demonstrates the basic usage of the tracer functionality in TinyNeuralNetwork.

```python
import torch
import torch.nn as nn
from tinygrad.tensor import Tensor
from tinyneuralnetwork.tracer import trace

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 20)
        self.layer2 = nn.Linear(20, 5)

    def forward(self, x):
        x = self.layer1(x)
        x = torch.relu(x)
        x = self.layer2(x)
        return x

model = Model()
input_tensor = torch.randn(1, 10)

# Trace the model
with trace(model) as tracer:
    output = model(input_tensor)

# Get the generated code
code = tracer.code()
print(code)

# You can also get the connections between nodes
connections = tracer.connections()
print(connections)
```

--------------------------------

### Quantization Preparation

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

Prepares the model for quantization by adding FakeQuantize nodes to requantizable operations.

```python
torch.quantization.prepare_qat(m, inplace=True)
print(m)
```

--------------------------------

### Quantization with the whole graph

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example code for performing quantization on the entire graph.

```python
# Quantization with the whole graph
with model_tracer():
    quantizer = QATQuantizer(model, dummy_input, work_dir='out')
    qat_model = quantizer.quantize()
```

--------------------------------

### Generated pruning configuration

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example of a generated configuration file with sparsity for each operator.

```yaml
# new yaml generated
sparsity:
  default: 0.25
  model_0_0: 0.25
  model_1_3: 0.25
  model_2_3: 0.25
  model_3_3: 0.25
  model_4_3: 0.25
  model_5_3: 0.25
  model_6_3: 0.25
  model_7_3: 0.25
  model_8_3: 0.25
  model_9_3: 0.25
  model_10_3: 0.25
  model_11_3: 0.25
  model_12_3: 0.25
  model_13_3: 0.25
metrics: l2_norm # Other supported values: random, l1_norm, l2_norm, fpgm
```

--------------------------------

### Reloading the model with modification for quantization

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example code for reloading a model with modifications for quantization.

```python
# Reload the model with modification
with model_tracer():
    quantizer = QATQuantizer(model, dummy_input, work_dir='out', config={'force_overwrite': False})
    qat_model = quantizer.quantize()
```

--------------------------------

### Explicit conversion to a quantized model

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

This code snippet shows how to explicitly convert the model with FakeQuantize nodes into an actual quantized model using `torch.quantization.convert`.

```python
quantized_m = torch.quantization.convert(m)

print(quantized_m)
```

--------------------------------

### Handling Inconsistent Training and Inference Graphs

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example of a model with conditional logic during training that is not present during inference.

```python
class FloatModel(nn.Module):
    def __init__(self):
        self.conv = nn.Conv2d()
        self.conv1 = nn.Conv2d()

    def forward(self, x):
        x = self.conv(x)
        if self.training:
            x = self.conv1(x)
        return x
```

--------------------------------

### Output of the converted quantized model

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/quantization/basic.ipynb

The output displays the structure of the model after conversion to a quantized format, replacing FakeQuantize nodes with actual quantization operations.

```python
Model(
  (conv): QuantizedConvReLU2d(3, 3, kernel_size=(1, 1), stride=(1, 1), scale=0.017137575894594193, zero_point=0)
  (bn): Identity()
  (relu): Identity()
  (fake_quant): Quantize(scale=tensor([0.0340]), zero_point=tensor([122]), dtype=torch.quint8)
  (fake_dequant): DeQuantize()
  (float_functional): QFunctional(
    scale=0.045997653156518936, zero_point=87
    (activation_post_process): Identity()
  )
)

```

--------------------------------

### Mixed quantization by operator type

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example code for specifying mixed quantization according to operator types.

```python
# For a model containing LSTM op, perform mixed quantization while retaining the quantization parameters of its inputs, facilitating subsequent quantization directly in the converter.
with model_tracer():
    quantizer = QATQuantizer(model, dummy_input, work_dir='out', config={ 'quantize_op_action': {nn.LSTM: 'rewrite'} })
    qat_model = quantizer.quantize()
```

--------------------------------

### Build with docker

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/README.md

Instructions for building the TinyNeuralNetwork framework with Docker.

```shell
sudo docker build -t tinynn:pytorch1.9.0-cuda11.1 .
```

--------------------------------

### Exporting Model for Debugging

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/docs/FAQ.md

Example demonstrating how to use export_converter_files to export model and configuration files for debugging purposes.

```python
from tinynn.util.converter_util import export_converter_files

model = Model()
model.cpu()
model.eval()

dummy_input = torch.randn(1, 3, 224, 224)
export_dir = 'out'
export_name = 'test_model'

export_converter_files(model, dummy_input, export_dir, export_name)
```

--------------------------------

### Compare TFLite Output with PyTorch Output

Source: https://github.com/alibaba/tinyneuralnetwork/blob/main/tutorials/model_conversion/basic.ipynb

Compares the output of the TFLite model with the output of the original PyTorch model to verify consistency.

```python
torch_input = torch.from_numpy(input_arr).permute((0, 3, 1, 2))

with torch.no_grad():
    torch_output = model(torch_input)

print('PyTorch out:', torch.argmax(torch_output))
```