### Install Dependencies and Setup Environment

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/colabs/Gemma3_270M_convertion.ipynb

Installs the necessary LiteRT Torch libraries and uninstalls TensorFlow. Use `force-reinstall` to ensure the correct version is installed.

```python
# @title Install dependencies and environment setup

!pip install ai-edge-litert-nightly==2.0.2.dev20250917
!pip uninstall -y tensorflow
!pip install litert-torch-nightly==0.7.0.dev20250920  --force-reinstall
```

--------------------------------

### Install LiteRT Torch (Stable)

Source: https://github.com/google-ai-edge/litert-torch/blob/main/README.md

Installs the latest stable release of litert-torch, including torchvision and ai-edge-litert for quickstart and CLI benchmarking.

```bash
pip install litert-torch torchvision ai-edge-litert
```

--------------------------------

### Install Libraries

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Installs required libraries including litert-torch, transformers, pillow, requests, matplotlib, and mediapipe.

```python
!pip install -r https://raw.githubusercontent.com/google-ai-edge/litert-torch/main/requirements.txt
!pip install --pre litert-torch-nightly
!pip install transformers pillow requests matplotlib mediapipe
```

--------------------------------

### Install Dependencies

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Installs the required libraries for the project, including litert-torch nightly build, transformers, and image processing tools.

```python
!pip install -r https://raw.githubusercontent.com/google-ai-edge/litert-torch/main/requirements.txt
!pip install --pre litert-torch-nightly
!pip install transformers pillow requests matplotlib
```

--------------------------------

### Install LiteRT Torch and Dependencies

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/getting_started.ipynb

Install the necessary packages for LiteRT Torch, including the nightly build and dependencies from a requirements file.

```python
!pip install -r https://raw.githubusercontent.com/google-ai-edge/litert-torch/main/requirements.txt
!pip install --pre litert-torch-nightly
```

--------------------------------

### Set Up Development Environment

Source: https://github.com/google-ai-edge/litert-torch/blob/main/CONTRIBUTING.md

Installs project dependencies and the project itself in editable mode. Ensure you are in the project's root directory before running.

```bash
cd litert-torch
python -m venv venv
source venv/bin/activate

pip install -r dev-requirements.txt
pip install -e .
```

--------------------------------

### Install Dependencies

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Installs required packages including litert-torch, pillow, requests, matplotlib, and mediapipe.

```python
!pip install -r https://raw.githubusercontent.com/google-ai-edge/litert-torch/main/requirements.txt
!pip install --pre litert-torch-nightly
!pip install pillow requests matplotlib mediapipe
```

--------------------------------

### Install LiteRT Torch (Nightly)

Source: https://github.com/google-ai-edge/litert-torch/blob/main/README.md

Installs the nightly build of litert-torch, along with torchvision, ai-edge-litert, and litert-cli-nightly for the latest features and CLI tools.

```bash
pip install --pre litert-torch-nightly torchvision ai-edge-litert litert-cli-nightly
```

--------------------------------

### Example: Dynamic INT4 Block32 Quantization

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/gemma3/README.md

Example command for converting Gemma 3 with dynamic int4 block32 quantization. Ensure the checkpoint path and output path are correctly set.

```bash
python convert_gemma3_to_tflite.py --quantize="dynamic_int4_block32" \
 --checkpoint_path=/tmp/gemma-3-pytorch-gemma-3-1b-pt-v1 --output_path="/tmp/" \
 --prefill_seq_lens=2048 --kv_cache_max_len=4096 --mask_as_input=True
```

--------------------------------

### Install Dependencies for PyTorch to LiteRT Conversion

Source: https://github.com/google-ai-edge/litert-torch/blob/main/SKILL.md

Installs necessary libraries for direct PyTorch to LiteRT conversion. Ensure you are using a Python 3.11 environment.

```bash
python3.11 -m pip install torch transformers timm litert-torch ai-edge-litert litert-cli-nightly
```

--------------------------------

### Install Gemma PyTorch Dependencies

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/README.md

Installs necessary packages and sets up the PYTHONPATH for using Gemma models. This is a prerequisite for verifying Gemma models.

```bash
pip install -q -U immutabledict sentencepiece
git clone https://github.com/google/gemma_pytorch.git
export PYTHONPATH=$PWD/gemma_pytorch:$PYTHONPATH
```

--------------------------------

### Install Dependencies

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Installs litert-torch nightly, Pillow, requests, and matplotlib. Ensure you have the correct requirements.txt for litert-torch.

```python
!pip install -r https://raw.githubusercontent.com/google-ai-edge/litert-torch/main/requirements.txt
!pip install --pre litert-torch-nightly
!pip install pillow requests matplotlib
```

--------------------------------

### Install Model Explorer Package

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/README.md

Install the Model Explorer package using pip to enable model visualization capabilities.

```bash
pip install ai-edge-model-explorer
```

--------------------------------

### Run Text Generator Example

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/cpp/README.md

Example command to run the text generator for an exported Gemma model. Ensure TensorFlow Lite, SentencePiece, and necessary model files are available. The `--weight_cache_path` flag can optimize initialization time and reduce peak memory.

```bash
bazel run -c opt //litert_torch/generative/examples/c++:text_generator_main -- --tflite_model=PATH/gemma_it.tflite  --sentencepiece_model=PATH/tokenizer.model --start_token="<bos>" --stop_token="<eos>" --num_threads=16 --prompt="Write an email:" --weight_cache_path=PATH/gemma.xnnpack_cache
```

--------------------------------

### Install litert-torch with Torch XLA Backend

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Install the nightly version of litert-torch with Torch XLA dependencies. This is useful for compatibility issues with the default backend.

```bash
pip install --pre litert-torch-nightly[torch-xla]

# Enable torch-xla as the LiteRT Torch backend
export USE_TORCH_XLA=1
```

--------------------------------

### Instantiate and Run a Sample PyTorch Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/getting_started.ipynb

Instantiate a ResNet18 model from torchvision, prepare sample inputs, and run the model to get PyTorch output.

```python
resnet18 = torchvision.models.resnet18(torchvision.models.ResNet18_Weights.IMAGENET1K_V1).eval()
sample_inputs = (torch.randn(1, 3, 224, 224),)
torch_output = resnet18(*sample_inputs)
```

--------------------------------

### Install and Use Model Explorer

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Install the Model Explorer tool using pip and then use it to visualize your exported TFLite model by providing the model file path.

```bash
pip install ai-edge-model-explorer
model-explorer 'resnet.tflite'
```

--------------------------------

### Setup Hugging Face Hub Credentials

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/colabs/Gemma3_270M_convertion.ipynb

Sets up Hugging Face Hub credentials by retrieving a token from Google Colab's user data and storing it as an environment variable.

```python
# Setup Hugging Face Hub credentials

import os
from google.colab import userdata
os.environ["HF_TOKEN"] = userdata.get('HF_TOKEN')
```

--------------------------------

### TF Lite Delegate Modification Example

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/doc/system_overview.md

Illustrates how the TF Lite delegate system traverses a model to identify compatible operations for acceleration. The XNNPack delegate specifically matches and optimizes operations like SDPA.

```c++
TfLiteStatus ModifyGraphWithDelegate(
    TfLiteContext* context,
    TfLiteDelegate* delegate,
    TfLiteGraph* graph) {
  // ... delegate traversal and op matching ...
  return kTfLiteOk;
}
```

--------------------------------

### Visualize the TFLite Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/getting_started.ipynb

Install the AI Edge Model Explorer and use it to visualize the generated TFLite flatbuffer file.

```python
!pip install ai-edge-model-explorer

import model_explorer
model_explorer.visualize('resnet.tflite')
```

--------------------------------

### Create Gemma 3 Task File for LLM Inference API

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/gemma3/README.md

This Python script creates a `.task` file for Gemma 3, required for using the converted model and tokenizer with the LLM Inference API. Ensure Mediapipe is installed and paths to the TFLite model and tokenizer are correct.

```python
TFLITE_MODEL = <path to your converted TF Lite model>
TOKENIZER_MODEL = <path to the Gemma 3 tokenizer from the Gemma Pytorch repo>
START_TOKEN="<bos>"
STOP_TOKENS=["<eos>", "<end_of_turn>"]
from mediapipe.tasks.python.genai import bundler
config = bundler.BundleConfig(
    tflite_model=TFLITE_MODEL,
    tokenizer_model=TOKENIZER_MODEL,
    start_token=START_TOKEN,
    stop_tokens=STOP_TOKENS,
    output_filename="/tmp/gemma3.task",
    prompt_prefix="<start_of_turn>user\n",
    prompt_suffix="<end_of_turn>\n<start_of_turn>model\n",
)
bundler.create_bundle(config)
```

--------------------------------

### Run PT2E Quantized Model and Get Probabilities

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Execute the PT2E quantized model and display the classification probabilities. Compare these results with the other quantization methods.

```python
pt2e_drq_output = pt2e_drq_model(input_tensor)

get_classification_probablities(torch.tensor(pt2e_drq_output))
```

--------------------------------

### Run MobileViT Inference and Get Prediction

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Runs the PyTorch MobileViT model on the prepared inputs and prints the predicted class.

```python
outputs = pt_model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
print(f'Predicted class: {pt_model.config.id2label[predicted_class_idx]}')
```

--------------------------------

### Run Quantized TFLite Model and Get Probabilities

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Run the post-training quantized TFLite model and display the classification probabilities. Note the potential change in accuracy compared to the unquantized model.

```python
tfl_drq_output = tfl_drq_model(input_tensor)

get_classification_probablities(torch.tensor(tfl_drq_output))
```

--------------------------------

### Get Prediction and Post-process Output

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Obtains the prediction from the PyTorch model, upsamples the result to the original image size, performs min-max normalization, and scales it to [0, 255].

```python
# Get output with the most accurate prediction.
pt_result = pt_model(pt_image)[0][0]

# Recover the prediction spatial size to the orignal image size.
pt_result = F.upsample(pt_result, image.shape[:2],  mode='bilinear')
pt_result = torch.squeeze(pt_result, 0)

# Min-max normalization.
ma = torch.max(pt_result)
mi = torch.min(pt_result)
pt_result = (pt_result - mi) / (ma - mi)

# Scale [0, 1] -> [0, 255].
pt_result = pt_result * 255

# BCHW -> BHWC.
pt_result = pt_result.permute(1, 2, 0)

```

--------------------------------

### Set up Python Virtual Environment

Source: https://github.com/google-ai-edge/litert-torch/blob/main/README.md

Creates and activates a Python virtual environment using Python 3.11, recommended for compatibility.

```bash
python3.11 -m venv --prompt litert-torch venv
source venv/bin/activate
```

--------------------------------

### Run TFLite Model and Get Probabilities

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Execute the converted TFLite model with an input tensor and then process the output to get classification probabilities.

```python
edge_model_output = edge_model(input_tensor)

get_classification_probablities(torch.tensor(edge_model_output))
```

--------------------------------

### Run Project Tests

Source: https://github.com/google-ai-edge/litert-torch/blob/main/CONTRIBUTING.md

Executes the test suite for the project. Navigate to the project's root directory before running.

```bash
cd litert-torch
bash ./run_tests.sh
```

--------------------------------

### Prepare and Run a PyTorch Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

This snippet demonstrates how to load a torchvision model, set it to evaluation mode, and prepare sample inputs for conversion.

```python
import torch
import torchvision
resnet18 = torchvision.models.resnet18(torchvision.models.ResNet18_Weights.IMAGENET1K_V1).eval()
sample_inputs = (torch.randn(1, 3, 224, 224),)
```

--------------------------------

### Apply Quantization with a Pre-defined Recipe

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/quantize/README.md

Use a pre-defined quantization recipe to convert a model to a quantized `.tflite` format. This is suitable for average users looking for a well-supported quantization scheme.

```python
quant_config = quant_recipes.full_int8_dynamic_recipe()
edge_model = litert_torch.convert(
    model, (tokens, input_pos), quant_config=quant_config
)
```

--------------------------------

### Run Inference and Get Prediction

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Runs the wrapped PyTorch model on the processed input tensor and prints the predicted class label.

```python
wrapped_pt_model_outputs = wrapped_pt_model(input_tensor)
predicted_class_idx = wrapped_pt_model_outputs.argmax(-1).item()
print(f'Predicted class: {hf_pt_model.config.id2label[predicted_class_idx]}')
```

--------------------------------

### Display Top 5 Predictions

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Applies softmax to the logits to get probabilities and prints the top 5 predicted classes and their probabilities.

```python
import torch
from torch import nn

probs, indices = torch.nn.functional.softmax(logits, dim=-1).flatten().topk(k=5)
for i in range(len(indices)):
  class_label = pt_model.config.id2label[indices[i].item()]
  prob = probs[i].item()
  print(f'{(prob * 100):4.1f}%  {class_label}')
```

--------------------------------

### Download Checkpoint

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/colabs/Gemma3_270M_convertion.ipynb

Downloads the Gemma-3-270m-it model checkpoint from Hugging Face Hub. This step is necessary before conversion.

```python
# @title Download checkpoint

checkpoint_dir = snapshot_download('google/gemma-3-270m-it')
```

--------------------------------

### Prepare Image Inputs for MobileViT

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Prepares the input image by resizing it to 256x256 pixels, disabling center cropping, and converting it to PyTorch tensors.

```python
inputs = processor(
    images=image,
    return_tensors='pt',
    size={'height': 256, 'width': 256},
    do_center_crop=False
)
```

--------------------------------

### Update LD_LIBRARY_PATH for Torch XLA

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Update the LD_LIBRARY_PATH environment variable to include the lib directory of your Python installation. This ensures that Torch XLA can link to the correct Python version.

```bash
export LD_LIBRARY_PATH=<path to Python installation>/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/getting_started.ipynb

Import the required libraries for PyTorch, NumPy, and LiteRT Torch.

```python
import numpy as np
import litert_torch
import torch
import torchvision
```

--------------------------------

### Clone IS-Net Repo and Download Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Clones the IS-Net DIS GitHub repository, downloads the pre-trained PyTorch model, and extracts it.

```bash
%cd /content
!rm -rf DIS sample_data

# Clone github repo and download the Pytorch model.
!git clone https://github.com/xuebinqin/DIS.git
%cd DIS/IS-Net/

!curl -o ./model.tar.gz -L https://www.kaggle.com/api/v1/models/paulruiz/dis/pyTorch/8-17-22/1/download
!tar -xvf 'model.tar.gz'
```

--------------------------------

### Build SentencePiece Model from Tokenizer Config

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/README.md

Use this script to build SentencePiece model protobuf files from tokenizer configuration JSON files when SentencePiece model files are not provided with PyTorch models. Note that generated models may have slight token ID mismatches compared to original BPE tokenizers.

```bash
python tokenizer_to_sentencepiece.py \
    --checkpoint=meta-llama/Llama-3.2-3B-Instruct \
    --output_path=llama3.spm.model
```

```log
I1011 tokenizer_to_sentencepiece.py:203] Not matched strictly 35/1000 pairs: 3.50%, loosely 9/1000 pairs: 0.90%
I1011 tokenizer_to_sentencepiece.py:274] Writing the SentencePieceModel protobuf file to: llama3.spm.model
```

--------------------------------

### Clone IS-Net Repo and Download Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Clones the IS-Net DIS repository, changes directory, downloads the PyTorch model weights, and extracts them.

```python
%cd /content
!rm -rf DIS sample_data

!git clone https://github.com/xuebinqin/DIS.git
%cd DIS/IS-Net/

!curl -o ./model.tar.gz -L https://www.kaggle.com/api/v1/models/paulruiz/dis/pyTorch/8-17-22/1/download
!tar -xvf 'model.tar.gz'
```

--------------------------------

### Perform Inference with the On-Device Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/getting_started.ipynb

Execute the converted on-device model with the sample inputs to obtain inference results.

```python
edge_output = edge_model(*sample_inputs)
```

--------------------------------

### Test with Prompt File

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/export_hf/experimental/validation/SKILL.md

Execute an equivalence test using prompts loaded from a specified file.

```bash
bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id=google/gemma-3-270m-it \
  --prompt_file=/path/to/prompts.txt
```

--------------------------------

### Load Safetensors Checkpoint

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/README.md

Loads a SafeTensors checkpoint file and prints its keys. This is the first step in loading pre-trained weights for a new model.

```python
import litert_torch.generative.utilities.loader as loading_utils

safetensors = loading_utils.load_safetensors("path_to_checkpoint")
print(safetensors.keys())
```

```text
dict_keys(['lm_head.weight', 'model.embed_tokens.weight', 'model.layers.0.input_layernorm.weight', 'model.layers.0.mlp.down_proj.weight', ...])
```

--------------------------------

### Visualize TFLite Model using CLI

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/README.md

Visualize a TFLite model using the Model Explorer command-line interface. Provide the path to the .tflite model file as an argument.

```bash
model-explorer 'gemma_seq512_kv1024.tflite'
```

--------------------------------

### Format Code Changes

Source: https://github.com/google-ai-edge/litert-torch/blob/main/CONTRIBUTING.md

Applies code formatting to your changes using the project's preconfigured script. Ensure you are in the project's root directory.

```bash
cd litert-torch
bash ./format.sh
```

--------------------------------

### Load PyTorch Model and Utilities

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Loads the pre-trained MobileViT model and its corresponding image processor from Hugging Face. Includes a utility function to display classification probabilities.

```python
from transformers import MobileViTImageProcessor, MobileViTForImageClassification


hf_model_path = 'apple/mobilevit-small'
image_processor = MobileViTImageProcessor.from_pretrained(hf_model_path)
hf_pt_model = MobileViTForImageClassification.from_pretrained(hf_model_path)

# Utility function to get %probablilities and corresponding class labels
# defined in MobileViTForImageClassification
# This utility applies softmax to convert logits to probabilities and shows
# top-5 of predicted classes.
def get_classification_probablities(logits):
  probs, indices = torch.nn.functional.softmax(logits, dim=-1).flatten().topk(k=5)
  for i in range(len(indices)):
    class_label = hf_pt_model.config.id2label[indices[i].item()]
    prob = probs[i].item()
    print(f'{(prob * 100):4.1f}%  {class_label}')

wrapped_pt_model = MobileViTForImageClassificationWrapper(hf_pt_model).eval()
```

--------------------------------

### Marking SDPA boundary with HLFB API

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/doc/system_overview.md

This code demonstrates how to use the high-level function boundary API (HLFB) to mark the input and output of an operation, enabling custom op conversion and optimization.

```python
builder.mark_inputs(inputs)
# ... computation ...
builder.mark_outputs(outputs)
```

--------------------------------

### Convert Model to LiteRT-LM Format

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/colabs/Gemma3_270M_convertion.ipynb

Converts the downloaded PyTorch model to LiteRT-LM format. This involves setting up export configurations, specifying tokenizer paths, token IDs, and prompt formats, and then initiating the conversion process with quantization.

```python
output_dir = '/content/'

# Import the weights and build the PyTorch model
pytorch_model = gemma3.build_model_270m(checkpoint_dir)

# Setup the export configurations and parameters for text generation models.
export_config = ExportConfig()
export_config.kvcache_layout = kv_cache.KV_LAYOUT_TRANSPOSED
export_config.mask_as_input = True

# Configs specific for text generation models.
litertlm_config = {
    "tokenizer_model_path": os.path.join(checkpoint_dir, 'tokenizer.model'),
    "start_token_id": 2,  # "<bos>"
    "stop_token_ids": [1, 106],  # ["<eos>", "<end_of_turn>"]
    "prompt_prefix": "<start_of_turn>user\n",
    "prompt_suffix": "<end_of_turn>\n<start_of_turn>model\n",
    "model_prompt_prefix": "<start_of_turn>model\n",
    "model_prompt_suffix": "<end_of_turn>\n",
    "user_prompt_prefix": "<start_of_turn>user\n",
    "user_prompt_suffix": "<end_of_turn>\n",
    "output_format": "litertlm",
}

# Convert to LiteRT or LiteRT-LM Format
converter.convert_to_litert(
    pytorch_model,
    output_path=output_dir,
    output_name_prefix="gemma",
    prefill_seq_len=2048,
    kv_cache_max_len=4096,
    quantize="dynamic_int8",
    export_config=export_config,
    **litertlm_config
)
```

--------------------------------

### Download and Display Test Image

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Downloads a sample image using curl and displays it using Matplotlib and PIL.

```python
!curl -H 'Accept: application/vnd.github.v3.raw'  -O   -L https://api.github.com/repos/google-ai-edge/litert-torch/contents/test/image_classification/test_data/astrid_happy_hike.jpg;

from matplotlib import pyplot as plt
from PIL import Image

image_path = 'astrid_happy_hike.jpg'
image = Image.open(image_path)
plt.figure(figsize=(7, 7))
plt.axis('off')
plt.imshow(image)
```

--------------------------------

### Preprocess Image for PyTorch Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Prepares the input image by resizing, normalizing, and converting it to a PyTorch tensor.

```python
from io import BytesIO
import numpy as np
from skimage import io

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.transforms.functional import normalize


input_size=[1024, 1024]

im = io.imread(IMAGE_PATH)
if len(im.shape) < 3:
  im = im[:, :, np.newaxis]
im_shp = im.shape[0:2]

im_tensor = torch.tensor(im, dtype=torch.float32).permute(2, 0, 1)
im_tensor = F.upsample(torch.unsqueeze(im_tensor, 0),
                       input_size, mode='bilinear').type(torch.uint8)
pt_image = torch.divide(im_tensor, 255.0)
pt_image = normalize(pt_image, mean=[0.5, 0.5, 0.5], std=[1.0, 1.0, 1.0])
```

--------------------------------

### Import Needed Packages

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/colabs/Gemma3_270M_convertion.ipynb

Imports essential libraries for model conversion, including Hugging Face Hub utilities, LiteRT Torch components, and export configurations.

```python
# @title Import needed packages.
from huggingface_hub import snapshot_download
from litert_torch.generative.examples.gemma3 import gemma3
from litert_torch.generative.utilities import converter
from litert_torch.generative.utilities.export_config import ExportConfig
from litert_torch.generative.layers import kv_cache
```

--------------------------------

### Download T5 Model Checkpoint

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/t5/t5_conversion_colab.ipynb

Downloads the PyTorch model checkpoint for T5 from Hugging Face. This checkpoint is required for the conversion process.

```bash
!curl -O -L https://huggingface.co/humarin/chatgpt_paraphraser_on_T5_base/resolve/main/pytorch_model.bin
```

--------------------------------

### Convert Gemma 3 to TFLite with Quantization

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/gemma3/README.md

Use this command to convert and quantize a Gemma 3 PyTorch model to TFLite. Specify desired quantization schemes, checkpoint path, output directory, and sequence lengths.

```bash
python convert_gemma3_to_tflite.py \
 --quantize=<string for the desired quantization schemes> \
 --checkpoint_path=<path to torch safetensor directory> \
 --output_path=<path the directory where the tflite file to be saved> \
 --prefill_seq_lens=<maximum length of supported input> \
 --kv_cache_max_len=<maximum of prefill + decode context length> \
 --mask_as_input=True
```

--------------------------------

### Multi-Signature Conversion with Litert Torch

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Convert multiple PyTorch modules into a single Edge model using signatures. This is useful when a model consists of components that share weights. Signatures allow for distinct entry points into the converted model.

```python
inputs_1 = (...,)
inputs_2 = (...,)

edge_model = litert_torch
            .signature("input1", model, inputs_1)
            .signature("input2", model, inputs_2)
            .convert()

# Run each signature separately by providing the signature_name as a keyword argument.
edge_model(*inputs_1, signature_name="input1")
edge_model(*inputs_2, signature_name="input2")
```

--------------------------------

### Download IS-Net DIS Model for Debug Test

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/android/README.md

Execute this command in the specified directory to download the TFLite model required for the debug test. The script uses a new converter tool.

```bash
cd app/src/androidTest
sh ./download_model.sh
```

--------------------------------

### Download Test Image

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Downloads a sample image 'astrid_l_shaped.jpg' using curl.

```bash
!curl -H 'Accept: application/vnd.github.v3.raw'  -O   -L https://api.github.com/repos/google-ai-edge/litert-torch/contents/test/image_segmentation/test_data/astrid_l_shaped.jpg;
```

--------------------------------

### Visualize PT2E Quantized Segmentation Results

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Display the original image, the PyTorch segmentation mask, and the PT2E quantized segmentation mask side-by-side for comparison.

```python
from matplotlib import pyplot as plt

f, ax = plt.subplots(1, 3, figsize = (10,10))
ax[0].imshow(image)  # Original image.
ax[1].imshow(pt_result, cmap = 'gray')  # PT segmentation mask.
ax[2].imshow(pt2e_drq_result, cmap = 'gray')  # TFL segmentation mask.
ax[0].set_title('Original Image')
ax[1].set_title('PT Mask')
ax[2].set_title('PT2E DRQ Mask')
ax[0].axis('off')
ax[1].axis('off')
ax[2].axis('off')
plt.tight_layout()
plt.show()
```

--------------------------------

### Load PyTorch MobileViT Model and Processor

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Loads the MobileViT model and its corresponding image processor from Hugging Face.

```python
from transformers import MobileViTImageProcessor, MobileViTForImageClassification


hf_model_path = 'apple/mobilevit-small'
processor = MobileViTImageProcessor.from_pretrained(hf_model_path)
pt_model = MobileViTForImageClassification.from_pretrained(hf_model_path)
```

--------------------------------

### Load Hugging Face MobileViT Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Loads a pre-trained MobileViT model and its processor from Hugging Face and wraps it for evaluation.

```python
hf_mobile_vit_processor = MobileViTImageProcessor.from_pretrained(hf_model_path)
hf_mobile_vit_model = MobileViTForImageClassification.from_pretrained(hf_model_path)
wrapped_pt_model = HF2MP_ImageClassificationModelWrapper(
hf_mobile_vit_model, hf_mobile_vit_processor).eval()
```

--------------------------------

### Run LiteRT Model Equivalence Test

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/export_hf/experimental/validation/SKILL.md

Use this command to run the equivalence test script. Specify the Hugging Face model ID and optionally provide prompts, token limits, and export configurations.

```bash
bazel run \
  //third_party/py/litert_torch/generative/export_hf/experimental/validation:equivalence_test \
  -- \
  --model_id={model_id} \
  [--prompt={prompt}] \
  [--prompt_file={prompt_file}] \
  [--max_new_tokens={max_new_tokens}] \
  [--max_num_tokens={max_num_tokens}] \
  [--work_dir={work_dir}] \
  [--externalize_embedder] \
  [--single_token_embedder] \
  [--split_cache] \
  [--backend={backend}]
```

--------------------------------

### Display Test Image

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Opens and displays the downloaded test image using Pillow and Matplotlib.

```python
from matplotlib import pyplot as plt
from PIL import Image

IMAGE_PATH = '/content/astrid_l_shaped.jpg'
image = Image.open(IMAGE_PATH)
plt.figure(figsize=(5, 5))
plt.axis('off')
plt.imshow(image);
```

--------------------------------

### Prepare Input Tensor for PyTorch Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Prepares the input image tensor for the PyTorch IS-Net model. This involves reading the image, converting it to a tensor, resizing, scaling, and normalizing.

```python
from io import BytesIO
import numpy as np
from skimage import io

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.transforms.functional import normalize


MODEL_INPUT_HW = (1024, 1024)
image = io.imread(IMAGE_PATH)

# BHWC -> BCHW.
image_tensor = torch.tensor(image, dtype=torch.float32).permute(2, 0, 1)

# Resize to meet model input size requirements.
image_tensor = F.upsample(torch.unsqueeze(image_tensor, 0),
                          MODEL_INPUT_HW, mode='bilinear').type(torch.uint8)

# Scale [0, 255] -> [0, 1].
pt_image = torch.divide(image_tensor, 255.0)

# Normalize.
pt_image = normalize(pt_image, mean=[0.5, 0.5, 0.5], std=[1.0, 1.0, 1.0])
```

--------------------------------

### Post-Training Dynamic-Range Quantization with PT2E

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Utilize PyTorch 2.0's PT2E quantization framework with litert-torch for dynamic range quantization. This involves capturing the graph, preparing for quantization, running with sample data, and converting the model.

```python
from litert_torch.quantize.pt2e_quantizer import get_symmetric_quantization_config
from litert_torch.quantize.pt2e_quantizer import PT2EQuantizer
from litert_torch.quantize.quant_config import QuantConfig

from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
from torch._export import capture_pre_autograd_graph

# PT2E is a framework-level quantization feature available in PyTorch 2.0.
# For more details see-
# https://pytorch.org/tutorials/prototype/quantization_in_pytorch_2_0_export_tutorial.html

# PT2EQuantizer is litert-torch backend specific and is configured to quantize models
# to leverage the quantized operators/kernels offered by the TFLite Runtime.
pt2e_quantizer = PT2EQuantizer().set_global(
    get_symmetric_quantization_config(is_per_channel=True, is_dynamic=True)
)

# Following are the required steps recommended in the pt2e quantization workflow.
autograd_torch_model = capture_pre_autograd_graph(wrapped_pt_model, sample_args)
# 1. Prepare for quantization
pt2e_torch_model = prepare_pt2e(autograd_torch_model, pt2e_quantizer)
# 2. Run the prepared model with sample input data to ensure that internal
# observers are populated with correct values
pt2e_torch_model(*sample_args)
# 3. Finally, convert(quantize) the prepared model
pt2e_torch_model = convert_pt2e(pt2e_torch_model, fold_quantize=False)

pt2e_drq_model = litert_torch.convert(pt2e_torch_model, sample_args, quant_config=QuantConfig(pt2e_quantizer=pt2e_quantizer))
```

--------------------------------

### Convert PyTorch Model with litert_torch

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

This snippet demonstrates the basic usage of the litert_torch.convert function to convert a PyTorch model. Ensure the model is in evaluation mode and provide necessary arguments.

```python
import torch
import torchaudio
import litert_torch

model = torchaudio.models.ConvTasNet()
args = (torch.rand((1, 1, 256)),)
litert_torch.convert(model.eval(), args)
```

--------------------------------

### Process Input Image

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_tfl.ipynb

Preprocesses the input image using the MobileViT image processor to prepare it for the model.

```python
processed_inputs = image_processor(
    images=image,
    return_tensors='pt',
)

input_tensor = processed_inputs['pixel_values']
```

--------------------------------

### Import and Run a Serialized TFLite Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Load a TFLite model that was previously serialized using litert_torch.export or any other TFLite Flatbuffers file. The imported model can then be used for inference.

```python
imported_edge_model = litert_torch.load('resnet.tflite')

# Once imported, you can run the model with an input.
imported_edge_model(*sample_inputs)
```

--------------------------------

### XNNPack Delegate Handling of SDPA

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/doc/system_overview.md

This C++ code snippet shows how the TF Lite XNNPack delegate specifically handles the Scaled Dot Product Attention (SDPA) computation by matching the composite node and constructing an XNNPack subgraph for acceleration.

```c++
TfLiteStatus VisitScaledDotAttentionCompositeNode(
    TfLiteContext* context,
    TfLiteNode* node,
    TfLiteDelegateParams* params) {
  // ... XNNPack computation subgraph construction ...
  return kTfLiteOk;
}
```

--------------------------------

### Load PyTorch IS-Net Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_mpt.ipynb

Loads the pre-trained IS-Net PyTorch model from a file and sets it to evaluation mode.

```python
import torch
from models import ISNetDIS


pytorch_model_filename = 'isnet-general-use.pth'
pt_model = ISNetDIS()
pt_model.load_state_dict(
    torch.load(pytorch_model_filename, map_location=torch.device('cpu')))
pt_model.eval();
```

--------------------------------

### Instantiate Pretrained Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/README.md

Instantiates a pretrained causal language model from Hugging Face Transformers. This is useful for inspecting the architecture of existing models.

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
print(model)
```

--------------------------------

### Quantize Model with PT2E Quantization

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

Quantizes a PyTorch model using PT2E quantization with the litert_torch backend. Requires specific imports and configuration for symmetric quantization. Note the conditional export method based on PyTorch version.

```python
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, convert_pt2e
from litert_torch.quantize.pt2e_quantizer import get_symmetric_quantization_config
from litert_torch.quantize.pt2e_quantizer import PT2EQuantizer
from litert_torch.quantize.quant_config import QuantConfig

pt2e_quantizer = PT2EQuantizer().set_global(
    get_symmetric_quantization_config(is_per_channel=True, is_dynamic=True)
)

# > For pytorch 2.6+
pt2e_torch_model = torch.export.export(torch_model, sample_args).module()
# > For pytorch 2.5 and before
# from torch._export import capture_pre_autograd_graph
# pt2e_torch_model = capture_pre_autograd_graph(torch_model, sample_args)

pt2e_torch_model = prepare_pt2e(pt2e_torch_model, pt2e_quantizer)

# Run the prepared model with sample input data to ensure that internal observers are populated with correct values
pt2e_torch_model(*sample_args)

# Convert the prepared model to a quantized model
pt2e_torch_model = convert_pt2e(pt2e_torch_model, fold_quantize=False)

# Convert to an litert_torch model
pt2e_drq_model = litert_torch.convert(pt2e_torch_model, sample_args, quant_config=QuantConfig(pt2e_quantizer=pt2e_quantizer))
```

--------------------------------

### Run PT2E Quantized Model and Process Results

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Execute the PT2E quantized model with sample input and process the segmentation mask output.

```python
pt2e_drq_output = pt2e_drq_model(np_image)

pt2e_drq_result = get_processed_isnet_result(pt2e_drq_output, INPUT_IMAGE_HW)
```

--------------------------------

### Load PyTorch IS-Net Model

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Loads the IS-Net PyTorch model from the downloaded weights file ('isnet-general-use.pth'). The model is set to evaluation mode.

```python
import torch
from models import ISNetDIS


pytorch_model_filename = 'isnet-general-use.pth'
pt_model = ISNetDIS()
pt_model.load_state_dict(
    torch.load(pytorch_model_filename, map_location=torch.device('cpu'))
)
pt_model.eval();
```

--------------------------------

### Create MediaPipe Image Classification Wrapper

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_classification/colab/mobile_vit_mpt.ipynb

Defines a PyTorch module wrapper to adapt the MobileViT model for MediaPipe Image Classification Tasks, handling channel format, color space, scaling, and adding a softmax layer.

```python
import torch
from torch import nn

class HF2MP_ImageClassificationModelWrapper(nn.Module):

  def __init__(self, hf_image_classification_model, hf_processor):
    super().__init__()
    self.model = hf_image_classification_model
    if hf_processor.do_rescale:
      self.rescale_factor = hf_processor.rescale_factor
    else:
      self.rescale_factor = 1.0

  def forward(self, image: torch.Tensor):
    # BHWC -> BCHW.
    image = image.permute(0, 3, 1, 2)
    # RGB -> BGR.
    image = image.flip(dims=(1,))
    # Scale [0, 255] -> [0, 1].
    image = image * self.rescale_factor
    logits = self.model(pixel_values=image).logits  # [B, 1000] float32.
    # Softmax is required for MediaPipe classification model.
    logits = torch.nn.functional.softmax(logits, dim=-1)

    return logits


hf_model_path = 'apple/mobilevit-small'
```

--------------------------------

### Run TFLite Model and Post-process

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Executes the TFLite model with the prepared input image and then applies post-processing to obtain the final segmentation mask. Compares the TFLite result with the original image and PyTorch mask.

```python
edge_model_output = edge_model(np_image)

tfl_result = get_processed_isnet_result(edge_model_output, INPUT_IMAGE_HW)
```

--------------------------------

### Visualize Segmentation Results

Source: https://github.com/google-ai-edge/litert-torch/blob/main/test/image_segmentation/colab/isnet_tfl.ipynb

Display the original image, the PyTorch segmentation mask, and the TFLite quantized segmentation mask side-by-side for comparison.

```python
from matplotlib import pyplot as plt

f, ax = plt.subplots(1, 3, figsize = (10,10))
ax[0].imshow(image)  # Original image.
ax[1].imshow(pt_result, cmap = 'gray')  # PT segmentation mask.
ax[2].imshow(tfl_drq_result, cmap = 'gray')  # TFL segmentation mask.
ax[0].set_title('Original Image')
ax[1].set_title('PT Mask')
ax[2].set_title('TFLQ DRQ Mask')
ax[0].axis('off')
ax[1].axis('off')
ax[2].axis('off')
plt.tight_layout()
plt.show()
```

--------------------------------

### Perform Inference and Verify Conversion

Source: https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md

After conversion, you can perform inference with the Edge model and compare its outputs against the original PyTorch model's outputs. This serves as a smoke test for the converted model's accuracy. Ensure numpy is imported for array comparison.

```python
import numpy as np

edge_output = edge_model(*sample_inputs)
assert np.allclose(torch_output.detach().numpy(), edge_output, atol=1e-5)
```

--------------------------------

### Convert PyTorch to TFLite

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/examples/stable_diffusion/README.md

Use this script to convert SafeTensors model checkpoints from the HuggingFace repository to TFLite format. Ensure you have cloned the original repository and have the safetensors file available. The encoder model conversion is not yet supported.

```bash
python litert_torch/generative/examples/stable_diffusion/convert_to_tflite.py \
--clip_ckpt=$HOME/stable-diffusion-v1-5/v1-5-pruned-emaonly.safetensors \
--diffusion_ckpt=$HOME/stable-diffusion-v1-5/v1-5-pruned-emaonly.safetensors \
--decoder_ckpt=$HOME/stable-diffusion-v1-5/v1-5-pruned-emaonly.safetensors \
--output_dir=/tmp/stable_diffusion_safetensors/
```

--------------------------------

### Custom Selective Quantization Recipe

Source: https://github.com/google-ai-edge/litert-torch/blob/main/litert_torch/generative/quantize/README.md

Define a custom quantization recipe for selective quantization of different model layers. This allows fine-grained control over quantization schemes for embedding, attention, and feedforward layers.

```python
def custom_selective_quantization_recipe() -> quant_config.QuantConfig:
  return quant_config.QuantConfig(
      generative_recipe=quant_recipe.GenerativeQuantRecipe(
          default=create_layer_quant_fp16(),
          embedding=create_layer_quant_int8_dynamic(),
          attention=create_layer_quant_int4_block(32),
          feedforward=create_layer_quant_int4_block(256),
      )
  )
```