### Install MobileCLIP with Conda

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Sets up a Conda environment and installs MobileCLIP. Use this for initial setup.

```bash
conda create -n clipenv python=3.10
conda activate clipenv
pip install -e .
```

--------------------------------

### Setup OpenCLIP with MobileCLIP2 Support

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Clones the OpenCLIP repository, applies a patch for MobileCLIP2, copies MobileCLIP2 models, and installs the package. Also installs PyTorch Image Models.

```bash
conda create -n clipenv python=3.10
conda activate clipenv

# Clone OpenCLIP repository, add MobileCLIP2 models, and install
git clone https://github.com/mlfoundations/open_clip.git
pushd open_clip
git apply ../mobileclip2/open_clip_inference_only.patch
cp -r ../mobileclip2/* ./src/open_clip/
pip install -e .
popd

pip install git+https://github.com/huggingface/pytorch-image-models
```

--------------------------------

### Example Usage of ConvFFN

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/repmixer-blocks.md

Demonstrates how to create and use the ConvFFN module with sample input. Verifies the output shape.

```python
import torch
from mobileclip.modules.text.repmixer import ConvFFN

# Create ConvFFN
ffn = ConvFFN(
    in_channels=512,
    context_size=11,
    hidden_channels=2048,
    out_channels=512,
    drop=0.1
)

x = torch.randn(2, 512, 1, 77)
output = ffn(x)
assert output.shape == (2, 512, 1, 77)
```

--------------------------------

### MCi Initialization and Forward Pass Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/image-encoder.md

Demonstrates how to create an MCi image encoder with a projection dimension and perform a forward pass with random images.

```python
import torch
from mobileclip.image_encoder import MCi

# Create image encoder with projection
encoder = MCi(model_name="mci0", projection_dim=512)
encoder.eval()

# Forward pass
images = torch.randn(4, 3, 256, 256)
with torch.no_grad():
    features = encoder(images)

print(features.shape)  # torch.Size([4, 512])
```

--------------------------------

### CLIP Model Initialization and Forward Pass Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/clip-model.md

Demonstrates how to load a configuration, initialize the CLIP model, and perform a forward pass with mock image and text inputs. Shows how to access output keys when output_dict is True.

```python
import torch
import mobileclip
from mobileclip.clip import CLIP
import json

# Load config
with open("mobileclip_s0.json") as f:
    cfg = json.load(f)

# Create model
model = CLIP(cfg=cfg, output_dict=True)
model.eval()

# Mock inputs
images = torch.randn(2, 3, 256, 256)
text_tokens = torch.randint(0, 49408, (2, 77))

# Forward pass
with torch.no_grad():
    output = model(image=images, text=text_tokens)
    
print(output.keys())
# dict_keys(['image_features', 'text_features', 'logit_scale'])

# Compute similarity
logits = output['image_features'] @ output['text_features'].T
logits *= output['logit_scale']
```

--------------------------------

### PositionalEmbedding Usage Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Demonstrates how to instantiate and use the PositionalEmbedding module. Shows how to get embeddings for both standard and different sequence lengths, including interpolation.

```python
import torch
from mobileclip.modules.common.transformer import PositionalEmbedding

pos_emb = PositionalEmbedding(
    num_embeddings=77,
    embedding_dim=512
)

# Get embeddings for standard length
emb = pos_emb(77)
print(emb.shape)  # torch.Size([1, 77, 512])

# Get embeddings for different length (with interpolation)
emb_short = pos_emb(50)
print(emb_short.shape)  # torch.Size([1, 50, 512])
```

--------------------------------

### Example Usage of RepMixerBlock

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/repmixer-blocks.md

Demonstrates how to create and use a RepMixerBlock for text encoding. It shows the expected input shape and verifies the output shape.

```python
import torch
from mobileclip.modules.text.repmixer import RepMixerBlock

# Create RepMixer block for text encoding
block = RepMixerBlock(
    dim=512,
    kernel_size=11,
    mlp_ratio=4.0,
    drop=0.1,
    drop_path=0.1
)

# Input: [batch_size, seq_len, dim]
x = torch.randn(2, 77, 512)
output = block(x)
assert output.shape == x.shape
```

--------------------------------

### Initialize and Use TextTransformer

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Demonstrates how to initialize a TextTransformer with a configuration dictionary and use it to get embeddings from tokenized text. Shows how to retrieve either the final EOT token embeddings or all token embeddings.

```python
import torch
from mobileclip.text_encoder import TextTransformer

cfg = {
    "dim": 512,
    "context_length": 77,
    "vocab_size": 49408,
    "n_transformer_layers": 4,
    "n_heads_per_layer": 8,
    "ffn_multiplier_per_layer": 4.0,
    "norm_layer": "layer_norm_fp32",
    "causal_masking": False,
    "model_name": "base"
}

encoder = TextTransformer(cfg=cfg, projection_dim=512)
encoder.eval()

# Tokenized text
text_tokens = torch.randint(0, 49408, (2, 77))

with torch.no_grad():
    # Get final EOT token embeddings
    embeddings = encoder(text_tokens)
    print(embeddings.shape)  # torch.Size([2, 512])
    
    # Get all token embeddings
    all_tokens = encoder(text_tokens, return_all_tokens=True)
    print(all_tokens.shape)  # torch.Size([2, 77, 512])
```

--------------------------------

### Install OpenCLIP with MobileCLIP Modifications (v2)

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This sequence of commands clones the MobileCLIP repository, then clones and sets up a specific version of OpenCLIP, applying a patch and copying necessary configuration and source files for DR training.

```bash
# Clone MobileCLIP repository
git clone git@github.com:apple/ml-mobileclip.git
cd ml-mobileclip/

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout 7260a46e7b4bcf518f5200fea06da5bc85aae025  # Mon Mar 17 18:18:30 2025 -0400
git apply ../open_clip_v2.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/open_clip_train/ -r
cp ../../mobileclip2/* ./src/open_clip/ -r

```

--------------------------------

### MultiHeadAttention Usage Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Demonstrates how to use the MultiHeadAttention module for both self-attention and cross-attention scenarios. Ensure torch and the module are imported.

```python
import torch
from mobileclip.modules.common.transformer import MultiHeadAttention

# Self-attention
attn = MultiHeadAttention(
    embed_dim=512,
    num_heads=8,
    attn_dropout=0.1
)

x = torch.randn(2, 77, 512)
output = attn(x)

# Cross-attention with different output dimension
cross_attn = MultiHeadAttention(
    embed_dim=512,
    num_heads=8,
    output_dim=256
)

x_q = torch.randn(2, 77, 512)
x_kv = torch.randn(2, 100, 512)
output = cross_attn(x_q, x_kv)
print(output.shape)  # torch.Size([2, 77, 256])
```

--------------------------------

### Train MobileCLIP-B on DFN-2B with DR

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This command starts the training process for a MobileCLIP-B model on the DFN-2B dataset, including DR-specific configurations.

```bash
cd open_clip/
bash configs/run_dfndr1B.sh  # Train a MobileCLIP-B on DFN-2B with DR

```

--------------------------------

### Image Preprocessing with Compose

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/types.md

Illustrates the use of torchvision.transforms.Compose for image preprocessing. It includes resizing, center cropping, and conversion to a tensor. The example shows how to load an image and apply the preprocessing steps.

```python
preprocess: torchvision.transforms.Compose
# Composition of:
# - Resize(size, interpolation=InterpolationMode.BILINEAR)
# - CenterCrop(size)
# - ToTensor()

# Usage:
from PIL import Image
image = Image.open("photo.jpg").convert("RGB")
image_tensor = preprocess(image)  # -> [3, H, W]
batch = image_tensor.unsqueeze(0)  # -> [1, 3, H, W]
```

--------------------------------

### Load and Use Pre-trained MobileCLIP Model

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

Demonstrates loading a pre-trained MobileCLIP model, preparing image and text inputs, and computing similarity scores. Ensure you have a checkpoint file and an image for this example.

```python
import torch
import mobileclip
from PIL import Image

# Create model
model, _, preprocess = mobileclip.create_model_and_transforms(
    model_name='mobileclip_s0',
    pretrained='/path/to/checkpoint.pt',
    reparameterize=True
)

# Get tokenizer
tokenizer = mobileclip.get_tokenizer('mobileclip_s0')

# Prepare inputs
image = preprocess(Image.open("photo.jpg").convert("RGB")).unsqueeze(0)
text = tokenizer(["a dog", "a cat"])

# Encode
with torch.no_grad():
    image_features = model.encode_image(image, normalize=True)
    text_features = model.encode_text(text, normalize=True)
    
    # Compute similarity
    logits = 100 * image_features @ text_features.T
    probs = logits.softmax(dim=-1)
```

--------------------------------

### TransformerEncoder Example Usage

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Demonstrates how to create and use a TransformerEncoder layer for both self-attention and cross-attention tasks. Ensure torch and the TransformerEncoder class are imported.

```python
import torch
from mobileclip.modules.common.transformer import TransformerEncoder

# Create single encoder layer
layer = TransformerEncoder(
    embed_dim=512,
    ffn_latent_dim=2048,
    num_heads=8,
    attn_dropout=0.1,
    dropout=0.1,
    transformer_norm_layer="layer_norm_fp32"
)

# Self-attention
x = torch.randn(2, 77, 512)  # [batch, seq_len, dim]
output = layer(x)

# Cross-attention
x_q = torch.randn(2, 77, 512)
x_kv = torch.randn(2, 100, 512)
output = layer(x_q, x_prev=x_kv)
```

--------------------------------

### Model State Dictionary Structure Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

Provides a Python code snippet demonstrating how to access the model's state dictionary and lists common key patterns for image encoder, text encoder, and other components.

```python
state_dict = model.state_dict()
# Keys pattern:
# "image_encoder.model.*"      # MCi and underlying TIMM model
# "image_encoder.model.head.*" # GlobalPool2D projection
# "text_encoder.embedding_layer.weight"
# "text_encoder.positional_embedding.pos_embed.pos_embed"
# "text_encoder.transformer.*.weight/bias"  # Transformer layers
# "text_encoder.final_layer_norm.weight/bias"
# "text_encoder.projection_layer"  # Final text projection
# "logit_scale"  # Temperature parameter
```

--------------------------------

### RepMixer Training Example

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/repmixer-blocks.md

Demonstrates how to create and use the RepMixer module in training mode. The input tensor shape is [batch, dim, 1, seq_len].

```python
import torch
from mobileclip.modules.text.repmixer import RepMixer

# Create RepMixer for training
mixer = RepMixer(
    dim=512,
    kernel_size=11,
    use_layer_scale=True,
    inference_mode=False
)

# Input: [batch, dim, 1, seq_len]
x = torch.randn(2, 512, 1, 77)
output = mixer(x)
assert output.shape == x.shape
```

--------------------------------

### Install OpenCLIP with MobileCLIP Modifications (v1)

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This procedure reverts specific changes for compatibility with an older OpenCLIP commit, then clones and applies a patch to OpenCLIP, along with copying configuration and source files for DR training.

```bash
# Revert changes for compatibility with older OpenCLIP commit
sed -i 's/open_clip_train/training/g' ../dr/transforms.py
find . --name "*.sh" | xargs sed -i 's/open_clip_train/training/g'

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout cf86ee7ec4658845f640858ecd34d0f15588271a  # Wed May 29 21:57:08 2024 +0700
git apply ../open_clip_v1.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/training/ -r

```

--------------------------------

### Example Usage of GlobalPool2D

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/image-encoder.md

Demonstrates how to create and use the GlobalPool2D module to transform feature maps into embeddings. Ensure the module is in evaluation mode and gradients are disabled for inference.

```python
import torch
from mobileclip.modules.image.image_projection import GlobalPool2D

# Create projection head
proj = GlobalPool2D(in_dim=768, out_dim=512)
proj.eval()

# Apply to feature maps
features = torch.randn(2, 768, 8, 8)
with torch.no_grad():
    embeddings = proj(features)

print(embeddings.shape)  # torch.Size([2, 512])

```

--------------------------------

### Source Code Reference Location

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

This indicates the file path and line numbers in the source code for verification. Ensure you have the correct version of the library installed to match these references.

```text
Location: mobileclip/clip.py:20-77
```

--------------------------------

### Get Kernel and Bias

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/model-reparameterization.md

Computes the fused kernel and bias by combining weights from all branches of the MobileOneBlock.

```python
def _get_kernel_bias(self) -> Tuple[torch.Tensor, torch.Tensor]
```

--------------------------------

### MobileCLIP Configuration Loading Pattern

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

Illustrates the flow from user call to configuration loading and model creation in MobileCLIP.

```text
User calls: mobileclip.create_model_and_transforms('mobileclip_s0')
    ↓
Looks for: mobileclip/configs/mobileclip_s0.json
    ↓
Loads: {"embed_dim": 512, "image_cfg": {...}, "text_cfg": {...}}
    ↓
Creates: CLIP(cfg=config)
    ↓
Returns: (model, None, preprocess_transforms)
```

--------------------------------

### Get Tokenizer

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

Function to retrieve the tokenizer for a given model name. This is a primary entry point for text processing.

```python
def get_tokenizer(model_name: str) -> nn.Module
```

--------------------------------

### Loading PyTorch Checkpoint

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/types.md

Demonstrates how to load a PyTorch checkpoint file and apply its state dictionary to a model.

```python
checkpoint = torch.load(path)
model.load_state_dict(checkpoint)
```

--------------------------------

### Get Current Timestamp

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/logger-utilities.md

Returns the current timestamp formatted as 'YYYY-MM-DD HH:MM:SS'. Useful for logging or time-stamping events.

```python
from mobileclip import logger

ts = logger.get_curr_time_stamp()
print(ts)  # "2024-06-17 14:30:45"
```

--------------------------------

### Get Input Feature Dimension

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/image-encoder.md

Extracts the input feature dimension from a given classification head module. Supports nn.Sequential and nn.Linear.

```python
@staticmethod
def _get_in_feature_dimension(image_classifier: nn.Module) -> int
```

--------------------------------

### Create MobileCLIP Model and Transforms

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/mobileclip-initialization.md

Instantiates a MobileCLIP model and its required preprocessing transforms. Use this as the primary entry point for loading a model. Specify the model name, an optional pretrained checkpoint path, whether to reparameterize for inference, and the target device.

```python
import torch
from PIL import Image
import mobileclip

# Create model and transforms
model, _, preprocess = mobileclip.create_model_and_transforms(
    model_name='mobileclip_s0',
    pretrained='/path/to/mobileclip_s0.pt',
    reparameterize=True,
    device='cuda' if torch.cuda.is_available() else 'cpu'
)

# Prepare image
image = preprocess(Image.open("photo.jpg").convert("RGB")).unsqueeze(0)

# Encode image
with torch.no_grad():
    image_features = model.encode_image(image)
    image_features /= image_features.norm(dim=-1, keepdim=True)
```

--------------------------------

### Download V1-only Pretrained Checkpoints

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Downloads pretrained checkpoints for V1 MobileCLIP models using a shell script. Files are saved to the 'checkpoints' directory.

```bash
source get_pretrained_models.sh   # Files will be downloaded to `checkpoints` directory.
```

--------------------------------

### Download CoCa Models (Context 77)

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Use this script to download CoCa models with a context length of 77. These models offer balanced performance.

```bash
# context=77 models
for model in \
  mscoco38k_s12m_context77 \
  gbc1m-short_context77 \
  docci_s12m_context77 \
  dci-short_s12m_context77 \
  dci-complete_s12m_context77 \
  dci-extended_s12m_context77 \
  recap-coco-30k_s12m_context77 \
do
  hf download apple/mobileclip2_coca_dfn2b_s13b_$model
done

```

--------------------------------

### Zero-Shot Classification with MobileCLIP

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/INDEX.md

Performs zero-shot classification by encoding images and text, then calculating similarity logits. Ensure you have the 'torch' and 'mobileclip' libraries installed.

```python
import torch
import mobileclip

model, _, preprocess = mobileclip.create_model_and_transforms('mobileclip_s0')
tokenizer = mobileclip.get_tokenizer('mobileclip_s0')

# Prepare images and class labels
images = torch.stack([preprocess(img) for img in image_list])
class_names = ["dog", "cat", "bird"]
text_tokens = tokenizer(class_names)

# Encode
with torch.no_grad():
    image_features = model.encode_image(images, normalize=True)
    text_features = model.encode_text(text_tokens, normalize=True)

# Classify
logits = 100 * image_features @ text_features.T
predictions = logits.argmax(dim=-1)
```

--------------------------------

### Download CoCa Models (Context 256)

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Use this script to download CoCa models with a context length of 256. Be aware that these models may have a higher chance of generating repeated output.

```bash
# Context=256 models. These models have a higher chance of generating repeated output
for model in \
  docci_s12m_context256 \
  dci-complete_s12m_context256 \
  dci-extended_s12m_context256 \
do
  hf download apple/mobileclip2_coca_dfn2b_s13b_$model
done

```

--------------------------------

### Building Custom CLIP Models

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/INDEX.md

This snippet demonstrates how to build a custom CLIP model using a configuration file and load a checkpoint. Ensure you have a valid config.json and checkpoint.pt.

```python
import json
from mobileclip.clip import CLIP

# Load or create config
with open("config.json") as f:
    cfg = json.load(f)

# Create model
model = CLIP(cfg=cfg, output_dict=False)

# Load checkpoint
import torch
checkpoint = torch.load("checkpoint.pt")
model.load_state_dict(checkpoint)
```

--------------------------------

### Using ClipTokenizer for Text Tokenization

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Demonstrates how to initialize and use the ClipTokenizer to tokenize single and multiple text inputs, and access tokenizer metadata.

```python
import torch
import mobileclip

# Get tokenizer for a model
tokenizer = mobileclip.get_tokenizer('mobileclip_s0')

# Tokenize single text
text = "a dog"
tokens = tokenizer(text)
print(tokens.shape)  # torch.Size([77])

# Tokenize multiple texts
texts = ["a dog", "a cat", "a diagram"]
multi_tokens = tokenizer(texts)
print(multi_tokens.shape)  # torch.Size([3, 77])

# Access metadata
print(f"Vocab size: {tokenizer.get_vocab_size()}")
print(f"EOT token: {tokenizer.get_eot_token()}")
print(f"SOT token: {tokenizer.get_sot_token()}")
```

--------------------------------

### Example Usage of LayerNormFP32 with Half Precision

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Demonstrates using LayerNormFP32 with a half-precision tensor. The computation is performed in FP32 for stability, but the output retains the input's half-precision dtype.

```python
import torch
from mobileclip.modules.common.transformer import LayerNormFP32

# Use with half precision
norm = LayerNormFP32(512)
x = torch.randn(2, 77, 512, dtype=torch.float16)
output = norm(x)
print(output.dtype)  # torch.float16 (but computed in FP32)
```

--------------------------------

### Create and Load Custom MobileCLIP Model

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

Shows how to define a custom model configuration and load a checkpoint into a CLIP model. This is useful for experimenting with different model architectures.

```python
import json
from mobileclip.clip import CLIP

# Create custom configuration
config = {
    "embed_dim": 512,
    "image_cfg": {"image_size": 256, "model_name": "mci0"},
    "text_cfg": {
        "context_length": 77,
        "vocab_size": 49408,
        "dim": 512,
        "n_transformer_layers": 4,
        "n_heads_per_layer": 8,
        "ffn_multiplier_per_layer": 4.0,
        "norm_layer": "layer_norm_fp32",
        "causal_masking": False,
        "model_name": "base"
    }
}

# Create and load model
model = CLIP(cfg=config)
checkpoint = torch.load("checkpoint.pt")
model.load_state_dict(checkpoint)
```

--------------------------------

### Join DataComp and DataCompDR Datasets

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This script joins the original DataComp-12M dataset with the DataCompDR-12M dataset (which lacks original images and captions) to create a complete dataset. It requires downloading both datasets from HuggingFace and processing them in tar archives.

```bash
#!/bin/bash
DATACOMP12M_PATH="./datasets/DataComp-12M/" # Download path of DataComp-12M from HF
DATACOMPDR12M_NOIMG_PATH="./datasets/DataCompDR-12M-noimage/" # Download path of DataCompDR-12M from HF
DATACOMPDR12M_PATH="./datasets/DataCompDR-12M/"
for  i in {00000000..00001023}
do
  mkdir tmp
  tar -xf $DATACOMP12M_PATH/${i}.tar -C tmp
  tar -xf $DATACOMPDR12M_NOIMG_PATH/${i}.tar -C tmp
  tar -cf $DATACOMPDR12M_PATH/${i}.tar -C tmp *.*
  rm -rf tmp
done

```

--------------------------------

### Automatic Configuration Loading via create_model_and_transforms

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/configuration.md

Automatically loads configuration based on the provided model name. The configuration file is typically located within the library's config directory. This method also returns the model and preprocessing transforms.

```python
import mobileclip

# Configuration loaded automatically from model name
model, _, preprocess = mobileclip.create_model_and_transforms(
    model_name='mobileclip_s0'
)
# Loads from: mobileclip/configs/mobileclip_s0.json
```

--------------------------------

### Load Custom Configuration into CLIP Model

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/configuration.md

Load a custom JSON configuration and initialize the CLIP model with it. Ensure the JSON file is correctly path-ed.

```python
import json
from mobileclip.clip import CLIP

with open("custom_config.json") as f:
    cfg = json.load(f)

model = CLIP(cfg=cfg)
```

--------------------------------

### Load Configuration from File

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/configuration.md

Loads configuration from a specified JSON file path using Python's built-in json library. Ensure the config_path variable points to a valid JSON configuration file.

```python
import json

config_path = "mobileclip/configs/mobileclip_s0.json"
with open(config_path, 'r') as f:
    cfg = json.load(f)
```

--------------------------------

### Get Tokenizer for MobileCLIP Model

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/mobileclip-initialization.md

Creates a tokenizer instance for a specified MobileCLIP model. This tokenizer is necessary for converting text strings into token indices compatible with the model's text encoder. It is used in conjunction with the model for text-based inference.

```python
import mobileclip

tokenizer = mobileclip.get_tokenizer('mobileclip_s0')

# Tokenize text samples
text_tokens = tokenizer(["a cat", "a dog", "a diagram"])

# Encode with model
with torch.no_grad():
    text_features = model.encode_text(text_tokens)
    text_features /= text_features.norm(dim=-1, keepdim=True)

# Compute similarity
logits = (image_features @ text_features.T) * 100
probs = logits.softmax(dim=-1)
```

--------------------------------

### Configuration Reference

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

Comprehensive reference for all configuration parameters, pre-configured models, and validation rules.

```APIDOC
## Configuration Reference

### Description
Provides a complete reference for all configuration parameters available in the MobileCLIP library. It details pre-configured model specifications (S0, S1, S2, B), outlines configuration validation rules, and offers guidance on creating custom models.

### Content
- Full reference of configuration parameters.
- Specifications for pre-configured models (e.g., S0, S1, S2, B).
- Configuration validation rules.
- Guide for custom model creation.
```

--------------------------------

### ClipTokenizer Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Initializes the ClipTokenizer with a configuration dictionary. It sets up the tokenizer based on the provided text configuration, including context length and optionally specifying the OpenCLIP tokenizer model.

```APIDOC
## ClipTokenizer Constructor

### Description
Initializes the ClipTokenizer with a configuration dictionary. It sets up the tokenizer based on the provided text configuration, including context length and optionally specifying the OpenCLIP tokenizer model.

### Parameters
#### Parameters
- **cfg** (dict) - Yes - Configuration dictionary containing `text_cfg` with keys `context_length` and optionally `open_clip_tokenizer`.
```

--------------------------------

### Train ViT-B/16 on DataComp-12M with DR

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This command launches the training of a ViT-B/16 model on the DataComp-12M dataset, incorporating DR-related training aspects.

```bash
cd open_clip/
bash configs/run_datacompdr12m.sh  # Train a ViT-B/16 on DataComp-12M with DR

```

--------------------------------

### MobileCLIP Initialization API

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/DELIVERABLES.txt

Documentation for functions related to initializing the MobileCLIP model and its components, including model creation, transformations, and tokenizers.

```APIDOC
## mobileclip-initialization.md

### Description
Provides documentation for the `create_model_and_transforms()` and `get_tokenizer()` functions, detailing their parameters, return types, and usage examples.

### Functions
#### `create_model_and_transforms()`
- **Purpose**: Creates the MobileCLIP model and associated image transformations.
- **Parameters**: (Details not specified in source)
- **Return Types**: (Details not specified in source)

#### `get_tokenizer()`
- **Purpose**: Retrieves the tokenizer for text encoding.
- **Parameters**: (Details not specified in source)
- **Return Types**: (Details not specified in source)

### Examples
(Usage examples are available within the documentation.)
```

--------------------------------

### CLIP Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/clip-model.md

Initializes the CLIP model with a configuration dictionary and an option to return outputs as a dictionary.

```python
def __init__(self, cfg: Dict, output_dict: bool = False, *args, **kwargs) -> None
```

--------------------------------

### Download MobileCLIP Pretrained Checkpoints

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Downloads pretrained checkpoints for various MobileCLIP models using the 'hf download' command. Models range from S0 to S4.

```bash
# MobileCLIP
for model in S0 S1 S2 B B-LT S3 L-14 S4
do
  hf download apple/MobileCLIP-$model
done
```

--------------------------------

### ClipTokenizer Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Initializes the ClipTokenizer with a configuration dictionary.

```python
def __init__(self, cfg, *args, **kwargs)
```

--------------------------------

### Download MobileCLIP2 Pretrained Checkpoints

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Downloads pretrained checkpoints for various MobileCLIP2 models using the 'hf download' command. Models range from S0 to L-14.

```bash
# MobileCLIP2
for model in S0 S2 B S3 L-14 S4
do
  hf download apple/MobileCLIP2-$model
done
```

--------------------------------

### Types and Configuration

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

Documentation on configuration structures, tensor shapes, return types, and checkpoint formats.

```APIDOC
## Types and Configuration Documentation

### Description
Details the various data structures, type conventions, and configuration formats used throughout the MobileCLIP library. This includes specifications for configuration dictionaries, tensor shapes, forward pass return types, and the format of checkpoint and state dictionaries.

### Topics
- Configuration dictionary structures.
- Tensor shapes and data conventions.
- Forward pass return types.
- Checkpoint and state dictionary formats.
```

--------------------------------

### Save and Load PyTorch Model State Dictionary

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/configuration.md

Demonstrates how to save the model's parameters to a state dictionary file and how to load them back into a model. This is crucial for resuming training or deploying pre-trained models.

```python
import torch

# Save
torch.save(model.state_dict(), "checkpoint.pt")

# Load
checkpoint = torch.load("checkpoint.pt")
model.load_state_dict(checkpoint)
```

--------------------------------

### Typical Usage of MobileCLIP Logger

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/logger-utilities.md

Demonstrates common logging patterns including printing headers, informational messages, warnings, progress updates, and error handling. Also shows how to disable and re-enable printing.

```python
from mobileclip import logger

# Print section header
logger.print_header("MobileCLIP Inference")

# Log informational messages
logger.info(f"Loading model from checkpoint")
logger.debug(f"Configuration: {config}")

# Issue warning if needed
if batch_size > 256:
    logger.warning(f"Large batch size {batch_size} may cause OOM")

# Progress logging with custom line ending
for i in range(num_batches):
    logger.log(f"Processing batch {i+1}/{num_batches}", end="\r")

logger.log("Done!")

# Error handling
try:
    result = process_data(data)
except ValueError as e:
    logger.error(f"Invalid data format: {e}")

# Suppress output for clean scripts
logger.disable_printing()
result = silently_run_function()
logger.enable_printing()
logger.info(f"Result: {result}")
```

--------------------------------

### Handling SystemExit from MobileCLIP Error()

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/logger-utilities.md

Shows how to catch `SystemExit` exceptions raised by the `logger.error()` function when a model fails to load due to invalid parameters.

```python
try:
    import mobileclip
    model = mobileclip.create_model_and_transforms('invalid_model')
except SystemExit as e:
    print(f"Model loading failed: {e}")
```

--------------------------------

### RepMixerBlock Constructor Parameters

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/repmixer-blocks.md

Defines the parameters for initializing a RepMixerBlock. Key parameters include embedding dimension, kernel size for token mixing, MLP ratio, activation layer, dropout rates, and layer scaling options.

```python
def __init__(
    self,
    dim: int,
    kernel_size: int = 11,
    mlp_ratio: float = 4.0,
    act_layer: nn.Module = nn.GELU,
    drop: float = 0.0,
    drop_path: float = 0.0,
    use_layer_scale: bool = True,
    layer_scale_init_value: float = 1e-5,
    inference_mode: bool = False,
    *args,
    **kwargs,
)
```

--------------------------------

### LayerNormFP32 Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Initializes LayerNormFP32 with standard LayerNorm parameters and optional epsilon and affine settings.

```python
def __init__(
    self,
    normalized_shape: Union[int, List[int], torch.Size],
    eps: Optional[float] = 1e-5,
    elementwise_affine: Optional[bool] = True,
    *args,
    **kwargs,
)
```

--------------------------------

### CLIP Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/clip-model.md

Initializes the CLIP model. It takes a configuration dictionary and an optional boolean to control the output format of the forward pass.

```APIDOC
## CLIP Constructor

### Description
Initializes the CLIP model with specified configuration and output format.

### Method Signature
`__init__(self, cfg: Dict, output_dict: bool = False, *args, **kwargs) -> None`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **cfg** (Dict) - Required - Configuration dictionary containing `embed_dim`, `image_cfg`, and `text_cfg`.
- **output_dict** (bool) - Optional - If True, `forward()` returns a dictionary. Defaults to False, returning a tuple.

### Raises
- **ValueError**: If `embed_dim` is missing or None in the config.
```

--------------------------------

### Package Entry Point Functions

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

These are the primary functions exposed by the MobileCLIP package for public use. They handle model creation, transformations, and tokenizer retrieval.

```APIDOC
## create_model_and_transforms

### Description
Creates a CLIP model instance along with associated transformations and tokenizer. This is the recommended way to instantiate models.

### Parameters
- **model_name** (str) - Required - The name of the model to create.
- **pretrained** (Optional[str]) - Optional - Path to pretrained weights.
- **reparameterize** (Optional[bool]) - Optional - Whether to reparameterize the model for inference. Defaults to True.
- **device** (Union[str, torch.device]) - Optional - The device to load the model on. Defaults to "cpu".

### Returns
- **Tuple[nn.Module, Any, Any]** - A tuple containing the model, image transformations, and text transformations.
```

```APIDOC
## get_tokenizer

### Description
Retrieves the tokenizer associated with a given model name.

### Parameters
- **model_name** (str) - Required - The name of the model for which to get the tokenizer.

### Returns
- **nn.Module** - The tokenizer module.
```

--------------------------------

### Custom Tokenization with MobileCLIP Tokenizer

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/INDEX.md

Demonstrates how to use the MobileCLIP tokenizer for single texts and batches, and how to access vocabulary size and EOT token. The tokenizer is obtained using 'mobileclip.get_tokenizer'.

```python
tokenizer = mobileclip.get_tokenizer('mobileclip_s0')

# Single text
tokens = tokenizer("a dog running in park")
print(tokens.shape)  # [77]

# Batch
texts = ["a dog", "a cat", "a bird"]
batch_tokens = tokenizer(texts)
print(batch_tokens.shape)  # [3, 77]

# Metadata
print(f"Vocab size: {tokenizer.get_vocab_size()}")
print(f"EOT token: {tokenizer.get_eot_token()}")
```

--------------------------------

### forward Method

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/clip-model.md

Performs the main forward pass of the CLIP model, supporting image, text, or both inputs. The return format depends on the `output_dict` initialization parameter.

```APIDOC
## forward Method

### Description
Main forward pass supporting image, text, or both inputs. Returns either a dictionary or a tuple based on `output_dict` setting.

### Method Signature
`forward(self, image: Optional[torch.Tensor] = None, text: Optional[torch.Tensor] = None, *args, **kwargs) -> Any`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **image** (Optional[torch.Tensor]) - Optional - Image tensor [batch_size, 3, H, W]. If provided, it will be encoded and normalized.
- **text** (Optional[torch.Tensor]) - Optional - Text token tensor [batch_size, context_length]. If provided, it will be encoded and normalized.

### Returns
- **Dict or Tuple**: If `output_dict` is True, returns a dictionary with keys `image_features`, `text_features`, and `logit_scale`. If `output_dict` is False, returns a tuple `(image_features, text_features, logit_scale)`.
```

--------------------------------

### TextTransformer Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Initializes the TextTransformer module. Requires a configuration dictionary and the desired projection dimension.

```python
def __init__(self, cfg: dict, projection_dim: int, *args, **kwargs) -> None

```

--------------------------------

### Import Public API for MobileCLIP

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

Recommended imports for primary entry points and direct model access in MobileCLIP.

```python
import mobileclip

# Primary entry points
model, _, preprocess = mobileclip.create_model_and_transforms(...)
tokenizer = mobileclip.get_tokenizer(...)

# Direct model access if needed
from mobileclip.clip import CLIP
```

--------------------------------

### Optimize MobileCLIP Model for Deployment

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/README.md

Converts a multi-branch training structure to a single-branch inference structure for faster execution and reduced parameter count. This optimization is typically done before deployment.

```python
from mobileclip.modules.common.mobileone import reparameterize_model

# Convert multi-branch training structure to single-branch inference
model = reparameterize_model(model)
model.eval()

# Now inference is faster with fewer parameters
with torch.no_grad():
    features = model.encode_image(images)
```

--------------------------------

### Create Model and Transforms

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/MODULES.md

Primary function to instantiate the CLIP model and associated transformations. Use this for all model creation.

```python
def create_model_and_transforms(
    model_name: str,
    pretrained: Optional[str] = None,
    reparameterize: Optional[bool] = True,
    device: Union[str, torch.device] = "cpu",
) -> Tuple[nn.Module, Any, Any]
```

--------------------------------

### TextTransformer Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/text-encoder.md

Initializes the TextTransformer model. It takes a configuration dictionary and the desired output projection dimension.

```APIDOC
## TextTransformer Constructor

### Description
Initializes the TextTransformer model. It takes a configuration dictionary and the desired output projection dimension.

### Method Signature
```python
def __init__(self, cfg: dict, projection_dim: int, *args, **kwargs) -> None
```

### Parameters
#### Parameters
- **cfg** (dict) - Required - Configuration dictionary with keys: `dim`, `norm_layer`, `context_length`, `vocab_size`, `model_name`, `n_transformer_layers`, `n_heads_per_layer`, `ffn_multiplier_per_layer`, `causal_masking`, and optional: `no_scale_embedding`, `no_pos_embedding`, `embed_dropout`.
- **projection_dim** (int) - Required - Output embedding dimension.

#### Config Dictionary Keys
- **dim** (int) - Required - Hidden dimension of transformer.
- **context_length** (int) - Required - Maximum sequence length in tokens.
- **vocab_size** (int) - Required - Size of tokenizer vocabulary.
- **n_transformer_layers** (int) - Required - Number of transformer encoder layers.
- **n_heads_per_layer** (int or list) - Required - Number of attention heads per layer. If int, same for all layers.
- **ffn_multiplier_per_layer** (float or list) - Required - FFN hidden dimension multiplier per layer.
- **norm_layer** (str) - Required - Normalization layer type: `layer_norm` or `layer_norm_fp32`.
- **model_name** (str) - Required - Variant: `base` (standard transformer) or `mct` (with RepMixer blocks).
- **causal_masking** (bool) - Required - If True, applies causal mask for autoregressive attention.
- **no_scale_embedding** (bool) - Optional - If True, disables embedding scale factor. Default: False.
- **no_pos_embedding** (bool) - Optional - If True, disables positional embeddings. Default: False.
- **embed_dropout** (float) - Optional - Dropout rate for embeddings. Default: 0.0.
```

--------------------------------

### Handle Model Configuration File Not Found

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/errors.md

Raises a ValueError if the model configuration file does not exist. Ensure the model name corresponds to an existing config file in the `mobileclip/configs/` directory.

```python
if not os.path.exists(model_cfg_file):
    raise ValueError(f"Unsupported model name: {model_name}")
```

--------------------------------

### create_model_and_transforms

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/mobileclip-initialization.md

Instantiates a MobileCLIP model and its associated preprocessing transforms. This is the main function for loading a model into your application.

```APIDOC
## create_model_and_transforms

### Description
Method to instantiate a MobileCLIP model and preprocessing transforms necessary for inference. This is the primary entry point for loading a model from the library.

### Method Signature
```python
def create_model_and_transforms(
    model_name: str,
    pretrained: Optional[str] = None,
    reparameterize: Optional[bool] = True,
    device: Union[str, torch.device] = "cpu",
) -> Tuple[nn.Module, Any, Any]
```

### Parameters
#### Path Parameters
- **model_name** (str) - Required - Model identifier. Choose from: `mobileclip_s0`, `mobileclip_s1`, `mobileclip_s2`, `mobileclip_b`. Configuration is loaded from JSON config file.
- **pretrained** (Optional[str]) - Optional - Path to pretrained checkpoint file (PyTorch .pt file). If None, model is initialized with random weights.
- **reparameterize** (Optional[bool]) - Optional - If True, reparameterizable branches are folded for faster inference by calling `reparameterize_model()`. Recommended for deployment. Default: True.
- **device** (Union[str, torch.device]) - Optional - Device identifier for model placement (e.g., "cpu", "cuda", "cuda:0"). Model is moved to this device and set to eval mode. Default: "cpu".

### Return Value
Returns a tuple of three elements:
- **model** (`nn.Module`): Instantiated CLIP model with image and text encoders. Always in eval mode.
- **None** (`Any`): Reserved for future use (always None in current implementation).
- **preprocess** (`Compose`): torchvision Compose object containing image preprocessing transforms: Resize, CenterCrop, and ToTensor. Image resolution matches the model's configured `image_size`.

### Behavior
1. Loads model configuration from `mobileclip/configs/{model_name}.json`
2. Builds preprocessing pipeline based on configured image size
3. Creates CLIP model instance with image and text encoders
4. Moves model to specified device and sets to eval mode
5. Loads checkpoint weights if pretrained path is provided
6. Optionally reparameterizes model by folding training-time branches into single-branch structure

### Exceptions
- **ValueError**: Raised if `model_name` is not found in configs directory or if `embed_dim` is not specified in model config.

### Example
```python
import torch
from PIL import Image
import mobileclip

# Create model and transforms
model, _, preprocess = mobileclip.create_model_and_transforms(
    model_name='mobileclip_s0',
    pretrained='/path/to/mobileclip_s0.pt',
    reparameterize=True,
    device='cuda' if torch.cuda.is_available() else 'cpu'
)

# Prepare image
image = preprocess(Image.open("photo.jpg").convert("RGB")).unsqueeze(0)

# Encode image
with torch.no_grad():
    image_features = model.encode_image(image)
    image_features /= image_features.norm(dim='-1', keepdim=True)
```
```

--------------------------------

### MCi Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/image-encoder.md

Initializes the MCi image encoder. Use 'projection_dim' to set the output embedding dimension.

```python
def __init__(self, model_name: str, *args, **kwargs) -> None
```

--------------------------------

### Train MobileCLIP-B on DFN-2B12M with DR

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This command initiates the training of a MobileCLIP-B model on the DFN-2B12M dataset, incorporating DR (likely referring to a specific training objective or data augmentation).

```bash
cd open_clip/
bash configs/run_dfndr12m.sh  # Train a MobileCILP-B on DFN-2B12M with DR

```

--------------------------------

### ConvFFN Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/repmixer-blocks.md

Initializes the ConvFFN module. Specify input/output channels, context size, hidden channels, activation layer, and dropout rate.

```python
def __init__(
    self,
    in_channels: int,
    context_size: int,
    hidden_channels: Optional[int] = None,
    out_channels: Optional[int] = None,
    act_layer: nn.Module = nn.GELU,
    drop: float = 0.0,
) -> None:
```

--------------------------------

### Run Zero-Shot Evaluation on ImageNet-1k

Source: https://github.com/apple/ml-mobileclip/blob/main/README.md

Use this script to perform zero-shot evaluation on the ImageNet-1k dataset with a specified model architecture and path. Requires a single GPU.

```bash
python eval/zeroshot_imagenet.py --model-arch mobileclip_s0 --model-path /path/to/mobileclip_s0.pt
```

--------------------------------

### MobileOneBlock Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/model-reparameterization.md

Initializes the MobileOneBlock with various parameters controlling its architecture for training and inference modes.

```python
def __init__(
    self,
    in_channels: int,
    out_channels: int,
    kernel_size: int,
    stride: int = 1,
    padding: int = 0,
    dilation: int = 1,
    groups: int = 1,
    inference_mode: bool = False,
    use_se: bool = False,
    use_act: bool = True,
    use_scale_branch: bool = True,
    num_conv_branches: int = 1,
    activation: nn.Module = nn.GELU(),
) -> None
```

--------------------------------

### Train ViT-B/16 on DataComp-12M without DR

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This script executes the training of a ViT-B/16 model on the DataComp-12M dataset, without the DR (Data Representation) component.

```bash
cd open_clip/
bash configs/run_datacomp12m.sh  # Train a ViT-B/16 on DataComp-12M without DR

```

--------------------------------

### GlobalPool2D Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/image-encoder.md

Initializes the GlobalPool2D module with input and output dimensions. It sets up the projection matrix and the global pooling module.

```python
def __init__(self, in_dim: int, out_dim: int, *args, **kwargs) -> None

```

--------------------------------

### PositionalEmbedding Constructor

Source: https://github.com/apple/ml-mobileclip/blob/main/_autodocs/api-reference/transformer-layers.md

Initializes the PositionalEmbedding module. Use this to set the maximum sequence length, embedding dimension, and optional padding index.

```python
def __init__(
    self,
    num_embeddings: int,
    embedding_dim: int,
    padding_idx: Optional[int] = None,
    is_learnable: Optional[bool] = False,
    interpolation_mode: Optional[str] = "bilinear",
    *args,
    **kwargs,
)
```

--------------------------------

### Train ViT-B/16 on DataComp-1B with DR

Source: https://github.com/apple/ml-mobileclip/blob/main/training/README.md

This script is used to train a ViT-B/16 model on the DataComp-1B dataset, including DR-specific training configurations.

```bash
cd open_clip/
bash configs/run_datacompdr1B.sh  # Train a ViT-B/16 on DataComp-1B with DR

```