### Initialize Project Environment

Source: https://github.com/pbonazzi/picosam3/blob/main/README.md

Run the automated setup script to download datasets, checkpoints, and install dependencies.

```bash
./init.sh
```

--------------------------------

### Start Services with Docker

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Build and start the frontend and backend services using Docker Compose.

```bash
docker compose up --build
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Install the interactive demo dependencies in editable mode.

```bash
pip install -e '.[interactive-demo]'
```

--------------------------------

### Install SAM 2 with Notebook Support

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Install SAM 2 and its notebook dependencies from the root of the repository. This command installs the package in editable mode.

```bash
pip install -e ".[notebooks]"
```

--------------------------------

### Install Yarn

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Global installation of the Yarn package manager.

```bash
npm install -g yarn
```

--------------------------------

### Install Frontend Dependencies

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Install project dependencies using Yarn.

```bash
yarn install
```

--------------------------------

### Start Frontend Development Server

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Launch the frontend development server on port 7262.

```bash
yarn dev --port 7262
```

--------------------------------

### Install dependencies and download assets

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Install required packages and download model checkpoints and sample videos if running in a Colab environment.

```python
if using_colab:
    import torch
    import torchvision
    print("PyTorch version:", torch.__version__)
    print("Torchvision version:", torchvision.__version__)
    print("CUDA is available:", torch.cuda.is_available())
    import sys
    !{sys.executable} -m pip install opencv-python matplotlib
    !{sys.executable} -m pip install 'git+https://github.com/facebookresearch/sam2.git'

    !mkdir -p videos
    !wget -P videos https://dl.fbaipublicfiles.com/segment_anything_2/assets/bedroom.zip
    !unzip -d videos videos/bedroom.zip

    !mkdir -p ../checkpoints/
    !wget -P ../checkpoints/ https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt
```

--------------------------------

### Start Backend with MPS

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Launch the Gunicorn server with environment variables configured for MPS support.

```bash
PYTORCH_ENABLE_MPS_FALLBACK=1 \
APP_ROOT="$(pwd)/../../../" \
API_URL=http://localhost:7263 \
MODEL_SIZE=base_plus \
DATA_PATH="$(pwd)/../../data" \
DEFAULT_VIDEO_PATH=gallery/05_default_juggle.mp4 \
gunicorn \
    --worker-class gthread app:app \
    --workers 1 \
    --threads 2 \
    --bind 0.0.0.0:7263 \
    --timeout 60
```

--------------------------------

### Install Evaluation Dependencies

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/README.md

Command to install the required dependencies for the evaluation script.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Dependencies and Download Data for Colab

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/image_predictor_example.ipynb

Installs necessary libraries, downloads sample images, and fetches the SAM 2 model checkpoint if running in Colab. Requires GPU.

```python
if using_colab:
    import torch
    import torchvision
    print("PyTorch version:", torch.__version__)
    print("Torchvision version:", torchvision.__version__)
    print("CUDA is available:", torch.cuda.is_available())
    import sys
    !{sys.executable} -m pip install opencv-python matplotlib
    !{sys.executable} -m pip install 'git+https://github.com/facebookresearch/sam2.git'

    !mkdir -p images
    !wget -P images https://raw.githubusercontent.com/facebookresearch/sam2/main/notebooks/images/truck.jpg
    !wget -P images https://raw.githubusercontent.com/facebookresearch/sam2/main/notebooks/images/groceries.jpg

    !mkdir -p ../checkpoints/
    !wget -P ../checkpoints/ https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt
```

--------------------------------

### Install FFmpeg

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Install the FFmpeg dependency via Conda.

```bash
conda install -c conda-forge ffmpeg
```

--------------------------------

### Knowledge Distillation Training Setup

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Imports necessary components for knowledge distillation training, including models, dataset utilities, and loss functions. This snippet sets up the environment for training a PicoSAM student model using a SAM teacher.

```python
import torch
from torch.utils.data import DataLoader, random_split
from model_compression.model import PicoSAM3
from model_compression.dataset import PicoSAMDataset, custom_collate
from model_compression.utils import mse_dice_loss, bce_dice_loss, area_loss, compute_iou
```

--------------------------------

### Install dependencies and download assets

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Install required libraries and download model checkpoints and sample images when running in a Colab environment.

```python
if using_colab:
    import torch
    import torchvision
    print("PyTorch version:", torch.__version__)
    print("Torchvision version:", torchvision.__version__)
    print("CUDA is available:", torch.cuda.is_available())
    import sys
    !{sys.executable} -m pip install opencv-python matplotlib
    !{sys.executable} -m pip install 'git+https://github.com/facebookresearch/sam2.git'

    !mkdir -p images
    !wget -P images https://raw.githubusercontent.com/facebookresearch/sam2/main/notebooks/images/cars.jpg

    !mkdir -p ../checkpoints/
    !wget -P ../checkpoints/ https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt
```

--------------------------------

### Install with No Build Isolation

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Use this flag if standard installation fails despite correct CUDA configuration.

```bash
pip install --no-build-isolation -e .
```

--------------------------------

### Verify CUDA Setup

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Check if PyTorch can access CUDA and verify the CUDA_HOME directory.

```bash
python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'
```

--------------------------------

### Install SAM 2 Skipping CUDA Extension

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Install SAM 2 while explicitly skipping the build of the CUDA extension by setting the SAM2_BUILD_CUDA environment variable to 0. This is useful if a GPU is not available or the extension is not needed.

```bash
SAM2_BUILD_CUDA=0 pip install -e ".[notebooks]"
```

--------------------------------

### Verify SAM 2 Installation Path

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Check the local file path of the SAM 2 base module to ensure the latest version is being used.

```python
from sam2.modeling import sam2_base

print(sam2_base.__file__)
```

--------------------------------

### Configure CUDA_HOME Environment Variable

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Explicitly set the CUDA_HOME path if the installation fails to locate the CUDA toolkit.

```bash
export CUDA_HOME=/usr/local/cuda  # change to your CUDA toolkit path
```

--------------------------------

### Batched Inference with Point Prompts

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/image_predictor_example.ipynb

Process batches of point prompts across multiple images for segmentation. This example demonstrates selecting the best mask per object using scores.

```python
image1 = image  # truck.jpg from above
image1_pts = np.array([
    [[500, 375]],
    [[650, 750]]
    ]) # Bx1x2 where B corresponds to number of objects 
image1_labels = np.array([[1], [1]])

image2_pts = np.array([
    [[400, 300]],
    [[630, 300]],
])
image2_labels = np.array([[1], [1]])

pts_batch = [image1_pts, image2_pts]
labels_batch = [image1_labels, image2_labels]
```

```python
masks_batch, scores_batch, _ = predictor.predict_batch(pts_batch, labels_batch, box_batch=None, multimask_output=True)

# Select the best single mask per object
best_masks = []
for masks, scores in zip(masks_batch,scores_batch):
    best_masks.append(masks[range(len(masks)), np.argmax(scores, axis=-1)])
```

```python
for image, points, labels, masks in zip(img_batch, pts_batch, labels_batch, best_masks):
    plt.figure(figsize=(10, 10))
    plt.imshow(image)   
    for mask in masks:
        show_mask(mask, plt.gca(), random_color=True)
    show_points(points, labels, plt.gca())
```

--------------------------------

### Navigate to Frontend Directory

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Change the working directory to the frontend location.

```bash
cd demo/frontend
```

--------------------------------

### Import libraries and configure environment

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Import necessary libraries and set environment variables for PyTorch compatibility.

```python
import os
# if using Apple MPS, fall back to CPU for unsupported ops
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
import numpy as np
import torch
import matplotlib.pyplot as plt
from PIL import Image
```

--------------------------------

### Navigate to Backend Directory

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Change the working directory to the backend server location.

```bash
cd demo/backend/server/
```

--------------------------------

### Initialize SAM 2 video predictor

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Load the model configuration and checkpoint to instantiate the video predictor.

```python
from sam2.build_sam import build_sam2_video_predictor

sam2_checkpoint = "../checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"

predictor = build_sam2_video_predictor(model_cfg, sam2_checkpoint, device=device)
```

--------------------------------

### Configure and Train PicoSAM3 Model

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Initializes the student model, optimizer, and training loop with weighted loss based on teacher confidence.

```python
IMAGE_SIZE = 96
BATCH_SIZE = 64
LEARNING_RATE = 3e-4
NUM_EPOCHS = 1

# Initialize model and optimizer
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
student_model = PicoSAM3().to(device)
optimizer = torch.optim.AdamW(student_model.parameters(), lr=LEARNING_RATE)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lambda step: min(1.0, step / 1000))

# Load dataset with pre-computed teacher logits
dataset = PicoSAMDataset(
    image_root="dataset/train2017",
    annotation_file="dataset/annotations/instances_train2017.json",
    image_size=IMAGE_SIZE,
    cache_dir="dataset/teacher_sam3_logits",
    require_cache=True
)

# Split into train/val
train_size = len(dataset) - len(dataset) // 20
train_set, val_set = random_split(dataset, [train_size, len(dataset) - train_size])
train_loader = DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True, collate_fn=custom_collate)

# Training loop
student_model.train()
for images, gt_masks, prompts, _, teacher_logits, teacher_scores in train_loader:
    images = images.to(device)
    gt_masks = gt_masks.to(device)
    teacher_logits = teacher_logits.to(device)
    teacher_scores = teacher_scores.to(device)

    # Forward pass
    pred_masks = student_model(images)

    # Compute losses
    confidence = teacher_scores.clamp(0.0, 1.0)
    loss_teacher = mse_dice_loss(pred_masks, teacher_logits)
    loss_gt = bce_dice_loss(pred_masks, gt_masks)
    loss_area = area_loss(pred_masks, gt_masks)

    # Weighted combination based on teacher confidence
    alpha = confidence.mean()
    loss = alpha * loss_teacher + (1 - alpha) * loss_gt + 0.4 * loss_area

    # Backward pass
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    scheduler.step()

    # Compute metrics
    batch_iou = compute_iou(pred_masks, gt_masks)
    print(f"Loss: {loss.item():.4f}, mIoU: {batch_iou:.4f}")
    break

# Save trained model
torch.save(student_model.state_dict(), "checkpoints/PicoSAM3_student.pt")
```

--------------------------------

### Reset Picosam3 Inference State

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Resets the current inference state. This is useful if you want to start a new segmentation or tracking session with the same initialized state.

```python
predictor.reset_state(inference_state)
```

--------------------------------

### Initialize SAM 2 Model and Predictor

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/image_predictor_example.ipynb

Load the SAM 2 model and instantiate the image predictor. Ensure the checkpoint path and configuration file are correctly specified.

```python
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

sam2_checkpoint = "../checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"

sam2_model = build_sam2(model_cfg, sam2_checkpoint, device=device)

predictor = SAM2ImagePredictor(sam2_model)
```

--------------------------------

### Initialize and Run PicoSAM2 Model

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Demonstrates initializing the PicoSAM2 model and performing inference on a sample input tensor. The output is a segmentation mask logits, which can be converted to probabilities and a binary mask.

```python
import torch
from model_compression.model import PicoSAM2

# Initialize the PicoSAM2 model
model = PicoSAM2(in_channels=3)
model.eval()

# Create sample input (batch_size=1, channels=3, height=96, width=96)
input_tensor = torch.randn(1, 3, 96, 96)

# Run inference
with torch.no_grad():
    segmentation_mask = model(input_tensor)
    # Output shape: (1, 1, 96, 96) - single channel mask logits

# Apply sigmoid to get probabilities
mask_probs = torch.sigmoid(segmentation_mask)
binary_mask = (mask_probs > 0.5).float()

print(f"Input shape: {input_tensor.shape}")
print(f"Output shape: {segmentation_mask.shape}")
print(f"Binary mask unique values: {binary_mask.unique()}")
```

--------------------------------

### Initialize SAM 2 Model

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Builds the SAM 2 model instance using a specified configuration and checkpoint file.

```python
from sam2.build_sam import build_sam2
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

sam2_checkpoint = "../checkpoints/sam2.1_hiera_large.pt"
model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"

sam2 = build_sam2(model_cfg, sam2_checkpoint, device=device, apply_postprocessing=False)

mask_generator = SAM2AutomaticMaskGenerator(sam2)
```

--------------------------------

### Download Checkpoints

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Download the required SAM 2 model checkpoints.

```bash
(cd ./checkpoints && ./download_ckpts.sh)
```

--------------------------------

### Initialize and Run PicoSAM3 Model

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Shows how to initialize the PicoSAM3 model, which includes an Efficient Channel Attention (ECA) block and dilated convolutions. Inference is performed similarly to PicoSAM2.

```python
import torch
from model_compression.model import PicoSAM3

# Initialize PicoSAM3 with enhanced attention mechanism
model = PicoSAM3(in_channels=3)
model.eval()

# PicoSAM3 has the same interface as PicoSAM2
input_tensor = torch.randn(1, 3, 96, 96)

with torch.no_grad():
    segmentation_mask = model(input_tensor)

# The ECA block applies channel attention before final refinement
# Architecture: Encoder -> Bottleneck (with dilation) -> Decoder -> ECA -> Refine
mask_probs = torch.sigmoid(segmentation_mask)
binary_mask = (mask_probs > 0.5).float()

print(f"PicoSAM3 output shape: {segmentation_mask.shape}")
```

--------------------------------

### Create Conda Environment

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Initialize a new Conda environment with Python 3.10.

```bash
conda create --name sam2-demo python=3.10 --yes
```

--------------------------------

### Run Evaluation Script

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/README.md

Commands to execute the evaluation script or display help information.

```bash
python sav_evaluator.py --gt_root {GT_ROOT} --pred_root {PRED_ROOT}
```

```bash
python sav_evaluator.py --help
```

--------------------------------

### Select computation device and optimize settings

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Detect available hardware (CUDA, MPS, or CPU) and apply performance optimizations like bfloat16 and TF32.

```python
# select the device for computation
if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")
print(f"using device: {device}")

if device.type == "cuda":
    # use bfloat16 for the entire notebook
    torch.autocast("cuda", dtype=torch.bfloat16).__enter__()
    # turn on tfloat32 for Ampere GPUs (https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices)
    if torch.cuda.get_device_properties(0).major >= 8:
        torch.backends.cuda.matmul.allow_tf32 = True
        torch.backends.cudnn.allow_tf32 = True
elif device.type == "mps":
    print(
        "\nSupport for MPS devices is preliminary. SAM 2 is trained with CUDA and might "
        "give numerically different outputs and sometimes degraded performance on MPS. "
        "See e.g. https://github.com/pytorch/pytorch/issues/84936 for a discussion."
    )
```

--------------------------------

### Deploy PicoSAM on IMX500 AI Camera for Real-time Segmentation

Source: https://context7.com/pbonazzi/picosam3/llms.txt

This script deploys a quantized PicoSAM model on a Raspberry Pi with an IMX500 AI camera, enabling real-time interactive ROI selection and segmentation visualization. Ensure the model path and display settings are correctly configured.

```python
import time
import cv2
import numpy as np
from picamera2 import Picamera2
from picamera2.devices import IMX500
from picam2.devices.imx500 import NetworkIntrinsics

# Configuration
MODEL_PATH = "/usr/share/imx500-models/picosam2.rpk"
DISPLAY_SIZE = (640, 480)
SENSOR_W, SENSOR_H = 4056, 3040

# Initialize IMX500 with PicoSAM model
imx500 = IMX500(MODEL_PATH)
intrinsics = imx500.network_intrinsics or NetworkIntrinsics()
intrinsics.task = "segmentation"
intrinsics.update_with_defaults()

# Initialize camera
picam2 = Picamera2(imx500.camera_num)
picam2.start(
    picam2.create_preview_configuration(
        controls={"FrameRate": intrinsics.inference_rate},
        buffer_count=8
    ),
    show_preview=False
)

# ROI state (initially full sensor)
roi_x, roi_y = 0, 0
roi_w, roi_h = SENSOR_W, SENSOR_H

# Set initial inference ROI
imx500.set_inference_roi_abs((roi_x, roi_y, roi_w, roi_h))

# Callback for processing inference results
def segmentation_callback(request):
    outputs = imx500.get_outputs(request.get_metadata())
    if outputs is None:
        return

    # Get binary segmentation mask
    mask = (outputs[0][0] > 0).astype(np.uint8) * 255

    # Get camera frame
    frame = request.make_array("main")
    frame = cv2.cvtColor(frame, cv2.COLOR_RGBA2BGR)
    frame = cv2.resize(frame, DISPLAY_SIZE)

    # Get scaled ROI coordinates
    rx, ry, rw, rh = imx500.get_roi_scaled(request)

    # Overlay mask on frame
    roi_mask = cv2.resize(mask, (rw, rh), interpolation=cv2.INTER_NEAREST)
    frame[ry:ry+rh, rx:rx+rw, 2] = np.clip(
        frame[ry:ry+rh, rx:rx+rw, 2].astype(np.float32) + roi_mask * 0.5,
        0, 255
    ).astype(np.uint8)

    # Draw ROI rectangle
    cv2.rectangle(frame, (rx, ry), (rx+rw, ry+rh), (0, 255, 0), 2)
    cv2.imshow("IMX500 Segmentation", frame)

picam2.pre_callback = segmentation_callback

# Main loop
cv2.namedWindow("IMX500 Segmentation")
try:
    while cv2.waitKey(1) != 27:  # ESC to exit
        time.sleep(0.01)
finally:
    picam2.stop()
    cv2.destroyAllWindows()
```

--------------------------------

### Train PicoSAM Student Models via Knowledge Distillation

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Train PicoSAM2 or PicoSAM3 student models using cached outputs from a teacher model. This script facilitates the knowledge distillation process.

```python
model_distillation.py
```

--------------------------------

### Configure MOSE Dataset Paths

Source: https://github.com/pbonazzi/picosam3/blob/main/training/README.md

Set the paths for the MOSE dataset in the configuration file. Ensure the img_folder and gt_folder point to the correct locations. The file_list_txt is optional for specifying a subset of videos.

```yaml
dataset:
    # PATHS to Dataset
    img_folder: null # PATH to MOSE JPEGImages folder
    gt_folder: null # PATH to MOSE Annotations folder
    file_list_txt: null # Optional PATH to filelist containing a subset of videos to be used for training
```

--------------------------------

### Fine-tune SAM 2 on MOSE (Single Node)

Source: https://github.com/pbonazzi/picosam3/blob/main/training/README.md

Launch the training script to fine-tune the base SAM 2 model on the MOSE dataset using a specified number of GPUs. The --use-cluster flag should be set to 0 for single-node training.

```python
python training/train.py \
    -c configs/sam2.1_training/sam2.1_hiera_b+_MOSE_finetune.yaml \
    --use-cluster 0 \
    --num-gpus 8
```

--------------------------------

### Load video frames and annotations

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/sav_visualization_example.ipynb

Initializes the SAVDataset and retrieves frames and both manual and automatic annotations for a specific video.

```python
sav_dataset = SAVDataset(sav_dir="example/")
frames, manual_annot, auto_annot = sav_dataset.get_frames_and_annotations("sav_000001")
```

--------------------------------

### Build SAM2 from Hugging Face

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Loads pre-trained SAM 2.1 models and initializes predictors directly from the Hugging Face Hub.

```python
from sam2.build_sam import build_sam2_hf
from sam2.sam2_image_predictor import SAM2ImagePredictor
from sam2.sam2_video_predictor import SAM2VideoPredictor

# Available Hugging Face model IDs
HF_MODELS = [
    "facebook/sam2.1-hiera-tiny",
    "facebook/sam2.1-hiera-small",
    "facebook/sam2.1-hiera-base-plus",
    "facebook/sam2.1-hiera-large",
]

# Load model directly from Hugging Face
model = build_sam2_hf(
    model_id="facebook/sam2.1-hiera-tiny",
    device="cuda",
    mode="eval"
)

# Create image predictor
predictor = SAM2ImagePredictor(model)

# Or use the from_pretrained class method
predictor = SAM2ImagePredictor.from_pretrained(
    model_id="facebook/sam2.1-hiera-tiny",
    device="cuda"
)

# For video prediction
video_predictor = SAM2VideoPredictor.from_pretrained(
    model_id="facebook/sam2.1-hiera-large",
    device="cuda"
)
```

--------------------------------

### Build and Use SAM2 Video Predictor

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Initializes and uses the SAM2VideoPredictor for tracking objects across video frames. Supports initialization with points or boxes on key frames and propagation throughout the video. Requires model and config files.

```python
import torch
from sam2.build_sam import build_sam2_video_predictor

# Build video predictor
predictor = build_sam2_video_predictor(
    config_file="configs/sam2.1/sam2.1_hiera_l.yaml",
    ckpt_path="checkpoints/sam2.1_hiera_large.pt",
    device="cuda",
    mode="eval"
)

# Initialize inference state with video
inference_state = predictor.init_state(
    video_path="video_frames/",  # Directory with JPEG frames
    offload_video_to_cpu=True,   # Save GPU memory
    offload_state_to_cpu=False
)

# Add object with point prompt on frame 0
frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=0,
    obj_id=1,
    points=[[500, 375]],
    labels=[1],  # 1 = foreground
    clear_old_points=True
)

# Or add object with box prompt
frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=0,
    obj_id=2,
    box=[100, 100, 400, 300],  # [x1, y1, x2, y2]
    clear_old_points=True
)

# Propagate masks forward through video
for frame_idx, obj_ids, video_res_masks in predictor.propagate_in_video(
    inference_state=inference_state,
    start_frame_idx=0,
    max_frame_num_to_track=None,  # Track all frames
    reverse=False
):
    print(f"Frame {frame_idx}: {len(obj_ids)} objects tracked")
    # video_res_masks shape: (num_objects, 1, H, W)
    for i, obj_id in enumerate(obj_ids):
        mask = (video_res_masks[i, 0] > 0).cpu().numpy()
        # Save or process mask...

# Reset state for new video
predictor.reset_state(inference_state)
```

--------------------------------

### Activate Conda Environment

Source: https://github.com/pbonazzi/picosam3/blob/main/demo/README.md

Switch to the project-specific Conda environment.

```bash
conda activate sam2-demo
```

--------------------------------

### Configure Advanced Mask Generation

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Demonstrates how to tune parameters for denser mask sampling and improved quality filtering.

```python
mask_generator_2 = SAM2AutomaticMaskGenerator(
    model=sam2,
    points_per_side=64,
    points_per_batch=128,
    pred_iou_thresh=0.7,
    stability_score_thresh=0.92,
    stability_score_offset=0.7,
    crop_n_layers=1,
    box_nms_thresh=0.7,
    crop_n_points_downscale_factor=2,
    min_mask_region_area=25.0,
    use_m2m=True,
)
```

```python
masks2 = mask_generator_2.generate(image)
```

```python
plt.figure(figsize=(20, 20))
plt.imshow(image)
show_anns(masks2)
plt.axis('off')
plt.show()
```

--------------------------------

### Execute Model Compression Scripts

Source: https://github.com/pbonazzi/picosam3/blob/main/README.md

Run distillation, training, benchmarking, or conversion tasks using the model_compression module.

```bash
python3 -m model_compression.picosam3.picosam3_model_distillation       # Distill student from SAM3
python3 -m model_compression.picosam3.picosam3_train_from_scratch       # Train supervised baseline
python3 -m model_compression.picosam3.benchmark                         # Evaluate mIoU, mAP 
python3 -m model_compression.picosam3.imx500_converter                  # Export ONNX for IMX500
```

--------------------------------

### Prepare Dataset with PicoSAMDataset and DataLoader

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Illustrates loading COCO-formatted data using PicoSAMDataset and setting up a DataLoader with a custom collate function. This prepares data for training, including images, masks, prompts, and optionally teacher logits.

```python
from model_compression.dataset import PicoSAMDataset, custom_collate
from torch.utils.data import DataLoader

# Initialize dataset with COCO annotations
dataset = PicoSAMDataset(
    image_root="dataset/train2017",
    annotation_file="dataset/annotations/instances_train2017.json",
    image_size=96,
    cache_dir="dataset/teacher_sam2_logits",
    require_cache=True  # Set False to skip loading teacher logits
)

print(f"Dataset size: {len(dataset)} annotations")

# Create DataLoader with custom collate function
dataloader = DataLoader(
    dataset,
    batch_size=8,
    shuffle=True,
    num_workers=4,
    collate_fn=custom_collate
)

# Each batch contains:
# - images: (B, 3, 96, 96) normalized RGB images
# - masks: (B, 1, 96, 96) ground truth binary masks
# - prompts: (B, 2) center point coordinates (x, y)
# - img_ids: list of image IDs
# - teacher_logits: (B, 1, 96, 96) pre-computed teacher predictions
# - teacher_scores: (B,) confidence scores from teacher model

for images, masks, prompts, img_ids, teacher_logits, teacher_scores in dataloader:
    print(f"Batch images: {images.shape}")
    print(f"Batch masks: {masks.shape}")
    print(f"Teacher logits: {teacher_logits.shape}")
    break
```

--------------------------------

### Deploy Model to IMX500 Hardware

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Script for deploying the quantized model to the IMX500 hardware. This is the final step in the edge deployment pipeline.

```python
imx500_deployment.py
```

--------------------------------

### Fine-tune SAM 2 on MOSE (Multi-Node Cluster)

Source: https://github.com/pbonazzi/picosam3/blob/main/training/README.md

Execute the training script for multi-node training on a cluster using SLURM. Specify the number of GPUs per node, number of nodes, and optionally SLURM partition, QoS, and account details.

```python
python training/train.py \
    -c configs/sam2.1_training/sam2.1_hiera_b+_MOSE_finetune.yaml \
    --use-cluster 1 \
    --num-gpus 8 \
    --num-nodes 2
    --partition $PARTITION \
    --qos $QOS \
    --account $ACCOUNT
```

--------------------------------

### Set PYTHONPATH for SAM 2

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Manually add the repository root to the Python path if the primary configuration file is not found.

```bash
export SAM2_REPO_ROOT=/path/to/sam2  # path to this repo
export PYTHONPATH="${SAM2_REPO_ROOT}:${PYTHONPATH}"
```

--------------------------------

### Visualize Input Points

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/image_predictor_example.ipynb

Display the image with the selected input points overlaid.

```python
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis('on')
plt.show()
```

--------------------------------

### Displaying Initial Frame with Points

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Displays the current frame with initial points and masks. Ensure necessary libraries like matplotlib, PIL, and os are imported.

```python
plt.figure(figsize=(9, 6))
plt.title(f"frame {ann_frame_idx}")
plt.imshow(Image.open(os.path.join(video_dir, frame_names[ann_frame_idx])))
show_points(points, labels, plt.gca())
for i, out_obj_id in enumerate(out_obj_ids):
    show_points(*prompts[out_obj_id], plt.gca())
    show_mask((out_mask_logits[i] > 0.0).cpu().numpy(), plt.gca(), obj_id=out_obj_id)
```

--------------------------------

### Configure Experiment Log Directory

Source: https://github.com/pbonazzi/picosam3/blob/main/training/README.md

Optionally set a custom path for the experiment log directory in the configuration file. If not specified, logs default to ./sam2_logs/${config_name}.

```yaml
experiment_log_dir: null # Path to log directory, defaults to ./sam2_logs/${config_name}
```

--------------------------------

### Configure Colab environment

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Set the flag to enable environment configuration for Google Colab.

```python
using_colab = False
```

--------------------------------

### GT_ROOT Directory Structures

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/README.md

Supported directory structures for the ground truth root folder.

```text
{GT_ROOT}  # gt root folder
├── {video_id}
│     ├── 000               # all masks associated with obj 000
│     │    ├── 00000.png    # mask for object 000 in frame 00000 (binary mask)
│     │    └── ...
│     ├── 001               # all masks associated with obj 001
│     ├── 002               # all masks associated with obj 002
│     └── ...
├── {video_id}
├── {video_id}
└── ...
```

```text
{GT_ROOT}  # gt root folder
├── {video_id}
│     ├── 00000.png        # annotations in frame 00000 (may contain multiple objects)
│     └── ...
├── {video_id}
├── {video_id}
└── ...
```

--------------------------------

### Load and Sort Video Frames in Python

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Scans a directory for JPEG frames, sorts them numerically by filename, and displays the first frame. Requires `os`, `PIL.Image`, and `matplotlib.pyplot`.

```python
# `video_dir` a directory of JPEG frames with filenames like `<frame_index>.jpg`
video_dir = "./videos/bedroom"

# scan all the JPEG frame names in this directory
frame_names = [
    p for p in os.listdir(video_dir)
    if os.path.splitext(p)[-1] in [".jpg", ".jpeg", ".JPG", ".JPEG"]
]
frame_names.sort(key=lambda p: int(os.path.splitext(p)[0]))

# take a look the first video frame
frame_idx = 0
plt.figure(figsize=(9, 6))
plt.title(f"frame {frame_idx}")
plt.imshow(Image.open(os.path.join(video_dir, frame_names[frame_idx])))
```

--------------------------------

### Initialize Picosam3 Inference State

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Initializes the inference state for Picosam3 by loading all JPEG frames from the specified video directory. This is a required step for stateful video segmentation.

```python
inference_state = predictor.init_state(video_path=video_dir)
```

--------------------------------

### Segment object with box prompt

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/video_predictor_example.ipynb

Initialize object segmentation using a bounding box coordinate array.

```python
ann_frame_idx = 0  # the frame index we interact with
ann_obj_id = 4  # give a unique id to each object we interact with (it can be any integers)

# Let's add a box at (x_min, y_min, x_max, y_max) = (300, 0, 500, 400) to get started
box = np.array([300, 0, 500, 400], dtype=np.float32)
_, out_obj_ids, out_mask_logits = predictor.add_new_points_or_box(
    inference_state=inference_state,
    frame_idx=ann_frame_idx,
    obj_id=ann_obj_id,
    box=box,
)

# show the results on the current (interacted) frame
plt.figure(figsize=(9, 6))
plt.title(f"frame {ann_frame_idx}")
plt.imshow(Image.open(os.path.join(video_dir, frame_names[ann_frame_idx])))
show_box(box, plt.gca())
show_mask((out_mask_logits[0] > 0.0).cpu().numpy(), plt.gca(), obj_id=out_obj_ids[0])
```

--------------------------------

### Load and Evaluate PicoSAM3 Model

Source: https://context7.com/pbonazzi/picosam3/llms.txt

Loads a pre-trained PicoSAM3 model and prepares the validation data loader for evaluation. Ensure the model checkpoint path and dataset directories are correct.

```python
# Load model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = PicoSAM3().to(device)
model.load_state_dict(torch.load("checkpoints/PicoSAM3_student.pt"))
model.eval()

# Load validation data
val_dataset = PicoSAMDataset(
    image_root="dataset/val2017",
    annotation_file="dataset/annotations/instances_val2017.json",
    image_size=96,
    cache_dir="dataset/teacher_sam3_logits",
    require_cache=False
)
val_loader = DataLoader(val_dataset, batch_size=1, shuffle=False, collate_fn=custom_collate)
```

--------------------------------

### Visualize SA-V annotations in frame 0 (auto + manual)

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/sav_visualization_example.ipynb

Displays the annotations for frame 0, including both manually and automatically generated masks. Ensure the video is downsampled to align with annotation frequency.

```python
sav_dataset.visualize_annotation(
    frames, manual_annot, auto_annot, 
    annotated_frame_id=0,
)
```

--------------------------------

### Convert Model for IMX500

Source: https://github.com/pbonazzi/picosam3/blob/main/README.md

Convert the ONNX model to the IMX500 format and package it for deployment.

```bash
imxconv-pt -i picosam2_student_quantized.onnx -o . --overwrite-output
```

```bash
imx500-package -i packerOut.zip -o .
```

--------------------------------

### Define Input Points

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/image_predictor_example.ipynb

Specify coordinates and labels for the model. Labels are 1 for foreground and 0 for background.

```python
input_point = np.array([[500, 375]])
input_label = np.array([1])
```

--------------------------------

### Load and Display Image

Source: https://github.com/pbonazzi/picosam3/blob/main/notebooks/automatic_mask_generator_example.ipynb

Loads an image from disk and displays it using matplotlib.

```python
image = Image.open('images/cars.jpg')
image = np.array(image.convert("RGB"))
```

```python
plt.figure(figsize=(20, 20))
plt.imshow(image)
plt.axis('off')
plt.show()
```

--------------------------------

### Add -allow-unsupported-compiler to nvcc

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Modify the `setup.py` file to include the '-allow-unsupported-compiler' argument in the nvcc compile arguments. This can resolve compilation errors related to unsupported Visual Studio versions when building CUDA extensions.

```python
def get_extensions():
    srcs = ["sam2/csrc/connected_components.cu"]
    compile_args = {
        "cxx": [],
        "nvcc": [
            "-DCUDA_HAS_FP16=1",
            "-D__CUDA_NO_HALF_OPERATORS__",
            "-D__CUDA_NO_HALF_CONVERSIONS__",
            "-D__CUDA_NO_HALF2_OPERATORS__",
            "-allow-unsupported-compiler"  # Add this argument
        ],
    }
    ext_modules = [CUDAExtension("sam2._C", srcs, extra_compile_args=compile_args)]
    return ext_modules
```

--------------------------------

### Configure mixed image and video datasets in YAML

Source: https://github.com/pbonazzi/picosam3/blob/main/training/README.md

Use this configuration to define a mixed dataset pipeline for training. Ensure that SA-V video datasets are pre-extracted into JPEG frames before use.

```yaml
data:
  train:
    _target_: training.dataset.sam2_datasets.TorchTrainMixedDataset 
    phases_per_epoch: ${phases_per_epoch} # Chunks a single epoch into smaller phases
    batch_sizes: # List of batch sizes corresponding to each dataset
    - ${bs1} # Batch size of dataset 1
    - ${bs2} # Batch size of dataset 2
    datasets:
    # SA1B as an example of an image dataset
    - _target_: training.dataset.vos_dataset.VOSDataset
      training: true
      video_dataset:
        _target_: training.dataset.vos_raw_dataset.SA1BRawDataset
        img_folder: ${path_to_img_folder}
        gt_folder: ${path_to_gt_folder}
        file_list_txt: ${path_to_train_filelist} # Optional
      sampler:
        _target_: training.dataset.vos_sampler.RandomUniformSampler
        num_frames: 1
        max_num_objects: ${max_num_objects_per_image}
      transforms: ${image_transforms}
    # SA-V as an example of a video dataset
    - _target_: training.dataset.vos_dataset.VOSDataset
      training: true
      video_dataset:
        _target_: training.dataset.vos_raw_dataset.JSONRawDataset
        img_folder: ${path_to_img_folder}
        gt_folder: ${path_to_gt_folder}
        file_list_txt: ${path_to_train_filelist} # Optional
        ann_every: 4
      sampler:
        _target_: training.dataset.vos_sampler.RandomUniformSampler
        num_frames: 8 # Number of frames per video
        max_num_objects: ${max_num_objects_per_video}
        reverse_time_prob: ${reverse_time_prob} # probability to reverse video
      transforms: ${video_transforms}
    shuffle: True
    num_workers: ${num_train_workers}
    pin_memory: True
    drop_last: True
    collate_fn:
    _target_: training.utils.data_utils.collate_fn
    _partial_: true
    dict_key: all
```

--------------------------------

### Force Rebuild of SAM 2 CUDA Extension

Source: https://github.com/pbonazzi/picosam3/blob/main/INSTALL.md

Reinstall SAM 2 and force the build of the CUDA extension by setting SAM2_BUILD_ALLOW_ERRORS to 0. This command also includes verbose output for debugging and ensures the previous extension is removed.

```bash
pip uninstall -y SAM-2 && \
rm -f ./sam2/*.so && \
SAM2_BUILD_ALLOW_ERRORS=0 pip install -v -e ".[notebooks]"
```

--------------------------------

### Display manual annotations and metadata as DataFrame

Source: https://github.com/pbonazzi/picosam3/blob/main/sav_dataset/sav_visualization_example.ipynb

Converts the manual annotation data into a pandas DataFrame for easier inspection and analysis of video metadata, mask details, and temporal information.

```python
pd.DataFrame([manual_annot])
```