### Install Gradio Demo Dependencies

Source: https://github.com/facebookresearch/edgetam/blob/main/README.md

Install the necessary dependencies to run the Gradio demo for EdgeTAM.

```bash
pip install -e ".[gradio]"
```

--------------------------------

### Install EdgeTAM with Notebook Support

Source: https://github.com/facebookresearch/edgetam/blob/main/README.md

Install EdgeTAM along with Jupyter and Matplotlib for running example notebooks.

```bash
pip install -e ".[notebooks]"
```

--------------------------------

### Install EdgeTAM and Set Up Environment

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/automatic_mask_generator_example.ipynb

Installs EdgeTAM and sets up PyTorch for MPS fallback to CPU for unsupported operations. This is necessary for local Jupyter environments.

```python
# Copyright (c) Meta Platforms, Inc. and affiliates.
# Lightly adapted from https://github.com/facebookresearch/sam2/blob/main/notebooks/automatic_mask_generator_example.ipynb
```

```python
import os
# if using Apple MPS, fall back to CPU for unsupported ops
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
import numpy as np
import torch
import matplotlib.pyplot as plt
from PIL import Image
```

--------------------------------

### Load and Display First Frame of Example Video

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/video_predictor_example.ipynb

Loads a list of JPEG frames from a specified directory and displays the first frame using matplotlib. Assumes frames are named sequentially starting from 0.

```python
# `video_dir` a directory of JPEG frames with filenames like `<frame_index>.jpg`
video_dir = "./videos/bedroom"

# scan all the JPEG frame names in this directory
frame_names = [
    p for p in os.listdir(video_dir)
    if os.path.splitext(p)[-1] in [".jpg", ".jpeg", ".JPG", ".JPEG"]
]
frame_names.sort(key=lambda p: int(os.path.splitext(p)[0]))

# take a look the first video frame
frame_idx = 0
plt.figure(figsize=(9, 6))
plt.title(f"frame {frame_idx}")
plt.imshow(Image.open(os.path.join(video_dir, frame_names[frame_idx])))
```

--------------------------------

### Environment Variables for SAM2 Build and PyTorch

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/configuration.md

Examples demonstrating how to set environment variables for SAM2 build configuration and PyTorch settings like GPU selection and cache directories.

```bash
# Use specific GPU
CUDA_VISIBLE_DEVICES=0 python script.py

# Skip CUDA build
SAM2_BUILD_CUDA=0 pip install -e .

# Cache Hugging Face models
HF_HOME=/path/to/cache python script.py
```

--------------------------------

### Run EdgeTAM Inference Example

Source: https://github.com/facebookresearch/edgetam/blob/main/coreml/README.md

Execute the inference example script for EdgeTAM. This script can run a default demo, use a custom video, or perform specific examples like single image segmentation or real-time tracking.

```bash
# Demo with default coffee video
python coreml/inference_example.py
```

```bash
# Use your own video
python coreml/inference_example.py --video path/to/your/video.mp4
```

```bash
# Run different examples
python coreml/inference_example.py --example segment  # Single image
python coreml/inference_example.py --example track   # Real-time tracking
python coreml/inference_example.py --example demo    # Video demo (default)
```

--------------------------------

### SAM2 Preprocessing Pipeline Example

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/transforms_and_utils.md

Demonstrates the typical image preprocessing flow using SAM2Transforms, including initialization, image loading, preprocessing, model inference, and post-processing.

```python
import torch
import numpy as np
from sam2.utils.transforms import SAM2Transforms

# Initialize transforms
transforms = SAM2Transforms(resolution=1024, mask_threshold=0.0)

# Load image
image = np.array(...)  # HxWx3 uint8

# Preprocess
with torch.no_grad():
    input_image = transforms(image)  # 1x3x1024x1024

    # Model inference
    masks = model(input_image)  # 1x256x256

    # Post-process
    output_masks = transforms.postprocess_masks(masks, orig_hw=image.shape[:2])
    # output_masks: 1xHxW at original resolution
```

--------------------------------

### Configuration File Structure

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/INDEX.md

This is the standard structure for configuration files. It includes sections for configuration names, descriptions, parameter tables, and examples or notes.

```markdown
## Section Name
Description

### Parameter Table
name | type | default | description

### Example/Notes
Configuration examples and important notes

```

--------------------------------

### SAM2VideoPredictor Initialization and Propagation Example

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Demonstrates initializing the video predictor state, adding prompts, and propagating segmentation masks through a video. This snippet shows the core workflow for video segmentation.

```python
# Initialize state on video
state = predictor.init_state("video.mp4")

# Add prompts
with torch.inference_mode():
    frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=[[100, 100]], labels=[1]
    )
    
    # Propagate through video
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        print(f"Frame {frame_idx}: {masks.shape}")
```

--------------------------------

### Install EdgeTAM

Source: https://github.com/facebookresearch/edgetam/blob/main/README.md

Clone the repository and install EdgeTAM using pip. This command also installs PyTorch and TorchVision dependencies.

```bash
git clone https://github.com/facebookresearch/EdgeTAM.git && cd EdgeTAM

pip install -e .
```

--------------------------------

### Prepare Image and Box Batches for Inference

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Load multiple images and their corresponding bounding box annotations into batches. This setup is required for end-to-end batched inference.

```python
image1 = image  # truck.jpg from above
image1_boxes = np.array([
    [75, 275, 1725, 850],
    [425, 600, 700, 875],
    [1375, 550, 1650, 800],
    [1240, 675, 1400, 750],
])

image2 = Image.open('images/groceries.jpg')
image2 = np.array(image2.convert("RGB"))
image2_boxes = np.array([
    [450, 170, 520, 350],
    [350, 190, 450, 350],
    [500, 170, 580, 350],
    [580, 170, 640, 350],
])

img_batch = [image1, image2]
boxes_batch = [image1_boxes, image2_boxes]
```

--------------------------------

### Install EdgeTAM with CoreML Support

Source: https://github.com/facebookresearch/edgetam/blob/main/README.md

Install EdgeTAM with dependencies required for CoreML export, enabling deployment on iOS/macOS devices.

```bash
pip install -e ".[coreml]"
```

--------------------------------

### Inference with bfloat16 Precision in PyTorch

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/configuration.md

Example showing how to use bfloat16 precision for inference with PyTorch, potentially saving memory and improving speed on compatible hardware.

```python
import torch

# Default precision (float32)
model = build_sam2(...)

# Use bfloat16 for inference
with torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(image)
    masks = predictor.predict(...)
```

--------------------------------

### SAM2 Predict Examples

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_image_predictor.md

Demonstrates various ways to use the `predict` method with different prompt types, including single points, multiple points, bounding boxes, and iterative refinement using previous mask logits.

```python
import torch
import numpy as np

with torch.inference_mode():
    predictor.set_image(image)
    
    # Single point prompt
    masks, iou, logits = predictor.predict(
        point_coords=np.array([[100, 100]]),
        point_labels=np.array([1])
    )
    
    # Multiple points
    masks, iou, logits = predictor.predict(
        point_coords=np.array([[100, 100], [200, 200]]),
        point_labels=np.array([1, 1])
    )
    
    # Box prompt
    masks, iou, logits = predictor.predict(
        box=np.array([50, 50, 150, 150])  # x1, y1, x2, y2
    )
    
    # Iterative refinement
    masks2, iou2, logits2 = predictor.predict(
        point_coords=np.array([[110, 110]]),
        point_labels=np.array([0]),  # Add negative point
        mask_input=logits[0:1]  # Use previous mask logits
    )
```

--------------------------------

### Install with CUDA Disabled

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Install the package with CUDA compilation explicitly disabled to bypass build failures. This may limit post-processing capabilities and fall back to slower CPU operations.

```bash
# When installing
pip install -e .

# Output may show:
# "Failed to build the SAM 2 CUDA extension"

# This is okay - library still works, but:
# - max_hole_area and max_sprinkle_area post-processing may be limited
# - Some connected components operations fall back to slower CPU version

# To force continuing without CUDA:
export SAM2_BUILD_CUDA=0
pip install -e .
```

--------------------------------

### Load SAM2ImagePredictor from Pretrained Weights

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_image_predictor.md

Load a pretrained SAM 2 model directly from Hugging Face Hub and initialize the image predictor. This is convenient for quickly starting with a known model.

```python
predictor = SAM2ImagePredictor.from_pretrained(
    "facebook/sam2-hiera-tiny",
    device="cuda"
)
```

--------------------------------

### API Reference File Structure

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/INDEX.md

This is the standard structure for API reference files in the documentation. It includes sections for class/function names, signatures, parameters, return values, exceptions, examples, and source references.

```markdown
# Class/Function Name
Brief description

## Constructor/Definition
Full signature with types

### Parameters
Table: name | type | default | description

### Returns
Description of return type(s)

### Raises
List of exceptions

### Example
Runnable code example

### Source
Reference to source file and line numbers

```

--------------------------------

### Initialize SAM2AutomaticMaskGenerator

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/automatic_mask_generator.md

Instantiate the SAM2AutomaticMaskGenerator with a SAM2Base model and configure generation parameters. Use 'points_per_side' for grid density or 'point_grids' for custom prompt locations. Ensure 'coco_rle' output mode has pycocotools installed.

```python
import torch
import numpy as np
from sam2.build_sam import build_sam2
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

model = build_sam2("configs/edgetam.yaml", "checkpoints/edgetam.pt")
mask_generator = SAM2AutomaticMaskGenerator(
    model,
    points_per_side=32,
    pred_iou_thresh=0.8,
    stability_score_thresh=0.95
)

image = np.array(...)  # HxWx3 uint8 image
masks = mask_generator.generate(image)
```

--------------------------------

### Load and Predict with CoreML Models

Source: https://github.com/facebookresearch/edgetam/blob/main/coreml/README.md

Load exported CoreML models for image encoder, prompt encoder, and mask decoder using coremltools. This example shows how to predict with the image encoder using a PIL Image.

```python
import coremltools as ct
from PIL import Image

# Load models
image_encoder = ct.models.MLModel("coreml_models/edgetam_image_encoder.mlpackage")
prompt_encoder = ct.models.MLModel("coreml_models/edgetam_prompt_encoder.mlpackage")
mask_decoder = ct.models.MLModel("coreml_models/edgetam_mask_decoder.mlpackage")

# Segment with point prompt
image = Image.open("image.jpg").resize((1024, 1024))
encoder_out = image_encoder.predict({"image": image})

# Add your point and generate mask
# See inference_example.py for complete video tracking example
```

--------------------------------

### Semi-supervised VOS Inference on SA-V with Per-Object Masks

Source: https://github.com/facebookresearch/edgetam/blob/main/tools/README.md

Generates per-object PNG files for VOS evaluation on the SA-V dataset. Use the `--per_obj_png_file` flag for this functionality. Ensure SAM 2 and its dependencies are installed.

```bash
python ./tools/vos_inference.py \
  --sam2_cfg configs/sam2.1/sam2.1_hiera_b+.yaml \
  --sam2_checkpoint ./checkpoints/sam2.1_hiera_base_plus.pt \
  --base_video_dir /path-to-sav-val/JPEGImages_24fps \
  --input_mask_dir /path-to-sav-val/Annotations_6fps \
  --video_list_file /path-to-sav-val/sav_val.txt \
  --per_obj_png_file \
  --output_mask_dir ./outputs/sav_val_pred_pngs
```

--------------------------------

### Initialize and Track Video with SAM2

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Initializes the predictor state with a video, adds an initial prompt, and then propagates tracking through the video. Allows for interactive refinement by adding prompts on specific frames.

```python
import torch

with torch.inference_mode():
    state = predictor.init_state("video.mp4")
    
    # Add initial prompt
    predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=[[100, 100]], labels=[1]
    )
    
    # Forward tracking
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        print(f"Frame {frame_idx}: {masks.shape}")
    
    # Add refinement on problematic frame
    predictor.add_new_points_or_box(
        state, frame_idx=50, obj_id=1,
        points=[[110, 110]], labels=[1]
    )
    
    # Continue tracking (from frame 50 backwards)
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(
        state, start_frame_idx=50, reverse=True
    ):
        pass
```

--------------------------------

### Initialize SAM2VideoPredictor

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Builds and initializes a SAM2VideoPredictor instance. Ensure the configuration file and checkpoint path are correctly specified.

```python
import torch
from sam2.build_sam import build_sam2_video_predictor

predictor = build_sam2_video_predictor(
    config_file="configs/edgetam.yaml",
    ckpt_path="checkpoints/edgetam.pt"
)
```

--------------------------------

### Runtime Device Selection for SAM2 Model

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/configuration.md

Demonstrates how to specify the computation device (GPU, CPU, MPS) when building the SAM2 model.

```python
from sam2.build_sam import build_sam2

# Use GPU (default)
model = build_sam2(..., device="cuda")

# Use specific GPU
model = build_sam2(..., device="cuda:1")

# Use CPU
model = build_sam2(..., device="cpu")

# Use MPS (Apple Silicon)
model = build_sam2(..., device="mps")
```

--------------------------------

### CoreML Export for iOS/macOS Deployment

Source: https://github.com/facebookresearch/edgetam/blob/main/tools/README.md

Exports EdgeTAM models to CoreML format for on-device inference on iOS and macOS. Install CoreML dependencies using `pip install -e ".[coreml]"` before running.

```bash
# Install CoreML dependencies
pip install -e " .[coreml]"

# Export EdgeTAM to CoreML
python ./tools/export_to_coreml.py \
  --sam2_cfg sam2/configs/edgetam.yaml \
  --sam2_checkpoint ./checkpoints/edgetam.pt \
  --output_dir ./coreml_models \
  --validate
```

--------------------------------

### Get Single Mask Shape

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Displays the shape of the single predicted mask when `multimask_output=False`.

```python
masks.shape
```

--------------------------------

### Video Tracking with SAM2

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/README.md

Initializes a SAM2 video predictor, sets up the tracking state for a video, adds an initial prompt, and then propagates the tracking through the video. Requires a video file and model configuration.

```python
import torch
from sam2.build_sam import build_sam2_video_predictor

# Create predictor
predictor = build_sam2_video_predictor(
    "configs/edgetam.yaml",
    "checkpoints/edgetam.pt"
)

# Initialize on video
state = predictor.init_state("video.mp4")

with torch.inference_mode():
    # Add initial prompt on frame 0
    frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=[[100, 100]], labels=[1]
    )
    
    # Track through video
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        print(f"Frame {frame_idx}: masks shape {masks.shape}")
```

--------------------------------

### device

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_image_predictor.md

Get the device where the model is loaded. This property returns the torch.device object indicating the current device of the model.

```APIDOC
## device

### Description
Get the device where the model is loaded.

### Returns
- **torch.device**: The device (e.g., `torch.device('cuda:0')`)

### Example
```python
print(predictor.device)  # device(type='cuda', index=0)
```
```

--------------------------------

### reset_state

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Resets all tracking state and prompts for the video, allowing for a fresh start on the same video without reloading frames.

```APIDOC
## reset_state

### Description
Reset all tracking state and prompts for the video.

### Parameters
#### Path Parameters
- **inference_state** (Dict) - Required - Inference state to reset

### Notes
- Removes all object IDs and tracking results
- Preserves loaded video frames
- Allows starting fresh tracking on the same video

### Example
```python
# Start over on same video
predictor.reset_state(state)

# Add new prompts and track
predictor.add_new_points_or_box(state, frame_idx=0, obj_id=1, ...)
for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
    pass
```
```

--------------------------------

### Build SAM 2 Model from Hugging Face Hub

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Instantiates a SAM 2 model by downloading weights directly from the Hugging Face Hub using a specified model ID. Additional arguments are passed to `build_sam2`.

```python
from sam2.build_sam import build_sam2_hf

model = build_sam2_hf("facebook/sam2-hiera-tiny", device="cuda")
```

--------------------------------

### Get Mask Shape

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Displays the shape of the predicted masks, indicating the number of masks and their dimensions (Height x Width).

```python
masks.shape  # (number_of_masks) x H x W
```

--------------------------------

### Image Segmentation with SAM2

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/README.md

Builds a SAM2 image model, initializes a predictor, sets an image, and performs segmentation using point prompts. Ensure you have the necessary model and configuration files.

```python
import torch
import numpy as np
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

# Load model
model = build_sam2("configs/edgetam.yaml", "checkpoints/edgetam.pt")
predictor = SAM2ImagePredictor(model)

# Set image
image = np.array(...)  # HxWx3 uint8 image
with torch.inference_mode():
    predictor.set_image(image)
    
    # Predict with point prompt
    masks, iou, logits = predictor.predict(
        point_coords=np.array([[100, 100]]),
        point_labels=np.array([1])
    )
```

--------------------------------

### Reset Inference State

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/video_predictor_example.ipynb

Resets the inference state. Use this if you have previously run tracking with the same inference state and need to start fresh.

```python
predictor.reset_state(inference_state)
```

--------------------------------

### Build EdgeTAM with Custom Memory and Image Size

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/configuration.md

Demonstrates how to build the EdgeTAM model using `build_sam2` and override configuration parameters such as the number of memory frames and internal image resolution.

```python
from sam2.build_sam import build_sam2

# Use EdgeTAM with custom memory size
model = build_sam2(
    config_file="configs/edgetam.yaml",
    ckpt_path="checkpoints/edgetam.pt",
    hydra_overrides_extra=[
        "model.num_maskmem=10",  # More frames in memory
        "model.image_size=768"   # Smaller internal size for speed
    ]
)
```

--------------------------------

### Get Connected Components

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/transforms_and_utils.md

Finds connected components in binary masks using 8-connectivity. Requires a CUDA extension and returns pixel labels and component areas.

```python
from sam2.utils.misc import get_connected_components

mask = torch.zeros((1, 1, 100, 100), dtype=torch.bool)
mask[0, 0, 10:30, 10:30] = True
mask[0, 0, 50:70, 50:70] = True

labels, areas = get_connected_components(mask)
print(f"Number of components: {labels.max()}")  # 2
```

--------------------------------

### Predicting Masks with Points

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/types.md

Use point coordinates and labels to predict segmentation masks, IoU scores, and logits. This is a common starting point for interactive segmentation.

```python
masks, iou, logits = predictor.predict(
    point_coords=points,
    point_labels=labels
)
```

--------------------------------

### Define Input Box and Point for Prediction

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Prepare input coordinates for a single box and a point to guide the image segmentation. This is useful for precise object selection.

```python
input_box = np.array([425, 600, 700, 875])
input_point = np.array([[575, 750]])
input_label = np.array([0])
```

--------------------------------

### Initializing and Tracking Objects with Object IDs

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/types.md

Initialize tracking state for a video and assign unique Object IDs to new objects. These IDs are used to identify masks corresponding to specific tracked entities.

```python
# Track three objects
state = predictor.init_state("video.mp4")

predictor.add_new_points_or_box(state, frame_idx=0, obj_id=1, points=[[100, 100]], labels=[1])
predictor.add_new_points_or_box(state, frame_idx=0, obj_id=2, points=[[200, 200]], labels=[1])
predictor.add_new_points_or_box(state, frame_idx=0, obj_id=3, points=[[300, 300]], labels=[1])

for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
    # obj_ids: [1, 2, 3]
    # masks: shape (3, H, W) where masks[0] corresponds to obj_id=1, etc.
    for obj_id, mask in zip(obj_ids, masks):
        print(f"Object {obj_id}")
```

--------------------------------

### Prepare Image and Point Batches for Inference

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Load multiple images and define point prompts with their labels for each image in a batch. This is used for batched inference with point-based prompts.

```python
image1 = image  # truck.jpg from above
image1_pts = np.array([
    [[500, 375]],
    [[650, 750]]
    ]) # Bx1x2 where B corresponds to number of objects
image1_labels = np.array([[1], [1]])

image2_pts = np.array([
    [[400, 300]],
    [[630, 300]],
])
image2_labels = np.array([[1], [1]])

pts_batch = [image1_pts, image2_pts]
labels_batch = [image1_labels, image2_labels]
```

--------------------------------

### Initialize Predictor State

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Initialize the predictor state, opting to keep video and state on the GPU if memory allows to avoid CPU offloading.

```python
state = predictor.init_state(
    "video.mp4",
    offload_video_to_cpu=False,  # Keep video on GPU if possible
    offload_state_to_cpu=False   # Keep state on GPU if possible
)
```

--------------------------------

### Semi-supervised VOS Inference on DAVIS

Source: https://github.com/facebookresearch/edgetam/blob/main/tools/README.md

Generates predictions for semi-supervised video object segmentation (VOS) evaluation on the DAVIS dataset. Ensure SAM 2 and its dependencies are installed.

```bash
python ./tools/vos_inference.py \
  --sam2_cfg configs/sam2.1/sam2.1_hiera_b+.yaml \
  --sam2_checkpoint ./checkpoints/sam2.1_hiera_base_plus.pt \
  --base_video_dir /path-to-davis-2017/JPEGImages/480p \
  --input_mask_dir /path-to-davis-2017/Annotations/480p \
  --video_list_file /path-to-davis-2017/ImageSets/2017/val.txt \
  --output_mask_dir ./outputs/davis_2017_pred_pngs
```

--------------------------------

### Build SAM 2 Model from Config

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Use this function to build a SAM 2 model from a local configuration file and optional checkpoint. It allows specifying the device, mode, and additional configuration overrides.

```python
import torch
from sam2.build_sam import build_sam2

# Build model with EdgeTAM config
model = build_sam2(
    config_file="configs/edgetam.yaml",
    ckpt_path="checkpoints/edgetam.pt",
    device="cuda"
)
model.eval()

# Use model for inference
with torch.inference_mode():
    output = model(image)
```

--------------------------------

### Reset Tracking State

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Resets the entire tracking state for a video, removing all prompts and tracking results. This is useful for starting fresh tracking on the same video without reloading it.

```python
# Start over on same video
predictor.reset_state(state)

# Add new prompts and track
predictor.add_new_points_or_box(state, frame_idx=0, obj_id=1, ...)
for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
    pass
```

--------------------------------

### Build SAM2VideoPredictor

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/types.md

Demonstrates how to build a SAM2VideoPredictor, which is a subclass of SAM2Base for video-specific tasks.

```python
video_predictor = build_sam2_video_predictor(...)  # Returns SAM2VideoPredictor(SAM2Base)
```

--------------------------------

### Check Model and Tensor Device Placement

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Verify that the model and tensors are placed on the correct devices (e.g., GPU) to ensure efficient computation.

```python
import torch

# Verify model device
print(f"Model device: {model.device}")

# Check tensor devices
state = predictor.init_state("video.mp4")
print(f"Video frames device: {state['images'][0].device}")
print(f"Storage device: {state['storage_device']}")
```

--------------------------------

### Get Predictor Device

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_image_predictor.md

Access the device where the SAM2 model is loaded using the `device` property. This returns a `torch.device` object, indicating the hardware (e.g., CPU or GPU) being used.

```python
print(predictor.device)  # device(type='cuda', index=0)
```

--------------------------------

### Load SAM 2 Model and Predictor

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Loads the SAM 2 model and initializes the predictor. Ensure the checkpoint path and model configuration are correctly set. Running on CUDA is recommended.

```python
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

checkpoint = "../checkpoints/edgetam.pt"
model_cfg = "edgetam.yaml"

model = build_sam2(model_cfg, checkpoint, device=device)

predictor = SAM2ImagePredictor(model)
```

--------------------------------

### build_sam2

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Build a SAM 2 model from a configuration file and optional checkpoint. This function allows for flexible model construction with options for device, mode, and postprocessing.

```APIDOC
## build_sam2

### Description
Build a SAM 2 model from a configuration file and optional checkpoint.

### Parameters
#### Path Parameters
- **config_file** (str) - Required - Path to the model configuration file (e.g., `configs/edgetam.yaml`)
- **ckpt_path** (Optional[str]) - Optional - Path to the model checkpoint file. If None, model weights are not loaded
- **device** (str) - Optional - Device to load the model on (`"cuda"`, `"cpu"`, etc.)
- **mode** (str) - Optional - Model mode; if `"eval"`, sets model to evaluation mode
- **hydra_overrides_extra** (List[str]) - Optional - Additional Hydra configuration overrides to apply
- **apply_postprocessing** (bool) - Optional - If True, applies default postprocessing settings for dynamic multimask and stability thresholds
- **kwargs** (dict) - Optional - Additional keyword arguments passed to model constructor

### Returns
- **SAM2Base**: A configured SAM 2 model in the specified device and mode.

### Raises
- **RuntimeError**: If checkpoint loading fails due to missing or unexpected keys in the model state dict.

### Example
```python
import torch
from sam2.build_sam import build_sam2

# Build model with EdgeTAM config
model = build_sam2(
    config_file="configs/edgetam.yaml",
    ckpt_path="checkpoints/edgetam.pt",
    device="cuda"
)
model.eval()

# Use model for inference
with torch.inference_mode():
    output = model(image)
```
```

--------------------------------

### Get Shape of Predicted Masks

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Check the dimensions of the output masks tensor. The shape indicates the batch size, number of masks per input, and image height and width.

```python
masks.shape  # (batch_size) x (num_predicted_masks_per_input) x H x W
```

--------------------------------

### Prevent Adding Object After Tracking Starts

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Catches RuntimeError when attempting to add a new object after the tracking process has begun. Demonstrates resetting the state to add new objects.

```python
state = predictor.init_state("video.mp4")

with torch.inference_mode():
    # Add first object
    predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=[[100, 100]], labels=[1]
    )
    
    # Start tracking
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        if frame_idx > 5:
            break
    
    # Now try to add new object - will fail
    try:
        predictor.add_new_points_or_box(
            state, frame_idx=10, obj_id=2,  # New object!
            points=[[200, 200]], labels=[1]
        )
    except RuntimeError as e:
        print(f"Cannot add object after tracking: {e}")
        
        # Solution: reset and start over
        predictor.reset_state(state)
        predictor.add_new_points_or_box(
            state, frame_idx=0, obj_id=1,
            points=[[100, 100]], labels=[1]
        )
        predictor.add_new_points_or_box(
            state, frame_idx=0, obj_id=2,
            points=[[200, 200]], labels=[1]
        )
```

--------------------------------

### Handle Config File Not Found

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Catch FileNotFoundError or RuntimeError if the configuration file is not found. Lists available configurations for user reference.

```python
try:
    model = build_sam2(
        config_file="configs/nonexistent.yaml",
        ckpt_path="checkpoint.pt"
    )
except (FileNotFoundError, RuntimeError) as e:
    print(f"Config not found: {e}")
    # Use valid config
    from sam2.build_sam import HF_MODEL_ID_TO_FILENAMES
    print(f"Available configs: {HF_MODEL_ID_TO_FILENAMES.keys()}")
```

--------------------------------

### Run CoreML Performance Benchmark

Source: https://github.com/facebookresearch/edgetam/blob/main/coreml/README.md

Execute the performance benchmark script for CoreML. Note that this is a limited test on synthetic data, and real-world performance may vary.

```bash
python coreml/benchmark_coreml.py
```

--------------------------------

### Error Reference File Structure

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/INDEX.md

This is the standard structure for error reference files. It includes sections for error categories, specific scenarios, error messages, trigger conditions, and example handling code.

```markdown
## Error Category
Related errors

### Scenario: Specific Case
- Error message
- Trigger condition
- Example handling code

```

--------------------------------

### Build SAM 2 Video Predictor from Hugging Face Hub

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Creates a SAM 2 video predictor by downloading weights from Hugging Face Hub. This function accepts a model ID and passes additional arguments to the underlying `build_sam2_video_predictor` function.

```python
from sam2.build_sam import build_sam2_video_predictor_hf

predictor = build_sam2_video_predictor_hf("facebook/sam2-hiera-tiny", device="cuda")
```

--------------------------------

### SAM2Transforms Preprocessing Pipeline

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/transforms_and_utils.md

Demonstrates a typical preprocessing pipeline using the SAM2Transforms class, including image loading, transformation, model inference, and post-processing of masks.

```APIDOC
## SAM2Transforms Preprocessing Pipeline

### Description
This section outlines a typical image processing flow using the SAM2Transforms class for preprocessing images and post-processing model outputs.

### Usage Example
```python
import torch
import numpy as np
from sam2.utils.transforms import SAM2Transforms

# Initialize transforms
transforms = SAM2Transforms(resolution=1024, mask_threshold=0.0)

# Load image
image = np.array(...)  # HxWx3 uint8

# Preprocess
with torch.no_grad():
    input_image = transforms(image)  # 1x3x1024x1024

    # Model inference
    masks = model(input_image)  # 1x256x256

    # Post-process
    output_masks = transforms.postprocess_masks(masks, orig_hw=image.shape[:2])
    # output_masks: 1xHxW at original resolution
```

### Components
- **SAM2Transforms**: Class for handling image transformations and mask post-processing.
- **transforms(image)**: Method to preprocess the input image.
- **postprocess_masks(masks, orig_hw)**: Method to post-process the generated masks to the original image resolution.
```

--------------------------------

### Initialize Inference State

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/video_predictor_example.ipynb

Initializes the inference state for a video by loading JPEG frames. This is a required first step for stateful inference in SAM 2.

```python
inference_state = predictor.init_state(video_path=video_dir)
```

--------------------------------

### Get Image Embedding

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_image_predictor.md

Retrieve cached image embeddings after setting an image with `set_image` or `set_image_batch`. This is useful for custom applications that need direct access to the embeddings. The embeddings have a shape of 1xCxHxW, where C is 256.

```python
import torch

with torch.inference_mode():
    predictor.set_image(image)
    embeddings = predictor.get_image_embedding()
    print(embeddings.shape)  # torch.Size([1, 256, 64, 64])
```

--------------------------------

### Initialize SAM2 Model and Mask Generator

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/automatic_mask_generator_example.ipynb

Builds the SAM2 model using a specified checkpoint and configuration, then initializes the SAM2AutomaticMaskGenerator.

```python
from sam2.build_sam import build_sam2
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator

checkpoint = "../checkpoints/edgetam.pt"
model_cfg = "edgetam.yaml"

model = build_sam2(model_cfg, checkpoint, device=device, apply_postprocessing=False)

mask_generator = SAM2AutomaticMaskGenerator(model)
```

--------------------------------

### Handle CUDA Out of Memory Errors

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Detects CUDA out of memory errors during the forward pass. Suggests solutions like processing smaller images, using CPU, offloading state, or using a smaller model. Includes an example of offloading video and state to CPU.

```python
# Occurs during forward pass when GPU memory exhausted
import torch

try:
    with torch.inference_mode():
        predictor.set_image(large_image)
        masks = predictor.predict(...)
except RuntimeError as e:
    if "out of memory" in str(e) or "CUDA out of memory" in str(e):
        print("GPU out of memory")
        
        # Solutions:
        # 1. Process smaller image
        # 2. Use CPU
        # 3. Offload state to CPU
        # 4. Use smaller model
        
        # For video predictor
        state = predictor.init_state(
            "video.mp4",
            offload_video_to_cpu=True,
            offload_state_to_cpu=True
        )
```

--------------------------------

### Build SAM2 Video Predictor from Hugging Face

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Instantiate a SAM2 video predictor using a Hugging Face model ID. Ensure the 'cuda' device is available if specified.

```python
from sam2.build_sam import build_sam2_video_predictor_hf

predictor = build_sam2_video_predictor_hf(
    "facebook/sam2-hiera-base-plus",
    device="cuda"
)
```

--------------------------------

### Build and Instantiate EdgeTAME Model

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/types.md

Instantiates the SAM2 model using a configuration file and checkpoint. Asserts the model type and shows how to use it with SAM2ImagePredictor.

```python
model = build_sam2("configs/edgetam.yaml", "checkpoints/edgetam.pt")

# Model is a SAM2Base instance
assert isinstance(model, SAM2Base)

# Use with SAM2ImagePredictor
predictor = SAM2ImagePredictor(model)
```

--------------------------------

### Import Necessary Libraries

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Imports essential libraries for numerical operations, PyTorch, plotting, and image handling. Includes a setting to enable MPS fallback for Apple Silicon.

```python
import os
# if using Apple MPS, fall back to CPU for unsupported ops
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
import numpy as np
import torch
import matplotlib.pyplot as plt
from PIL import Image
```

--------------------------------

### Check State Offloading for Slow Inference

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Checks the configuration of state offloading to CPU, which can be a cause of slow inference. Prints whether state and video are offloaded to CPU.

```python
# Possible causes:
# 1. State offloaded to CPU (slower but memory-efficient)
# 2. Memory cache not working
# 3. Too many frames in memory bank

# Check state configuration
state = predictor.init_state("video.mp4")
print(f"Offload state to CPU: {state['offload_state_to_cpu']}")
print(f"Offload video to CPU: {state['offload_video_to_cpu']}")
```

--------------------------------

### Run EdgeTAM Gradio Demo

Source: https://github.com/facebookresearch/edgetam/blob/main/README.md

Execute the Gradio application to interact with the EdgeTAM demo. The demo will be available at http://127.0.0.1:7860/ by default.

```bash
python gradio_app.py
```

--------------------------------

### Enable Debug Logging

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/errors.md

Enable detailed debug logging to observe model building and image processing steps for troubleshooting.

```python
import logging

# Enable debug logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger()

model = build_sam2(...)  # Will show detailed logs
predictor.set_image(image)  # Will log image processing steps
```

--------------------------------

### Load Pretrained SAM2VideoPredictor

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Loads a SAM2VideoPredictor model from Hugging Face Hub. Specify the model ID and optionally the device for loading.

```python
predictor = SAM2VideoPredictor.from_pretrained(
    "facebook/sam2-hiera-tiny",
    device="cuda"
)
```

--------------------------------

### Visualize Input Point

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Displays the input image with the selected point highlighted. This helps verify the input prompt.

```python
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis('on')
plt.show()
```

--------------------------------

### Build SAM 2 Video Predictor from Config

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Constructs a SAM 2 video predictor for segmentation and tracking tasks using a local configuration file and optional checkpoint. Supports specifying device, mode, and postprocessing options.

```python
import torch
from sam2.build_sam import build_sam2_video_predictor

# Build video predictor
predictor = build_sam2_video_predictor(
    config_file="configs/edgetam.yaml",
    ckpt_path="checkpoints/edgetam.pt",
    device="cuda"
)

# Initialize state on video
state = predictor.init_state("path/to/video.mp4")

# Add prompts and propagate
with torch.inference_mode():
    frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=1,
        points=[[100, 100]], labels=[1]
    )
    
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        pass  # Process results
```

--------------------------------

### build_sam2_hf

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/build_sam.md

Build a SAM 2 model by downloading weights directly from the Hugging Face Hub. This function simplifies loading models using their repository IDs.

```APIDOC
## build_sam2_hf

### Description
Build a SAM 2 model by downloading weights from Hugging Face Hub.

### Parameters
#### Path Parameters
- **model_id** (str) - Required - Hugging Face model repository ID (e.g., `"facebook/sam2-hiera-tiny"`)
- **kwargs** (dict) - Optional - Additional arguments passed to `build_sam2()`

### Returns
- **SAM2Base**: A SAM 2 model with weights loaded from Hugging Face.

### Supported Model IDs

- `facebook/sam2-hiera-tiny`
- `facebook/sam2-hiera-small`
- `facebook/sam2-hiera-base-plus`
- `facebook/sam2-hiera-large`
- `facebook/sam2.1-hiera-tiny`
- `facebook/sam2.1-hiera-small`
- `facebook/sam2.1-hiera-base-plus`
- `facebook/sam2.1-hiera-large`

### Example
```python
from sam2.build_sam import build_sam2_hf

model = build_sam2_hf("facebook/sam2-hiera-tiny", device="cuda")
```
```

--------------------------------

### Initialize State for Large Videos

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/configuration.md

Configures the predictor state for processing large videos by offloading video and state data to CPU and enabling asynchronous frame loading to save GPU memory.

```python
state = predictor.init_state(
    "video.mp4",
    offload_video_to_cpu=True,      # Save GPU memory on video
    offload_state_to_cpu=True,       # Save GPU memory on state
    async_loading_frames=True        # Load frames asynchronously
)
```

--------------------------------

### Initialize Video Inference State

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Initializes the inference state for a given video path. This state is required for subsequent tracking operations. Note that the video is loaded at the model's internal image size.

```python
import torch

predictor = build_sam2_video_predictor(...)

# Initialize on video
with torch.inference_mode():
    state = predictor.init_state("path/to/video.mp4")
    print(f"Video frames: {state['num_frames']}")
    print(f"Video size: {state['video_height']}x{state['video_width']}")
```

--------------------------------

### Select Computation Device and Configure PyTorch

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/image_predictor_example.ipynb

Selects the computation device (CUDA, MPS, or CPU) and configures PyTorch for optimal performance, including mixed-precision and TF32 settings for CUDA devices. Provides a warning for MPS device usage.

```python
# select the device for computation
if torch.cuda.is_available():
    device = torch.device("cuda")
elif torch.backends.mps.is_available():
    device = torch.device("mps")
else:
    device = torch.device("cpu")
print(f"using device: {device}")

if device.type == "cuda":
    # use bfloat16 for the entire notebook
    torch.autocast("cuda", dtype=torch.bfloat16).__enter__()
    # turn on tfloat32 for Ampere GPUs (https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices)
    if torch.cuda.get_device_properties(0).major >= 8:
        torch.backends.cuda.matmul.allow_tf32 = True
        torch.backends.cudnn.allow_tf32 = True
elif device.type == "mps":
    print(
        "\nSupport for MPS devices is preliminary. EdgeTAM is trained with CUDA and might "
        "give numerically different outputs and sometimes degraded performance on MPS. "
        "See e.g. https://github.com/pytorch/pytorch/issues/84936 for a discussion."
    )
```

--------------------------------

### Load Image

Source: https://github.com/facebookresearch/edgetam/blob/main/notebooks/automatic_mask_generator_example.ipynb

Loads an image from a file path and converts it to an RGB numpy array.

```python
image = Image.open('images/cars.jpg')
image = np.array(image.convert("RGB"))
```

--------------------------------

### SAM2VideoPredictor Initialization

Source: https://github.com/facebookresearch/edgetam/blob/main/_autodocs/api-reference/sam2_video_predictor.md

Initializes the video predictor with a SAM 2 model. Allows configuration of various parameters for mask processing and memory management during video tracking.

```APIDOC
## SAM2VideoPredictor

### Description
Initializes the video predictor with a SAM 2 model. Supports interactive prompting, mask refinement, and multi-object tracking throughout video sequences.

### Parameters
- **fill_hole_area** (int) - Optional - Default: 0 - If > 0, fill holes in predicted masks up to this area size (in pixels)
- **non_overlap_masks** (bool) - Optional - Default: False - If True, apply non-overlapping constraints to multi-object masks
- **clear_non_cond_mem_around_input** (bool) - Optional - Default: False - If True, clear memory around frames receiving user input
- **clear_non_cond_mem_for_multi_obj** (bool) - Optional - Default: False - If True, also clear memory for multi-object tracking
- **add_all_frames_to_correct_as_cond** (bool) - Optional - Default: False - If True, treat all frames with user corrections as conditioning frames
- **kwargs** (dict) - Optional - Additional arguments passed to SAM2Base
```