### Install and Run Next.js SAM Project

Source: https://github.com/karlorz/next-sam/blob/main/README.md

Instructions for cloning the repository, installing dependencies, and starting the development server for the Next.js SAM application. Ensure Node.js and npm are installed.

```bash
git clone https://github.com/geronimi73/next-sam
cd next-sam
npm install
npm run dev
```

--------------------------------

### Clone and Install segment-anything-2 Repository

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This snippet first clones the official 'segment-anything-2' repository from GitHub using `git clone`. It then navigates into the cloned directory and installs the package using `pip3 install .`. This process requires `git` and `pip3` to be available, and installs necessary dependencies specified in the repository's `pyproject.toml`.

```bash
!git clone https://github.com/facebookresearch/segment-anything-2.git
!cd segment-anything-2; pip3 install .
```

--------------------------------

### Download SAM2 Weights using wget

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This code snippet uses the `wget` command to download the official SAM2 weights ('sam2_hiera_tiny.pt') from a provided URL. Ensure `wget` is installed in your environment. The output shows the download progress and confirmation.

```bash
!wget "https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_tiny.pt"
```

--------------------------------

### Install SAM-2 and Dependencies with Pip

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This command installs the SAM-2 project along with its core Python dependencies such as antlr4-python3-runtime, iopath, torch, and torchvision. It also handles the uninstallation of potentially conflicting older versions of packages like typing_extensions, triton, sympy, pillow, numpy, torch, and torchvision. Note the warning about running pip as root.

```bash
!pip3 install -U onnx onnxscript onnxsim onnxruntime
```

--------------------------------

### Convert SAM2 Image Encoder to ONNX in PyTorch

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This snippet demonstrates how to export the SAM2 image encoder to the ONNX format using PyTorch. It loads a pre-trained model, performs a forward pass to get tensor shapes, and then uses `torch.onnx.export` to save the model. Dependencies include `torch` and `sam2` library.

```python
import torch
from sam2.build_sam import build_sam2
from sam2.modeling.sam_image_encoder import SAM2ImageEncoder

model_type = "sam2_hiera_tiny"
model_cfg = "sam2_hiera_t.yaml"
input_size = 1024
multimask_output = True

sam2_checkpoint = f"./{model_type}.pt"
sam2_model = build_sam2(model_cfg, sam2_checkpoint, device="cpu")

img=torch.randn(1, 3, input_size, input_size).cpu()

sam2_encoder = SAM2ImageEncoder(sam2_model).cpu()
high_res_feats_0, high_res_feats_1, image_embed = sam2_encoder(img)
print(high_res_feats_0.shape)
print(high_res_feats_1.shape)
print(image_embed.shape)

torch.onnx.export(sam2_encoder,
      img,
      f"{model_type}_encoder.onnx",
      export_params=True,
      opset_version=17,
      do_constant_folding=True,
      input_names = ['image'],
      output_names = ['high_res_feats_0', 'high_res_feats_1', 'image_embed'],
    )
```

--------------------------------

### Extracting Masks from Tensor with sliceTensor (JavaScript)

Source: https://context7.com/karlorz/next-sam/llms.txt

Demonstrates how to use the `sliceTensor` function to extract individual mask candidates from a multi-mask tensor produced by the SAM model. It shows the process of getting mask tensors, logging their dimensions and lengths, converting them to canvas elements for visualization, and selecting the best mask using IoU scores.

```javascript
import { sliceTensor } from "./lib/imageutils";

// After decoding, we get multiple mask candidates
const results = await sam.decode(points);
const maskTensor = results.masks;

console.log(maskTensor.dims);  // [1, 3, 256, 256] for SAM2
// Batch=1, Masks=3, Width=256, Height=256

// Extract each mask candidate
const mask0 = sliceTensor(maskTensor, 0);  // First mask
const mask1 = sliceTensor(maskTensor, 1);  // Second mask
const mask2 = sliceTensor(maskTensor, 2);  // Third mask

console.log(mask0.length);  // 65536 (256 * 256)
console.log(mask0 instanceof Float32Array);  // true

// Convert to canvas for visualization
const canvas0 = float32ArrayToCanvas(mask0, 256, 256);
const canvas1 = float32ArrayToCanvas(mask1, 256, 256);
const canvas2 = float32ArrayToCanvas(mask2, 256, 256);

// Display all candidates
document.body.appendChild(canvas0);
document.body.appendChild(canvas1);
document.body.appendChild(canvas2);

// Typically used with IoU scores to select best mask
const scores = results.iou_predictions.cpuData;
let bestIdx = 0;
let bestScore = -Infinity;
for (let i = 0; i < scores.length; i++) {
  if (scores[i] > bestScore) {
    bestScore = scores[i];
    bestIdx = i;
  }
}
const bestMask = sliceTensor(maskTensor, bestIdx);
```

--------------------------------

### SAM2 Class Constructor and Model Initialization

Source: https://context7.com/karlorz/next-sam/llms.txt

This section details how to initialize the SAM2 class, load ONNX models, and create inference sessions. It supports automatic model downloads, caching via OPFS, and WebGPU/CPU fallback.

```APIDOC
## SAM2 Class Constructor and Model Initialization

### Description
Initializes the SAM2 class for image segmentation, handling model loading, caching, and inference session creation with device fallback (WebGPU/CPU).

### Method
`new SAM2(modelConfig)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **modelConfig** (object) - Required - Configuration object specifying the model to load (e.g., `MODEL_CONFIG.sam2_tiny`, `MODEL_CONFIG.mobilesam_tiny`).

### Request Example
```javascript
import { SAM2 } from "./app/SAM2";
import { MODEL_CONFIG } from "./app/modelConfig";

// Initialize with SAM2 Tiny model (default)
const sam = new SAM2(MODEL_CONFIG.sam2_tiny);

// Or initialize with MobileSAM for faster inference
const mobileSam = new SAM2(MODEL_CONFIG.mobilesam_tiny);

// Download models (with automatic OPFS caching)
try {
  await sam.downloadModels();
  console.log("Models downloaded and cached successfully");
} catch (error) {
  console.error("Model download failed:", error);
}

// Create inference sessions (tries WebGPU, falls back to CPU)
const result = await sam.createSessions();
if (result.success) {
  console.log(`Sessions created, running on: ${result.device}`);
} else {
  console.error("Failed to create sessions");
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if session creation was successful.
- **device** (string) - The device used for inference ('webgpu' or 'cpu').

#### Response Example
```json
{
  "success": true,
  "device": "webgpu"
}
```
```

--------------------------------

### Initialize SAM2/MobileSAM and Download Models (JavaScript)

Source: https://context7.com/karlorz/next-sam/llms.txt

Initializes the SAM2 class with different model configurations (SAM2 Tiny or MobileSAM Tiny). It handles downloading the ONNX models, automatically caching them using the Origin Private File System (OPFS), and creating inference sessions with WebGPU/CPU fallback.

```javascript
import { SAM2 } from "./app/SAM2";
import { MODEL_CONFIG } from "./app/modelConfig";

// Initialize with SAM2 Tiny model (default)
const sam = new SAM2(MODEL_CONFIG.sam2_tiny);

// Or initialize with MobileSAM for faster inference
const mobileSam = new SAM2(MODEL_CONFIG.mobilesam_tiny);

// Download models (with automatic OPFS caching)
try {
  await sam.downloadModels();
  console.log("Models downloaded and cached successfully");
} catch (error) {
  console.error("Model download failed:", error);
}

// Create inference sessions (tries WebGPU, falls back to CPU)
const result = await sam.createSessions();
if (result.success) {
  console.log(`Sessions created, running on: ${result.device}`);
  // Output: "Sessions created, running on: webgpu" or "cpu"
} else {
  console.error("Failed to create sessions");
}
```

--------------------------------

### MobileSAM WebGPU Porting Analysis and Recommendations

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Discusses challenges encountered when porting MobileSAM to WebGPU, including mask quality issues and the handling of IoU scores greater than 1.0. It contrasts CPU WASM success with WebGPU specifics and provides recommendations for the next-sam project.

```text
The original analysis deemed MobileSAM unreliable for WebGPU due to poor mask quality and invalid scores. However, this is challenged based on successful alternatives:

- **CPU WASM Success (candle-segment-anything-wasm)**: This repo uses Rust-based Candle framework compiled to WASM, running MobileSAM on CPU with excellent results—no IoU >1 issues or fragmentation reported. Inference ~500ms on average hardware, proving the model architecture (TinyViT encoder + SAM decoder) is sound. Differences: Candle faithfully replicates PyTorch behavior without ONNX conversion losses; no WebGPU, so avoids hardware-specific bugs (e.g., Intel iGPU inaccuracies in ONNX WebGPU).

- **IoU >1.0 Not Invalid**: In facebookresearch/segment-anything issues, users report predicted scores >1 in original SAM, attributed to model calibration limits rather than bugs. Treating them as invalid skips good masks; challenge: Select max score instead, as in successful ports.

- **ONNX Export Issues**: Acly/MobileSAM may have conversion artifacts; official ChaoningZhang/MobileSAM export script (with onnx==1.12.0) or vietanhdev/samexporter yields better models. Quantized decoders reduce size/latency without quality loss.

- **WebGPU-Specific Challenges**: ONNX Runtime Web on WebGPU can produce unstable predictions on Intel GPUs (microsoft/onnxruntime issues). Test with CPU fallback or Chrome flags for precision (e.g., f32 strict). Successful examples like akbartus/MobileSAM-in-the-Browser run MobileSAM fully in-browser with ONNX Web, achieving clean segments under 300ms.

- **Other Factors**: Fragmentation often from unnormalized inputs or resize artifacts; add post-processing. Compared to SAM2, MobileSAM is 5-7x faster per paper, viable post-fixes.

**Porting Recommendations for next-sam**:
- Use official ONNX: Run ChaoningZhang's export script for fresh models.
- Refactor: In modelConfig.js, add variant with quantized decoder.
- Test: Re-run with updated selection; expect 80-90% quality match to SAM2.
```

--------------------------------

### Prepare SAM2 Decoder Inputs in PyTorch

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This snippet prepares input tensors for the SAM2 image decoder, including point coordinates, point labels, mask input, and original image size. It uses PyTorch and assumes the model and input parameters are already defined.

```python
import torch
from sam2.build_sam import build_sam2
from sam2.modeling.sam_image_decoder import SAM2ImageDecoder

# Assuming model and input_size are already defined from previous snippets
# model_type = "sam2_hiera_tiny"
# model_cfg = "sam2_hiera_t.yaml"
# input_size = 1024
# multimask_output = True
# sam2_model = build_sam2(model_cfg, f"./{model_type}.pt", device="cpu")

sam2_decoder = SAM2ImageDecoder(sam2_model, multimask_output=multimask_output).cpu()

embed_dim = sam2_model.sam_prompt_encoder.embed_dim
embed_size = (sam2_model.image_size // sam2_model.backbone_stride, sam2_model.image_size // sam2_model.backbone_stride)
mask_input_size = [4 * x for x in embed_size]
# mask_input_size = [1024, 1024]
print(embed_dim, embed_size, mask_input_size)

point_coords = torch.randint(low=0, high=input_size, size=(1, 5, 2), dtype=torch.float)
point_labels = torch.randint(low=0, high=1, size=(1, 5), dtype=torch.float)
mask_input = torch.randn(1, 1, *mask_input_size, dtype=torch.float)
has_mask_input = torch.tensor([0], dtype=torch.float)
orig_im_size = torch.tensor([input_size, input_size], dtype=torch.int32)
```

--------------------------------

### Convert ONNX to ORT using Python

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This script utilizes the `onnxruntime` Python package to convert an ONNX model file to the ORT (ONNX Runtime) format. It takes the ONNX model path as input and produces one or more ORT format model files, along with configuration files. The conversion process includes options for different optimization styles and levels.

```python
!python3 -m onnxruntime.tools.convert_onnx_models_to_ort sam2_hiera_tiny_encoder.onnx
```

--------------------------------

### Run SAM2 Decoder ONNX Model Inference

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Performs inference using a SAM2 decoder model loaded with ONNX Runtime. It provides the necessary input tensors and prints the names and shapes of the resulting output tensors, which include masks and IoU predictions.

```python
import onnxruntime as ort
import numpy as np

ort_sess = ort.InferenceSession('sam2_hiera_tiny_decoder.onnx', {})

outputs = ort_sess.run(None, {
    'image_embed': image_embed.detach().numpy(), 
    'high_res_feats_0': high_res_feats_0.detach().numpy(), 
    'high_res_feats_1': high_res_feats_1.detach().numpy(), 
    'point_coords': point_coords.numpy(), 
    'point_labels': point_labels.numpy(), 
    'mask_input': mask_input.numpy(), 
    'has_mask_input': has_mask_input.numpy(), 
    # 'orig_im_size': orig_im_size.numpy()
})

for i in range(len(outputs)):
    print (ort_sess.get_outputs()[i].name)
    print (outputs[i].shape)
```

--------------------------------

### Inspect SAM2 Decoder ONNX Model Inputs

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Loads a SAM2 decoder model exported to ONNX format using onnxruntime and prints the details of its inputs. This is useful for understanding the expected input tensor names, types, and shapes for inference.

```python
import onnxruntime as ort
import numpy as np

ort_sess = ort.InferenceSession('sam2_hiera_tiny_decoder.onnx', {})

for i in ort_sess.get_inputs():
    print(str(i))
```

--------------------------------

### MobileSAM Configuration with ONNX

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Defines the configuration for the MobileSAM model, including URLs for ONNX encoder and decoder, image/mask sizes, and processing parameters. It specifies the use of official ONNX exports for improved quality and mentions quantization for the decoder.

```javascript
export const MODEL_CONFIG = {
  mobilesam_tiny: {
    id: "mobilesam_tiny",
    name: "Mobile SAM Tiny",
    description: "Mobile SAM Tiny (45 MB, TinyViT encoder) - Improved with official ONNX",
    encoderUrl: "https://example.com/official_mobile_sam_encoder.onnx", // Use official export
    decoderUrl: "https://example.com/official_sam_decoder_multi.onnx", // Quantized preferred
    imageSize: { w: 1024, h: 1024 },
    maskSize: { w: 256, h: 256 },
    modelType: "mobilesam",
    encoderInputName: "input_image",
    useBatchDimension: false,
    tensorFormat: "HWC",
    postProcess: true // Enable erosion/dilation
  },
  // SAM2 unchanged
};

export const DEFAULT_MODEL = "sam2_tiny"; // Still default, but MobileSAM viable

```

--------------------------------

### Inspect SAM2 Decoder ONNX Model Inputs (Different Shape)

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Loads a SAM2 decoder model exported to ONNX format and prints the details of its inputs. This inspection reveals different input shapes compared to a previous check, specifically for 'mask_input' and 'has_mask_input', indicating potential variations in model versions or export configurations.

```python
import onnxruntime as ort

ort_sess = ort.InferenceSession('sam2_hiera_tiny_decoder.onnx', {})

for i in ort_sess.get_inputs():
    print(str(i))
```

--------------------------------

### Configure and Access SAM Models in JavaScript

Source: https://context7.com/karlorz/next-sam/llms.txt

Defines and accesses configurations for various SAM models, including URLs, dimensions, and tensor formats. Enables easy switching between different segmentation architectures and custom model definitions.

```javascript
import { MODEL_CONFIG, DEFAULT_MODEL } from "./app/modelConfig";

// Access SAM2 configuration
const sam2Config = MODEL_CONFIG.sam2_tiny;
console.log(sam2Config.name);  // "Meta's SAM2 Tiny"
console.log(sam2Config.encoderUrl);  // Hugging Face model URL
console.log(sam2Config.imageSize);  // { w: 1024, h: 1024 }
console.log(sam2Config.tensorFormat);  // "CHW" (Channels, Height, Width)

// Access MobileSAM configuration
const mobileConfig = MODEL_CONFIG.mobilesam_tiny;
console.log(mobileConfig.modelType);  // "mobilesam"
console.log(mobileConfig.tensorFormat);  // "HWC" (Height, Width, Channels)
console.log(mobileConfig.inputRange);  // 255 (expects 0-255 input)

// Use default model
const defaultModelId = DEFAULT_MODEL;  // "sam2_tiny"
const sam = new SAM2(MODEL_CONFIG[defaultModelId]);

// Custom model configuration
const customConfig = {
  id: "custom_model",
  name: "Custom SAM Model",
  encoderUrl: "https://example.com/encoder.onnx",
  decoderUrl: "https://example.com/decoder.onnx",
  imageSize: { w: 1024, h: 1024 },
  maskSize: { w: 256, h: 256 },
  modelType: "sam2",
  encoderInputName: "image",
  useBatchDimension: true,
  tensorFormat: "CHW"
};

```

--------------------------------

### Test SAM2 Encoder ONNX Model Inference

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Tests the inference of a SAM2 encoder model exported to ONNX format using onnxruntime. It generates random input image data, runs the inference session, and prints the names and shapes of the output tensors.

```python
import onnxruntime as ort
import numpy as np

ort_sess = ort.InferenceSession('sam2_hiera_tiny_encoder.onnx', {})

img = torch.randn(1, 3, input_size, input_size).cpu()

outputs = ort_sess.run(None, {
    'image': img.numpy(),
})

for i in range(len(outputs)):
    print (ort_sess.get_outputs()[i].name)
    print (outputs[i].shape)
```

--------------------------------

### Canvas Resizing and Padding Utility

Source: https://context7.com/karlorz/next-sam/llms.txt

Handles image preprocessing by resizing canvases and calculating padding boxes to maintain aspect ratios when fitting images into square dimensions. It requires the original canvas element and target dimensions, outputting a new canvas or padding box coordinates.

```javascript
import { resizeCanvas, resizeAndPadBox } from "./lib/imageutils";

// Resize canvas to specific dimensions
const originalCanvas = document.createElement("canvas");
originalCanvas.width = 800;
originalCanvas.height = 600;
// ... draw image on originalCanvas ...

const resized = resizeCanvas(originalCanvas, { w: 1024, h: 1024 });
console.log(resized.width);   // 1024
console.log(resized.height);  // 1024

// Calculate padding box to preserve aspect ratio
const sourceDim = { w: 800, h: 600 };  // Landscape
const targetDim = { w: 1024, h: 1024 };  // Square
const box = resizeAndPadBox(sourceDim, targetDim);

console.log(box);
// { x: 0, y: 192, w: 1024, h: 640 }
// Image will be centered vertically with 192px padding top and bottom

// Portrait example
const portraitDim = { w: 600, h: 800 };
const portraitBox = resizeAndPadBox(portraitDim, targetDim);
console.log(portraitBox);
// { x: 128, y: 0, w: 768, h: 1024 }
// Image centered horizontally with 128px padding left and right

// Use box to draw with proper aspect ratio
const paddedCanvas = document.createElement("canvas");
paddedCanvas.width = targetDim.w;
paddedCanvas.height = targetDim.h;
const ctx = paddedCanvas.getContext("2d");
ctx.drawImage(img, 0, 0, img.width, img.height,
              box.x, box.y, box.w, box.h);

```

--------------------------------

### Export SAM2 Decoder to ONNX

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Exports the SAM2 decoder model to ONNX format using torch.onnx.export. This function allows for the conversion of PyTorch models to the ONNX standard, facilitating interoperability with other platforms. It defines input and output names, dynamic axes for variable input dimensions, and the ONNX opset version.

```python
torch.onnx.export(sam2_decoder,
      (image_embed, high_res_feats_0, high_res_feats_1, point_coords, point_labels, mask_input, has_mask_input),
      # (image_embed, high_res_feats_0, high_res_feats_1, point_coords, point_labels, mask_input, has_mask_input, orig_im_size),
      f"{model_type}_decoder.onnx",
      export_params=True,
      opset_version=16,
      do_constant_folding=True,
      input_names = ['image_embed', 'high_res_feats_0', 'high_res_feats_1', 'point_coords', 'point_labels', 'mask_input', 'has_mask_input'],
      # input_names = ['image_embed', 'high_res_feats_0', 'high_res_feats_1', 'point_coords', 'point_labels', 'mask_input', 'has_mask_input', 'img_size'],
      output_names = ['masks', 'iou_predictions'],
      dynamic_axes = {"point_coords": {0: "num_labels", 1: "num_points"},
                      "point_labels": {0: "num_labels", 1: "num_points"},
                      "mask_input": {0: "bs", 1: "num_masks"},
      }
    )
```

--------------------------------

### MobileSAM Test Results Summary (Improved)

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Presents the test results for an improved MobileSAM implementation, noting its functional and reliable status after addressing previous issues. It covers mask quality, handling of IoU scores >1.0, performance metrics, and its suitability for specific use cases.

```text
### MobileSAM (Improved, Viable Alternative) ✅
- **Status:** Functional and reliable with fixes
- **Mask Quality:** Good after updates - consistent segmentation, reduced fragmentation via post-processing
- **IoU Scores:** Includes >1.0 (now treated as valid high-confidence); e.g., [0.9058, 0.9233, 0.9566, 1.0024] → select index 3
- **Decoder Output:** 1024x1024 masks
- **Issues Resolved:**
  - Single-mask: Scores >1.0 now valid; outputs tight masks.
  - Multi-mask: Select max score; post-process fixes strips/background.
- **Performance:** ~270ms encode, ~80ms decode on WebGPU (faster than SAM2)
- **Use Case:** Recommended for low-resource/mobile; experimental no longer.

**Updated Test Output Example:**
```
// Multi decoder with fixes
Decoder output: masks: dims=[1,4,1024,1024]
IoU scores: [0.9058, 0.9233, 0.9566, 1.0024]
Selected: index 3 (max, treated valid)
Result: Clean object segmentation after post-process
```
```

--------------------------------

### Image Encoder for SAM2 Model

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

The SAM2ImageEncoder class takes a SAM2Base model and extracts the image encoder. It processes input tensors through the backbone and FPN, then reshapes and prepares the features and positional embeddings for further use. It returns three levels of feature maps.

```python
from typing import Optional, Tuple, Any
import torch
from torch import nn
import torch.nn.functional as F
from torch.nn.init import trunc_normal_

from sam2.modeling.sam2_base import SAM2Base

class SAM2ImageEncoder(nn.Module):
    def __init__(self, sam_model: SAM2Base) -> None:
        super().__init__()
        self.model = sam_model
        self.image_encoder = sam_model.image_encoder
        self.no_mem_embed = sam_model.no_mem_embed

    def forward(self, x: torch.Tensor) -> tuple[Any, Any, Any]:
        backbone_out = self.image_encoder(x)
        backbone_out["backbone_fpn"][0] = self.model.sam_mask_decoder.conv_s0(
            backbone_out["backbone_fpn"][0]
        )
        backbone_out["backbone_fpn"][1] = self.model.sam_mask_decoder.conv_s1(
            backbone_out["backbone_fpn"][1]
        )

        feature_maps = backbone_out["backbone_fpn"][-self.model.num_feature_levels:]
        vision_pos_embeds = backbone_out["vision_pos_enc"][-self.model.num_feature_levels:]

        feat_sizes = [(x.shape[-2], x.shape[-1]) for x in vision_pos_embeds]

        # flatten NxCxHxW to HWxNxC
        vision_feats = [x.flatten(2).permute(2, 0, 1) for x in feature_maps]
        vision_pos_embeds = [x.flatten(2).permute(2, 0, 1) for x in vision_pos_embeds]

        vision_feats[-1] = vision_feats[-1] + self.no_mem_embed

        feats = [feat.permute(1, 2, 0).reshape(1, -1, *feat_size)
                 for feat, feat_size in zip(vision_feats[::-1], feat_sizes[::-1])][::-1]

        return feats[0], feats[1], feats[2]
```

--------------------------------

### Web Worker Message Protocol for SAM2 Inference

Source: https://context7.com/karlorz/next-sam/llms.txt

Implements a message-based protocol for offloading SAM2 inference to a background thread using Web Workers. It handles model initialization, image encoding, and mask decoding asynchronously, preventing UI blocking. Communication is done via `postMessage` and `addEventListener`.

```javascript
// Main thread - Initialize worker with model selection
const samWorker = new Worker(new URL("./worker.js", import.meta.url), {
  type: "module"
});

// Listen for worker messages
samWorker.addEventListener("message", (event) => {
  const { type, data } = event.data;

  if (type === "pong") {
    console.log(`Model loaded, device: ${data.device}`);
    // data: { success: true, device: "webgpu" }
  } else if (type === "encodeImageDone") {
    console.log(`Encoding took ${data.durationMs}ms`);
  } else if (type === "decodeMaskResult") {
    // data contains masks tensor and iou_predictions
    const maskTensor = data.masks;
    const scores = data.iou_predictions.cpuData;
    console.log(`Decoded ${maskTensor.dims[1]} masks`);
  } else if (type === "stats") {
    console.log("Performance stats:", data);
    // Shows download times, encoding times, decode times
  }
});

// Initialize with model selection
samWorker.postMessage({
  type: "ping",
  data: { modelId: "sam2_tiny" }  // or "mobilesam_tiny"
});

// Encode image
const { float32Array, shape } = canvasToFloat32Array(canvas);
samWorker.postMessage({
  type: "encodeImage",
  data: { float32Array, shape }
});

// Decode mask with points
const points = [
  { x: 512, y: 512, label: 1 }
];
samWorker.postMessage({
  type: "decodeMask",
  data: {
    points: points,
    maskArray: null,      // null for first decode
    maskShape: null
  }
});

// Refinement decode with previous mask
samWorker.postMessage({
  type: "decodeMask",
  data: {
    points: points,
    maskArray: previousMaskFloat32Array,
    maskShape: [1, 1, 256, 256]
  }
});

// Request performance statistics
samWorker.postMessage({ type: "stats" });

// Cleanup
samWorker.terminate();

```

--------------------------------

### Mask Decoding with Point Prompts

Source: https://context7.com/karlorz/next-sam/llms.txt

Generates segmentation masks based on point prompts (positive or negative). Supports iterative refinement by providing previous masks.

```APIDOC
## Mask Decoding with Point Prompts

### Description
Generates segmentation masks using point prompts (positive/negative) and supports iterative refinement by incorporating previous mask outputs.

### Method
`await sam.decode(points, maskInput)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **points** (array of objects) - Required - An array of points, where each object has `x`, `y` coordinates and a `label` (1 for positive, 0 for negative).
- **maskInput** (ort.Tensor) - Optional - A tensor representing a previous mask for iterative refinement. Shape [1, 1, 256, 256].

### Request Example
```javascript
// Initial mask decoding with positive and negative points
const points = [
  { x: 512, y: 300, label: 1 },  // Positive point (include)
  { x: 200, y: 150, label: 0 }   // Negative point (exclude)
];

let results = await sam.decode(points);

// Results contain multiple mask candidates with scores
const maskTensor = results.masks;  // Tensor [1, 3 or 4, 256, 256]
const scores = results.iou_predictions.cpuData;  // Float32Array of IoU scores

// Find best mask by highest score
let bestIdx = 0;
let bestScore = -Infinity;
for (let i = 0; i < scores.length; i++) {
  if (scores[i] > bestScore) {
    bestScore = scores[i];
    bestIdx = i;
  }
}
console.log(`Best mask index: ${bestIdx}, score: ${bestScore}`);

// Iterative refinement - add another point and reuse previous mask
points.push({ x: 600, y: 400, label: 1 });

// Convert previous best mask to tensor for refinement
// Assuming a helper function sliceTensor exists to extract a slice
// const maskArray = sliceTensor(results.masks, bestIdx);
// const maskTensor = new ort.Tensor("float32", maskArray, [1, 1, 256, 256]);

// Decode with mask input for refinement
// results = await sam.decode(points, maskTensor);
// console.log("Refined mask generated");
```

### Response
#### Success Response (200)
- **masks** (ort.Tensor) - Tensor containing multiple mask candidates. Shape [1, N, 256, 256] where N is the number of masks.
- **iou_predictions** (ort.Tensor) - Tensor of IoU scores for each mask candidate.
- **low_res_masks** (ort.Tensor) - Tensor of low-resolution masks.

#### Response Example
```json
{
  "masks": {
    "data": [ /* ... mask data ... */ ],
    "dims": [1, 4, 256, 256]
  },
  "iou_predictions": {
    "data": [0.85, 0.72, 0.65, 0.58],
    "dims": [4]
  },
  "low_res_masks": {
    "data": [ /* ... low res mask data ... */ ],
    "dims": [1, 4, 256, 256]
  }
}
```
```

--------------------------------

### MobileSAM Decoder Input Interface (JavaScript)

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Defines the expected input structure for the MobileSAM decoder. It includes image embeddings from the encoder, point coordinates and labels for guidance, an optional mask input, a flag indicating mask input presence, and the original image dimensions.

```javascript
{
  image_embeddings: <encoder output>,
  point_coords: Tensor([1, N, 2]),
  point_labels: Tensor([1, N]),
  mask_input: Tensor([1, 1, 256, 256]),
  has_mask_input: Tensor([1]),
  orig_im_size: Tensor([2]) // [height, width]
}
```

--------------------------------

### SAM2 Test Results Summary

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Summarizes the test results for the SAM2 model, highlighting its status as fully functional and reliable with excellent mask quality and consistent performance. It details typical IoU scores, decoder output dimensions, and processing times.

```text
### SAM2 (Recommended Default) ✅
- **Status:** Fully functional, reliable
- **Mask Quality:** Excellent - accurately segments objects
- **IoU Scores:** Valid range (0.08 - 0.13 for 3 masks)
- **Decoder Output:** 256x256 masks, 3 candidates
- **Performance:** ~700ms encode, ~100ms decode on WebGPU
- **Reliability:** Consistent results across different images
- **Use Case:** Production-ready default model

**Test Output Example:**
```
Decoder output: masks: dims=[1,3,256,256]
IoU scores: [0.0850, 0.1012, 0.1321]
Selected: index 0 (smallest/tightest mask)
Result: Clean flamingo segmentation
```
```

--------------------------------

### Check SAM2 Encoder ONNX Model Validity

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Loads a SAM2 encoder model exported to ONNX format and checks its structural validity using onnx.checker.check_model. This ensures the exported ONNX model is correctly formatted and can be used for inference.

```python
import onnx 

onnx_model = onnx.load("sam2_hiera_tiny_encoder.onnx")
onnx.checker.check_model(onnx_model)
```

--------------------------------

### Image Decoder for SAM2 Model

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

The SAM2ImageDecoder class reconstructs segmentation masks from image embeddings and various prompts. It embeds points and masks, then uses the mask decoder to predict masks and IoU scores. It supports multi-mask output based on a flag.

```python
class SAM2ImageDecoder(nn.Module):
    def __init__(
            self, 
            sam_model: SAM2Base,
            multimask_output: bool
    ) -> None:
        super().__init__()
        self.mask_decoder = sam_model.sam_mask_decoder
        self.prompt_encoder = sam_model.sam_prompt_encoder
        self.model = sam_model
        self.multimask_output = multimask_output

    @torch.no_grad()
    def forward(
            self,
            image_embed: torch.Tensor,
            high_res_feats_0: torch.Tensor,
            high_res_feats_1: torch.Tensor,
            point_coords: torch.Tensor,
            point_labels: torch.Tensor,
            mask_input: torch.Tensor,
            has_mask_input: torch.Tensor,
            # img_size: torch.Tensor
    ):
        sparse_embedding = self._embed_points(point_coords, point_labels)
        self.sparse_embedding = sparse_embedding
        if has_mask_input.item() == 1:
            mask_input = mask_input[:,0,None]
        dense_embedding = self._embed_masks(mask_input, has_mask_input)

        high_res_feats = [high_res_feats_0, high_res_feats_1]
        image_embed = image_embed

        masks, iou_predictions, _, _ = self.mask_decoder.predict_masks(
            image_embeddings=image_embed,
            image_pe=self.prompt_encoder.get_dense_pe(),
            sparse_prompt_embeddings=sparse_embedding,
            dense_prompt_embeddings=dense_embedding,
            repeat_image=False,
            high_res_features=high_res_feats,
        )

        if self.multimask_output:
            masks = masks[:, 1:, :, :] 
            iou_predictions = iou_predictions[:, 1:]
        else:
            masks, iou_predictions = self.mask_decoder._dynamic_multimask_via_stability(masks, iou_predictions)

        masks = torch.clamp(masks, -32.0, 32.0)
        # print("original masks shape", masks.shape, iou_predictions.shape, img_size.shape)

        # masksDummy = F.interpolate(masks, (img_size[0], img_size[1]), mode="bilinear", align_corners=False)

        return masks, iou_predictions

    def _embed_points(self, point_coords: torch.Tensor, point_labels: torch.Tensor) -> torch.Tensor:

        point_coords = point_coords + 0.5

        padding_point = torch.zeros((point_coords.shape[0], 1, 2), device=point_coords.device)
        padding_label = -torch.ones((point_labels.shape[0], 1), device=point_labels.device)
        point_coords = torch.cat([point_coords, padding_point], dim=1)
        point_labels = torch.cat([point_labels, padding_label], dim=1)

        point_coords[:, :, 0] = point_coords[:, :, 0] / self.model.image_size
        point_coords[:, :, 1] = point_coords[:, :, 1] / self.model.image_size

        point_embedding = self.prompt_encoder.pe_layer._pe_encoding(point_coords)
        point_labels = point_labels.unsqueeze(-1).expand_as(point_embedding)

        point_embedding = point_embedding * (point_labels != -1)
        point_embedding = point_embedding + self.prompt_encoder.not_a_point_embed.weight * (
                point_labels == -1
        )

        for i in range(self.prompt_encoder.num_point_embeddings):
            point_embedding = point_embedding + self.prompt_encoder.point_embeddings[i].weight * (point_labels == i)

        return point_embedding

    def _embed_masks(self, input_mask: torch.Tensor, has_mask_input: torch.Tensor) -> torch.Tensor:

```

--------------------------------

### SAM2 Mask Embedding Calculation in PyTorch

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

This Python code calculates mask embeddings for the SAM2 model. It handles cases with and without a mask input, incorporating mask downscaling and a no-mask embedding. The output is a tensor representing the mask embedding.

```python
# Assuming 'has_mask_input', 'self.prompt_encoder.mask_downscaling', 
# 'input_mask', 'self.prompt_encoder.no_mask_embed.weight' are defined.
# Example placeholder definitions:
# has_mask_input = torch.tensor([1.0])
# class MockPromptEncoder:
#     def __init__(self):
#         self.mask_downscaling = lambda x: x * 0.5 # Mock function
#         self.no_mask_embed = type('obj', (object,), {'weight': torch.zeros(1, 256, 1, 1)})()
# self.prompt_encoder = MockPromptEncoder()
# input_mask = torch.randn(1, 1, 256, 256)

mask_embedding = has_mask_input * self.prompt_encoder.mask_downscaling(input_mask)
mask_embedding = mask_embedding + (
        1 - has_mask_input
) * self.prompt_encoder.no_mask_embed.weight.reshape(1, -1, 1, 1)
return mask_embedding
```

--------------------------------

### Post-Process MobileSAM Masks with Thresholding

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

This JavaScript function provides a basic post-processing step for masks generated by MobileSAM. It applies a simple threshold to convert raw mask values into binary (0.0 or 1.0) predictions, helping to clean up fragmented edges after resizing for refinement.

```javascript
// Post-process mask (example: simple threshold)
function postProcessMask(maskArray, threshold = 0.5) {
  return maskArray.map(val => val > threshold ? 1.0 : 0.0);
}

// After resizing for refinement
// const refinementMaskArray = postProcessMask(maskCanvasToFloat32Array(refinementMaskCanvas));

```

--------------------------------

### SAM2 Decoder Input Interface (JavaScript)

Source: https://github.com/karlorz/next-sam/blob/main/MOBILESAM_ANALYSIS.md

Defines the input structure for the SAM2 decoder. Similar to MobileSAM, it takes encoder outputs (including high-resolution features), point coordinates and labels, an optional mask input, and a flag for its presence. It does not explicitly require original image size.

```javascript
{
  image_embed: <encoder output>,
  high_res_feats_0: <encoder output>,
  high_res_feats_1: <encoder output>,
  point_coords: Tensor([1, N, 2]),
  point_labels: Tensor([1, N]),
  mask_input: Tensor([1, 1, 256, 256]),
  has_mask_input: Tensor([1])
}
```

--------------------------------

### Check SAM2 Decoder ONNX Model Validity

Source: https://github.com/karlorz/next-sam/blob/main/notebooks/SAM2-to-ONNX.ipynb

Loads a SAM2 decoder model exported to ONNX format and checks its structural validity using onnx.checker.check_model. This step is crucial to ensure the exported ONNX model is free from structural errors before proceeding with inference.

```python
import onnx 

onnx_model = onnx.load("sam2_hiera_tiny_decoder.onnx")
onnx.checker.check_model(onnx_model)
```

--------------------------------

### Encode Image for Segmentation (JavaScript)

Source: https://context7.com/karlorz/next-sam/llms.txt

Prepares an image for segmentation by resizing it to 1024x1024 and converting it into an ONNX Tensor. This tensor is then passed to the `encodeImage` method, which generates embeddings necessary for subsequent mask decoding. It assumes the existence of `canvasToFloat32Array` and `resizeCanvas` utility functions.

```javascript
import * as ort from "onnxruntime-web";
import { canvasToFloat32Array, resizeCanvas } from "./lib/imageutils";

// Load and prepare image
const img = new Image();
img.src = "path/to/image.jpg";
img.onload = async function() {
  // Resize to 1024x1024 (required by both models)
  const canvas = document.createElement("canvas");
  canvas.width = 1024;
  canvas.height = 1024;
  canvas.getContext("2d").drawImage(img, 0, 0, 1024, 1024);

  // Convert to Float32Array tensor
  const { float32Array, shape } = canvasToFloat32Array(canvas);
  const inputTensor = new ort.Tensor("float32", float32Array, shape);

  // Encode image (stores embeddings internally)
  await sam.encodeImage(inputTensor);
  console.log("Image encoded, ready for mask decoding");
};

```

--------------------------------

### Decode Segmentation Mask with Point Prompts (JavaScript)

Source: https://context7.com/karlorz/next-sam/llms.txt

Generates segmentation masks using point prompts (positive or negative) and optionally refines a previous mask. The `decode` method returns mask candidates and their Intersection over Union (IoU) scores. The code demonstrates how to select the best mask and perform iterative refinement by including a previous mask as input.

```javascript
// Initial mask decoding with positive and negative points
const points = [
  { x: 512, y: 300, label: 1 },  // Positive point (include)
  { x: 200, y: 150, label: 0 }   // Negative point (exclude)
];

let results = await sam.decode(points);

// Results contain multiple mask candidates with scores
const maskTensor = results.masks;  // Tensor [1, 3 or 4, 256, 256]
const scores = results.iou_predictions.cpuData;  // Float32Array of IoU scores

// Find best mask by highest score
let bestIdx = 0;
let bestScore = -Infinity;
for (let i = 0; i < scores.length; i++) {
  if (scores[i] > bestScore) {
    bestScore = scores[i];
    bestIdx = i;
  }
}
console.log(`Best mask index: ${bestIdx}, score: ${bestScore}`);

// Iterative refinement - add another point and reuse previous mask
points.push({ x: 600, y: 400, label: 1 });

// Convert previous best mask to tensor for refinement
const maskArray = sliceTensor(results.masks, bestIdx); // Assuming sliceTensor exists
const maskTensor = new ort.Tensor("float32", maskArray, [1, 1, 256, 256]);

// Decode with mask input for refinement
results = await sam.decode(points, maskTensor);
console.log("Refined mask generated");

```

--------------------------------

### Convert Float32Array Mask to HTML Canvas for Visualization (JavaScript)

Source: https://context7.com/karlorz/next-sam/llms.txt

Creates an HTML canvas visually representing a segmentation mask from a Float32Array. Positive mask values are rendered in green, while zero or negative values are transparent.

```javascript
import { float32ArrayToCanvas } from "./lib/imageutils";

// Decode mask and convert to canvas
const results = await sam.decode(points);
const maskTensor = results.masks;
const [, , width, height] = maskTensor.dims;  // [1, 3, 256, 256]

// Extract best mask (assuming index 0)
const maskArray = sliceTensor(maskTensor, 0); // Assuming sliceTensor is defined elsewhere

// Convert to canvas for visualization
const maskCanvas = float32ArrayToCanvas(maskArray, width, height);

// Display on page
document.body.appendChild(maskCanvas);

// Or overlay on original image
const displayCanvas = document.getElementById("display");
const ctx = displayCanvas.getContext("2d");
ctx.drawImage(originalImage, 0, 0);
ctx.globalAlpha = 0.7;
ctx.drawImage(maskCanvas, 0, 0);
ctx.globalAlpha = 1.0;

// Canvas properties
console.log(maskCanvas.width);   // 256
console.log(maskCanvas.height);  // 256
// Pixels where mask > 0 are green (#32cd32), others transparent

```

--------------------------------

### Mask Image Canvas Compositing

Source: https://context7.com/karlorz/next-sam/llms.txt

Applies a mask to an image by compositing two canvases, creating a cropped version where only masked pixels remain visible. This function takes an image canvas and a resized mask canvas as input, returning a new canvas with the mask applied.

```javascript
import { maskImageCanvas } from "./lib/imageutils";

// Original image canvas
const imageCanvas = document.createElement("canvas");
const ctx = imageCanvas.getContext("2d");
imageCanvas.width = 1024;
imageCanvas.height = 1024;
ctx.drawImage(originalImage, 0, 0, 1024, 1024);

// Mask canvas (from segmentation result)
const maskCanvas = float32ArrayToCanvas(maskArray, 256, 256);
const resizedMask = resizeCanvas(maskCanvas, { w: 1024, h: 1024 });

// Apply mask to image
const croppedCanvas = maskImageCanvas(imageCanvas, resizedMask);

// Export as image
const croppedDataURL = croppedCanvas.toDataURL("image/png");

// Download cropped image
const link = document.createElement("a");
link.href = croppedDataURL;
link.download = "segmented_object.png";
document.body.appendChild(link);
link.click();
document.body.removeChild(link);

// Or display in page
document.getElementById("result").appendChild(croppedCanvas);

```