HQ-SAM

https://github.com/syscv/sam-hq
Admin
HQ-SAM is an upgraded Segment Anything Model that provides high-quality zero-shot segmentation for...
Tokens:97,170
Snippets:532
Trust Score:7.8
License:Apache-2.0
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Environment setup for SegInW

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Commands to set up the environment for SegInW. Run from the seginw directory, it installs the GroundingDINO package in editable mode.

```bash
cd seginw
python -m pip install -e GroundingDINO
```

--------------------------------

### Clone and install HQ-SAM from source

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Clone the repository and install it in editable mode. Requires Python >=3.8, PyTorch >=1.7, and TorchVision >=0.8.

```bash
git clone https://github.com/SysCV/sam-hq.git
cd sam-hq; pip install -e .
```

--------------------------------

### Install optional dependencies for HQ-SAM

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Install optional dependencies for mask post-processing, COCO format saving, example notebooks, and ONNX export. Also requires jupyter for running notebooks.

```bash
pip install opencv-python pycocotools matplotlib onnxruntime onnx timm
```

--------------------------------

### Run the HQ-SAM pip example

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Run the example script that demonstrates using the pip-installed HQ-SAM with a vit-l model. Set PYTHONPATH to the current directory before running.

```bash
export PYTHONPATH=$(pwd)
python demo/demo_hqsam_pip_example.py
```

--------------------------------

### HQ-SAM 2 video predictor usage example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.build_sam.md

Usage example from the sam-hq2/README.md video example. Builds the predictor with a config and checkpoint, then runs inference with torch.inference_mode and autocast, adding points and propagating masks through the video.

```python
import torch
from sam2.build_sam import build_sam2_hq_video_predictor

predictor = build_sam2_hq_video_predictor(
    "configs/sam2.1/sam2.1_hq_hiera_l.yaml",
    "./checkpoints/sam2.1_hq_hiera_large.pt",
    device="cuda",
)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state("<your_video>")
    frame_idx, object_ids, masks = predictor.add_new_points_or_box(state, 0, 1, points=..., labels=...)
    for frame_idx, object_ids, masks in predictor.propagate_in_state(state):
        ...
```

--------------------------------

### Usage example with manual preprocessing

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Shows how to use SamPredictor with a manual, batched pre-processing path using ResizeLongestSide and set_torch_image.

```python
import torch
from segment_anything import SamPredictor, sam_model_registry
from segment_anything.utils.transforms import ResizeLongestSide

sam = sam_model_registry["vit_b"](checkpoint="sam_hq_vit_b.pth")
predictor = SamPredictor(sam)

# Manual, batched pre-processing path
transform = ResizeLongestSide(1024)
img = transform.apply_image(image_np)                      # HWC uint8
t = torch.as_tensor(img, device="cuda").permute(2, 0, 1).contiguous()[None]
predictor.set_torch_image(t, image_np.shape[:2])
```

--------------------------------

### Example evaluation script on a single dataset

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Example command to evaluate a single dataset (Airplane-Parts) using the GroundingDINO and SAM-HQ models, saving results as JSON.

```bash
python test_ap_on_seginw.py -c GroundingDINO/groundingdino/config/GroundingDINO_SwinB.py -p pretrained_checkpoint/groundingdino_swinb_cogcoor.pth --anno_path data/seginw/Airplane-Parts/valid/_annotations_min1cat.coco.json --image_dir data/seginw/Airplane-Parts/valid/ --use_sam_hq --save_json
```

--------------------------------

### HQ-SAM-B training script

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Example command to train the HQ-SAM-B model using 8 GPUs with the specified checkpoint and output directory.

```bash
python -m torch.distributed.launch --nproc_per_node=8 train.py --checkpoint ./pretrained_checkpoint/sam_vit_b_01ec64.pth --model-type vit_b --output work_dirs/hq_sam_b
```

--------------------------------

### HQ-SAM-H training script

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Example command to train the HQ-SAM-H model using 8 GPUs with the specified checkpoint and output directory.

```bash
python -m torch.distributed.launch --nproc_per_node=8 train.py --checkpoint ./pretrained_checkpoint/sam_vit_h_4b8939.pth --model-type vit_h --output work_dirs/hq_sam_h
```

--------------------------------

### HQ-SAM-L training script

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Example command to train the HQ-SAM-L model using 8 GPUs with the specified checkpoint and output directory.

```bash
python -m torch.distributed.launch --nproc_per_node=8 train.py --checkpoint ./pretrained_checkpoint/sam_vit_l_0b3195.pth --model-type vit_l --output work_dirs/hq_sam_l
```

--------------------------------

### ImageEncoderViT usage example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Example of instantiating and running the encoder on a random input. Shows the output shapes for the final feature map and intermediate embeddings.

```python
from segment_anything.modeling import ImageEncoderViT

encoder = ImageEncoderViT(img_size=1024, embed_dim=768, depth=12,
                          num_heads=12, global_attn_indexes=(2, 5, 8, 11),
                          window_size=14, out_chans=256)
out, interm = encoder(torch.randn(1, 3, 1024, 1024))
print(out.shape, [f.shape for f in interm])  # (1, 256, 64, 64), 4 entries @ 64x64x768
```

--------------------------------

### Conda environment setup for HQ-SAM

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Creates a conda environment named 'sam_hq' with Python 3.8, installs PyTorch 1.10.0, torchvision 0.11.0, and CUDA 11.1, then installs additional dependencies and clones the HQ-SAM repository.

```bash
conda create --name sam_hq python=3.8 -y
conda activate sam_hq
conda install pytorch==1.10.0 torchvision==0.11.0 cudatoolkit=11.1 -c pytorch -c nvidia
pip install opencv-python pycocotools matplotlib onnxruntime onnx timm

# under your working directory
git clone https://github.com/SysCV/sam-hq.git
cd sam-hq
pip install -e .
export PYTHONPATH=$(pwd)
```

--------------------------------

### Usage example for build_sam2_hf and build_sam2_video_predictor_hf

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.build_sam.md

Imports and uses both HF builder functions with the facebook/sam2.1-hiera-large model.

```python
from sam2.build_sam import build_sam2_hf, build_sam2_video_predictor_hf

model = build_sam2_hf("facebook/sam2.1-hiera-large", device="cuda")
vpred = build_sam2_video_predictor_hf("facebook/sam2.1-hiera-large", device="cuda")
```

--------------------------------

### Install and use HQ-SAM via pip

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Install the segment-anything-hq package and load a model checkpoint using the sam_model_registry. Replace <model_type> with vit_l, vit_b, vit_h, or vit_tiny, and <path/to/checkpoint> with your checkpoint path.

```bash
pip install segment-anything-hq
```

```python
python
from segment_anything_hq import sam_model_registry
model_type = "<model_type>" #"vit_l/vit_b/vit_h/vit_tiny"
sam_checkpoint = "<path/to/checkpoint>"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
```

--------------------------------

### Install Grounding DINO with pip

Source: https://github.com/syscv/sam-hq/blob/main/seginw/GroundingDINO/README.md

Install the package in editable mode. If a CUDA environment is available, ensure the CUDA_HOME environment variable is set; otherwise, the package will be compiled under CPU-only mode.

```bash
pip install -e .
```

--------------------------------

### Usage example: instantiate and generate masks

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.automatic_mask_generator.md

Example of creating a SAM2AutomaticMaskGenerator with a pretrained model and generating masks for an image. Uses build_sam2 to load the model, sets points_per_side=16, pred_iou_thresh=0.7, stability_score_thresh=0.9, and enables use_m2m. The generate method returns a list of mask dictionaries.

```python
from sam2.automatic_mask_generator import SAM2AutomaticMaskGenerator
from sam2.build_sam import build_sam2

mask_generator = SAM2AutomaticMaskGenerator(
    build_sam2("configs/sam2.1/sam2.1_hq_hiera_l.yaml", "checkpoints/sam2.1_hq_hiera_large.pt"),
    points_per_side=16,
    pred_iou_thresh=0.7,
    stability_score_thresh=0.9,
    use_m2m=True,
)
anns = mask_generator.generate(image)
```

--------------------------------

### Usage example with error handling and HQ token

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_image_predictor.md

Example showing how to instantiate SAM2ImagePredictor, handle the RuntimeError when predict is called before set_image, and perform multi-click prediction with hq_token_only=True.

```python
import numpy as np
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor(model)

# error handling: predict before set_image
try:
    predictor.predict(point_coords=np.array([[10, 10]]), point_labels=np.array([1]))
except RuntimeError as e:
    print(e)  # "An image must be set with .set_image(...) before mask prediction."

predictor.set_image(image_rgb)
# multi-click with HQ token only
masks, scores, logits = predictor.predict(
    point_coords=np.array([[500, 375], [600, 400]]),
    point_labels=np.array([1, 1]),
    multimask_output=False,
    hq_token_only=True,
)
```

--------------------------------

### predict usage example with NumPy

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Example of using predictor.predict with a single click and box prompt. Shows HQ-only mode and error handling for missing image.

```python
import numpy as np
from segment_anything import SamPredictor, sam_model_registry

predictor = SamPredictor(sam_model_registry["vit_l"](checkpoint="sam_hq_vit_l.pth"))
predictor.set_image(image_rgb)  # HWC uint8

# single click, HQ-only mode
masks, scores, low_res = predictor.predict(
    point_coords=np.array([[500, 375]]),
    point_labels=np.array([1]),
    multimask_output=False,
    hq_token_only=True,
)

# box prompt with error handling
try:
    masks, scores, low_res = predictor.predict(box=np.array([4, 13, 1007, 1023]))
except RuntimeError as e:
    print("set an image first:", e)
```

--------------------------------

### RLE Round-trip Example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.utils.md

Example usage of the RLE conversion functions.

```APIDOC
## RLE Round-trip Example

### Description
Example demonstrating how to convert a mask to RLE, back to mask, and compute area.

### Code
```python
from segment_anything.utils.amg import mask_to_rle_pytorch, rle_to_mask, area_from_rle
import torch

rles = mask_to_rle_pytorch(torch.tensor(mask_bhw).bool())  # list of RLE dicts
mask = rle_to_mask(rles[0])
area = area_from_rle(rles[0])
```
```

--------------------------------

### SAM2Transforms

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.utils.md

Documentation for the SAM2Transforms class, including constructor parameters, callable methods, and usage example.

```APIDOC
## SAM2Transforms

### Description
Transforms images, coordinates, boxes, and masks for SAM 2 models. Handles resizing, normalization, coordinate/box scaling, and mask post-processing (hole filling, sprinkle removal, and resizing).

### Constructor
`SAM2Transforms(resolution, mask_threshold, max_hole_area=0.0, max_sprinkle_area=0.0)`

#### Parameters
- **resolution** (int) - Required - Square target resolution (model `image_size`, 1024 for HQ configs)
- **mask_threshold** (float) - Required - Binarization threshold for output masks
- **max_hole_area** (float) - Optional - Fill holes with area ≤ this value post-mask (requires CUDA extension)
- **max_sprinkle_area** (float) - Optional - Remove sprinkles with area ≤ this value (requires CUDA extension)

### Methods

#### `__call__(self, x)`
- **Description**: Converts a single PIL image to a normalized tensor of shape `3xHxW`.
- **Parameters**:
  - **x** (PIL Image) - Required - Input image
- **Returns**: `torch.Tensor` - Normalized image tensor

#### `forward_batch(self, img_list)`
- **Description**: Converts a list of images to a batched tensor of shape `Bx3xHxW`.
- **Parameters**:
  - **img_list** (list) - Required - List of PIL images
- **Returns**: `torch.Tensor` - Batched normalized tensor

#### `transform_coords(self, coords, normalize=False, orig_hw=None) -> torch.Tensor`
- **Description**: Transforms coordinates to model-frame absolute pixels. If `normalize=True`, divides by `orig_hw` (H, W) first, then multiplies by `resolution`.
- **Parameters**:
  - **coords** (torch.Tensor) - Required - Coordinates with last dim 2
  - **normalize** (bool) - Optional - Whether to normalize coordinates
  - **orig_hw** (tuple) - Optional - Original (H, W) dimensions; required if `normalize=True`
- **Returns**: `torch.Tensor` - Transformed coordinates

#### `transform_boxes(self, boxes, normalize=False, orig_hw=None) -> torch.Tensor`
- **Description**: Transforms bounding boxes of shape `Bx4` by reshaping to `-1,2,2`, applying `transform_coords`, and reshaping back.
- **Parameters**:
  - **boxes** (torch.Tensor) - Required - Bounding boxes
  - **normalize** (bool) - Optional - Whether to normalize boxes
  - **orig_hw** (tuple) - Optional - Original (H, W) dimensions; required if `normalize=True`
- **Returns**: `torch.Tensor` - Transformed boxes

#### `postprocess_masks(self, masks: torch.Tensor, orig_hw) -> torch.Tensor`
- **Description**: Post-processes float masks: optionally fills holes and removes sprinkles using connected components (CUDA extension), then interpolates to `orig_hw` (bilinear). On any exception, warns and returns input masks unchanged.
- **Parameters**:
  - **masks** (torch.Tensor) - Required - Float masks
  - **orig_hw** (tuple) - Required - Original (H, W) dimensions
- **Returns**: `torch.Tensor` - Post-processed masks

### Example
```python
import torch
from sam2.utils.transforms import SAM2Transforms
from PIL import Image

tf = SAM2Transforms(resolution=1024, mask_threshold=0.0, max_hole_area=8.0)
img = Image.open("frame.jpg").convert("RGB")
x = tf(img)                                   # 3x1024x1024, normalized
coords = tf.transform_coords(torch.tensor([[0.5, 0.5]]), normalize=True, orig_hw=(1080, 1920))
```
```

--------------------------------

### MaskDecoderHQ usage example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Example of instantiating MaskDecoderHQ with a TwoWayTransformer and calling it with image embeddings, prompts, and intermediate embeddings. Demonstrates the hq_token_only parameter.

```python
from segment_anything.modeling import MaskDecoderHQ, TwoWayTransformer

decoder = MaskDecoderHQ(
    num_multimask_outputs=3,
    transformer=TwoWayTransformer(depth=2, embedding_dim=256, mlp_dim=2048, num_heads=8),
    transformer_dim=256,
    iou_head_depth=3,
    iou_head_hidden_dim=256,
    vit_dim=768,
)
masks, iou = decoder(
    image_embeddings=emb, image_pe=pe, sparse_prompt_embeddings=se,
    dense_prompt_embeddings=de, multimask_output=False,
    hq_token_only=True, interm_embeddings=interm,
)
```

--------------------------------

### HQ-SAM-L evaluation script

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Example evaluation command for the HQ-SAM-L model using 1 GPU, specifying the checkpoint, model type, output, and restore-model path.

```bash
python -m torch.distributed.launch --nproc_per_node=1 train.py --checkpoint ./pretrained_checkpoint/sam_vit_l_0b3195.pth --model-type vit_l --output work_dirs/hq_sam_l --eval --restore-model work_dirs/hq_sam_l/epoch_11.pth
```

--------------------------------

### Builder usage with checkpoint

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Example of loading a SAM model using the registry and a checkpoint. The mask decoder's hf_token weight is frozen by default, as asserted. Use this pattern to instantiate a model for inference or fine-tuning.

```python
import torch
from segment_anything import sam_model_registry

model_type = "vit_l"
sam_checkpoint = "pretrained_checkpoint/sam_hq_vit_l.pth"
sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
sam.to(device="cuda" if torch.cuda.is_available() else "cpu")

assert not sam.mask_decoder.hf_token.weight.requires_grad  # frozen by default
```

--------------------------------

### HQ-SAM-L visualization script

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Example command to visualize results for the HQ-SAM-L model, adding the --visualize flag to the evaluation command.

```bash
python -m torch.distributed.launch --nproc_per_node=1 train.py --checkpoint ./pretrained_checkpoint/sam_vit_l_0b3195.pth --model-type vit_l --output work_dirs/hq_sam_l --eval --restore-model work_dirs/hq_sam_l/epoch_11.pth --visualize
```

--------------------------------

### Usage example: SAM2ImagePredictor with build_sam2

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_image_predictor.md

Shows how to instantiate the predictor with a SAM 2.1 HQ model and run inference on a single image. Uses torch.inference_mode and autocast with bfloat16, then calls set_image and predict with a box prompt and multimask_output=False.

```python
import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor(
    build_sam2("configs/sam2.1/sam2.1_hq_hiera_l.yaml", "checkpoints/sam2.1_hq_hiera_large.pt")
)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(image_rgb)
    masks, scores, low_res = predictor.predict(box=box_xyxy, multimask_output=False)
```

--------------------------------

### TwoWayTransformer usage example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Example of instantiating TwoWayTransformer and calling forward. Uses depth=2, embedding_dim=256, num_heads=8, mlp_dim=2048. The forward call passes image_embedding, image_pe, and point_embedding tensors.

```python
from segment_anything.modeling import TwoWayTransformer

t = TwoWayTransformer(depth=2, embedding_dim=256, num_heads=8, mlp_dim=2048)
q, k = t(image_embedding=torch.randn(1, 256, 64, 64),
         image_pe=pe, point_embedding=torch.randn(1, 5, 256))
```

--------------------------------

### propagate_in_video_preflight method signature

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_video_predictor.md

Method signature for the internal preflight step called at the start of propagate_in_video. Sets tracking_has_started, consolidates per-object temporary outputs, and asserts frame indices match input frames.

```python
def propagate_in_video_preflight(self, inference_state):
```

--------------------------------

### Forward method raises NotImplementedError

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.modeling.sam2_hq_base.md

The forward method is not implemented and always raises NotImplementedError. Use SAM2HQVideoPredictor for inference or the SAM 2 training code for training/fine-tuning. An inference example is available in notebooks/video_predictor_example.ipynb.

```python
def forward(self, *args, **kwargs):
    raise NotImplementedError(
        "Please use the corresponding methods in SAM2VideoPredictor for inference or SAM2Train for training/fine-tuning"
        "See notebooks/video_predictor_example.ipynb for an inference example."
    )
```

--------------------------------

### RLE round-trip example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.utils.md

Shows how to convert a binary mask tensor to RLE, decode the first RLE back to a mask, and compute its area. Requires torch and the amg module functions.

```python
from segment_anything.utils.amg import mask_to_rle_pytorch, rle_to_mask, area_from_rle
import torch

rles = mask_to_rle_pytorch(torch.tensor(mask_bhw).bool())  # list of RLE dicts
mask = rle_to_mask(rles[0])
area = area_from_rle(rles[0])
```

--------------------------------

### Using SAM2Transforms

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.utils.md

Usage example for SAM2Transforms: creates a transform with resolution 1024, mask_threshold 0.0, and max_hole_area 8.0; loads an image; applies the transform to get a normalized 3x1024x1024 tensor; transforms normalized coordinates to model-frame absolute pixels.

```python
import torch
from sam2.utils.transforms import SAM2Transforms
from PIL import Image

tf = SAM2Transforms(resolution=1024, mask_threshold=0.0, max_hole_area=8.0)
img = Image.open("frame.jpg").convert("RGB")
x = tf(img)                                   # 3x1024x1024, normalized
coords = tf.transform_coords(torch.tensor([[0.5, 0.5]]), normalize=True, orig_hw=(1080, 1920))

```

--------------------------------

### Launch distributed training with torch.distributed.launch

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/training.md

Launches distributed training with torch.distributed.launch. Requires --checkpoint, --model-type, and --output; other options have defaults.

```bash
python -m torch.distributed.launch --nproc_per_node=<num_gpus> train.py \
    --checkpoint <path/to/checkpoint> --model-type <model_type> --output <path/to/output>
```

--------------------------------

### ResizeLongestSide usage example

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.utils.md

Example of using ResizeLongestSide with target_length 1024. It applies the transform to an image, coordinates, and boxes. This is used internally by SamPredictor.set_image.

```python
from segment_anything.utils.transforms import ResizeLongestSide

transform = ResizeLongestSide(1024)
img = transform.apply_image(image_np)                 # HWC uint8
coords = transform.apply_coords(np.array([[100, 200]]), image_np.shape[:2])
boxes = transform.apply_boxes(np.array([[4, 13, 1007, 1023]]), image_np.shape[:2])

```

--------------------------------

### Run baseline SAM demo

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Runs the demo script for baseline SAM. Requires downloading the original SAM checkpoint and placing it in the pretrained_checkpoint folder.

```bash
python demo/demo_sam.py
```

--------------------------------

### Instantiate HQ-SAM model from registry

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Shows how to load a HQ-SAM model using the sam_model_registry with a specified model type and checkpoint path.

```python
from segment_anything import sam_model_registry
sam = sam_model_registry["<model_type>"](checkpoint="<path/to/checkpoint>")
```

--------------------------------

### Download and unzip SegInW dataset

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Commands to download and extract the SegInW dataset from Hugging Face. Run from the data directory.

```bash
cd data
wget https://huggingface.co/sam-hq-team/SegInW/resolve/main/seginw.zip
unzip seginw.zip
```

--------------------------------

### Usage example with SamAutomaticMaskGenerator

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Example of creating a mask generator with a registered SAM model and generating masks. Requires opencv-python for min_mask_region_area and pycocotools for coco_rle output mode.

```python
from segment_anything import SamAutomaticMaskGenerator, sam_model_registry

mask_generator = SamAutomaticMaskGenerator(
    sam_model_registry["vit_h"](checkpoint="sam_hq_vit_h.pth"),
    points_per_side=16,
    pred_iou_thresh=0.86,
    stability_score_thresh=0.92,
    min_mask_region_area=100,   # requires opencv-python
    output_mode="coco_rle",     # requires pycocotools
)
anns = mask_generator.generate(image)
print(anns[0]["bbox"], anns[0]["predicted_iou"])
```

--------------------------------

### build_sam2_hf(model_id, **kwargs)

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.build_sam.md

Downloads a SAM 2 checkpoint from Hugging Face and builds the model using the provided configuration and checkpoint path.

```APIDOC
## build_sam2_hf(model_id, **kwargs)

### Description
Downloads the checkpoint from the Hugging Face hub for the given `model_id` and delegates to `build_sam2` to construct the model.

### Parameters
- **model_id** (str) - Required - The Hugging Face model identifier (e.g., "facebook/sam2.1-hiera-large").
- **kwargs** - Additional keyword arguments passed to `build_sam2` (e.g., `device`).

### Returns
- The constructed SAM 2 model object.

### Example
```python
from sam2.build_sam import build_sam2_hf
model = build_sam2_hf("facebook/sam2.1-hiera-large", device="cuda")
```
```

--------------------------------

### Training folder structure

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Shows the expected directory layout for the training folder, including subfolders for data, pretrained checkpoints, training scripts, and utilities.

```text
train
|____data
|____pretrained_checkpoint
|____train.py
|____utils
| |____dataloader.py
| |____misc.py
| |____loss_mask.py
|____segment_anything_training
|____work_dirs
```

--------------------------------

### Run Light HQ-SAM demo

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Runs the demo script to obtain Light HQ-SAM's visual result.

```bash
python demo/demo_hqsam_light.py
```

--------------------------------

### build_sam2_video_predictor_hf(model_id, **kwargs)

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.build_sam.md

Downloads a SAM 2 checkpoint from Hugging Face and builds a video predictor model.

```APIDOC
## build_sam2_video_predictor_hf(model_id, **kwargs)

### Description
HF variant of `build_sam2_video_predictor`. Downloads the checkpoint from the Hugging Face hub for the given `model_id` and delegates to `build_sam2_video_predictor`.

### Parameters
- **model_id** (str) - Required - The Hugging Face model identifier (e.g., "facebook/sam2.1-hiera-large").
- **kwargs** - Additional keyword arguments passed to `build_sam2_video_predictor` (e.g., `device`).

### Returns
- The constructed SAM 2 video predictor object.

### Example
```python
from sam2.build_sam import build_sam2_video_predictor_hf
vpred = build_sam2_video_predictor_hf("facebook/sam2.1-hiera-large", device="cuda")
```
```

--------------------------------

### Expected dataset structure for SegInW

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Shows the expected directory structure after unzipping the SegInW dataset, listing the 25 category folders.

```text
data
|____seginw
| |____Airplane-Parts
| |____Bottles
| |____Brain-Tumor
| |____Chicken
| |____Cows
| |____Electric-Shaver
| |____Elephants
| |____Fruits
| |____Garbage
| |____Ginger-Garlic
| |____Hand
| |____Hand-Metal
| |____House-Parts
| |____HouseHold-Items
| |____Nutterfly-Squireel
| |____Phones
| |____Poles
| |____Puppies
| |____Rail
| |____Salmon-Fillet
| |____Strawberry
| |____Tablets
| |____Toolkits
| |____Trash
| |____Watermelon
```

--------------------------------

### TwoWayTransformer.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Initializes the TwoWayTransformer with the given configuration parameters.

```APIDOC
## TwoWayTransformer.__init__

### Description
Initializes the TwoWayTransformer module with the specified depth, embedding dimension, number of attention heads, MLP dimension, activation function, and attention downsample rate.

### Method
Constructor

### Parameters
- **depth** (`int`) - Required - Number of `TwoWayAttentionBlock` layers.
- **embedding_dim** (`int`) - Required - Channel dimension; `num_heads` must divide it (for the final `Attention` layer).
- **num_heads** (`int`) - Required - Number of attention heads.
- **mlp_dim** (`int`) - Required - MLP hidden dimension.
- **activation** (`Type[nn.Module]`) - Optional - MLP activation function. Default: `nn.ReLU`.
- **attention_downsample_rate** (`int`) - Optional - Internal dimension = `embedding_dim // rate`. Default: `2`.

### Request Example
```python
from segment_anything.modeling import TwoWayTransformer

t = TwoWayTransformer(depth=2, embedding_dim=256, num_heads=8, mlp_dim=2048)
```

### Response
#### Success Response
- Returns an instance of `TwoWayTransformer`.

#### Response Example
```python
# No explicit response body; the constructor returns the module instance.
```
```

--------------------------------

### CXBlock.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.modeling.memory.md

Initializes a ConvNeXt block with depthwise convolution, LayerNorm, linear layers, and GELU activation.

```APIDOC
## CXBlock.__init__

### Description
Initializes a ConvNeXt block with depthwise convolution, LayerNorm, linear layers, and GELU activation.

### Method
Constructor

### Signature
```python
CXBlock(
    dim,
    drop_path=0.0,
    layer_scale_init_value=1e-6,
    ...
)
```

### Parameters
- **dim** (int) - Required - Input and output channel dimension.
- **drop_path** (float) - Optional - Default: `0.0` - Drop path rate for stochastic depth.
- **layer_scale_init_value** (float) - Optional - Default: `1e-6` - Initial value for layer scale.

### Notes
- Architecture: depthwise conv → channels-last LayerNorm → Linear → GELU → Linear.
- Config: `dim=256, kernel_size=7, padding=3, layer_scale_init_value=1e-6, use_dwconv=True`.
```

--------------------------------

### propagate_in_video_preflight(inference_state)

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_video_predictor.md

Internal preflight called automatically at the start of propagate_in_video. Sets tracking state and consolidates per-object outputs.

```APIDOC
## propagate_in_video_preflight(inference_state)

### Description
Called automatically at the start of `propagate_in_video`. Sets `tracking_has_started = True` (which blocks new object ids), consolidates per-object temporary outputs (`temp_output_dict_per_obj`) into `output_dict` for conditioning and non-conditioning frames — running the memory encoder — removes prior non-conditioning outputs on frames that now have conditioning outputs, and finally **asserts** that the consolidated frame indices exactly equal the frames with point/mask inputs.

### Method
instance method

### Signature
```python
def propagate_in_video_preflight(self, inference_state):
```

### Parameters
- **inference_state** (dict) - Required - State from `init_state`.

### Returns
- None (implicitly).

### Throws
- `AssertionError` if consolidated frame indices do not equal input frames.

### Source
`sam-hq2/sam2/sam2_hq_video_predictor.py:593`
```

--------------------------------

### ImageEncoderViT.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Initializes the ImageEncoderViT model with configurable architecture parameters.

```APIDOC
## ImageEncoderViT.__init__

### Description
Initializes the ImageEncoderViT model with the given configuration parameters.

### Method
Constructor

### Parameters
- **img_size** (int) - Optional - Input image size. Default: `1024`.
- **patch_size** (int) - Optional - Patch size. Default: `16`.
- **in_chans** (int) - Optional - Input channels. Default: `3`.
- **embed_dim** (int) - Optional - Patch embedding dimension. Default: `768`.
- **depth** (int) - Optional - ViT depth (number of blocks). Default: `12`.
- **num_heads** (int) - Optional - Number of attention heads. Default: `12`.
- **mlp_ratio** (float) - Optional - MLP hidden dimension / embed dimension ratio. Default: `4.0`.
- **out_chans** (int) - Optional - Neck output channels. Default: `256`.
- **qkv_bias** (bool) - Optional - Whether to use learnable QKV bias. Default: `True`.
- **norm_layer** (Type[nn.Module]) - Optional - Normalization layer type. Default: `nn.LayerNorm`.
- **act_layer** (Type[nn.Module]) - Optional - Activation layer type. Default: `nn.GELU`.
- **use_abs_pos** (bool) - Optional - Whether to use absolute positional embeddings. Default: `True`.
- **use_rel_pos** (bool) - Optional - Whether to use relative positional embeddings in attention. Default: `False`.
- **rel_pos_zero_init** (bool) - Optional - Whether to zero-initialize relative position parameters. Default: `True`.
- **window_size** (int) - Optional - Window size for window attention; `0` means global attention. Default: `0`.
- **global_attn_indexes** (Tuple[int, ...]) - Optional - Indices of blocks that use global attention. Default: `()`.

### Returns
None
```

--------------------------------

### Folder structure of seginw

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Shows the expected directory layout for the seginw folder, including data, pretrained_checkpoint, GroundingDINO, segment_anything, test scripts, and logs.

```text
seginw
|____data
|____pretrained_checkpoint
|____GroundingDINO
|____segment_anything
|____test_ap_on_seginw.py
|____test_seginw.sh
|____test_seginw_hq.sh
|____logs
```

--------------------------------

### build_all_layer_point_grids

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.utils.md

Documentation for the build_all_layer_point_grids helper function.

```APIDOC
## `build_all_layer_point_grids`

### Description
Builds point grids for multiple layers.

### Signature
```python
def build_all_layer_point_grids(n_per_side: int, n_layers: int, scale_per_layer: int) -> List[np.ndarray]
```

### Parameters
- `n_per_side` (int) - Required - Number of points per side.
- `n_layers` (int) - Required - Number of layers.
- `scale_per_layer` (int) - Required - Scale factor per layer.

### Returns
- `List[np.ndarray]` - List of point grids.

### Source
`segment_anything/utils/amg.py:190`
```

--------------------------------

### Fuser.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.modeling.memory.md

Initializes a stack of CXBlocks with optional 1x1 input projection.

```APIDOC
## Fuser.__init__

### Description
Initializes a stack of CXBlocks with optional 1x1 input projection.

### Method
Constructor

### Signature
```python
Fuser(
    layer,
    num_layers,
    dim=None,
    input_projection=False
)
```

### Parameters
- **layer** (nn.Module) - Required - The layer type to stack (e.g., CXBlock).
- **num_layers** (int) - Required - Number of layers in the stack.
- **dim** (int) - Optional - Default: `None` - Dimension of the input projection.
- **input_projection** (bool) - Optional - Default: `False` - Whether to apply a 1x1 input projection.

### Notes
- Config: 2 layers of `CXBlock`.
```

--------------------------------

### Create pretrained_checkpoint directory

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Creates a directory named 'pretrained_checkpoint' to store downloaded model checkpoints.

```bash
mkdir pretrained_checkpoint
```

--------------------------------

### SamPredictor class constructor

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Defines the SamPredictor class and its __init__ method. The constructor takes a Sam model and calls reset_image(), so the predictor starts with no image set.

```python
class SamPredictor:
    def __init__(
        self,
        sam_model: Sam,
    ) -> None:

```

--------------------------------

### Download pretrained checkpoints

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Commands to download the GroundingDINO, SAM-HQ, and HQ-SAM pretrained checkpoints into the 'pretrained_checkpoint' directory.

```bash
cd pretrained_checkpoint
wget https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swinb_cogcoor.pth
wget https://huggingface.co/sam-hq-team/sam-hq-training/resolve/main/pretrained_checkpoint/sam_vit_h_4b8939.pth
wget https://huggingface.co/lkeab/hq-sam/resolve/main/sam_hq_vit_h.pth
```

--------------------------------

### Change to training directory

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Changes the current folder path to the 'train' directory for training instructions.

```bash
cd train
```

--------------------------------

### Evaluate on 25 seginw datasets

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Commands to run the evaluation scripts for the baseline Grounded SAM and Grounded HQ-SAM on the SegInW dataset.

```bash
# baseline Grounded SAM
bash test_seginw.sh

# Grounded HQ-SAM
bash test_seginw_hq.sh
```

--------------------------------

### ImageEncoder.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.modeling.backbones.md

Constructor for the ImageEncoder module. Requires a trunk (e.g., Hiera) and a neck (e.g., FpnNeck). The optional scalp parameter skips trunk layers at the start. Raises AssertionError if trunk.channel_list does not match neck.backbone_channel_list.

```python
class ImageEncoder(nn.Module):
    def __init__(
        self,
        trunk: nn.Module,
        neck: nn.Module,
        scalp: int = 0,
    ):
```

--------------------------------

### Expected checkpoint directory structure

Source: https://github.com/syscv/sam-hq/blob/main/seginw/README.md

Shows the expected layout of the 'pretrained_checkpoint' directory after downloading the checkpoints.

```text
pretrained_checkpoint
|____groundingdino_swinb_cogcoor.pth
|____sam_hq_vit_h.pth
|____sam_vit_h_4b8939.pth
```

--------------------------------

### Expected checkpoint files

Source: https://github.com/syscv/sam-hq/blob/main/train/README.md

Lists the required pretrained checkpoint files for different model types (vit_b, vit_l, vit_h) and their mask decoder counterparts.

```text
pretrained_checkpoint
|____sam_vit_b_maskdecoder.pth
|____sam_vit_b_01ec64.pth
|____sam_vit_l_maskdecoder.pth
|____sam_vit_l_0b3195.pth
|____sam_vit_h_maskdecoder.pth
|____sam_vit_h_4b8939.pth
```

--------------------------------

### propagate_in_video: Propagate prompts over video

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_video_predictor.md

Yields (frame_idx, obj_ids, video_res_masks) per tracked frame. Requires inference_state with conditioning frames; throws RuntimeError if no points are provided. The example shows initialization with init_state and adding points before propagation.

```python
def propagate_in_video(
    self,
    inference_state,
    start_frame_idx=None,
    max_frame_num_to_track=None,
    reverse=False,
):
```

```python
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state("video.mp4", offload_video_to_cpu=True)
    frame_idx, obj_ids, masks = predictor.add_new_points_or_box(
        state, frame_idx=0, obj_id=0,
        points=[[210, 350]], labels=[1],
    )
    for frame_idx, obj_ids, masks in predictor.propagate_in_video(state):
        # masks: (num_objects, 1, H, W) float scores at video resolution
        ...
```

--------------------------------

### add_new_points_or_box method signature

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_video_predictor.md

Method signature for adding or refining click/box prompts on a frame. Returns masks at original video resolution. Throws ValueError for invalid prompt combinations and RuntimeError for new object ids after tracking starts.

```python
@torch.inference_mode()
def add_new_points_or_box(
    self,
    inference_state,
    frame_idx,
    obj_id,
    points=None,
    labels=None,
    clear_old_points=True,
    normalize_coords=True,
    box=None,
):
```

--------------------------------

### Sam.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.modeling.md

Initializes the Sam model with an image encoder, prompt encoder, mask decoder, and optional pixel mean/std normalization values.

```APIDOC
## Sam.__init__

### Description
Initializes the Sam model with the given components and normalization parameters.

### Method
Constructor

### Parameters
- **image_encoder** (ImageEncoderViT) - Required - Backbone producing image embeddings
- **prompt_encoder** (PromptEncoder) - Required - Encodes point/box/mask prompts
- **mask_decoder** (MaskDecoder) - Required - Predicts masks from embeddings + prompts
- **pixel_mean** (List[float]) - Optional - Default: [123.675, 116.28, 103.53] - Registered as non-persistent buffer 1x1x1-shaped
- **pixel_std** (List[float]) - Optional - Default: [58.395, 57.12, 57.375] - Registered as non-persistent buffer

### Class Attributes
- **mask_threshold** (float) - Default: 0.0 - Used for binarizing logits
- **image_format** (str) - Default: "RGB" - Image format

### Source
segment_anything/modeling/sam.py:30
```

--------------------------------

### Run inference on a single image

Source: https://github.com/syscv/sam-hq/blob/main/seginw/GroundingDINO/README.md

Run inference on a single image using the demo script. Provide paths to the config and checkpoint, an input image, an output directory, and a text prompt. Add the --cpu-only flag to run in CPU mode.

```bash
CUDA_VISIBLE_DEVICES=6 python demo/inference_on_a_image.py \
  -c /path/to/config \
  -p /path/to/checkpoint \
  -i .asset/cats.png \
  -o "outputs/0" \
  -t "cat ear." \
  [--cpu-only] # open it for cpu mode
```

--------------------------------

### FpnNeck.__init__

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.modeling.backbones.md

Constructor for the FpnNeck module. Takes a position encoding module, model dimension, and backbone channel list. Defaults for kernel_size, stride, padding, fpn_interp_model, fuse_type, and fpn_top_down_levels are provided. The d_model attribute drives SAM2HQBase.hidden_dim.

```python
class FpnNeck(nn.Module):
    def __init__(
        self,
        position_encoding: nn.Module,
        d_model: int,
        backbone_channel_list: List[int],
        kernel_size: int = 1,
        stride: int = 1,
        padding: int = 0,
        fpn_interp_model: str = "bilinear",
        fuse_type: str = "sum",
        fpn_top_down_levels: Optional[List[int]] = None,
    ):
```

--------------------------------

### init_state method signature

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.sam2_video_predictor.md

Initializes the inference state for a video. Accepts a video path (MP4 file, bytes, or directory of JPEG frames) and optional flags for offloading video/state to CPU and async frame loading.

```python
@torch.inference_mode()
def init_state(
    self,
    video_path,
    offload_video_to_cpu=False,
    offload_state_to_cpu=False,
    async_loading_frames=False,
):
```

--------------------------------

### Load a SAM model with checkpoint

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/segment_anything.md

Instantiate a SAM model using the registry. The builder accepts an optional checkpoint path; when provided, the state dict is loaded with strict=False and the printed info from load_state_dict is sent to stdout. All builders call eval() and freeze parameters not related to HQ-SAM components.

```python
sam = sam_model_registry["vit_l"](checkpoint="pretrained_checkpoint/sam_hq_vit_l.pth")
```

--------------------------------

### load_video_frames() function signature

Source: https://github.com/syscv/sam-hq/blob/main/_autodocs/api-reference/sam2.utils.md

Dispatcher: bytes or .mp4/.MP4 path → load_video_frames_from_video_file (decord); directory → load_video_frames_from_jpg_images (JPEG frames named <frame_index>.jpg). Throws NotImplementedError("Only MP4 video and JPEG folder are supported at this moment") otherwise. Returns (images, video_height, video_width) — images is num_frames x 3 x image_size x image_size, normalized and (unless offload_video_to_cpu) on compute_device; with async_loading_frames=True returns the AsyncVideoFrameLoader instead of a dense tensor.

```python
def load_video_frames(
    video_path,
    image_size,
    offload_video_to_cpu,
    img_mean=(0.485, 0.456, 0.406),
    img_std=(0.229, 0.224, 0.225),
    async_loading_frames=False,
    compute_device=torch.device("cuda"),
):
```

--------------------------------

### Run HQ-SAM demo

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Runs the demo script to obtain HQ-SAM's visual result.

```bash
python demo/demo_hqsam.py
```

--------------------------------

### Change to seginw directory

Source: https://github.com/syscv/sam-hq/blob/main/README.md

Changes the current folder path to the 'seginw' directory for evaluation instructions.

```bash
cd seginw
```
Tokens:
Raw