### Install Sapiens2 and Download Checkpoints

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Installs Sapiens2 from source and downloads pre-trained checkpoints for various tasks from HuggingFace. Requires Python >= 3.12 and PyTorch >= 2.7. The pointmap visualization requires additional dependencies.

```bash
git clone https://github.com/facebookresearch/sapiens2.git
cd sapiens2
export SAPIENS_ROOT=$(pwd)
pip install -e .

# Optional: 3D pointmap visualization requires open3d
pip install -e .[pointmap]

# Download checkpoints from HuggingFace, e.g.:
pip install huggingface_hub
python -c "
from huggingface_hub import hf_hub_download
import os, pathlib

root = pathlib.Path.home() / 'sapiens2_host'
tasks = {
    'pretrain': ('facebook/sapiens2-pretrain-1b',  'sapiens2_1b_pretrain.safetensors'),
    'pose':     ('facebook/sapiens2-pose-1b',       'sapiens2_1b_pose.safetensors'),
    'seg':      ('facebook/sapiens2-seg-1b',        'sapiens2_1b_seg.safetensors'),
    'normal':   ('facebook/sapiens2-normal-1b',     'sapiens2_1b_normal.safetensors'),
    'pointmap': ('facebook/sapiens2-pointmap-1b',   'sapiens2_1b_pointmap.safetensors'),
}
for task, (repo, filename) in tasks.items():
    dest = root / task
    dest.mkdir(parents=True, exist_ok=True)
    hf_hub_download(repo_id=repo, filename=filename, local_dir=str(dest))
    print(f'Downloaded {filename}')
"
```

--------------------------------

### Install Pointmap Visualization Dependencies

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/POINTMAP.md

Install the necessary dependencies for Pointmap visualization, including open3d. This command installs the package in editable mode with the pointmap extra.

```bash
pip install -e .[pointmap]
```

--------------------------------

### Clone Repository and Set Environment Variable (Bash)

Source: https://github.com/facebookresearch/sapiens2/blob/main/README.md

Instructions for cloning the Sapiens2 repository and setting the `SAPIENS_ROOT` environment variable. This is a prerequisite for installation and further setup.

```bash
git clone https://github.com/facebookresearch/sapiens2.git
cd sapiens2
export SAPIENS_ROOT=$(pwd)
```

--------------------------------

### Launch Sapiens2 Pose Estimation Training

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/POSE.md

Execute the training script for Sapiens2. Adjust parameters like DE VICES, TRAIN_BATCH_SIZE_PER_GPU, mode, LOAD_FROM, and RESUME_FROM within the node.sh script as needed for your training setup.

```bash
cd $SAPIENS_ROOT/sapiens/pose
./scripts/keypoints308/train/sapiens2_1b/node.sh
```

--------------------------------

### Install Sapiens2 Package (Bash)

Source: https://github.com/facebookresearch/sapiens2/blob/main/README.md

Command to install the Sapiens2 package using pip. Requires Python version 3.12 or higher and PyTorch version 2.7 or higher.

```bash
pip install -e .
```

--------------------------------

### Sapiens2 Standalone Backbone Forward Pass

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Demonstrates how to use the standalone Sapiens2 ViT backbone for inference. This backbone can be used without installing the full Sapiens2 framework.

```APIDOC
## Sapiens2 Standalone Backbone Forward Pass

Self-contained ViT backbone (no framework dependencies). Supports five architectures (`sapiens2_0.1b` → `sapiens2_5b`). Input is a NCHW tensor at 1024×768 (H×W). Returns a tuple of feature tensors from the requested output indices; index 0 of the tuple corresponds to the last block by default (`out_indices=-1`).

```python
import os
import torch
from safetensors.torch import load_file

# Copy sapiens/backbones/standalone/sapiens2.py into your project, then:
from sapiens2 import Sapiens2  # standalone copy
# OR, with the full package installed:
from sapiens.backbones.standalone.sapiens2 import Sapiens2

# ── Build and load 1B model ──────────────────────────────────────────────────
model = Sapiens2(
    arch="sapiens2_1b",
    img_size=(1024, 768),   # (H, W)
    patch_size=16,
    out_type="raw",         # "raw" | "cls_token" | "featmap"
    out_indices=-1,         # last layer; or e.g. [20, 39] for intermediate layers
    with_cls_token=True,
    n_storage_tokens=8,
).eval().cuda()

ckpt_path = os.path.expanduser("~/sapiens2_host/pretrain/sapiens2_1b_pretrain.safetensors")
model.load_state_dict(load_file(ckpt_path))

# ── Inference ────────────────────────────────────────────────────────────────
x = torch.randn(1, 3, 1024, 768, device="cuda")   # ImageNet normalization recommended
with torch.no_grad():
    (features,) = model(x)   # out_indices=-1 → one-element tuple

# out_type="raw"     → (B, 1 + 8 + H/p * W/p, C) = (1, 4049, 1536)
# out_type="featmap" → (B, C, H/p, W/p)           = (1, 1536, 64, 48)
# out_type="cls_token"→ (B, C)                     = (1, 1536)
print("Feature shape:", features.shape)   # → torch.Size([1, 4049, 1536])

# ── 4 K resolution (use_tokenizer=True) ─────────────────────────────────────
model_4k = Sapiens2(
    arch="sapiens2_1b",
    img_size=(4096, 3072),
    patch_size=16,
    use_tokenizer=True,   # required for 4K; reduces spatial tokens via window pooling
    window_size=4,
).eval().cuda()
```
```

--------------------------------

### 3D Point Cloud Generation from Image

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

This script preprocesses an image, performs inference to generate a point map, and exports the result as a PLY point cloud file. Ensure necessary libraries like OpenCV and PyTorch are installed.

```python
import cv2, numpy as np
from torch import nn
image_bgr = cv2.imread("demo/data/000001.png")
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
img_t = torch.from_numpy(image_rgb).float().permute(2, 0, 1).unsqueeze(0)
mean  = torch.tensor([123.675, 116.28, 103.53]).view(1, 3, 1, 1)
std   = torch.tensor([58.395,  57.12,  57.375]).view(1, 3, 1, 1)
img_t = nn.functional.interpolate((img_t - mean) / std, size=(1024, 768), mode="bilinear", align_corners=False)

with torch.no_grad():
    pointmap = model(img_t.cuda())       # 1 x 3 x 1024 x 768

pts = pointmap[0].cpu().permute(1, 2, 0).reshape(-1, 3).numpy()  # (N, 3)

# Export as PLY point cloud
import open3d as o3d
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(pts)
o3d.io.write_point_cloud("person_3d.ply", pcd)
print(f"Point cloud saved: {pts.shape[0]} points, depth range {pts[:,2].min():.2f} – {pts[:,2].max():.2f} m")
```

--------------------------------

### Run Pretrained Backbone Forward Pass (Python)

Source: https://github.com/facebookresearch/sapiens2/blob/main/README.md

Demonstrates how to load a pretrained Sapiens2 backbone and perform a forward pass on a single image. Requires `torch` and `safetensors`. The model is built with a specified architecture, image size, and patch size, then loaded with a checkpoint. Ensure the checkpoint path is correct.

```python
import os
import torch
from safetensors.torch import load_file
from sapiens.backbones.standalone.sapiens2 import Sapiens2

# Build the model and load a pretrained checkpoint
model = Sapiens2(arch="sapiens2_1b", img_size=(1024, 768), patch_size=16).eval().cuda()  # img_size is (H, W)
ckpt = os.path.expanduser("~/sapiens2_host/pretrain/sapiens2_1b_pretrain.safetensors")
model.load_state_dict(load_file(ckpt))

# Forward pass on a single image (RGB; ImageNet normalization recommended)
x = torch.randn(1, 3, 1024, 768).cuda()
with torch.no_grad():
    features = model(x)[0]  # dense backbone features
```

--------------------------------

### Run Sapiens2 Pose Estimation Demo

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/POSE.md

Navigate to the pose directory and execute the demo script for keypoint estimation. This script runs on a default demo dataset and can be customized by modifying its internal variables.

```bash
cd $SAPIENS_ROOT/sapiens/pose
./scripts/demo/keypoints308.sh
```

--------------------------------

### Launch Sapiens2 Pointmap Training

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/POINTMAP.md

Navigate to the Sapiens dense directory and execute the training script for the Sapiens2 1B model. Adjust parameters within the node.sh script for device selection, batch size, training mode, and checkpoint loading.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/pointmap/train/sapiens2_1b/node.sh
```

--------------------------------

### Sapiens2.__init__ Architecture Configuration

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Details the parameters and available architecture configurations for initializing the Sapiens2 Vision Transformer model.

```APIDOC
## `Sapiens2.__init__` — Architecture Configuration

Constructs the Sapiens2 ViT with RoPE positional encoding, optional `Tokenizer` module for high-resolution inputs, grouped-query attention in middle layers, and learnable storage (register) tokens. `frozen_stages` freezes the first N transformer blocks to speed up fine-tuning.

```python
from sapiens.backbones.standalone.sapiens2 import Sapiens2
import torch

# ── Available arch sizes ──────────────────────────────────────────────────────
# sapiens2_0.1b  →  embed_dims=768,  layers=12,  heads=12
# sapiens2_0.4b  →  embed_dims=1024, layers=24,  heads=16
# sapiens2_0.8b  →  embed_dims=1280, layers=32,  heads=16
```
```

--------------------------------

### Launch Sapiens2 Normal Training

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/NORMAL.md

Execute the training script for Sapiens2 with a 1.1B parameter model. Adjust `DEVICES`, `TRAIN_BATCH_SIZE_PER_GPU`, `mode`, `LOAD_FROM`, and `RESUME_FROM` within the `node.sh` script as needed for your environment and training strategy.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/normal/train/sapiens2_1b/node.sh
```

--------------------------------

### Launch Sapiens2 Segmentation Training

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/SEG.md

Navigate to the Sapiens2 dense directory and execute the training script for the 1b model size. Adjustments for devices, batch size, and loading/resuming checkpoints can be made within the node.sh script.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/seg/train/sapiens2_1b/node.sh
```

--------------------------------

### Initialize and Use Sapiens2 for Feature Extraction

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Initializes a Sapiens2 model for feature extraction and demonstrates inference. Ensure the model architecture and parameters match your requirements.

```python
model = Sapiens2(
    arch="sapiens2_0.4b",
    img_size=(1024, 768),
    patch_size=16,
    out_indices=[-1],                       # collect final layer only
    out_type="featmap",                     # spatial (B, C, H/p, W/p) feature map
    with_cls_token=True,
    n_storage_tokens=8,
    final_norm=True,                        # apply RMSNorm after last block
    frozen_stages=12,                       # freeze patch embed + first 12 blocks
    layer_scale_init_value=1e-4,            # LayerScale initializer
    pos_embed_rope_base=100.0,
    pos_embed_rope_normalize_coords="separate",  # "min" | "max" | "separate"
    pos_embed_rope_dtype="bf16",
).eval()

x = torch.randn(2, 3, 1024, 768)
with torch.no_grad():
    (fmap,) = model(x)
print(fmap.shape)   # torch.Size([2, 1024, 64, 48])  (B, C, H/patch, W/patch)
```

--------------------------------

### Run Albedo Inference Script

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Estimates per-pixel albedo (diffuse reflectance) from a single image. Outputs include albedo visualizations and predicted albedo maps.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/demo/albedo.sh
```

--------------------------------

### Run Surface Normal Estimation Demo

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/NORMAL.md

Execute the demo script for surface normal estimation. Adjust INPUT, OUTPUT, MODEL_NAME, JOBS_PER_GPU, and GPU_IDS variables within the script for customization. For optimal results, consider running body-part segmentation first and using the foreground mask.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/demo/normal.sh
```

--------------------------------

### Train Pose Estimation Model

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Launches distributed multi-GPU training for pose estimation on a custom dataset. Requires specific data layout and configuration file edits. Supports resuming from checkpoints or fine-tuning from released task checkpoints.

```bash
export SAPIENS_ROOT=/path/to/sapiens2
export DATA_ROOT=/path/to/pose_data
# Required layout:
#   $DATA_ROOT/images/          RGB images
#   $DATA_ROOT/annotations/train.json   COCO-style annotations in 308-kpt format

cd $SAPIENS_ROOT/sapiens/pose

# Edit the config file (configs/keypoints308/shutterstock_goliath_3po/sapiens2_1b_*.py):
#   pretrained_checkpoint = "~/sapiens2_host/pretrain/sapiens2_1b_pretrain.safetensors"
#   ann_file = f"{DATA_ROOT}/annotations/train.json"

# Launch training (8 GPUs, batch 7 per GPU)
./scripts/keypoints308/train/sapiens2_1b/node.sh

# Resume from checkpoint:
RESUME_FROM=/path/to/iter_10000.pth ./scripts/keypoints308/train/sapiens2_1b/node.sh

# Fine-tune from a released task checkpoint:
LOAD_FROM=~/sapiens2_host/pose/sapiens2_1b_pose.safetensors \
    ./scripts/keypoints308/train/sapiens2_1b/node.sh

# Output: Outputs/keypoints308/train/sapiens2_1b_keypoints308_shutterstock_goliath_3po-1024x768/node/<timestamp>/
```

--------------------------------

### Download Person Detector Model

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/POSE.md

Use this command to download the DETR ResNet-101 DC5 model for person detection from HuggingFace. Ensure the local directory is set correctly.

```bash
hf download facebook/detr-resnet-101-dc5 \
    --local-dir "${SAPIENS_CHECKPOINT_ROOT}/detector/detr-resnet-101-dc5"
```

--------------------------------

### Run Sapiens2 Segmentation Demo

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/SEG.md

Execute the segmentation demo script. Adjust INPUT, OUTPUT, MODEL_NAME, JOBS_PER_GPU, and GPU_IDS variables within the script to customize the inference process.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/demo/seg.sh
```

--------------------------------

### Load and Use SegEstimator for Body-Part Segmentation

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Loads a SegEstimator model using a configuration file and checkpoint, preprocesses an image, and performs per-pixel segmentation. Ensure the config and ckpt paths are correct.

```python
import torch
import cv2
import numpy as np
from sapiens.dense.models import init_model  # from sapiens/dense package

# ── Load model from config + checkpoint ──────────────────────────────────────
config   = "sapiens/dense/configs/seg/shutterstock_goliath/sapiens2_1b_seg_shutterstock_goliath-1024x768.py"
ckpt     = "~/sapiens2_host/seg/sapiens2_1b_seg.safetensors"
model    = init_model(config, ckpt, device="cuda:0").eval()

# ── Preprocess an image ───────────────────────────────────────────────────────
image_bgr = cv2.imread("demo/data/000001.png")                  # H x W x 3
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
img_t     = torch.from_numpy(image_rgb).float().permute(2, 0, 1)   # 3 x H x W

# ImageNet normalization
mean = torch.tensor([123.675, 116.28, 103.53]).view(3, 1, 1)
std  = torch.tensor([58.395,  57.12,  57.375]).view(3, 1, 1)
img_t = (img_t - mean) / std

img_t = torch.nn.functional.interpolate(
    img_t.unsqueeze(0), size=(1024, 768), mode="bilinear", align_corners=False
)  # 1 x 3 x 1024 x 768

# ── Inference ─────────────────────────────────────────────────────────────────
with torch.no_grad():
    logits = model(img_t.cuda())                 # 1 x 29 x 1024 x 768
    pred_classes = logits.argmax(dim=1)[0]       # 1024 x 768, values 0-28

# Class indices:
# 0=Background, 3=Face_Neck, 4=Hair, 22=Torso, 23=Upper_Clothing, ...
print("Unique predicted classes:", pred_classes.unique().cpu().tolist())
```

--------------------------------

### Run Pointmap Inference Script

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/POINTMAP.md

Execute the default inference script for Pointmap estimation on the demo dataset. Adjust INPUT, OUTPUT, MODEL_NAME, JOBS_PER_GPU, and GPU_IDS variables within the script for custom configurations.

```bash
cd $SAPIENS_ROOT/sapiens/dense
./scripts/demo/pointmap.sh
```

--------------------------------

### Pose Estimation Inference Loop (Python)

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

This Python script demonstrates the core inference loop for pose estimation, including initializing the pose and detection models, processing images, and decoding keypoints. It requires specific configuration and checkpoint paths.

```python
# Python equivalent of the core inference loop (single image)
import cv2, numpy as np, torch, json
from sapiens.pose.datasets import parse_pose_metainfo, UDPHeatmap
from sapiens.pose.models import init_model
from transformers import DetrForObjectDetection, DetrImageProcessor
from PIL import Image

device = "cuda:0"
config  = f"{SAPIENS_ROOT}/sapiens/pose/configs/keypoints308/shutterstock_goliath_3po/sapiens2_1b_keypoints308_shutterstock_goliath_3po-1024x768.py"
ckpt    = "~/sapiens2_host/pose/sapiens2_1b_pose.safetensors"
det_dir = "~/sapiens2_host/detector/detr-resnet-101-dc5"

model = init_model(config, ckpt, device=device)
model.pose_metainfo = parse_pose_metainfo(dict(from_file="configs/_base_/keypoints308.py"))
model.codec = UDPHeatmap(**model.cfg.codec)

# DETR detector
proc = DetrImageProcessor.from_pretrained(det_dir)
det  = DetrForObjectDetection.from_pretrained(det_dir).eval().to(device)

image_bgr = cv2.imread("demo/data/000001.png")
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
pil_img   = Image.fromarray(image_rgb)
inputs_d  = proc(images=pil_img, return_tensors="pt").to(device)
with torch.no_grad():
    out_d = det(**inputs_d)
results = proc.post_process_object_detection(
    out_d, target_sizes=torch.tensor([image_rgb.shape[:2]], device=device), threshold=0.3
)[0]
bboxes = results["boxes"][results["labels"] == 1].cpu().numpy()

# Pose inference for each detected person
for bbox in bboxes:
    data_info = dict(img=image_bgr, bbox=bbox[None], bbox_score=np.ones(1, dtype=np.float32))
    data = model.pipeline(data_info)
    data = model.data_preprocessor(data)
    with torch.no_grad():
        pred = model(data["inputs"])   # 1 x 308 x heatmap_H x heatmap_W
    kpts, scores = model.codec.decode(pred[0].cpu().numpy())
    # kpts: (1, 308, 2) in crop-image coords
    print(f"Detected {kpts.shape[1]} keypoints, max score: {scores.max():.3f}")
```

--------------------------------

### Run Pose Estimation Inference (Bash)

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

This script executes the pose estimation inference using a DETR person detector and a pose model. Ensure the SAPIENS_ROOT and SAPIENS_CHECKPOINT_ROOT environment variables are set correctly. Customize input, output, model name, and parallelism settings within the script.

```bash
export SAPIENS_ROOT=/path/to/sapiens2
export SAPIENS_CHECKPOINT_ROOT=~/sapiens2_host

# Download the DETR detector (required for pose only)
hf download facebook/detr-resnet-101-dc5 \
    --local-dir "${SAPIENS_CHECKPOINT_ROOT}/detector/detr-resnet-101-dc5"

# Run inference on the bundled demo images (100 PNG frames in demo/data/)
cd $SAPIENS_ROOT/sapiens/pose
./scripts/demo/keypoints308.sh

# Customize inside keypoints308.sh:
#   INPUT='../../demo/data'                            # image directory
#   OUTPUT="${HOME}/Desktop/sapiens2/pose/Outputs/vis" # output directory
#   MODEL_NAME='sapiens2_1b'                           # or 0.4b / 0.8b / 5b
#   JOBS_PER_GPU=2; GPU_IDS=(0 1 2 3 4 5 6 7)         # parallelism

# Output per image:
#   <OUTPUT>/<model>/<image_name>.jpg   — visualization with skeleton overlay
#   <OUTPUT>/<model>/<seq>_predictions.json — keypoints JSON
```

--------------------------------

### Set Data Root Directory

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/NORMAL.md

Set the environment variable `$DATA_ROOT` to the root directory of your dataset before proceeding with training.

```bash
export DATA_ROOT=/path/to/your/normal_data
```

--------------------------------

### Set Data Root Directory

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/SEG.md

Set the DATA_ROOT environment variable to the path of your dataset. This is a prerequisite for data preparation.

```bash
export DATA_ROOT=/path/to/your/seg_data
```

--------------------------------

### Sapiens2 Architecture Configuration Options

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Defines the available architecture sizes for the Sapiens2 ViT model, specifying embed dimensions, number of layers, and attention heads for each size.

```python
from sapiens.backbones.standalone.sapiens2 import Sapiens2
import torch

# ── Available arch sizes ──────────────────────────────────────────────────────
# sapiens2_0.1b  →  embed_dims=768,  layers=12,  heads=12
# sapiens2_0.4b  →  embed_dims=1024, layers=24,  heads=16
# sapiens2_0.8b  →  embed_dims=1280, layers=32,  heads=16

```

--------------------------------

### Set Dataset Root Directory

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/POSE.md

Set the environment variable DATA_ROOT to the path of your dataset. This is required before proceeding with data preparation.

```bash
export DATA_ROOT=/path/to/your/pose_data
```

--------------------------------

### Load and Use NormalEstimator for Surface Normal Estimation

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Loads a NormalEstimator model and preprocesses an image identically to SegEstimator. It then performs inference to predict surface normals and saves a visualizable RGB representation. Ensure correct paths for config and checkpoint.

```python
import torch
from sapiens.dense.models import init_model

config = "sapiens/dense/configs/normal/metasim_render_people/sapiens2_1b_normal_metasim_render_people-1024x768.py"
ckpt   = "~/sapiens2_host/normal/sapiens2_1b_normal.safetensors"
model  = init_model(config, ckpt, device="cuda:0").eval()

# Preprocess identical to SegEstimator above
import cv2, numpy as np
from torch import nn
image_bgr = cv2.imread("demo/data/000001.png")
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
img_t = torch.from_numpy(image_rgb).float().permute(2, 0, 1).unsqueeze(0)
mean = torch.tensor([123.675, 116.28, 103.53]).view(1, 3, 1, 1)
std  = torch.tensor([58.395,  57.12,  57.375]).view(1, 3, 1, 1)
img_t = nn.functional.interpolate((img_t - mean) / std, size=(1024, 768), mode="bilinear", align_corners=False)

with torch.no_grad():
    normals = model(img_t.cuda())        # 1 x 3 x 1024 x 768

# Convert to visualizable RGB: normal_rgb = (normal * 0.5 + 0.5) * 255
normal_np = normals[0].cpu().permute(1, 2, 0).numpy()   # 1024 x 768 x 3
vis_rgb   = np.clip((normal_np * 0.5 + 0.5) * 255, 0, 255).astype(np.uint8)
cv2.imwrite("normal_vis.png", cv2.cvtColor(vis_rgb, cv2.COLOR_RGB2BGR))
print("Normal map range:", normal_np.min(), "→", normal_np.max())
```

--------------------------------

### Run Segmentation Inference (Bash)

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

This script performs per-pixel 29-class body-part segmentation on a directory of images. It saves visualizations, probability arrays, and foreground masks. Key variables like model name, input/output paths, and parallelism can be edited within the script.

```bash
cd $SAPIENS_ROOT/sapiens/dense

# Default: runs on demo/data with the 1B model across 8 GPUs
./scripts/demo/seg.sh

# Key variables (edit inside seg.sh):
#   MODEL_NAME='sapiens2_1b'       # or 0.4b / 0.8b / 5b
#   JOBS_PER_GPU=3                 # parallel workers per GPU
#   GPU_IDS=(0 1 2 3 4 5 6 7)
#   INPUT='../../demo/data'
#   OUTPUT="${HOME}/Desktop/sapiens2/seg/Outputs/vis"

# Outputs written per image:
#   <OUTPUT>/<model>/<image_name>.jpg      — color segmentation overlay
#   <OUTPUT>/<model>/<image_name>_pred.npy — (29, H, W) class probabilities
#   <OUTPUT>/<model>/<image_name>_fg.npy   — foreground binary mask
```

--------------------------------

### Export and Benchmark PyTorch Model

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Exports a task model using torch.export and benchmarks its performance with torch.compile. Supports dynamic batch and spatial dimensions. Requires models to be moved to CUDA for benchmarking.

```python
from sapiens.dense.models import init_model
from tools.deployment.torch_optimization import (
    revert_sync_batchnorm,
    create_demo_inputs,
    compile_and_export_model,
    benchmark_model,
)
import torch

config = "sapiens/dense/configs/seg/shutterstock_goliath/sapiens2_1b_seg_shutterstock_goliath-1024x768.py"
ckpt   = "~/sapiens2_host/seg/sapiens2_1b_seg.safetensors"

model = init_model(config, ckpt, device="cpu").eval()
model = revert_sync_batchnorm(model)         # SyncBN → BN for single-device export
model = model.to(torch.bfloat16)

demo_inputs = create_demo_inputs((4, 3, 1024, 768))   # (N, C, H, W)
demo_inputs["imgs"] = demo_inputs["imgs"].to(torch.bfloat16)

# Export with dynamic batch (1..32) and dynamic spatial dims (1024–2048 × 768–1536)
compile_and_export_model(
    model,
    demo_inputs,
    output_file="sapiens2_1b_seg_exported.pt",
    max_batch_size=32,
    dtype=torch.bfloat16,
)
# Reloads from .pt, applies _ToDeviceTransformer (CUDA), and benchmarks torch.compile

# Or benchmark an already-loaded model directly:
model_cuda = model.cuda()
with torch.no_grad():
    mean_ms = benchmark_model(
        model_cuda,
        {"imgs": demo_inputs["imgs"].cuda()},
        model_name="baseline",
        num_warmup=3,
        num_iterations=10,
    )
print(f"Mean latency: {mean_ms:.1f} ms/sample")
```

--------------------------------

### Set Data Root Directory

Source: https://github.com/facebookresearch/sapiens2/blob/main/docs/train/POINTMAP.md

Set the DATA_ROOT environment variable to the root directory of your dataset. This is required before proceeding with data preparation.

```bash
export DATA_ROOT=/path/to/your/pointmap_data
```

--------------------------------

### Load and Use PointmapEstimator for Per-Pixel 3D Pointmap

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Loads a PointmapEstimator model using its configuration and checkpoint. This model predicts per-pixel 3D coordinates in camera frame, useful for generating dense point clouds. Ensure correct paths for config and checkpoint.

```python
import torch
from sapiens.dense.models import init_model

config = "sapiens/dense/configs/pointmap/render_people/sapiens2_1b_pointmap_render_people-1024x768.py"
ckpt   = "~/sapiens2_host/pointmap/sapiens2_1b_pointmap.safetensors"
model  = init_model(config, ckpt, device="cuda:0").eval()
```

--------------------------------

### Train Pointmap Estimation

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Trains per-pixel 3D pointmap prediction. Requires ground-truth coordinate maps and RGB images organized in a specified directory structure. Outputs are saved to a timestamped directory.

```bash
export DATA_ROOT=/path/to/pointmap_data
# Required layout:
#   $DATA_ROOT/images/         RGB images
#   $DATA_ROOT/pointmaps/      (H, W, 3) coordinate maps (.npy)
#   $DATA_ROOT/annotations/train.json

cd $SAPIENS_ROOT/sapiens/dense
./scripts/pointmap/train/sapiens2_1b/node.sh
# Output: Outputs/pointmap/train/sapiens2_1b_pointmap_render_people-1024x768/node/<timestamp>/
```

--------------------------------

### Train Body-Part Segmentation Model

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Trains a 29-class segmentation model on custom data with per-pixel mask annotations. Requires specific data layout and configuration edits. Supports a debug mode for single-GPU testing.

```bash
export DATA_ROOT=/path/to/seg_data
# Required layout:
#   $DATA_ROOT/images/                RGB images
#   $DATA_ROOT/masks/                 per-pixel class-index PNGs (0–28)
#   $DATA_ROOT/annotations/train.json image/mask pair index

cd $SAPIENS_ROOT/sapiens/dense

# Edit config: sapiens/dense/configs/seg/shutterstock_goliath/sapiens2_1b_seg_*.py
#   pretrained_checkpoint = "~/sapiens2_host/pretrain/sapiens2_1b_pretrain.safetensors"
#   num_classes = 29   # change to match your taxonomy if needed

./scripts/seg/train/sapiens2_1b/node.sh

# Debug mode (single GPU, batch=1, workers=0):
# Edit node.sh: mode='debug'

# Output: Outputs/seg/train/sapiens2_1b_seg_shutterstock_goliath-1024x768/node/<timestamp>/
```

--------------------------------

### Train Surface Normal Estimation

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Trains per-pixel normal estimation. Requires ground-truth normal maps and RGB images in a specific directory layout. The script outputs results to a timestamped directory.

```bash
export DATA_ROOT=/path/to/normal_data
# Required layout:
#   $DATA_ROOT/images/       RGB images
#   $DATA_ROOT/normals/      (H, W, 3) .npy arrays OR RGB .png (n = 2*rgb - 1)
#   $DATA_ROOT/annotations/train.json

cd $SAPIENS_ROOT/sapiens/dense
./scripts/normal/train/sapiens2_1b/node.sh
# Output: Outputs/normal/train/sapiens2_1b_normal_metasim_render_people-1024x768/node/<timestamp>/
```

--------------------------------

### Sapiens2 Standalone Backbone Forward Pass

Source: https://context7.com/facebookresearch/sapiens2/llms.txt

Performs a forward pass using the standalone Sapiens2 ViT backbone. Supports multiple architectures and output types. Input is an NCHW tensor, and output is a tuple of feature tensors. Recommended to use ImageNet normalization for input.

```python
import os
import torch
from safetensors.torch import load_file

# Copy sapiens/backbones/standalone/sapiens2.py into your project, then:
from sapiens2 import Sapiens2  # standalone copy
# OR, with the full package installed:
from sapiens.backbones.standalone.sapiens2 import Sapiens2

# ── Build and load 1B model ──────────────────────────────────────────────────
model = Sapiens2(
    arch="sapiens2_1b",
    img_size=(1024, 768),   # (H, W)
    patch_size=16,
    out_type="raw",         # "raw" | "cls_token" | "featmap"
    out_indices=-1,         # last layer; or e.g. [20, 39] for intermediate layers
    with_cls_token=True,
    n_storage_tokens=8,
).eval().cuda()

ckpt_path = os.path.expanduser("~/sapiens2_host/pretrain/sapiens2_1b_pretrain.safetensors")
model.load_state_dict(load_file(ckpt_path))

# ── Inference ────────────────────────────────────────────────────────────────
x = torch.randn(1, 3, 1024, 768, device="cuda")   # ImageNet normalization recommended
with torch.no_grad():
    (features,) = model(x)   # out_indices=-1 → one-element tuple

# out_type="raw"     → (B, 1 + 8 + H/p * W/p, C) = (1, 4049, 1536)
# out_type="featmap" → (B, C, H/p, W/p)           = (1, 1536, 64, 48)
# out_type="cls_token"→ (B, C)                     = (1, 1536)
print("Feature shape:", features.shape)   # → torch.Size([1, 4049, 1536])

# ── 4 K resolution (use_tokenizer=True) ─────────────────────────────────────
model_4k = Sapiens2(
    arch="sapiens2_1b",
    img_size=(4096, 3072),
    patch_size=16,
    use_tokenizer=True,   # required for 4K; reduces spatial tokens via window pooling
    window_size=4,
).eval().cuda()

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.