### Install VGGT Dependencies

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Clone the repository and install the required Python packages using the provided requirements file. This is the initial setup step for using VGGT.

```bash
git clone git@github.com:facebookresearch/vggt.git 
cd vggt
pip install -r requirements.txt
```

--------------------------------

### Default Configuration Example

Source: https://github.com/facebookresearch/vggt/blob/main/training/README.md

Example of the default.yaml configuration file, specifying dataset paths and checkpoint resume path for training.

```yaml
data:
  train:
    dataset:
      dataset_configs:
        - _target_: data.datasets.co3d.Co3dDataset
          split: train
          CO3D_DIR: /YOUR/PATH/TO/CO3D
          CO3D_ANNOTATION_DIR: /YOUR/PATH/TO/CO3D_ANNOTATION
# ... same for val ...

checkpoint:
  resume_checkpoint_path: /YOUR/PATH/TO/CKPT
```

--------------------------------

### Run Demo with uv

Source: https://github.com/facebookresearch/vggt/blob/main/docs/package.md

Execute a demo script using uv for fast package installation and resolution. Ensure uv is installed first.

```bash
uv run --extra demo demo_gradio.py
```

--------------------------------

### Install Demo Dependencies

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Installs the necessary Python packages for running the interactive demo visualizations.

```bash
pip install -r requirements_demo.txt
```

--------------------------------

### Complete VGGT Pipeline Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

An end-to-end example demonstrating model initialization, image loading, inference, and accessing predictions.

```python
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
import torch

# 1. Initialize model
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# 2. Load images
image_paths = [
    "scene/view1.jpg",
    "scene/view2.jpg",
    "scene/view3.jpg"
]
images = load_and_preprocess_images(image_paths, mode="pad").to(device)

# 3. Run inference
with torch.no_grad():
    with torch.cuda.amp.autocast(dtype=torch.bfloat16):
        predictions = model(images)

# 4. Access predictions
depth_maps = predictions["depth"]       # [1, 3, 518, 518, 1]
world_points = predictions["world_points"]  # [1, 3, 518, 518, 3]
camera_poses = predictions["pose_enc"]  # [1, 3, 9]
```

--------------------------------

### VGGT Model Inference Quick Start

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/README.md

Demonstrates how to load a pre-trained VGGT model, preprocess images, and perform inference to obtain predictions like depth and world points. Ensure you have PyTorch and the vggt library installed.

```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images

# Initialize model
model = VGGT.from_pretrained("facebook/VGGT-1B").cuda().eval()

# Load images
image_paths = ["view1.png", "view2.png", "view3.png"]
images = load_and_preprocess_images(image_paths).cuda()

# Inference
with torch.no_grad():
    predictions = model(images)

# Access predictions
depth = predictions["depth"]              # [1, 3, 518, 518, 1]
world_points = predictions["world_points"]  # [1, 3, 518, 518, 3]
cameras = predictions["pose_enc"]         # [1, 3, 9]
```

--------------------------------

### Run Demo with Pixi

Source: https://github.com/facebookresearch/vggt/blob/main/docs/package.md

Execute a demo script using pixi to manage the environment. Ensure pixi is installed first.

```bash
pixi run -e python demo_gradio.py
```

--------------------------------

### Install VGGT with Pip

Source: https://github.com/facebookresearch/vggt/blob/main/docs/package.md

Install VGGT in editable mode using pip. This is the simplest installation method.

```bash
pip install -e .
```

--------------------------------

### VGGT Integration Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Pose-Encoding-Utilities.md

A complete example demonstrating VGGT model initialization, image loading, inference, and decoding pose predictions into camera matrices. This snippet shows how to access translation, quaternion, and field of view components from the pose encoding.

```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
from vggt.utils.pose_enc import pose_encoding_to_extri_intri
from vggt.utils.rotation import quat_to_mat

# Initialize model
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# Load images
image_paths = ["img1.png", "img2.png", "img3.png"]
images = load_and_preprocess_images(image_paths).to(device)
H, W = images.shape[-2:]

# Inference
with torch.no_grad():
    predictions = model(images)

# Get pose encoding (shape: [1, 3, 9])
pose_enc = predictions["pose_enc"]

# Decode to camera matrices
extrinsics, intrinsics = pose_encoding_to_extri_intri(
    pose_enc, image_size_hw=(H, W)
)

# Access individual components
translation = pose_enc[..., 0:3]  # [1, 3, 3]
quaternion = pose_enc[..., 3:7]   # [1, 3, 4]
fov_h = pose_enc[..., 7]          # [1, 3]
fov_w = pose_enc[..., 8]          # [1, 3]

# Decode rotation from quaternion
rotation_matrices = quat_to_mat(quaternion)  # [1, 3, 3, 3]

print(f"Translation shape: {translation.shape}")
print(f"Extrinsics shape: {extrinsics.shape}")
print(f"Intrinsics shape: {intrinsics.shape}")
```

--------------------------------

### Complete VGGT Inference Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

A comprehensive example showing how to load a VGGT model, preprocess images, run inference with query points, and extract predictions from all available heads. It also decodes camera parameters from the pose encoding output.

```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
from vggt.utils.pose_enc import pose_encoding_to_extri_intri

# Initialize
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# Load images
image_paths = ["img1.png", "img2.png", "img3.png"]
images = load_and_preprocess_images(image_paths).to(device)
H, W = images.shape[-2:]

# Define query points for tracking
query_points = torch.tensor([
    [100.0, 150.0],
    [250.0, 300.0],
    [400.0, 250.0]
]).to(device)

# Run inference
with torch.no_grad():
    predictions = model(images, query_points=query_points)

# Extract predictions from each head
pose_enc = predictions["pose_enc"]      # CameraHead output
depth = predictions["depth"]            # DPTHead (depth)
points_3d = predictions["world_points"] # DPTHead (points)
tracks = predictions["track"]           # TrackHead output
visibility = predictions["vis"]
confidence = predictions["conf"]

# Decode camera parameters
extrinsics, intrinsics = pose_encoding_to_extri_intri(
    pose_enc, image_size_hw=(H, W)
)

print(f"Cameras: {extrinsics.shape}")
print(f"Depth: {depth.shape}")
print(f"Points: {points_3d.shape}")
print(f"Tracks: {tracks.shape}")
```

--------------------------------

### Complete VGGT Integration Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Rotation-Utilities.md

Demonstrates initializing the VGGT model, loading and preprocessing images, running inference, and extracting/converting rotation representations from quaternions to rotation matrices.

```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
from vggt.utils.pose_enc import pose_encoding_to_extri_intri
from vggt.utils.rotation import quat_to_mat

# 1. Initialize model
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# 2. Load images
image_paths = ["img1.png", "img2.png", "img3.png"]
images = load_and_preprocess_images(image_paths).to(device)

# 3. Run inference
with torch.no_grad():
    predictions = model(images)

# 4. Extract and convert rotation representations
pose_enc = predictions["pose_enc"]  # [1, 3, 9]
quaternions = pose_enc[..., 3:7]  # [1, 3, 4]

# Convert quaternions to rotation matrices
rotation_matrices = quat_to_mat(quaternions)  # [1, 3, 3, 3]

# Use rotation matrices
R = rotation_matrices[0, 0]  # Rotation for first frame
print(f"Rotation matrix:\n{R}")

# Verify orthogonality
should_be_eye = R @ R.T
print(f"R @ R.T ≈ I:\n{should_be_eye}")

# Convert back to quaternion
from vggt.utils.rotation import mat_to_quat
quat_reconstructed = mat_to_quat(R)
print(f"Original quaternion: {quaternions[0, 0]}")
print(f"Reconstructed quaternion: {quat_reconstructed}")
```

--------------------------------

### Initialize VGGT Model

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Initializes the VGGT model and moves it to the appropriate device (GPU or CPU). This example assumes weights will be downloaded automatically.

```python
import torch
from vggt.models.vggt import VGGT

# Initialize model (automatically downloads weights)
model = VGGT.from_pretrained("facebook/VGGT-1B")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()
```

--------------------------------

### VGGT Tracking Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Performs inference with specified query points for tracking. This example demonstrates how to obtain track predictions, visibility, and confidence scores.

```python
# Specify points to track
query_points = torch.tensor([[100.0, 200.0], [60.72, 259.94]]).to(device)

with torch.no_grad():
    with torch.cuda.amp.autocast(dtype=dtype):
        predictions = model(images, query_points=query_points)

# Access track predictions
tracks = predictions["track"]  # [1, 3, 2, 2]
visibility = predictions["vis"]  # [1, 3, 2]
confidence = predictions["conf"]  # [1, 3, 2]
```

--------------------------------

### Load and Preprocess Single Image

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

This example demonstrates loading and preprocessing a single image using default settings. It shows how to import the function, specify the image path, and print the resulting tensor shape.

```python
import torch
from vggt.utils.load_fn import load_and_preprocess_images

image_paths = ["image.png"]
images = load_and_preprocess_images(image_paths)
print(images.shape)  # torch.Size([1, 3, 518, 518])
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://github.com/facebookresearch/vggt/blob/main/docs/package.md

Install PyTorch and torchvision with a specific version and CUDA support. Ensure your system has compatible CUDA drivers.

```bash
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu121
```

--------------------------------

### Launch Gradio Web Interface

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Starts the Gradio-based web interface for interactive 3D reconstruction and scene exploration. Can be run locally or accessed via Hugging Face Spaces.

```bash
python demo_gradio.py
```

--------------------------------

### Initialize VGGT Model and Load Images

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Load the VGGT model and preprocess input images. This snippet demonstrates the basic setup for running inference with VGGT, including device and data type selection.

```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images

device = "cuda" if torch.cuda.is_available() else "cpu"
# bfloat16 is supported on Ampere GPUs (Compute Capability 8.0+)
dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16


```

--------------------------------

### Training on Multiple Datasets Configuration

Source: https://github.com/facebookresearch/vggt/blob/main/training/README.md

Configuration example for training on multiple datasets, such as Co3D and VKitti, by composing them using ComposedDataset. Adjust `len_train` to control sampling ratios.

```yaml
data:
  train:
    dataset:
      _target_: data.composed_dataset.ComposedDataset
      dataset_configs:
        - _target_: data.datasets.co3d.Co3dDataset
          split: train
          CO3D_DIR: /YOUR/PATH/TO/CO3D
          CO3D_ANNOTATION_DIR: /YOUR/PATH/TO/CO3D_ANNOTATION
          len_train: 100000
        - _target_: data.datasets.vkitti.VKittiDataset
          split: train
          VKitti_DIR: /YOUR/PATH/TO/VKitti
          len_train: 100000
          expand_ratio: 8 

```

--------------------------------

### DPTHead Usage Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

Demonstrates how to use the DPTHead for depth and point map prediction. It shows the expected output shapes for predictions and confidence scores.

```python
# Depth prediction
depth, depth_conf = model.depth_head(
    aggregated_tokens_list, images, patch_start_idx
)

# Point map prediction
points_3d, points_conf = model.point_head(
    aggregated_tokens_list, images, patch_start_idx
)

print(f"Depth shape: {depth.shape}")       # [B, S, 518, 518, 1]
print(f"Confidence shape: {depth_conf.shape}")  # [B, S, 518, 518]
```

--------------------------------

### TrackHead Usage Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

Illustrates the usage of the TrackHead for tracking query points across frames. It shows how to define query points and access the resulting tracks, visibility, and confidence scores.

```python
# Define query points
B, S, N = 1, 3, 5
query_points = torch.tensor([
    [100.0, 150.0],
    [200.0, 250.0],
    [350.0, 400.0],
    [450.0, 100.0],
    [300.0, 300.0]
])

# Run tracking
tracks, vis_scores, conf_scores = model.track_head(
    aggregated_tokens_list,
    images,
    patch_start_idx,
    query_points=query_points
)

print(f"Tracks shape: {tracks.shape}")              # [1, 3, 5, 2]
print(f"Visibility shape: {vis_scores.shape}")      # [1, 3, 5]
print(f"Confidence shape: {conf_scores.shape}")     # [1, 3, 5]

# Access track for specific point and frame
point_track = tracks[0, :, 0, :]  # Track for point 0 across all frames
```

--------------------------------

### VGGT Inference Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Performs inference using the VGGT model on preprocessed images. It utilizes mixed-precision casting for efficiency on compatible hardware.

```python
from vggt.utils.load_fn import load_and_preprocess_images

# Load and preprocess images
device = "cuda"
dtype = torch.bfloat16  # or torch.float16 for older GPUs

image_paths = ["image1.png", "image2.png", "image3.png"]
images = load_and_preprocess_images(image_paths).to(device)

# Run inference
with torch.no_grad():
    with torch.cuda.amp.autocast(dtype=dtype):
        predictions = model(images)

# Access predictions
pose_enc = predictions["pose_enc"]  # [1, 3, 9]
depth = predictions["depth"]        # [1, 3, H, W, 1]
points_3d = predictions["world_points"]  # [1, 3, H, W, 3]
```

--------------------------------

### Configuration Validation Checks

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Configuration-Guide.md

Example assertions for validating key configuration parameters such as image size, patch size, depth divisibility, and pose encoding type. These checks ensure compatibility and prevent errors.

```python
# Validate image size
assert img_size in [224, 256, 384, 518, 1024], "Unsupported image size"

# Validate patch size
assert patch_size in [14, 16], "Unsupported patch size"

# Validate depth divisibility
assert depth % aa_block_size == 0, "Depth must be divisible by aa_block_size"

# Validate pose encoding type
assert pose_encoding_type == "absT_quaR_FoV", "Unsupported encoding type"
```

--------------------------------

### Load and Preprocess Multiple Images with Padding and GPU Transfer

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

This example shows how to load and preprocess multiple images using 'pad' mode and then transfer the resulting tensor to a CUDA-enabled GPU. It's suitable for batch processing when preserving full image content is important.

```python
image_paths = [
    "scene/image1.jpg",
    "scene/image2.jpg",
    "scene/image3.jpg"
]
images = load_and_preprocess_images(image_paths, mode="pad")
device = torch.device("cuda")
images = images.to(device)
```

--------------------------------

### Integrate with Gaussian Splatting

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Train a Gaussian Splatting model using exported COLMAP files. Ensure gsplat is installed and navigate to its directory.

```bash
cd gsplat
python examples/simple_trainer.py  default --data_factor 1 --data_dir /YOUR/SCENE_DIR/ --result_dir /YOUR/RESULT_DIR/
```

--------------------------------

### Fine-tune VGGT on Co3D

Source: https://github.com/facebookresearch/vggt/blob/main/training/README.md

Command to start fine-tuning the pre-trained VGGT model on the Co3D dataset using PyTorch Distributed Data Parallel (DDP) across 4 GPUs.

```bash
torchrun --nproc_per_node=4 launch.py
```

--------------------------------

### Load Pretrained VGGT Model

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Configuration-Guide.md

Load a pretrained VGGT model using its identifier. This is the simplest way to get started with a model trained on a large dataset.

```python
# Load pretrained model with default config
model = VGGT.from_pretrained("facebook/VGGT-1B")
```

--------------------------------

### Aggregator Forward Pass Example

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Aggregator.md

Performs a forward pass with the Aggregator model using preprocessed images. It's recommended to use torch.no_grad() for inference. The output includes cached token tensors and the starting index for patch tokens.

```python
from vggt.utils.load_fn import load_and_preprocess_images

# Prepare images
device = "cuda"
image_paths = ["image1.png", "image2.png", "image3.png"]
images = load_and_preprocess_images(image_paths).to(device)

# Forward pass
with torch.no_grad():
    aggregated_tokens_list, patch_start_idx = aggregator(images)

# Access cached layers
for i, tokens in enumerate(aggregated_tokens_list):
    if tokens is not None:
        print(f"Layer {i}: {tokens.shape}")  # e.g., torch.Size([3, 1333, 1024])
```

--------------------------------

### Run Viser 3D Viewer

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Launches the Viser 3D viewer to visualize point clouds from reconstruction. Requires a folder path containing images and supports using point maps.

```bash
python demo_viser.py --image_folder path/to/your/images/folder
```

--------------------------------

### img_from_cam

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Geometry-Utilities.md

Applies intrinsic parameters to camera coordinates to get pixel coordinates, optionally applying distortion.

```APIDOC
## Function: img_from_cam()

Applies intrinsic parameters to camera coordinates to get pixel coordinates.

### Signature

```python
def img_from_cam(
    cam_intrinsics: torch.Tensor,
    cam_points: torch.Tensor,
    distortion_params: torch.Tensor = None,
    default: float = 0.0
) -> torch.Tensor
```

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| cam_intrinsics | torch.Tensor | Yes | — | Intrinsic matrices of shape [B, 3, 3] |
| cam_points | torch.Tensor | Yes | — | Camera coordinates of shape [B, 3, N] |
| distortion_params | torch.Tensor | No | None | Distortion parameters of shape [B, N] |
| default | float | No | 0.0 | Value to replace NaNs |

### Return Type

Returns `torch.Tensor` of pixel coordinates with shape [B, N, 2].
```

--------------------------------

### Full VGGT Configuration

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Configuration-Guide.md

Initializes the VGGT model with all branches enabled and default parameters.

```python
model = VGGT(
    img_size=518,
    patch_size=14,
    embed_dim=1024,
    enable_camera=True,
    enable_depth=True,
    enable_point=True,
    enable_track=True
)
```

--------------------------------

### Define Query Points for Tracking

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Example of defining query points for the point tracking functionality. These points are used as input for the TrackHead.

```python
# Define query points
query_points = torch.tensor([
    [100.0, 150.0],
    [250.0, 300.0]
]).to(device)
```

--------------------------------

### VGGT Model Initialization

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

Shows how to load a pre-trained VGGT model and set it to evaluation mode. This is the standard way to integrate and use the head modules.

```python
from vggt.models.vggt import VGGT

model = VGGT.from_pretrained("facebook/VGGT-1B").eval()
```

--------------------------------

### Initialize CameraHead

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Initialize the CameraHead for predicting camera parameters. Requires the input dimension of the aggregated tokens.

```python
from vggt.heads.camera_head import CameraHead

camera_head = CameraHead(dim_in=2048)
pose_enc_list = camera_head(aggregated_tokens_list, num_iterations=4)
```

--------------------------------

### Load Pretrained VGGT Model

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Demonstrates how to automatically download and load pretrained weights for the VGGT model. Includes a fallback for manual loading if automatic download fails.

```python
# Automatic download and loading
model = VGGT.from_pretrained("facebook/VGGT-1B")

# Manual loading if automatic download fails
model = VGGT()
url = "https://huggingface.co/facebook/VGGT-1B/resolve/main/model.pt"
state_dict = torch.hub.load_state_dict_from_url(url)
model.load_state_dict(state_dict)
```

--------------------------------

### Get Camera Parameters from Predictions

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Extract camera pose encoding from prediction results and convert it into extrinsic and intrinsic matrices. This is useful for subsequent geometric operations.

```python
# Method 1: From predictions
pose_enc = predictions["pose_enc"]
extrinsics, intrinsics = pose_encoding_to_extri_intri(pose_enc, (H, W))

# Method 2: Extract from pose encoding
quaternion = predictions["pose_enc"][..., 3:7]
translation = predictions["pose_enc"][..., 0:3]
fov_h = predictions["pose_enc"][..., 7]
fov_w = predictions["pose_enc"][..., 8]
```

--------------------------------

### Run Inference and Get Tracking Results

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Perform inference on images and extract tracking and visibility information. Ensure the model is in evaluation mode and gradients are disabled.

```python
import torch

# Assuming model, images, and query_points are defined
# model.eval() # Ensure model is in evaluation mode
with torch.no_grad():
    predictions = model(images, query_points=query_points)

tracks = predictions["track"]        # [1, 3, 2, 2]
visibility = predictions["vis"]      # [1, 3, 2]
confidence = predictions["conf"]     # [1, 3, 2]

for t, frame_tracks in enumerate(tracks[0]):
    print(f"Frame {t}: {frame_tracks}")
```

--------------------------------

### Initialize VGGT Model with Pretrained Weights

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Initializes the VGGT model and loads pretrained weights from Hugging Face. The weights are downloaded automatically on the first run.

```python
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device)
```

--------------------------------

### CameraHead Initialization

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

Initializes the CameraHead module for predicting camera extrinsic and intrinsic parameters.

```python
class CameraHead(nn.Module):
    def __init__(
        self,
        dim_in: int = 2048,
        trunk_depth: int = 4,
        pose_encoding_type: str = "absT_quaR_FoV",
        num_heads: int = 16,
        mlp_ratio: int = 4,
        init_values: float = 0.01,
        trans_act: str = "linear",
        quat_act: str = "linear",
        fl_act: str = "relu"
    )
```

--------------------------------

### Minimal VGGT Configuration (Camera Only)

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Configuration-Guide.md

Initializes the VGGT model with only the camera prediction branch enabled for memory savings.

```python
from vggt.models.vggt import VGGT

model = VGGT(
    enable_camera=True,
    enable_depth=False,
    enable_point=False,
    enable_track=False
)
```

--------------------------------

### Basic VGGT Inference Pipeline

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Demonstrates a typical inference workflow: setting up the model, loading images, performing inference, and accessing prediction results.

```python
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
import torch

# Setup
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# Load images
image_paths = ["img1.png", "img2.png", "img3.png"]
images = load_and_preprocess_images(image_paths).to(device)

# Inference
with torch.no_grad():
    predictions = model(images)

# Access results
depth = predictions["depth"]           # [1, 3, 518, 518, 1]
world_points = predictions["world_points"]  # [1, 3, 518, 518, 3]
pose_enc = predictions["pose_enc"]     # [1, 3, 9]
```

--------------------------------

### DPTHead forward()

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Head-Modules.md

The forward method of the DPTHead is used for depth and point map prediction. It takes cached tokens, images, and a patch start index to produce predictions and confidence scores.

```APIDOC
## DPTHead forward()

### Description
Performs depth and point map prediction using the Dense Prediction Transformer architecture. It processes aggregated tokens and images to output predictions and their associated confidence scores.

### Method
`forward(aggregated_tokens_list: list, images: torch.Tensor, patch_start_idx: int) -> tuple[torch.Tensor, torch.Tensor]`

### Parameters
#### Path Parameters
* `aggregated_tokens_list` (list) - Required - Cached token tensors from aggregator.
* `images` (torch.Tensor) - Required - Original input images [B, S, 3, H, W].
* `patch_start_idx` (int) - Required - Starting index of patch tokens.

### Return Type
Returns a tuple containing:
- `predictions` (torch.Tensor): Output predictions with shape [B, S, H, W, output_dim].
- `confidence` (torch.Tensor): Confidence scores with shape [B, S, H, W].

### Usage Example
```python
# Depth prediction
depth, depth_conf = model.depth_head(
    aggregated_tokens_list,
    images,
    patch_start_idx
)

# Point map prediction
points_3d, points_conf = model.point_head(
    aggregated_tokens_list,
    images,
    patch_start_idx
)

print(f"Depth shape: {depth.shape}")       # [B, S, 518, 518, 1]
print(f"Confidence shape: {depth_conf.shape}")  # [B, S, 518, 518]
```
```

--------------------------------

### Multi-GPU Inference Setup

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Advanced-Usage-Guide.md

Distribute inference workload across multiple GPUs using PyTorch's distributed package. Each process handles a subset of scenes on its assigned GPU. Requires `torch.distributed` and `torch.multiprocessing`.

```python
import torch.distributed as dist
import torch.multiprocessing as mp

def inference_multi_gpu(rank, world_size, scene_list):
    """Run inference on multiple GPUs."""
    # Setup
    dist.init_process_group(
        backend="nccl",
        rank=rank,
        world_size=world_size
    )
    
    device = torch.device(f"cuda:{rank}")
    model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()
    
    # Process subset of scenes
    scenes_per_gpu = len(scene_list) // world_size
    my_scenes = scene_list[rank * scenes_per_gpu:(rank+1) * scenes_per_gpu]
    
    for scene in my_scenes:
        images = load_and_preprocess_images(scene).to(device)
        with torch.no_grad():
            predictions = model(images)
        
        # Process results
        # ...
    
    dist.destroy_process_group()

# Launch
if __name__ == "__main__":
    scenes = [...]  # List of scene image paths
    mp.spawn(inference_multi_gpu, args=(torch.cuda.device_count(), scenes), 
             nprocs=torch.cuda.device_count())
```

--------------------------------

### Load and Initialize VGGT Model

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Load the main VGGT inference model from pre-trained weights and prepare it for evaluation. Ensure CUDA is available if using GPU.

```python
from vggt.models.vggt import VGGT

model = VGGT.from_pretrained("facebook/VGGT-1B").cuda().eval()
```

--------------------------------

### Memory Profiling Function

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Advanced-Usage-Guide.md

Profiles peak CPU and GPU memory usage during model inference. Ensure `tracemalloc` is started before inference and stopped afterward. Clears CUDA cache and resets peak memory stats before profiling.

```python
import torch
import tracemalloc

def profile_memory(model, images):
    """Profile peak memory usage."""
    tracemalloc.start()
    
    # Clear cache
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    
    # Inference
    with torch.no_grad():
        predictions = model(images)
    
    # Check memory
    current, peak = tracemalloc.get_traced_memory()
    gpu_peak = torch.cuda.max_memory_allocated() / 1e9
    
    print(f"CPU current: {current / 1e6:.1f}MB")
    print(f"CPU peak: {peak / 1e6:.1f}MB")
    print(f"GPU peak: {gpu_peak:.2f}GB")
    
    tracemalloc.stop()
    return predictions

# Usage
predictions = profile_memory(model, images)
```

--------------------------------

### Initialize TrackHead

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Initialize the TrackHead for point tracking across frames. Requires input dimension and patch size.

```python
from vggt.heads.track_head import TrackHead

track_head = TrackHead(dim_in=2048, patch_size=14)
tracks, vis, conf = track_head(tokens_list, images, patch_start_idx, query_points)
```

--------------------------------

### Import Pose Encoding Utilities

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Import functions for converting between camera pose encodings and extrinsic/intrinsic matrices. These are crucial for handling camera calibration and pose information.

```python
from vggt.utils.pose_enc import (
    extri_intri_to_pose_encoding,
    pose_encoding_to_extri_intri
)
```

--------------------------------

### Guide NeRF Training with VGGT Predictions

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Advanced-Usage-Guide.md

Use VGGT's depth and confidence predictions to supervise NeRF training. This involves obtaining VGGT predictions and computing depth losses alongside standard rendering losses.

```python
class VGGTGuidedNeRF:
    def __init__(self, nerf_model, vggt_model):
        self.nerf = nerf_model
        self.vggt = vggt_model
    
    def training_step(self, images, ray_batch):
        """
        NeRF training step with VGGT guidance.
        """
        # Get VGGT predictions
        with torch.no_grad():
            vggt_predictions = self.vggt(images)
        
        vggt_depth = vggt_predictions["depth"]
        vggt_confidence = vggt_predictions["depth_conf"]
        
        # NeRF forward pass
        nerf_output = self.nerf(ray_batch)
        
        # Compute losses
        render_loss = self.nerf_loss(nerf_output, ray_batch)
        
        # Depth supervision from VGGT
        # (requires proper coordinate transformation)
        # depth_loss = MSE(nerf_depth, vggt_depth)
        
        # Total loss
        # total_loss = render_loss + 0.1 * depth_loss
        
        return nerf_output
```

--------------------------------

### VGGT 3D Reconstruction Pipeline

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Geometry-Utilities.md

This snippet outlines a complete 3D reconstruction pipeline using VGGT predictions. It covers getting predictions, converting depth to world coordinates, and projecting world points back to camera coordinates.

```python
# 1. Get VGGT predictions
predictions = model(images)
depth = predictions["depth"]
extrinsics, intrinsics = pose_encoding_to_extri_intri(
    predictions["pose_enc"], image_size_hw=(H, W)
)

# 2. Convert depth to world coordinates
world_points = unproject_depth_map_to_point_map(
    depth.squeeze(0).cpu().numpy(),
    extrinsics.squeeze(0).cpu().numpy(),
    intrinsics.squeeze(0).cpu().numpy()
)

# 3. Project world points to all cameras
image_coords, _ = project_world_points_to_cam(
    torch.from_numpy(world_points[0].reshape(-1, 3)),
    extrinsics,
    intrinsics
)
```

--------------------------------

### VGGT Model Inference

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/README.md

Demonstrates how to initialize the VGGT model and perform inference on a batch of images.

```APIDOC
## VGGT.forward()

### Description
Performs inference using the VGGT model to generate predictions such as depth, world points, and camera poses.

### Method
`VGGT.forward(images)`

### Parameters
#### Input
- **images** (Tensor) - Preprocessed images to perform inference on.

### Request Example
```python
import torch
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images

# Initialize model
model = VGGT.from_pretrained("facebook/VGGT-1B").cuda().eval()

# Load images
image_paths = ["view1.png", "view2.png", "view3.png"]
images = load_and_preprocess_images(image_paths).cuda()

# Inference
with torch.no_grad():
    predictions = model(images)
```

### Response
#### Success Response
- **predictions** (dict) - A dictionary containing the model's predictions.
  - **depth** (Tensor) - Predicted depth map.
  - **world_points** (Tensor) - Predicted 3D world points.
  - **pose_enc** (Tensor) - Predicted camera pose encoding.

#### Response Example
```json
{
  "depth": "[1, 3, 518, 518, 1]",
  "world_points": "[1, 3, 518, 518, 3]",
  "pose_enc": "[1, 3, 9]"
}
```
```

--------------------------------

### Load and Preprocess Images

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Loads and preprocesses a list of image paths for model input. Ensure images are moved to the appropriate device.

```python
image_names = ["path/to/imageA.png", "path/to/imageB.png", "path/to/imageC.png"]
images = load_and_preprocess_images(image_names).to(device)
```

--------------------------------

### Load and Preprocess Images for VGGT Model Inference

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

This snippet demonstrates the complete workflow of loading images, preprocessing them, and feeding them into a VGGT model for inference. It includes model loading, image preprocessing, and running inference within a `torch.no_grad()` context.

```python
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images

model = VGGT.from_pretrained("facebook/VGGT-1B").cuda().eval()

image_paths = ["image1.png", "image2.png"]
images = load_and_preprocess_images(image_paths).cuda()

with torch.no_grad():
    predictions = model(images)
```

--------------------------------

### Load Custom VGGT Architecture with Weights

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Configuration-Guide.md

Create a custom VGGT architecture by specifying parameters like image size, and then load custom model weights from a saved state dictionary.

```python
# Create custom architecture and load weights
model = VGGT(img_size=384)
state_dict = torch.load("model_weights.pt")
model.load_state_dict(state_dict)
```

--------------------------------

### Initialize Gaussian Splatting with VGGT Predictions

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Advanced-Usage-Guide.md

Use VGGT's depth and confidence predictions to initialize the means, scales, colors, and opacities for Gaussian Splatting. This requires pre-processing images and filtering points based on confidence.

```python
class VGGTInitializer:
    def __init__(self, model, device="cuda"):
        self.model = model
        self.device = device
    
    def get_initial_gaussians(self, image_paths, scene_scale=1.0):
        """
        Initialize Gaussian splatting from VGGT predictions.
        
        Returns:
            means: [N, 3] 3D positions
            scales: [N, 3] isotropic scales
            colors: [N, 3] RGB colors
            opacities: [N] alpha values
        """
        # Load images
        from vggt.utils.load_fn import load_and_preprocess_images
        images = load_and_preprocess_images(image_paths).to(self.device)
        
        # Get predictions
        with torch.no_grad():
            predictions = self.model(images)
        
        # Extract 3D points
        world_points = predictions["world_points"][0]  # [S, H, W, 3]
        depth_conf = predictions["depth_conf"][0]  # [S, H, W]
        
        # Flatten and filter
        points_flat = world_points.reshape(-1, 3)
        conf_flat = depth_conf.reshape(-1)
        
        # Keep high-confidence points
        valid_mask = conf_flat > 0.5
        means = points_flat[valid_mask] * scene_scale
        
        # Initialize scales (based on depth uncertainty)
        scales = torch.ones_like(means) * 0.1
        
        # Sample colors from images
        colors = torch.ones_like(means)  # Placeholder
        
        # Initialize opacities
        opacities = conf_flat[valid_mask]
        
        return means, scales, colors, opacities
```

--------------------------------

### Import Image Preprocessing Utility

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Import the function for loading and preprocessing images. This utility handles necessary transformations before feeding images to the model.

```python
from vggt.utils.load_fn import load_and_preprocess_images
```

--------------------------------

### Basic Inference Pattern

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Demonstrates a typical inference workflow using the VGGT model, including loading images and accessing prediction results.

```APIDOC
## Common Usage Patterns

### Pattern 1: Basic Inference
```python
from vggt.models.vggt import VGGT
from vggt.utils.load_fn import load_and_preprocess_images
import torch

# Setup
device = "cuda"
model = VGGT.from_pretrained("facebook/VGGT-1B").to(device).eval()

# Load images
image_paths = ["img1.png", "img2.png", "img3.png"]
images = load_and_preprocess_images(image_paths).to(device)

# Inference
with torch.no_grad():
    predictions = model(images)

# Access results
depth = predictions["depth"]           # [1, 3, 518, 518, 1]
world_points = predictions["world_points"]  # [1, 3, 518, 518, 3]
pose_enc = predictions["pose_enc"]     # [1, 3, 9]
```
```

--------------------------------

### Load and Preprocess Images

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Load images from file paths and preprocess them for the model. The 'mode' parameter can be set to 'crop' or other options.

```python
from vggt.utils.load_fn import load_and_preprocess_images

# Load and preprocess images
images = load_and_preprocess_images(image_paths, mode="crop")
# Returns: torch.Tensor of shape [N, 3, H, W]
```

--------------------------------

### Load and Preprocess Images (Pad Mode)

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

Use this snippet to load and preprocess images using the 'pad' mode. This mode makes the largest dimension 518px while maintaining aspect ratio, then pads to create square images, preserving all image content without cropping. It's ideal when you need to keep the entire image.

```python
images = load_and_preprocess_images(image_paths, mode="pad")
```

--------------------------------

### VGGT Initialization

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Initializes the VGGT model. The model can be initialized with various configurations for predicting different scene attributes. Weights are automatically downloaded when using `from_pretrained`.

```APIDOC
## VGGT Class

### Description
Initializes the VGGT model with specified parameters.

### Parameters
* **img_size** (int) - Optional - Default: 518 - Height and width of input images in pixels.
* **patch_size** (int) - Optional - Default: 14 - Size of patches for tokenization.
* **embed_dim** (int) - Optional - Default: 1024 - Dimension of token embeddings and feature representations.
* **enable_camera** (bool) - Optional - Default: True - Whether to enable camera parameter prediction.
* **enable_point** (bool) - Optional - Default: True - Whether to enable 3D point map prediction.
* **enable_depth** (bool) - Optional - Default: True - Whether to enable depth map prediction.
* **enable_track** (bool) - Optional - Default: True - Whether to enable point tracking.

### Initialization Example
```python
import torch
from vggt.models.vggt import VGGT

# Initialize model (automatically downloads weights)
model = VGGT.from_pretrained("facebook/VGGT-1B")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()
```
```

--------------------------------

### Import Rotation Utilities

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Import functions for converting between quaternion and rotation matrix representations. These utilities help manage different rotation formats used in 3D transformations.

```python
from vggt.utils.rotation import (
    quat_to_mat,
    mat_to_quat,
    standardize_quaternion
)
```

--------------------------------

### Import Geometry Utilities

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Import various geometry utility functions for 3D computer vision tasks, including point unprojection, depth conversion, and projection to camera coordinates.

```python
from vggt.utils.geometry import (
    unproject_depth_map_to_point_map,
    depth_to_world_coords_points,
    project_world_points_to_cam,
    cam_from_img
)
```

--------------------------------

### VGGT Camera Head Initialization

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Initializes the Camera Head submodule of the VGGT model. This head predicts camera extrinsics and intrinsics through iterative refinement.

```python
self.camera_head = CameraHead(dim_in=2 * embed_dim)
```

--------------------------------

### Run COLMAP with Bundle Adjustment

Source: https://github.com/facebookresearch/vggt/blob/main/README.md

Execute COLMAP reconstruction with bundle adjustment enabled. Ensure images are in the specified scene directory.

```bash
python demo_colmap.py --scene_dir=/YOUR/SCENE_DIR/ --use_ba
```

--------------------------------

### Initialize Track Head

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/VGGT-Model.md

Initializes the track head for tracking specified query points across frames using feature correlation. Requires aggregated tokens, images, patch indices, and query points as input.

```python
self.track_head = TrackHead(
    dim_in=2 * embed_dim,
    patch_size=patch_size
)
```

--------------------------------

### Load and Preprocess Images (Crop Mode)

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

Use this snippet to load and preprocess images using the 'crop' mode, which resizes images to a fixed width while maintaining aspect ratio and center-cropping height if necessary. This is useful when uniform image dimensions prioritizing width are required.

```python
images = load_and_preprocess_images(image_paths, mode="crop")
```

--------------------------------

### load_and_preprocess_images

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Image-Loading-Utilities.md

Loads a list of images, preprocesses them according to the specified mode ('crop' or 'pad'), and returns them as a PyTorch tensor.

```APIDOC
## Function: load_and_preprocess_images()

### Signature

```python
def load_and_preprocess_images(
    image_path_list: list,
    mode: str = "crop"
) -> torch.Tensor
```

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| image_path_list | list | Yes | — | List of file paths to image files (PNG, JPG, etc.) |
| mode | str | No | "crop" | Preprocessing mode: "crop" or "pad" |

### Return Type

Returns a `torch.Tensor` of preprocessed images with shape: [N, 3, H, W].

### Preprocessing Modes

#### Mode: "crop"

Resizes images to width=518px while maintaining aspect ratio. Height is center-cropped if exceeds 518px.

```python
images = load_and_preprocess_images(image_paths, mode="crop")
```

#### Mode: "pad"

Makes the largest dimension 518px while maintaining aspect ratio, then pads to create square images.

```python
images = load_and_preprocess_images(image_paths, mode="pad")
```

### Usage Examples

#### Single Image

```python
import torch
from vggt.utils.load_fn import load_and_preprocess_images

image_paths = ["image.png"]
images = load_and_preprocess_images(image_paths)
print(images.shape)  # torch.Size([1, 3, 518, 518])
```

#### Multiple Images

```python
image_paths = [
    "scene/image1.jpg",
    "scene/image2.jpg",
    "scene/image3.jpg"
]
images = load_and_preprocess_images(image_paths, mode="pad")
device = torch.device("cuda")
images = images.to(device)
```

### Image Format Handling

Supported formats: PNG, JPEG, BMP, and any format supported by PIL.Image.open(). Alpha channels are composited onto a white background and converted to RGB.
```

--------------------------------

### Initialize DPTHead for Depth Prediction

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Initialize the DPTHead for dense depth prediction. Specify input dimension, output dimension (2 for depth and confidence), and activation function.

```python
from vggt.heads.dpt_head import DPTHead

# For depth
depth_head = DPTHead(dim_in=2048, output_dim=2, activation="exp")
depth, conf = depth_head(tokens_list, images, patch_start_idx)
```

--------------------------------

### Image Loading Utilities

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/README.md

Provides functions for loading and preprocessing images for the VGGT model.

```APIDOC
## Image Loading Utilities

### Description
Functions to load and preprocess images, preparing them for input into the VGGT model.

### Functions
- `load_and_preprocess_images(image_paths)`: Loads and preprocesses a list of images.
- `load_and_preprocess_images_square(image_paths)`: Loads and preprocesses images, ensuring they are square.

### Request Example
```python
from vggt.utils.load_fn import load_and_preprocess_images

image_paths = ["view1.png", "view2.png"]
images = load_and_preprocess_images(image_paths)
```
```

--------------------------------

### Import VGGT Model

Source: https://github.com/facebookresearch/vggt/blob/main/_autodocs/Quick-API-Reference.md

Import the main VGGT model class for use in your project. This is the primary entry point for using the VGGT architecture.

```python
from vggt.models.vggt import VGGT
```