### Install UniDepth Environment

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Set up a virtual environment and install UniDepth along with its dependencies. Ensure your CUDA version is compatible.

```shell
export VENV_DIR=<YOUR-VENVS-DIR>
export NAME=Unidepth

python -m venv $VENV_DIR/$NAME
source $VENV_DIR/$NAME/bin/activate

# Install UniDepth and dependencies, cuda >11.8 work fine, too.
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### Run UniDepth Demo

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Execute the demo script to test the UniDepth installation. Expected output is 'ARel: 7.45%'.

```shell
python ./scripts/demo.py
```

--------------------------------

### Environment Setup: Conda vs. Venv

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Instructions for adapting environment setup commands when using Conda instead of a virtual environment (venv).

```shell
python -m venv $VENV_DIR/$NAME -> conda create -n $NAME python=3.11
source $VENV_DIR/$NAME/bin/activate -> conda activate $NAME
```

--------------------------------

### Install KNN

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Navigate to the KNN directory and compile the necessary scripts. This is for evaluation purposes only.

```shell
cd unidepth/ops/knn;bash compile.sh;cd ../../../
```

--------------------------------

### Set Up Training Environment Variables

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/README.md

Configure essential environment variables before running the training script. Adapt paths and settings to your specific setup.

```bash
export REPO=`pwd`
export PYTHONPATH=${REPO}:${PYTHONPATH}

# Adapt all this to your setup
export TMPDIR="/tmp"
export TORCH_HOME=${TMPDIR}
export HUGGINGFACE_HUB_CACHE=${TMPDIR}
export WANDB_HOME=${TMPDIR}
export DATAROOT=<where-you-stored-the-hdf5>


export MASTER_PORT=$((( RANDOM % 600 ) + 29400 ))
if [ $NNODES -gt 1 ]; then
    export MASTER_PORT=29400
fi

# this is the config will be used
export CFG="train_v1_vitl14.json"
```

--------------------------------

### Install Pillow-SIMD (Optional)

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Optionally install Pillow-SIMD for potential performance improvements. This involves uninstalling the current Pillow and reinstalling with specific compiler flags.

```shell
pip uninstall pillow
CC="cc -mavx2" pip install -U --force-reinstall pillow-simd
```

--------------------------------

### Run Training Script (Non-SLURM)

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/README.md

Execute the training script using torchrun for multi-node or single-node setups without SLURM. Ensure CUDA_VISIBLE_DEVICES is set.

```bash
# make the following input-dependent for multi-node
export NNODES=1
export RANK=0
export MASTER_ADDR=127.0.0.1
export CUDA_VISIBLE_DEVICES="0" # set yours

export GPUS=$(echo ${CUDA_VISIBLE_DEVICES} | tr ',' '\n' | wc -l)
echo "Start script with python from: `which python`"
torchrun --rdzv-backend=c10d --nnodes=${NNODES} --nproc_per_node=${GPUS} --rdzv-endpoint ${MASTER_ADDR}:${MASTER_PORT} ${REPO}/scripts/train.py --config-file ${REPO}/configs/${CFG} --distributed
```

--------------------------------

### Run Training Script (SLURM)

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/README.md

Execute the training script when using SLURM. SLURM handles most of the distributed training setup, requiring fewer manual exports.

```bash
srun -c ${SLURM_CPUS_PER_TASK} --kill-on-bad-exit=1 python -u ${REPO}/scripts/train.py --config-file ${REPO}/configs/${CFG} --master-port ${MASTER_PORT} --distributed
```

--------------------------------

### Compile KNN Operation

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/scripts/README.md

Compile the KNN operation for Pytorch3D if installation issues arise. This is required for chamfer distance evaluation.

```bash
bash compile.sh
```

--------------------------------

### Distributed Training with torchrun

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Initiates distributed training for UniDepthV2 using torchrun. Ensure environment variables for repository path, data root, configuration file, and distributed settings are correctly set.

```bash
export REPO=$(pwd)
export PYTHONPATH=${REPO}:${PYTHONPATH}
export DATAROOT=/path/to/hdf5/datasets
export CFG="configs/train_v1_vitl14.json"
export MASTER_ADDR=127.0.0.1
export MASTER_PORT=29400
export CUDA_VISIBLE_DEVICES="0,1,2,3"
export GPUS=4
export NNODES=1
export RANK=0

torchrun \
    --rdzv-backend=c10d \
    --nnodes=${NNODES} \
    --nproc_per_node=${GPUS} \
    --rdzv-endpoint ${MASTER_ADDR}:${MASTER_PORT} \
    ${REPO}/scripts/train.py \
    --config-file ${CFG} \
    --distributed
```

--------------------------------

### Initialize Pinhole Camera Model

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Initializes a Pinhole camera model using either a 3x3 K matrix or camera parameters. Supports projection, unprojection, and reconstruction. Requires 'torch'. The camera's device can be checked.

```python
import torch
from unidepth.utils.camera import Pinhole

# From K matrix
K = torch.tensor([[800.0, 0.0, 320.0],
                  [0.0,   800.0, 240.0],
                  [0.0,   0.0,   1.0]]).unsqueeze(0)  # (1, 3, 3)
cam = Pinhole(K=K)

# From params
cam2 = Pinhole(params=torch.tensor([[800.0, 800.0, 320.0, 240.0]]))

# Unproject 2D pixel grid to 3D rays
from unidepth.utils.coordinate import coords_grid
uv = coords_grid(1, 480, 640, device="cpu")  # (1, 2, H, W)
rays = cam.unproject(uv)                     # (1, 3, H, W)

# Project 3D points back to 2D
pcd = torch.randn(1, 3, 480, 640)
pixels = cam.project(pcd)                    # (1, 2, H, W)

# Reconstruct 3D from depth
depth = torch.rand(1, 1, 480, 640) * 5.0    # meters
xyz = cam.reconstruct(depth)                 # (1, 3, H, W)
print("Point cloud shape:", xyz.shape)       # torch.Size([1, 3, 480, 640])
print("Camera device:", cam.device)
```

--------------------------------

### UniDepthV2.from_pretrained

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads a pretrained UniDepthV2 model from the Hugging Face Hub. Supports different backbone sizes and can be moved to a specified device. Optional settings for resolution level and interpolation mode are available.

```APIDOC
## UniDepthV2.from_pretrained

### Description
Downloads and instantiates a UniDepthV2 model from the Hugging Face Hub. Available backbones are `vits14`, `vitb14`, and `vitl14`. The model inherits from `PyTorchModelHubMixin`, so standard HF Hub loading applies.

### Method Signature
`UniDepthV2.from_pretrained(model_name: str)`

### Parameters
- **model_name** (str) - Required - The name of the pretrained model to load from Hugging Face Hub (e.g., "lpiccinelli/unidepth-v2-vitl14").

### Request Example
```python
import torch
from unidepth.models import UniDepthV2

# Load ViT-L backbone (largest, most accurate)
model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()

# Optional V2-only settings
model.resolution_level = 9          # 0 (low-res) to 9 (high-res); default range used if unset
model.interpolation_mode = "bilinear"  # output upsampling mode

print(f"Model loaded on {device}")
```

### Response
- **model** (object) - An instantiated UniDepthV2 model ready for inference.
```

--------------------------------

### Export UniDepthV2 to ONNX (CLI)

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Exports a UniDepthV2 model to ONNX format using the command line. Supports standard export and export with camera ray inputs. Specifies model version, backbone, input shape, and output path.

```bash
# Standard export (model predicts intrinsics)
python unidepth/models/unidepthv2/export.py \
    --version v2 \
    --backbone vitl \
    --shape 462 630 \
    --output-path unidepthv2_vitl.onnx

# Export with camera rays input
python unidepth/models/unidepthv2/export.py \
    --version v2 \
    --backbone vitb \
    --shape 462 630 \
    --output-path unidepthv2_vitb_cam.onnx \
    --with-camera
```

--------------------------------

### Load UniDepthV2 Model from Hugging Face

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Downloads and instantiates a UniDepthV2 model from the Hugging Face Hub. Supports different backbone sizes and optional V2-only settings for resolution and interpolation.

```python
import torch
from unidepth.models import UniDepthV2

# Load ViT-L backbone (largest, most accurate)
model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14")

# Load ViT-S backbone (smallest, fastest)
# model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vits14")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()

# Optional V2-only settings
model.resolution_level = 9          # 0 (low-res) to 9 (high-res); default range used if unset
model.interpolation_mode = "bilinear"  # output upsampling mode

print(f"Model loaded on {device}")
# Model loaded on cuda
```

--------------------------------

### Load UniDepth via TorchHub

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads any UniDepth version and backbone through the PyTorch Hub interface without directly importing the package. Ensure `trust_repo=True` for security.

```python
import torch

model = torch.hub.load(
    "lpiccinelli-eth/UniDepth",
    "UniDepth",
    version="v2",         # "v1" or "v2"
    backbone="vitl14",    # "vitl14", "vitb14", "vits14" for v2; "vitl14", "cnvnxtl" for v1
    pretrained=True,
    trust_repo=True,
    force_reload=True,
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()
# UniDepth_v2_vitl14 is loaded with:
#     missing keys: []
#     additional keys: []
```

--------------------------------

### UniDepthV1.from_pretrained

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads a pretrained UniDepthV1 model. Supports ViT-L and ConvNext-L backbones. It shares the same `infer` interface as V2 but lacks certain V2-specific features.

```APIDOC
## UniDepthV1.from_pretrained

### Description
Loads a UniDepthV1 model. Available backbones: `vitl14` and `cnvnxtl`. V1 uses the same `infer` interface but does not support `resolution_level`, confidence output, or new camera classes.

### Method Signature
`UniDepthV1.from_pretrained(model_name: str)`

### Parameters
- **model_name** (str) - Required - The name of the pretrained model to load (e.g., "lpiccinelli/unidepth-v1-vitl14").

### Request Example
```python
import torch
from unidepth.models import UniDepthV1

# ViT-L backbone
model_v1 = UniDepthV1.from_pretrained("lpiccinelli/unidepth-v1-vitl14")

# ConvNext-L backbone
# model_v1 = UniDepthV1.from_pretrained("lpiccinelli/unidepth-v1-cnvnxtl")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_v1 = model_v1.to(device).eval()
print("UniDepthV1 loaded")
```

### Response
- **model_v1** (object) - An instantiated UniDepthV1 model ready for inference.
```

--------------------------------

### Distributed Training with SLURM

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Configures distributed training for UniDepthV2 on a SLURM cluster. This command assumes SLURM environment variables are set.

```bash
srun -c ${SLURM_CPUS_PER_TASK} --kill-on-bad-exit=1 \
    python -u ${REPO}/scripts/train.py \
    --config-file ${REPO}/configs/train_v1_vitl14.json \
    --master-port ${MASTER_PORT} \
    --distributed
```

--------------------------------

### Export UniDepthV2 to ONNX (Programmatic)

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Programmatically exports a UniDepthV2 model to ONNX format. Loads a configuration, modifies it for export (disabling efficient attention), initializes the model, and loads pretrained weights.

```python
import torch, json
from unidepth.models.unidepthv2.export import UniDepthV2ONNX, export

with open("configs/config_v2_vitb14.json") as f:
    config = json.load(f)
config["training"]["export"] = True   # disables efficient attention (not ONNX-exportable)

model = UniDepthV2ONNX(config)
model.load_pretrained("pytorch_model.bin")
```

--------------------------------

### Load and Train UniDepthV2 Model

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads a pre-trained UniDepthV2 model and prepares it for training. Requires 'torch' and 'json'. Inputs must be ImageNet-normalized and provided in a dictionary format. The forward pass returns outputs and losses.

```python
import torch
from unidepth.models import UniDepthV2
import json

with open("configs/config_v2_vitl14.json") as f:
    config = json.load(f)

model = UniDepthV2(config)
model.load_pretrained("pytorch_model.bin")
model.train()

# Inputs must be pre-processed:
# - ImageNet normalized, shape (B, C, H, W)
# - "K" is the intrinsics tensor (B, 3, 3); optional if unknown
B, C, H, W = 2, 3, 480, 640
data = {
    "image": torch.randn(B, C, H, W),   # ImageNet-normalized
    "depth": torch.rand(B, 1, H, W),    # GT depth map
    "depth_mask": torch.ones(B, 1, H, W, dtype=torch.bool),
}
image_metas = [{"si": False, "flip": False}] * B

outputs, losses = model(data, image_metas)
print("Losses:", {k: v.item() for k, v in losses["opt"].items()})
# Losses: {'SILog': 0.42, 'Regression': 0.07, 'SelfDistill': 0.03, ...}
```

--------------------------------

### Initialize Fisheye624 Camera Model

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Initializes a Fisheye624 camera model with up to 16 distortion parameters. Compatible with the V2 infer interface. Requires 'torch'. Can be used with UniDepthV2.infer for depth prediction.

```python
import torch
from unidepth.utils.camera import Fisheye624

# params: [fx, fy, cx, cy, k1, k2, k3, k4, k5, k6, t1, t2, s1, s2, s3, s4]
params = torch.tensor([[ 
    500.0, 500.0, 320.0, 240.0,   # intrinsics
    -0.2,  0.1,   0.0,   0.0,    # radial k1-k4
    0.0,   0.0,                  # radial k5-k6
    0.0,   0.0,                  # tangential t1, t2
    0.0,   0.0,   0.0,   0.0,   # thin prism s1-s4
]])
cam = Fisheye624(params=params)

from unidepth.utils.coordinate import coords_grid
uv = coords_grid(1, 480, 640, device="cpu")
rays = cam.unproject(uv)   # (1, 3, H, W)
print("Fisheye rays shape:", rays.shape)  # torch.Size([1, 3, 480, 640])

# Use with UniDepthV2.infer
from unidepth.models import UniDepthV2
model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14").eval()
import numpy as np
from PIL import Image
# rgb = torch.from_numpy(np.array(Image.open("assets/demo/rgb.png"))).permute(2, 0, 1)
# predictions = model.infer(rgb, cam)
# print("Depth shape:", predictions["depth"].shape)
```

--------------------------------

### Custom Training with Forward Method

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Prepare input data for custom training using the model's forward method. This involves ImageNet normalization, resizing, padding, and formatting the input as a dictionary with 'image' and 'K' (intrinsics).

```python
data = {"image": rgb, "K": intrinsics}
predictions = model(data, {})
```

--------------------------------

### Load Pre-trained UniDepth Model

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Load a pre-trained UniDepth model from Hugging Face. You can choose between the ViT-Large/14 or ConvNeXt-Large backbone.

```python
from unidepth.models import UniDepthV1

model = UniDepthV1.from_pretrained("lpiccinelli/unidepth-v1-vitl14") # or "lpiccinelli/unidepth-v1-cnvnxtl" for the ConvNext backbone
```

--------------------------------

### Create Spherical Camera Rays

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Initializes a Spherical camera model and generates unit sphere direction vectors for a given UV coordinate grid. Useful for unprojecting image pixels into 3D space.

```python
params = torch.tensor([[ 
    1024/np.pi, 512/(np.pi/2),   # fx, fy (pixel/rad)
    512.0, 256.0,                # cx, cy
    1024.0, 512.0,               # image width, height
    np.pi, np.pi/2               # half hfov, half vfov (full 360° x 180°)
]])
cam = Spherical(params=params)

from unidepth.utils.coordinate import coords_grid
uv = coords_grid(1, 512, 1024, device="cpu")
rays = cam.unproject(uv)   # unit sphere directions (1, 3, H, W)
print("Spherical rays shape:", rays.shape)
print("hfov (rad):", cam.hfov)
```

--------------------------------

### Load UniDepth Models from Hugging Face

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Load UniDepth V1 or V2 models using their respective classes from Hugging Face. Ensure the 'name' variable matches a model available in the model zoo.

```python
from unidepth.models import UniDepthV1, UniDepthV2

model_v1 = UniDepthV1.from_pretrained(f"lpiccinelli/{name}")
model_v2 = UniDepthV2.from_pretrained(f"lpiccinelli/{name}")
```

--------------------------------

### Initialize Spherical Camera Model

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Initializes a Spherical camera model for equirectangular images. Supports full 360° views. Parameters include intrinsics, image dimensions, and field-of-view angles. Requires 'torch' and 'numpy'.

```python
import torch
from unidepth.utils.camera import Spherical
import numpy as np

# Parameters: [fx, fy, cx, cy, width, height, hfov/2, vfov/2]
# Example initialization (parameters need to be defined based on actual camera)
# params = torch.tensor([[...]]) 
# cam = Spherical(params=params)

```

--------------------------------

### Load UniDepthV1 Model from Hugging Face

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads a UniDepthV1 model from the Hugging Face Hub. V1 models do not support `resolution_level`, confidence output, or new camera classes.

```python
import torch
from unidepth.models import UniDepthV1

# ViT-L backbone
model_v1 = UniDepthV1.from_pretrained("lpiccinelli/unidepth-v1-vitl14")

# ConvNext-L backbone
# model_v1 = UniDepthV1.from_pretrained("lpiccinelli/unidepth-v1-cnvnxtl")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_v1 = model_v1.to(device).eval()
print("UniDepthV1 loaded")
# UniDepthV1 loaded
```

--------------------------------

### torch.hub.load

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Loads any UniDepth version (V1 or V2) and backbone through the PyTorch Hub interface. This method allows loading models without directly importing the `unidepth` package.

```APIDOC
## torch.hub.load

### Description
Loads any UniDepth version and backbone through the PyTorch Hub interface without directly importing the package.

### Method Signature
`torch.hub.load(repo_or_dir: str, model: str, *args, **kwargs)`

### Parameters
- **repo_or_dir** (str) - Required - The repository path, e.g., "lpiccinelli-eth/UniDepth".
- **model** (str) - Required - The model name, e.g., "UniDepth".
- **version** (str) - Optional - "v1" or "v2".
- **backbone** (str) - Optional - Backbone architecture (e.g., "vitl14", "vitb14", "vits14" for v2; "vitl14", "cnvnxtl" for v1).
- **pretrained** (bool) - Optional - Whether to load pretrained weights. Defaults to True.
- **trust_repo** (bool) - Optional - Whether to trust the repository. Defaults to True.
- **force_reload** (bool) - Optional - Whether to force reload the model. Defaults to True.

### Request Example
```python
import torch

model = torch.hub.load(
    "lpiccinelli-eth/UniDepth",
    "UniDepth",
    version="v2",         # "v1" or "v2"
    backbone="vitl14",    # "vitl14", "vitb14", "vits14" for v2; "vitl14", "cnvnxtl" for v1
    pretrained=True,
    trust_repo=True,
    force_reload=True,
)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()
```

### Response
- **model** (object) - The loaded UniDepth model.
```

--------------------------------

### Load UniDepth Model from TorchHub

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Load UniDepth models, specifically V2, using torch.hub.load. This method requires specifying the version, backbone, and other parameters for loading.

```python
version = "v2"
backbone = "vitl14"

model = torch.hub.load("lpiccinelli-eth/UniDepth", "UniDepth", version=version, backbone=backbone, pretrained=True, trust_repo=True, force_reload=True)
```

--------------------------------

### model.forward (Training Mode)

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Details on the `model.forward` method used during custom training. It expects pre-processed data and returns model outputs along with training losses.

```APIDOC
## `model.forward` — Training-mode forward pass

### Description
Used during custom training. Expects a pre-processed data dictionary with ImageNet-normalized images in `B×C×H×W` format. Dispatches to `forward_train` (returns outputs + losses) or `forward_test` (returns matched test outputs).

### Usage
```python
import torch
from unidepth.models import UniDepthV2
import json

# Load configuration
with open("configs/config_v2_vitl14.json") as f:
    config = json.load(f)

# Initialize and load model
model = UniDepthV2(config)
model.load_pretrained("pytorch_model.bin")
model.train() # Set model to training mode

# Prepare input data (must be pre-processed)
# - ImageNet normalized, shape (B, C, H, W)
# - "K" is the intrinsics tensor (B, 3, 3); optional if unknown
B, C, H, W = 2, 3, 480, 640
data = {
    "image": torch.randn(B, C, H, W),   # ImageNet-normalized
    "depth": torch.rand(B, 1, H, W),    # GT depth map
    "depth_mask": torch.ones(B, 1, H, W, dtype=torch.bool),
}
image_metas = [{"si": False, "flip": False}] * B

# Perform forward pass for training
outputs, losses = model(data, image_metas)

# Print training losses
print("Losses:", {k: v.item() for k, v in losses["opt"].items()})
# Example output: Losses: {'SILog': 0.42, 'Regression': 0.07, 'SelfDistill': 0.03, ...}
```

### Inputs
- **data**: A dictionary containing pre-processed training data, including:
    - `image`: Image tensor (Batch, Channels, Height, Width), normalized.
    - `depth`: Ground truth depth map.
    - `depth_mask`: Boolean mask for valid depth pixels.
- **image_metas**: A list of dictionaries containing metadata for each image in the batch.
```

--------------------------------

### Perform Inference with UniDepth

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Generate metric depth estimation and intrinsic prediction from an RGB image using the loaded UniDepth model. Ensure the image is in BxCxHxW format and moved to the appropriate device (CUDA or CPU).

```python
import numpy as np
from PIL import Image

# Move to CUDA, if any
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)

# Load the RGB image and the normalization will be taken care of by the model
rgb = torch.from_numpy(np.array(Image.open(image_path))).permute(2, 0, 1) # C, H, W

predictions = model.infer(rgb)

# Metric Depth Estimation
depth = predictions["depth"]

# Point Cloud in Camera Coordinate
xyz = predictions["points"]

# Intrinsics Prediction
intrinsics = predictions["intrinsics"]
```

--------------------------------

### Run Inference with UniDepthV2

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Performs depth estimation on a single RGB image using the `infer` method. Automatically handles image preprocessing and returns a dictionary of predictions. Requires an RGB tensor and optionally accepts a camera model.

```python
import numpy as np
import torch
from PIL import Image
from unidepth.models import UniDepthV2
from unidepth.utils.camera import Pinhole

model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()
model.resolution_level = 9
model.interpolation_mode = "bilinear"

# Load RGB image as C×H×W tensor (uint8, no normalization needed)
rgb = torch.from_numpy(np.array(Image.open("assets/demo/rgb.png"))).permute(2, 0, 1)

# --- Option A: predict intrinsics automatically (no camera given) ---
predictions = model.infer(rgb)


```

--------------------------------

### Export Model to ONNX

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Exports the UniDepthV2 model to ONNX format. Specify the output path, input shape, and whether to include camera intrinsics.

```python
export(model, path="unidepthv2.onnx", shape=(462, 630), with_camera=False)
```

--------------------------------

### Model Inference with Known Intrinsics

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Demonstrates how to perform depth prediction using a pre-trained model when camera intrinsics (K matrix) are known. The output includes depth, point cloud, refined intrinsics, confidence, rays, and radius.

```APIDOC
## Model Inference with Known Intrinsics

### Description
This example shows how to use the `model.infer` method with a provided camera intrinsics matrix (K). It takes an RGB image and a `Pinhole` camera object as input and returns a dictionary containing depth, 3D points, predicted intrinsics, confidence scores, rays, and radius.

### Usage
```python
import torch
import numpy as np
from unidepth.models import UniDepthV2
from unidepth.utils.camera import Pinhole

# Load pre-trained model (assuming model is already loaded and in eval mode)
# model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14").eval()

# Assume 'rgb' is a pre-processed tensor (e.g., from PIL Image, shape [1, 3, H, W])
# rgb = ... 

# Load known intrinsics (3x3 K matrix)
K = torch.from_numpy(np.load("assets/demo/intrinsics.npy"))  # shape (3, 3)

# Create a Pinhole camera object
camera = Pinhole(K=K.unsqueeze(0)) # Add batch dimension

# Perform inference
predictions = model.infer(rgb, camera)

# Access predictions
depth      = predictions["depth"]       # (1, 1, H, W) metric depth in meters
points     = predictions["points"]      # (1, 3, H, W) 3D point cloud in camera coords
intrinsics = predictions["intrinsics"]  # (1, 3, 3)   predicted/refined K matrix
confidence = predictions["confidence"]  # (1, 1, H, W) per-pixel confidence
rays       = predictions["rays"]        # (1, 3, H, W) unit direction vectors
radius     = predictions["radius"]      # (1, 1, H, W) Euclidean distance from camera

print("Depth shape:", depth.shape)
print("Mean depth:", depth.mean().item(), "m")
print("Predicted fx:", intrinsics[0, 0, 0].item())
```

### Outputs
- **depth**: Metric depth map in meters.
- **points**: 3D point cloud in camera coordinates.
- **intrinsics**: Predicted or refined camera intrinsic matrix.
- **confidence**: Per-pixel confidence score.
- **rays**: Unit direction vectors for each pixel.
- **radius**: Euclidean distance from the camera center.
```

--------------------------------

### Tile Images into a Grid

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Assembles a list of numpy images of the same height and width into a grid layout. Useful for visualizing multiple predictions or comparisons side-by-side.

```python
import numpy as np
from PIL import Image
from unidepth.utils.visualization import colorize, image_grid

rgb   = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8)
depth = np.random.rand(480, 640) * 10.0

depth_col = colorize(depth, vmin=0.01, vmax=10.0, cmap="magma_r")
error     = np.abs(depth - depth * 0.9)
error_col = colorize(error, vmin=0.0,  vmax=1.0,  cmap="coolwarm")

# 2×2 grid: [rgb, depth_gt, depth_pred, error]
grid = image_grid([rgb, depth_col, depth_col, error_col], rows=2, cols=2)
Image.fromarray(grid).save("output_grid.png")
print("Grid shape:", grid.shape)  # (960, 1280, 3)
```

--------------------------------

### Pinhole Camera Model

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Documentation for the `Pinhole` camera class, which represents a standard perspective camera. It supports projection, unprojection, depth reconstruction, and image transformations.

```APIDOC
## `Pinhole` — Standard pinhole camera model

### Description
Represents a standard perspective camera. Supports `project` (3D→2D), `unproject` (2D→3D rays), `reconstruct` (depth→point cloud), `resize`, `crop`, and `flip`. Can be constructed from either a `params` tensor `[fx, fy, cx, cy]` or a 3×3 `K` matrix.

### Initialization
Can be initialized using a 3x3 intrinsic matrix `K` or a parameter tensor `[fx, fy, cx, cy]`.

### Methods
- **`unproject(uv)`**: Converts 2D pixel coordinates to 3D rays.
- **`project(pcd)`**: Projects 3D points to 2D pixel coordinates.
- **`reconstruct(depth)`**: Reconstructs a 3D point cloud from a depth map.

### Usage
```python
import torch
from unidepth.utils.camera import Pinhole
from unidepth.utils.coordinate import coords_grid

# Initialize from K matrix
K = torch.tensor([[800.0, 0.0, 320.0],
                  [0.0,   800.0, 240.0],
                  [0.0,   0.0,   1.0]]).unsqueeze(0)  # (1, 3, 3)
cam = Pinhole(K=K)

# Initialize from params [fx, fy, cx, cy]
cam2 = Pinhole(params=torch.tensor([[800.0, 800.0, 320.0, 240.0]]))

# Unproject 2D pixel grid to 3D rays
uv = coords_grid(1, 480, 640, device="cpu")  # (1, 2, H, W)
rays = cam.unproject(uv)                     # (1, 3, H, W)

# Project 3D points back to 2D
pcd = torch.randn(1, 3, 480, 640)
pixels = cam.project(pcd)                    # (1, 2, H, W)

# Reconstruct 3D from depth
depth = torch.rand(1, 1, 480, 640) * 5.0    # meters
xyz = cam.reconstruct(depth)                 # (1, 3, H, W)
print("Point cloud shape:", xyz.shape)       # torch.Size([1, 3, 480, 640])
print("Camera device:", cam.device)
```
```

--------------------------------

### Evaluate 2D Depth Metrics

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Computes standard 2D depth evaluation metrics including threshold accuracy, RMSE, AbsRel, and SILog. Automatically rescales predictions to match ground truth resolution. Requires ground truth depth, predicted depth, and a valid pixel mask.

```python
import torch
from unidepth.utils import eval_depth

B, H, W = 4, 480, 640
gt_depth   = torch.rand(B, 1, H, W) * 10.0 + 0.5   # GT metric depth (meters)
pred_depth = gt_depth * (1 + 0.05 * torch.randn_like(gt_depth))  # ~5% noise
mask       = (gt_depth > 0.1).squeeze(1)             # valid pixel mask

metrics = eval_depth(gt_depth, pred_depth, mask, max_depth=80.0)

for name, vals in metrics.items():
    print(f"{name}: {vals.mean().item():.4f}")
# d1: 0.9781
# arel: 0.0312
# rmse: 0.2847
# silog: 4.2310
# ...
```

--------------------------------

### Infer Depth with Pinhole Camera Intrinsics

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Infers depth from an RGB image using a Pinhole camera model with known intrinsics. Requires the 'torch' and 'numpy' libraries. The model outputs depth, 3D points, refined intrinsics, confidence, rays, and radius.

```python
import torch
import numpy as np
from unidepth.utils.camera import Pinhole

# Assuming 'model' and 'rgb' are already defined
# K = torch.from_numpy(np.load("assets/demo/intrinsics.npy"))  # shape (3, 3)
# camera = Pinhole(K=K.unsqueeze(0))
# predictions = model.infer(rgb, camera)

# depth      = predictions["depth"]       # (1, 1, H, W) metric depth in meters
# points     = predictions["points"]      # (1, 3, H, W) 3D point cloud in camera coords
# intrinsics = predictions["intrinsics"]  # (1, 3, 3)   predicted/refined K matrix
# confidence = predictions["confidence"]  # (1, 1, H, W) per-pixel confidence
# rays       = predictions["rays"]        # (1, 3, H, W) unit direction vectors
# radius     = predictions["radius"]      # (1, 1, H, W) Euclidean distance from camera

# print("Depth shape:", depth.shape)            # torch.Size([1, 1, H, W])
# print("Mean depth:", depth.mean().item(), "m")
# print("Predicted fx:", intrinsics[0, 0, 0].item())
```

--------------------------------

### ONNX Export

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Exports UniDepthV2 models to ONNX format (opset 14) for deployment in various inference runtimes. Supports both standard export and a variant that accepts camera rays as input, with dynamic batch size.

```APIDOC
## ONNX Export

### Description
Exports a UniDepthV2 model to ONNX (opset 14) for deployment in inference runtimes. Two variants: standard (intrinsic-free) and `with_camera` (accepts pre-computed rays as second input). Dynamic batch axis is supported; spatial dimensions are fixed.

### Command Line Usage
```bash
# Standard export (model predicts intrinsics)
python unidepth/models/unidepthv2/export.py \
    --version v2 \
    --backbone vitl \
    --shape 462 630 \
    --output-path unidepthv2_vitl.onnx

# Export with camera rays input
python unidepth/models/unidepthv2/export.py \
    --version v2 \
    --backbone vitb \
    --shape 462 630 \
    --output-path unidepthv2_vitb_cam.onnx \
    --with-camera
```

### Programmatic Usage
```python
import torch, json
from unidepth.models.unidepthv2.export import UniDepthV2ONNX, export

with open("configs/config_v2_vitb14.json") as f:
    config = json.load(f)
config["training"]["export"] = True   # disables efficient attention (not ONNX-exportable)

model = UniDepthV2ONNX(config)
model.load_pretrained("pytorch_model.bin")

# Further steps would involve calling an export function with the model and desired output path.
```
```

--------------------------------

### Evaluate 3D Point Cloud Metrics

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Computes 3D point cloud evaluation metrics such as Chamfer distance, F1 score, and MSE. It subsamples spatially to manage memory usage for large point clouds. Requires ground truth points, predicted points, and a mask.

```python
import torch
from unidepth.utils import eval_3d

B, H, W = 2, 240, 320
gt_pts   = torch.rand(B, 3, H, W)  # GT 3D points (B, 3, H, W)
pred_pts = gt_pts + 0.01 * torch.randn_like(gt_pts)
mask     = torch.ones(B, 1, H, W, dtype=torch.bool)

metrics_3d = eval_3d(gt_pts, pred_pts, mask, thresholds=[0.05, 0.1, 0.2])
for name, vals in metrics_3d.items():
    print(f"{name}: {vals.mean().item():.5f}")
# chamfer: 0.00712
# F1: 0.89341
# MSE_3d: 0.01034
```

--------------------------------

### Provide Ground Truth Intrinsics

Source: https://github.com/lpiccinelli-eth/unidepth/blob/main/README.md

Optionally, provide ground truth intrinsics to the model. The input can be a 3x3 tensor, which will be converted to a Pinhole camera model, or a specific camera class instance.

```python
intrinsics_path = "assets/demo/intrinsics.npy"

# Load the intrinsics if available
intrinsics = torch.from_numpy(np.load(intrinsics_path)) # 3 x 3

# For V2, we defined camera classes. If you pass a 3x3 tensor (as above)
# it will convert to Pinhole, but you can pass classes from camera.py.
# The `Camera` class is meant as an abstract, use only child classes as e.g.:
from unidepth.utils.camera import Pinhole, Fisheye624

camera = Pinhole(K=intrinsics) # pinhole 
# fill in fisheye, params: fx,fy,cx,cy,d1,d2,d3,d4,d5,d6,t1,t2,s1,s2,s3,s4
camera = Fisheye624(params=torch.tensor([...]))
predictions = model.infer(rgb, camera)
```

--------------------------------

### image_grid

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Assembles a list of numpy images of the same height and width into a grid. This function is useful for organizing multiple images, such as predictions and ground truth, into a single image for comparison or saving.

```APIDOC
## image_grid

### Description
Assembles a list of numpy images (all same H×W) into a rows×cols grid. Useful for saving prediction comparisons.

### Parameters
- **images** (list of np.ndarray) - A list of numpy images to arrange in the grid.
- **rows** (int) - The number of rows in the grid.
- **cols** (int) - The number of columns in the grid.

### Request Example
```python
import numpy as np
from PIL import Image
from unidepth.utils.visualization import colorize, image_grid

rgb   = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8)
depth = np.random.rand(480, 640) * 10.0

depth_col = colorize(depth, vmin=0.01, vmax=10.0, cmap="magma_r")
error     = np.abs(depth - depth * 0.9)
error_col = colorize(error, vmin=0.0,  vmax=1.0,  cmap="coolwarm")

# 2×2 grid: [rgb, depth_gt, depth_pred, error]
grid = image_grid([rgb, depth_col, depth_col, error_col], rows=2, cols=2)
Image.fromarray(grid).save("output_grid.png")
print("Grid shape:", grid.shape)  # (960, 1280, 3)
```
```

--------------------------------

### model.infer

Source: https://context7.com/lpiccinelli-eth/unidepth/llms.txt

Performs depth estimation inference on a single RGB image. Accepts a tensor and an optional camera model, returning predictions including depth, 3D points, and intrinsics. Handles image preprocessing automatically.

```APIDOC
## model.infer

### Description
The primary inference method. Accepts a `C×H×W` uint8 RGB tensor and an optional camera model. Returns a dictionary with `depth`, `points`, `intrinsics`, `rays`, `radius`, `confidence`, and `depth_features`. Automatically handles ImageNet normalization, aspect-ratio-preserving padding, and resizing.

### Method Signature
`model.infer(rgb_tensor: torch.Tensor, camera_model: Optional[object] = None)`

### Parameters
- **rgb_tensor** (torch.Tensor) - Required - Input RGB image as a `C×H×W` uint8 tensor.
- **camera_model** (Optional[object]) - Optional - Camera model object (e.g., `Pinhole` from `unidepth.utils.camera`). If not provided, intrinsics are predicted automatically.

### Request Example
```python
import numpy as np
import torch
from PIL import Image
from unidepth.models import UniDepthV2
from unidepth.utils.camera import Pinhole

# Assuming 'model' is already loaded and configured
# model = UniDepthV2.from_pretrained("lpiccinelli/unidepth-v2-vitl14")
# device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# model = model.to(device).eval()
# model.resolution_level = 9
# model.interpolation_mode = "bilinear"

# Load RGB image as C×H×W tensor (uint8, no normalization needed)
rgb = torch.from_numpy(np.array(Image.open("assets/demo/rgb.png"))).permute(2, 0, 1)

# --- Option A: predict intrinsics automatically (no camera given) ---
predictions = model.infer(rgb)

# --- Option B: provide camera intrinsics ---
# camera = Pinhole(fx=800., fy=800., cx=320., cy=240.) # Example intrinsics
# predictions = model.infer(rgb, camera_model=camera)

print(predictions.keys())
# dict_keys(['depth', 'points', 'intrinsics', 'rays', 'radius', 'confidence', 'depth_features'])
```

### Response
- **predictions** (dict) - A dictionary containing:
  - **depth** (torch.Tensor) - Predicted depth map.
  - **points** (torch.Tensor) - 3D point cloud.
  - **intrinsics** (torch.Tensor) - Predicted camera intrinsics.
  - **rays** (torch.Tensor) - Ray directions.
  - **radius** (torch.Tensor) - Radius of pixels.
  - **confidence** (torch.Tensor) - Confidence map (V2 only).
  - **depth_features** (torch.Tensor) - Intermediate depth features.
```