InfiniDepth

https://github.com/zju3dv/infinidepth
Admin
InfiniDepth is a neural implicit field framework for arbitrary-resolution monocular depth...
Tokens:97,046
Snippets:557
Trust Score:7.9
Update:1 week ago
Show doc for...
Tokens:
Raw
### Run InfiniDepth Gradio demo locally

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Starts the Gradio Space entrypoint for the InfiniDepth demo. Requires the repository to be installed and dependencies available.

```bash
python app.py
```

--------------------------------

### Install dependencies with pip

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Installs the required Python packages from requirements.txt using pip.

```shell
pip install -r requirements.txt
```

--------------------------------

### Run Waymo multi-view inference example scripts

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Run the three multi-view inference example scripts for Waymo sequences: standard InfiniDepth, InfiniDepth_DepthSensor with depth input, and explicit camera mode using intrinsics/extrinsics directories.

```bash
bash example_scripts/infer_depth/waymo_multi_view_infinidepth.sh
bash example_scripts/infer_depth/waymo_multi_view_infinidepth_depthsensor.sh
bash example_scripts/infer_depth/waymo_multi_view_infinidepth_explicit_camera.sh
```

--------------------------------

### Example Scripts for Depth and Gaussian Inference

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Example scripts for running depth and Gaussian inference on ETH3D and Waymo datasets with the InfiniDepth_DepthSensor model.

```bash
bash example_scripts/infer_depth/eth3d_infinidepth_depthsensor.sh
bash example_scripts/infer_depth/waymo_infinidepth_depthsensor.sh
bash example_scripts/infer_gs/eth3d_infinidepth_depthsensor_gs.sh
bash example_scripts/infer_gs/waymo_infinidepth_depthsensor_gs.sh
```

--------------------------------

### Fast setup: training DINOv3 ViT-L/16 on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Launch DINOv3 ViT-L/16 pretraining on 4 H100-80GB nodes (32 GPUs) in a SLURM cluster with submitit. Training takes approximately 14 hours and should reach 82.0% k-NN eval and 83.5% linear eval. The teacher weights are saved every 12500 iterations.

```shell
 PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 4 \
  --config-file dinov3/configs/train/vitl_im1k_lin834.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=ImageNet22k:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Run example 3D Gaussian inference scripts

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Example scripts for running 3D Gaussian inference on different input images. Each script wraps the inference_gs.py command with appropriate arguments.

```bash
bash example_scripts/infer_gs/courtyard_infinidepth_gs.sh
bash example_scripts/infer_gs/camera_infinidepth_gs.sh
bash example_scripts/infer_gs/fruit_infinidepth_gs.sh
bash example_scripts/infer_gs/eth3d_infinidepth_gs.sh
```

--------------------------------

### Install dependencies with uv

Source: https://github.com/zju3dv/infinidepth/blob/main/INSTALL.md

Installs uv, PyTorch with CUDA 12.8, package dependencies, MoGe for scale restoration, and gsplat for Gaussian Splatting.

```bash
# Create environment
pip install uv

# Install PyTorch with CUDA 12.8
uv pip install torch==2.9.0 torchvision==0.24.0 torchaudio==2.9.0 xformers==0.0.33.post1 --index-url https://download.pytorch.org/whl/cu128

# Install package dependencies
uv pip install -r requirements.txt

# Install MoGe (For scale restoration)
uv pip install git+https://github.com/microsoft/MoGe.git

# Install gsplat (For Gaussian Splatting)
uv pip install git+https://github.com/nerfstudio-project/gsplat.git
```

--------------------------------

### Run example depth inference scripts

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Example scripts for running relative depth inference on different input images. Each script wraps the inference_depth.py command with appropriate arguments.

```bash
bash example_scripts/infer_depth/courtyard_infinidepth.sh
bash example_scripts/infer_depth/camera_infinidepth.sh
bash example_scripts/infer_depth/eth3d_infinidepth.sh
bash example_scripts/infer_depth/waymo_infinidepth.sh
```

--------------------------------

### Full DINOv3 segmentor inference example

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Complete example that loads a DINOv3 segmentor, processes an image, and generates a segmentation map using sliding window inference. Requires a CUDA GPU and the 'dinov3' package. The example downloads an image from COCO and uses bfloat16 autocasting.

```python
import sys
sys.path.append(REPO_DIR)

from PIL import Image
import torch
from torchvision import transforms
import matplotlib.pyplot as plt
from matplotlib import colormaps
from functools import partial
from dinov3.eval.segmentation.inference import make_inference


def get_img():
    import requests
    url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
    return image

def make_transform(resize_size: int | list[int] = 768):
    to_tensor = transforms.ToTensor()
    resize = transforms.Resize((resize_size, resize_size), antialias=True)
    normalize = transforms.Normalize(
        mean=(0.485, 0.456, 0.406),
        std=(0.229, 0.224, 0.225),
    )
    return transforms.Compose([to_tensor, resize, normalize])

segmentor = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_ms', source="local", weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)

img_size = 896
img  = get_img()
transform = make_transform(img_size)
with torch.inference_mode():
    with torch.autocast('cuda', dtype=torch.bfloat16):
        batch_img = transform(img)[None]
        pred_vit7b = segmentor(batch_img)  # raw predictions  
        # actual segmentation map
        segmentation_map_vit7b = make_inference(
            batch_img,
            segmentor,
            inference_mode="slide",
            decoder_head_type="m2f",
            rescale_to=(img.size[-1], img.size[-2]),
            n_output_channels=150,
            crop_size=(img_size, img_size),
            stride=(img_size, img_size),
            output_activation=partial(torch.nn.functional.softmax, dim=1),
        ).argmax(dim=1, keepdim=True)
plt.figure(figsize=(12, 6))
plt.subplot(121)
plt.imshow(img)
plt.axis("off")
plt.subplot(122)
plt.imshow(segmentation_map_vit7b[0,0].cpu(), cmap=colormaps["Spectral"])
plt.axis("off")
```

--------------------------------

### Import PersistenceManager and state utilities

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Import the PersistenceManager class and utility functions for getting distributed model and optimizer states.

```python
# Persistence Manager
from common.persistence import PersistenceManager

# Utility functions for getting distributed states.
from common.persistence import get_model_states, get_optimizer_states
```

--------------------------------

### Load DINOv3 model with torch.hub

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/MODEL_CARD.md

Load a DINOv3 model from the facebookresearch/dinov3 torch.hub repository. Replace <MODEL_NAME> with one of the listed model names (e.g., dinov3_vits16) and provide a checkpoint path or URL. The example shows a concrete call for dinov3_vits16.

```python
import torch

model = torch.hub.load(
    repo_or_dir='facebookresearch/dinov3',
    model='<MODEL_NAME>',
    weights='<PATH/OR/URL/TO/CHECKPOINT>',
)

# where MODEL_NAME can be one of:
# - dinov3_vits16
# - dinov3_vits16plus
# - dinov3_vitb16
# - dinov3_vitl16
# - dinov3_vith16plus
# - dinov3_vit7b16
# - dinov3_convnext_tiny
# - dinov3_convnext_small
# - dinov3_convnext_base
# - dinov3_convnext_large

# For instance
dinov3_vits16 = torch.hub.load(
    repo_or_dir='facebookresearch/dinov3',
    model='dinov3_vits16',
    weights='<PATH/OR/URL/TO/DINOV3/VITS16/LVD1689M/CHECKPOINT>',
)
```

--------------------------------

### Load DINOv3 backbones via PyTorch Hub

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Loads DINOv3 pretrained backbones from a local clone of the DINOv3 repository using torch.hub.load with source='local'. Requires PyTorch installed (CUDA support recommended) and a local directory path for REPO_DIR. The weights argument accepts a checkpoint URL or path.

```python
import torch

REPO_DIR = <PATH/TO/A/LOCAL/DIRECTORY/WHERE/THE/DINOV3/REPO/WAS/CLONED>

# DINOv3 ViT models pretrained on web images
dinov3_vits16 = torch.hub.load(REPO_DIR, 'dinov3_vits16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vits16plus = torch.hub.load(REPO_DIR, 'dinov3_vits16plus', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vitb16 = torch.hub.load(REPO_DIR, 'dinov3_vitb16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vitl16 = torch.hub.load(REPO_DIR, 'dinov3_vitl16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vith16plus = torch.hub.load(REPO_DIR, 'dinov3_vith16plus', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vit7b16 = torch.hub.load(REPO_DIR, 'dinov3_vit7b16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)

# DINOv3 ConvNeXt models pretrained on web images
dinov3_convnext_tiny = torch.hub.load(REPO_DIR, 'dinov3_convnext_tiny', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_convnext_small = torch.hub.load(REPO_DIR, 'dinov3_convnext_small', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_convnext_base = torch.hub.load(REPO_DIR, 'dinov3_convnext_base', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_convnext_large = torch.hub.load(REPO_DIR, 'dinov3_convnext_large', source='local', weights=<CHECKPOINT/URL/OR/PATH>)

# DINOv3 ViT models pretrained on satellite imagery
dinov3_vitl16 = torch.hub.load(REPO_DIR, 'dinov3_vitl16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
dinov3_vit7b16 = torch.hub.load(REPO_DIR, 'dinov3_vit7b16', source='local', weights=<CHECKPOINT/URL/OR/PATH>)
```

--------------------------------

### Create conda environment with mamba

Source: https://github.com/zju3dv/infinidepth/blob/main/INSTALL.md

Creates a new conda environment named 'infinidepth' with Python 3.10 and activates it. Includes optional commands to install gxx and CUDA toolkit if gsplat compilation fails.

```bash
mamba create -n infinidepth python=3.10
mamba activate infinidepth

# Optional: When gsplat compilation fails due to g++ version or CUDA toolkit issues.
# mamba install gxx=10
# mamba install nvidia/label/cuda-12.8.0::cuda-toolkit -c nvidia/label/cuda-12.8.0
# export CUDA_HOME=$CONDA_PREFIX
```

--------------------------------

### Load DINOv2 models via PyTorch Hub

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Loads the four DINOv2 backbone models (ViT-S/14, ViT-B/14, ViT-L/14, ViT-g/14) via PyTorch Hub. Requires PyTorch installed; CUDA support is strongly recommended. The models are pretrained on 142M images without labels.

```python
import torch

dinov2_vits14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vits14')
dinov2_vitb14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
dinov2_vitl14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
dinov2_vitg14 = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitg14')
```

--------------------------------

### Train text alignment with dinov3/eval/text/train_dinotxt.py

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Trains text alignment on 4 nodes with 8 GPUs each (32 GPUs total) using the DINOv3 text alignment script. The example config uses the CocoCaptions dataset for illustration; adapt the dataset class as needed.

```shell
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/eval/text/train_dinotxt.py \
   --nodes 4 \
  # An example config for text alignment is here: dinov3/eval/text/configs/dinov3_vitl_text.yaml \ 
  trainer_config_file="<PATH/TO/DINOv3/TEXT/CONFIG>" \
  output-dir=<PATH/TO/OUTPUT/DIR>
```

--------------------------------

### Get image embedding with DINOv3 AutoModel

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Loads a DINOv3 backbone with AutoModel and AutoImageProcessor, processes an image, and prints the pooled output shape. The pretrained_model_name can be any of the listed DINOv3 model identifiers.

```python
import torch
from transformers import AutoImageProcessor, AutoModel
from transformers.image_utils import load_image

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = load_image(url)

pretrained_model_name = "facebook/dinov3-convnext-tiny-pretrain-lvd1689m"
processor = AutoImageProcessor.from_pretrained(pretrained_model_name)
model = AutoModel.from_pretrained(
    pretrained_model_name, 
    device_map="auto", 
)

inputs = processor(images=image, return_tensors="pt").to(model.device)
with torch.inference_mode():
    outputs = model(**inputs)

pooled_output = outputs.pooler_output
print("Pooled output shape:", pooled_output.shape)
```

--------------------------------

### Launch fine-tuning via provided shell scripts

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Equivalent launch scripts for fine-tuning the RGB-only and RGB + depth sensor InfiniDepth models. These scripts wrap the corresponding main.py commands.

```bash
bash launch_scripts/train/infinidepth.sh
bash launch_scripts/train/infinidepth_depthsensor.sh
```

--------------------------------

### Train InfiniDepth_DepthSensor from scratch

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Trains the RGB + depth sensor InfiniDepth model from scratch without loading a .ckpt checkpoint. The DINOv3 backbone is loaded from ${commonspace}/pretrained_models/dinov3/, which must be downloaded and placed there beforehand. Use a new exp_name for a clean run; setting resume_training=False deletes the old output directory.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  exp_name=train_infinidepth_depthsensor_from_scratch \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2
```

--------------------------------

### Train InfiniDepth RGB-only from scratch

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Trains the RGB-only InfiniDepth model from scratch without loading a .ckpt checkpoint. The DINOv3 backbone is loaded from ${commonspace}/pretrained_models/dinov3/, which must be downloaded and placed there beforehand. Use a new exp_name for a clean run; setting resume_training=False deletes the old output directory.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  exp_name=train_infinidepth_from_scratch \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2
```

--------------------------------

### Run reference evaluation launch scripts

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Reference launch scripts for evaluating the RGB-only and RGB + depth sensor InfiniteDepth models. Run these scripts to execute the corresponding validation pipelines.

```bash
bash launch_scripts/eval/infinidepth.sh
bash launch_scripts/eval/infinidepth_depthsensor.sh
```

--------------------------------

### Fine-tune InfiniDepth RGB-only from checkpoint

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Fine-tunes the RGB-only InfiniDepth model from an existing checkpoint using the Hypersim training set. Requires workspace and commonspace environment variables, and ckpt_path must point to an existing checkpoint. Training resumes from the latest checkpoint in the exp_name output directory if present.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  ckpt_path=checkpoints/depth/infinidepth.ckpt \
  exp_name=finetune_infinidepth_on_hypersim \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=8
```

--------------------------------

### Create PersistenceManager and save config

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Create a PersistenceManager with the save path and save the training config. Must be called by all ranks before training.

```python
persistence = PersistenceManager("hdfs://path/to/save")
persistence.save_config(trainer_config)
```

--------------------------------

### Train DINOv2 ViT-L/16 on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Launches DINOv2 training on 4 A100-80GB nodes (32 GPUs) using submitit. Training takes about 1 day and should reach 81.6% k-NN and 82.9% linear eval. Teacher weights are saved every 12500 iterations in the eval folder.

```shell
python dinov2/run/train/train.py \
    --nodes 4 \
    --config-file dinov2/configs/train/vitl16_short.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Fine-tune InfiniDepth_DepthSensor from checkpoint

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Fine-tunes the RGB + depth sensor InfiniDepth model from an existing checkpoint using the Hypersim training set. Requires workspace and commonspace environment variables, and ckpt_path must point to an existing checkpoint. Training resumes from the latest checkpoint in the exp_name output directory if present.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/train/infinidepth_train_hypersim.yaml \
  ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  exp_name=finetune_infinidepth_depthsensor_on_hypersim \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=8
```

--------------------------------

### Gram anchoring for DINOv3 ViT-7B/16

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Launch the gram anchoring stage for DINOv3 ViT-7B/16 on 32 nodes (256 GPUs) with submitit. Requires the gram teacher checkpoint from the previous pretraining step.

```shell
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_gram_anchor.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
  gram.ckpt=<PATH/TO/GRAM_TEACHER_FROM_PREVIOUS_STEP>   
```

--------------------------------

### Train DINOv2 ViT-L/14 on ImageNet-22k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Launches DINOv2 training on 12 A100-80GB nodes (96 GPUs) using submitit. Training takes about 3.3 days and should reach 82.0% k-NN and 84.5% linear eval. Teacher weights are saved every 12500 iterations in the eval folder.

```shell
python dinov2/run/train/train.py \
    --nodes 12 \
    --config-file dinov2/configs/train/vitl14.yaml \
    --output-dir <PATH/TO/OUTPUT/DIR> \
    train.dataset_path=ImageNet22k:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### High-resolution adaptation for DINOv3 ViT-7B/16

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Launch the high-resolution adaptation stage for DINOv3 ViT-7B/16 on 32 nodes (256 GPUs) with submitit. Requires the teacher checkpoint from the gram anchoring step and resumes the student from that teacher checkpoint.

```shell
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_high_res_adapt.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
  gram.ckpt=<PATH/TO/TEACHER_FROM_GRAM> \
  student.resume_from_teacher_chkpt=<PATH/TO/TEACHER_FROM_GRAM>
```

--------------------------------

### Run multi-distillation training

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Launches multi-distillation training on a single node using the DINOv3 training script. Requires the multi_distillation_test.yaml config and dataset paths.

```shell
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 1 \
  --config-file dinov3/configs/train/multi_distillation_test.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  --multi-distillation \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Pretraining DINOv3 ViT-7B/16

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Launch DINOv3 ViT-7B/16 pretraining on 32 nodes (256 GPUs) in a SLURM cluster with submitit. This is the first stage of the three-stage training process.

```shell
PYTHONPATH=${PWD} python -m dinov3.run.submit dinov3/train/train.py \
  --nodes 32 \
  --config-file dinov3/configs/train/dinov3_vit7b16_pretrain.yaml \
  --output-dir <PATH/TO/OUTPUT/DIR> \
  train.dataset_path=<DATASET>:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Create and activate conda environment

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Creates a conda environment named 'dinov2' from the provided conda.yaml and activates it. Recommended for setting up dependencies.

```shell
conda env create -f conda.yaml
conda activate dinov2
```

--------------------------------

### Run 3D Gaussian inference with inference_gs.py

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Run 3D Gaussian inference from a single RGB image using inference_gs.py. Requires the depth and Gaussian checkpoints, and optionally the MoGe normal model for metric scale recovery and the sky segmentation ONNX model for sky filtering. Outputs a .ply file and an optional novel-view video; if --render_size is omitted, the video is rendered at the original input resolution.

```bash
python inference_gs.py \
  --input_image_path=example_data/image/courtyard.jpg \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
  --gs_model_path=checkpoints/gs/infinidepth_gs.ckpt
```

--------------------------------

### Load model utilities with load_model_from_path and load_model_from_task

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Load a persisted model from a path or task ID, optionally specifying name and step. Access config and states, or create the model instance.

```python
from common.persistence import load_model_from_path, load_model_from_task

persisted_model = load_model_from_path("hdfs://path/to/save")
persisted_model = load_model_from_path("hdfs://path/to/save", name="model")
persisted_model = load_model_from_path("hdfs://path/to/save", name="model", step=1000)

persisted_model = load_model_from_task("d5ad7ed1c6acd81f")
persisted_model = load_model_from_task("d5ad7ed1c6acd81f", name="model")
persisted_model = load_model_from_task("d5ad7ed1c6acd81f", name="model", step=1000)

# You can access all the information.
persisted_model.config.path          # str
persisted_model.config.load()        # DictConfig
persisted_model.states.path          # str
persisted_model.states.load("cpu")   # state_dict

# Or instantiate the model.
model = persisted_model.create("cpu") # nn.Module
```

--------------------------------

### Load DINOv3 dino.txt model via torch.hub

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Loads the DINOv3 ViT-L/16 distilled dino.txt model and its tokenizer. Requires specifying the model checkpoint and backbone weights as URLs or paths.

```python
import torch
# DINOv3
dinov3_vitl16_dinotxt_tet1280d20h24l, tokenizer = torch.hub.load(REPO_DIR, 'dinov3_vitl16_dinotxt_tet1280d20h24l', weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
```

--------------------------------

### Create and activate dinov3 environment with micromamba

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Creates and activates a conda environment named 'dinov3' using the provided conda.yaml file. Requires micromamba and a Linux environment.

```shell
micromamba env create -f conda.yaml
micromamba activate dinov3
```

--------------------------------

### Run relative depth inference with inference_depth.py

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Run relative depth inference from a single RGB image using the InfiniDepth model. Requires the depth checkpoint and optionally the MoGe normal model for metric scale recovery and the sky segmentation ONNX model for sky filtering. Outputs a colorized depth map to example_data/pred_depth/ and a point cloud to example_data/pred_pcd/ when --save_pcd=True.

```bash
python inference_depth.py \
  --input_image_path=example_data/image/courtyard.jpg \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
  --output_resolution_mode=upsample \
  --upsample_ratio=2
```

--------------------------------

### Explicit Camera-Parameter Multi-View Command

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs multi-view depth inference with explicit camera intrinsics and extrinsics, skipping DA3 processing. Expects Waymo-style text files in the intrinsics and extrinsics directories, matched lexicographically to the sorted image list. Supports image inputs only, not video.

```bash
python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --camera_intrinsics_dir=/path/to/intrinsics \
  --camera_extrinsics_dir=/path/to/extrinsics \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
```

--------------------------------

### Check resume step with load_last_step

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Check if a previous training step exists and access the step number, models, and optimizers from the resume object.

```python
resume = persistence.load_last_step()
if resume:
    print(resume.step)          # 500
    print(resume.models)        # {"model": PersistedModel}
    print(resume.optimizers)    # {"optimizer": PersistedOptimizer}
```

--------------------------------

### RGB + Depth Sensor Multi-View / Video Command

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs multi-view/video depth inference with RGB and depth sensor input, using the InfiniDepth_DepthSensor model. The depth input must have the same number of frames as the RGB input when using video.

```bash
python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --input_depth_path=example_data/multi-view/waymo/depth \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
```

--------------------------------

### RGB-Only Multi-View / Video Command

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs multi-view/video depth inference with RGB-only input, using the InfiniDepth model. Requires the checkpoint checkpoints/depth/infinidepth.ckpt.

```bash
python inference_multi_view_depth.py \
  --input_path=example_data/multi-view/waymo/image \
  --model_type=InfiniDepth \
  --depth_model_path=checkpoints/depth/infinidepth.ckpt \
```

--------------------------------

### PersistenceManager.load_last_step

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Loads the last saved step information for resuming training. Returns a resume object with step, models, and optimizers.

```APIDOC
## PersistenceManager.load_last_step

### Description
Loads the last saved step information for resuming training. Returns a resume object with step, models, and optimizers.

### Method
load_last_step

### Parameters
None

### Request Example
```python
resume = persistence.load_last_step()
```

### Response
- **step** (int) - The last saved step number.
- **models** (dict) - Dictionary of persisted models.
- **optimizers** (dict) - Dictionary of persisted optimizers.

### Response Example
```python
resume.step  # 500
resume.models  # {"model": PersistedModel}
resume.optimizers  # {"optimizer": PersistedOptimizer}
```
```

--------------------------------

### Generate ImageNet metadata files with ImageNet.dump_extra()

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Generates the metadata files (class-ids, class-names, entries) for all ImageNet splits. Requires the dinov3 package and the dataset root and extra directories. Run once before training or evaluation.

```python
from dinov3.data.datasets import ImageNet

for split in ImageNet.Split:
    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
    dataset.dump_extra()
```

--------------------------------

### Load DINOv3 segmentor head via torch.hub

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/dinov3/README.md

Loads the DINOv3 ViT-7B/16 segmentor head trained on ADE20K. Requires specifying the segmentor checkpoint and backbone weights as URLs or paths.

```python
segmentor = torch.hub.load(REPO_DIR, 'dinov3_vit7b16_ms', source="local", weights=<SEGMENTOR/CHECKPOINT/URL/OR/PATH>, backbone_weights=<BACKBONE/CHECKPOINT/URL/OR/PATH>)
```

--------------------------------

### 3D Gaussian Inference Command

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs 3D Gaussian inference using the InfiniDepth_DepthSensor model, requiring both depth and Gaussian checkpoints. Camera intrinsics are provided explicitly.

```bash
python inference_gs.py \
  --input_image_path=example_data/image/eth3d_office.png \
  --input_depth_path=example_data/depth/eth3d_office.npz \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  --gs_model_path=checkpoints/gs/infinidepth_depthsensor_gs.ckpt \
  --fx_org=866.39 \
  --fy_org=866.04 \
  --cx_org=791.5 \
  --cy_org=523.81
```

--------------------------------

### Evaluate pretrained ViT-g/14 weights on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Runs linear evaluation on the released pretrained ViT-g/14 model weights using the provided config and URL. This command evaluates the model's performance on ImageNet-1k.

```shell
python dinov2/run/eval/linear.py \
    --config-file dinov2/configs/eval/vitg14_pretrain.yaml \
    --pretrained-weights https://dl.fbaipublicfiles.com/dinov2/dinov2_vitg14/dinov2_vitg14_pretrain.pth \
    --train-dataset ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
    --val-dataset ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Metric Depth Inference Command

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs metric depth inference with InfiniDepth_DepthSensor, providing camera intrinsics and an input depth map. The output resolution is set to upsample with ratio 1.

```bash
python inference_depth.py \
  --input_image_path=example_data/image/eth3d_office.png \
  --input_depth_path=example_data/depth/eth3d_office.npz \
  --model_type=InfiniDepth_DepthSensor \
  --depth_model_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  --fx_org=866.39 \
  --fy_org=866.04 \
  --cx_org=791.5 \
  --cy_org=523.81 \
  --output_resolution_mode=upsample \
  --upsample_ratio=1
```

--------------------------------

### Evaluate with linear classification with data augmentation on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Runs linear evaluation with data augmentation on a single node using the teacher checkpoint saved during training. Requires the config.yaml and teacher_checkpoint.pth from the training output.

```shell
python dinov2/run/eval/linear.py \
    --config-file <PATH/TO/OUTPUT/DIR>/config.yaml \
    --pretrained-weights <PATH/TO/OUTPUT/DIR>/eval/training_24999/teacher_checkpoint.pth \
    --output-dir <PATH/TO/OUTPUT/DIR>/eval/training_24999/linear \
    --train-dataset ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
    --val-dataset ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Access PersistedModel APIs

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Access the config and states of a PersistedModel, load them, or create the model instance on a specified device.

```python
resume.models["model"].config.path          # str
resume.models["model"].config.load()        # DictConfig
resume.models["model"].states.path          # str
resume.models["model"].states.load("cpu")   # state_dict
resume.models["model"].create("cpu")        # nn.Module
```

--------------------------------

### Evaluate with k-NN classification on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Runs k-NN evaluation on a single node using the teacher checkpoint saved during training. Requires the config.yaml and teacher_checkpoint.pth from the training output.

```shell
python dinov2/run/eval/knn.py \
    --config-file <PATH/TO/OUTPUT/DIR>/config.yaml \
    --pretrained-weights <PATH/TO/OUTPUT/DIR>/eval/training_24999/teacher_checkpoint.pth \
    --output-dir <PATH/TO/OUTPUT/DIR>/eval/training_24999/knn \
    --train-dataset ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
    --val-dataset ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Export workspace and commonspace environment variables

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Sets the environment variables required by main.py before running any training or validation command. workspace stores experiment outputs under outputs/<task>/<exp_name>/, and commonspace stores datasets and pretrained weights shared across experiments.

```bash
export workspace=/path/to/your/experiments
export commonspace=/path/to/your/common_space
```

--------------------------------

### PersistedModel and PersistedOptimizer APIs

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

APIs for accessing persisted model and optimizer information, including config, states, and model instantiation.

```APIDOC
## PersistedModel and PersistedOptimizer APIs

### Description
APIs for accessing persisted model and optimizer information, including config, states, and model instantiation.

### Methods
- **config.path** (str) - Path to the config.
- **config.load()** (DictConfig) - Load the config.
- **states.path** (str) - Path to the states.
- **states.load("cpu")** (state_dict) - Load the state dict.
- **create("cpu")** (nn.Module) - Instantiate the model.

### Request Example
```python
resume.models["model"].config.path          # str
resume.models["model"].config.load()        # DictConfig
resume.models["model"].states.path          # str
resume.models["model"].states.load("cpu")   # state_dict
resume.models["model"].create("cpu")        # nn.Module
```

### Response
No return value.
```

--------------------------------

### Validate RGB-only InfiniteDepth model

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs validation for the RGB-only InfiniteDepth model using the specified config, checkpoint, and evaluation settings. Sets absolute-metric computation, saves original predictions and metrics, and uses 2 GPUs.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth.yaml \
  --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
  --entry val \
  ckpt_path=checkpoints/depth/infinidepth.ckpt \
  exp_name=eval_infinidepth \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2
```

--------------------------------

### PersistenceManager.save_config

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Saves the training configuration before training begins. Must be called by all ranks.

```APIDOC
## PersistenceManager.save_config

### Description
Saves the training configuration to the persistence storage. Must be called by all ranks before training.

### Method
save_config

### Parameters
- **config** (object) - Required - The training configuration to save.

### Request Example
```python
persistence = PersistenceManager("hdfs://path/to/save")
persistence.save_config(trainer_config)
```

### Response
No return value.
```

--------------------------------

### Validate RGB + depth sensor InfiniteDepth model

Source: https://github.com/zju3dv/infinidepth/blob/main/README.md

Runs validation for the RGB + depth sensor variant of InfiniteDepth, using the depthsensor config and checkpoint. Enables absolute-metric evaluation, saves original predictions and metrics, and uses 2 GPUs.

```bash
python3 main.py \
  --cfg_file training/exp_configs/exps/infinidepth_depthsensor.yaml \
  --include training/exp_configs/components/data/test/infinidepth_mix_data.yaml \
  --entry val \
  ckpt_path=checkpoints/depth/infinidepth_depthsensor.ckpt \
  exp_name=eval_infinidepth_depthsensor \
  model.compute_abs_metric=True \
  model.save_orig_pred=True \
  model.save_metrics=True \
  pl_trainer.devices=2
```

--------------------------------

### Evaluate with logistic regression classification on ImageNet-1k

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Runs logistic regression evaluation on a single node using the teacher checkpoint saved during training. Requires the config.yaml and teacher_checkpoint.pth from the training output.

```shell
python dinov2/run/eval/log_regression.py \
    --config-file <PATH/TO/OUTPUT/DIR>/config.yaml \
    --pretrained-weights <PATH/TO/OUTPUT/DIR>/eval/training_24999/teacher_checkpoint.pth \
    --output-dir <PATH/TO/OUTPUT/DIR>/eval/training_24999/logreg \
    --train-dataset ImageNet:split=TRAIN:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET> \
    --val-dataset ImageNet:split=VAL:root=<PATH/TO/DATASET>:extra=<PATH/TO/DATASET>
```

--------------------------------

### Generate ImageNet metadata files

Source: https://github.com/zju3dv/infinidepth/blob/main/InfiniDepth/model/block/torchhub/facebookresearch_dinov2_main/README.md

Generates the metadata files (class-ids, class-names, entries) for all ImageNet splits. Requires the dinov2 package in the Python path.

```python
from dinov2.data.datasets import ImageNet

for split in ImageNet.Split:
    dataset = ImageNet(split=split, root="<ROOT>", extra="<EXTRA>")
    dataset.dump_extra()
```

--------------------------------

### Save optimizer with save_optimizer

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Save an optimizer checkpoint. Must be called by all ranks. Multiple optimizers can be saved with different names. get_optimizer_states unwraps ZeroRedundancyOptimizer; get_fsdp_optimizer_states handles FSDP optimizers.

```python
persistence.save_optimizer(
    step=1000,                              # Current step.
    name="optimizer",                       # Optimizer name. (Default to "optimizer")
    states=get_optimizer_states(optimizer), # Optimizer states.
    dtype=None,                             # dtype conversion. Default None to disable.
    blocking=False,                         # Async copy. Default no blocking.
)
```

--------------------------------

### load_model_from_path

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Loads a persisted model from a given path. Optionally specify model name and step.

```APIDOC
## load_model_from_path

### Description
Loads a persisted model from a given path. Optionally specify model name and step.

### Method
load_model_from_path

### Parameters
- **path** (str) - Required - The path to the persisted model.
- **name** (str) - Optional - Model name. Default to "model".
- **step** (int) - Optional - Step number to load.

### Request Example
```python
persisted_model = load_model_from_path("hdfs://path/to/save")
persisted_model = load_model_from_path("hdfs://path/to/save", name="model")
persisted_model = load_model_from_path("hdfs://path/to/save", name="model", step=1000)
```

### Response
- **config.path** (str) - Path to the model config.
- **config.load()** (DictConfig) - Loaded config.
- **states.path** (str) - Path to the model states.
- **states.load("cpu")** (state_dict) - Loaded state dict.
- **create("cpu")** (nn.Module) - Instantiated model.
```

--------------------------------

### Save model with save_model

Source: https://github.com/zju3dv/infinidepth/blob/main/training/utils/common/persistence/README.md

Save a model checkpoint. Must be called by all ranks. Multiple models can be saved with different names. get_model_states unwraps DDP, FSDP, and compile.

```python
persistence.save_model(
    step=1000,                              # Current step.
    name="model",                           # Model name. (Default to "model")
    config=trainer_config.unet.model,       # Model config.
    states=get_model_states(unet),          # Model states.
    dtype=None,                             # dtype conversion. Default None to disable.
    blocking=False,                         # Async copy. Default no blocking.
)
```
Tokens:
Raw