### Install Project Dependencies

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Installs all other project dependencies listed in the requirements.txt file.

```bash
pip3 install -r requirements.txt
```

--------------------------------

### Render Single Point Cloud Example

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Renders a single point cloud for a given example and resolution. Supports '384p' and '720p' resolutions. To output point clouds without temporal persistence, set RENDER_ONLY_NECESSARY=false.

```bash
EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/preprocess/example_render_single.sh
```

```bash
RENDER_ONLY_NECESSARY=false EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/preprocess/example_render_single.sh
```

--------------------------------

### Launch Camera UI

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Starts the Viser server, FastAPI backend, and React frontend for the camera UI. Access the UI by opening http://localhost:9999 in your browser.

```bash
bash cam_ui/startup.sh
```

--------------------------------

### Launch Full Camera UI Stack

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Starts the complete camera UI stack, including Viser, FastAPI, and React. This command opens the UI in a web browser.

```bash
# Launch the full camera UI stack (Viser + FastAPI + React)
bash cam_ui/startup.sh
# Opens: http://localhost:9999
```

--------------------------------

### Download Evaluation Dataset

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Downloads the official Vista4D evaluation dataset from Hugging Face. Ensure you have the huggingface-cli installed.

```bash
# Download evaluation dataset
huggingface-cli download Eyeline-Labs/Vista4D-Eval-Data \
    --repo-type dataset --local-dir eval_data
tar -xvf eval_data/eval_data.tar -C eval_data/
```

--------------------------------

### Install Flash Attention and XFuser

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Installs Flash Attention version 2.8.3 and XFuser with Flash Attention support version 0.4.5. The FLASH_ATTENTION_FORCE_BUILD variable can be set to TRUE to force compilation from source.

```bash
export FLASH_ATTENTION_FORCE_BUILD="TRUE"  # Optional: If you want to force a compilation from source
pip3 install flash-attn==2.8.3 --no-build-isolation  # This might take a while if you are compiling from source
pip3 install "xfuser[flash-attn]==0.4.5"
```

--------------------------------

### Install Node.js for React Frontend

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Installs Node.js version 25 using Conda Forge. This is required for the React-based frontend of the camera UI.

```bash
conda install conda-forge::nodejs=25  # Slightly older versions probably work, this is just the newest one at the time of writing
```

--------------------------------

### Render Point Cloud with Edits

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Renders a scene's point cloud after applying specified edits. Configure EXAMPLE and RESOLUTION. Output is saved to a specified directory.

```bash
EXAMPLE=hike_cow RESOLUTION=720p bash scripts/preprocess/example_render_edit.sh
```

--------------------------------

### Install PyTorch

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Installs PyTorch version 2.10.0 and Torchvision version 0.25.0 for CUDA 12.8.

```bash
pip3 install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
```

--------------------------------

### Run Vista4D Inference

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Performs Vista4D inference on the rendered point cloud for a given example and resolution. Outputs are stored in './results/single/$EXAMPLE/vista4d_$RESOLUTION/'.

```bash
EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh
```

--------------------------------

### Dynamic Scene Expansion (DSE) Setup

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Supplements a source video with environmental captures to reduce hallucination and expand the scene. The joint reconstruction is stored in a `recon_and_seg` folder with a `clips.json` file detailing frame ranges for source and DSE content.

```bash
# Dynamic Scene Expansion (DSE) Setup
# ... (details about clips.json and folder structure)
```

--------------------------------

### Install CUDA Toolkit and C++ Compiler

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Installs CUDA toolkit version 12.8 and a compatible C++ compiler into the Conda environment. Sets CUDA_HOME and LD_LIBRARY_PATH environment variables.

```bash
conda install -c nvidia cuda-toolkit=12.8
conda install -c conda-forge gxx_linux-64  # Ensure a compatible C++ compiler
export CUDA_HOME=$CONDA_PREFIX  # Required for the environment setup
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH
```

--------------------------------

### Render Point Cloud with DSE

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Unprojects source clips and DSE frames into a single 4D point cloud, then renders it. Configure EXAMPLE, RESOLUTION, and DSE_FRAME_INTERVAL. Renders are saved to a specified directory.

```bash
EXAMPLE=lounge-cup RESOLUTION=720p bash scripts/preprocess/example_render_dse.sh
```

--------------------------------

### API: Check Load Status and Get First-Frame Camera

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Retrieves the current loading status of a reconstruction folder and provides details about the first frame's camera intrinsics and other properties.

```bash
# API: Check load status + get first-frame camera
curl http://localhost:9998/api/load-status
# {"status":"loaded","loaded":true,"num_frames":49,"fps":30.0,
#  "first_frame_camera":{"intrinsics":{"fx":...,"fy":...,"cx":...,"cy":...},...}}
```

--------------------------------

### Download and Install Cloudflared for Tunneling

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Downloads the cloudflared executable and makes it executable. This is used to create secure tunnels for accessing the UI on remote GPU nodes without direct port access.

```bash
curl -L --output cloudflared https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
chmod +x cloudflared
```

--------------------------------

### Vista4D Inference with Edits

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Performs Vista4D inference on a scene with applied edits. Supports USP multi-GPU inference. Configure EXAMPLE and RESOLUTION.

```bash
EXAMPLE=hike_cow RESOLUTION=720p bash scripts/inference/example_inference_edit.sh
```

--------------------------------

### Run 4D Reconstruction and Segmentation

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Executes a script for 4D reconstruction and dynamic mask segmentation. Configurable by 'EXAMPLE' for video selection and 'RECON_METHOD' for reconstruction method (pi3 or da3).

```bash
EXAMPLE=couple-newspaper RECON_METHOD=pi3 bash scripts/preprocess/example_recon_and_seg_single.sh
```

--------------------------------

### Vista4D Inference with DSE

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Performs Vista4D inference using Dynamic Scene Expansion (DSE) data. Supports USP multi-GPU inference. Configure EXAMPLE and RESOLUTION. Outputs are saved to a specified directory.

```bash
EXAMPLE=lounge-cup RESOLUTION=720p bash scripts/inference/example_inference_dse.sh
```

--------------------------------

### 4D Reconstruction with DSE

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Jointly reconstructs and segments source clips and paired scene captures using Dynamic Scene Expansion (DSE). Results are saved to a specified directory. Configure EXAMPLE and RECON_METHOD.

```bash
EXAMPLE=lounge-cup RECON_METHOD=pi3 bash scripts/preprocess/example_recon_and_seg_dse.sh
```

--------------------------------

### Start Cloudflared Tunnels for UI Access

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Opens two separate tunnels using cloudflared: one for the React frontend (port 9999) and one for the Viser backend (port 9997). This is for remote GPU nodes without inbound port access.

```bash
./cloudflared tunnel --url http://localhost:9999  # React frontend
```

```bash
./cloudflared tunnel --url http://localhost:9997  # Viser backend (for the iframe)
```

--------------------------------

### Initialize Vista4DPipeline for Novel-View Synthesis

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

This Python code shows how to initialize the `Vista4DPipeline` by loading configurations and model checkpoints from local paths. It specifies model paths, data types, and device for GPU acceleration.

```python
import torch
import yaml
import numpy as np
from PIL import Image
from diffsynth.pipelines.wan_video_vista4d import Vista4DPipeline, ModelConfig
from utils.media import load_video, load_cameras, load_masks

# Build pipeline from local checkpoints
vista4d_config_path = "./checkpoints/vista4d/720p49_step=3000/config.yaml"
with open(vista4d_config_path) as f:
    vista4d_config = yaml.safe_load(f)

wan_name = "Wan2.1-T2V-14B"
model_configs = [
    ModelConfig(
        path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/diffusion_pytorch_model*.safetensors")),
        skip_download=True,
    ),
    ModelConfig(
        path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/models_t5_umt5-xxl-enc-bf16.pth")),
        skip_download=True,
    ),
    ModelConfig(
        path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/Wan2.1_VAE.pth")),
        skip_download=True,
    ),
]
tokenizer_config = ModelConfig(
    path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/google/*")),
    skip_download=True,
)

pipe = Vista4DPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=model_configs,
    tokenizer_config=tokenizer_config,
    vista4d_config=vista4d_config,
    vista4d_checkpoint="./checkpoints/vista4d/720p49_step=3000/dit.pth",
    use_usp=False,            # Set True for multi-GPU unified sequence parallel
    vram_limit=40.0,          # Available VRAM in GB minus 2
)

```

--------------------------------

### Download Wan2.1-T2V-14B Base Model

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Downloads the Wan2.1-T2V-14B base model checkpoint to the specified local directory.

```bash
hf download Wan-AI/Wan2.1-T2V-14B --local-dir ./checkpoints/wan/Wan2.1-T2V-14B
```

--------------------------------

### Download Vista4D Evaluation Dataset

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Use the huggingface-cli to download the Vista4D evaluation dataset. Ensure you extract the contents after downloading.

```bash
huggingface-cli download Eyeline-Labs/Vista4D-Eval-Data --repo-type dataset --local-dir eval_data
```

```bash
tar -xvf eval_data/eval_data.tar -C eval_data/
```

--------------------------------

### Download Vista4D Checkpoints

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Downloads the Vista4D checkpoints to the specified local directory. Use --include to download specific variants.

```bash
hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d
```

```bash
hf download Eyeline-Labs/Vista4D --local-dir ./checkpoints/vista4d --include "384p49_step=30000/*" OR "720p49_step=3000/*"
```

--------------------------------

### Render Point Cloud with Vista4D Core API

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

This Python code demonstrates loading inputs and rendering a point cloud from target cameras using the `render_video` function. It supports basic rendering and double reprojection for diagnostics.

```python
import numpy as np
import torch
from utils.point_cloud.render_video import render_video
from utils.media import load_video, load_cameras, load_masks, load_depths

# Load inputs
video, fps = load_video("results/single/parkour/recon_and_seg/video.mp4")  # (F,H,W,3) uint8
depths = load_depths("results/single/parkour/recon_and_seg/depths", dtype=np.float16).astype(np.float32)
cam_c2w_src, intrinsics_src = load_cameras("results/single/parkour/recon_and_seg/cameras.npz")
cam_c2w_tgt, intrinsics_tgt = load_cameras("media/single/parkour_truck-left.npz")
dynamic_mask = load_masks("results/single/parkour/recon_and_seg/dynamic_mask")  # bool (F,H,W)
static_mask = ~dynamic_mask

# Render point cloud from target cameras (basic)
video_pc, depths_pc, alpha_mask_pc, dynamic_mask_pc = render_video(
    video=video[:49],
    depths=depths[:49],
    cam_c2w_src=cam_c2w_src[:49],
    cam_c2w_tgt=cam_c2w_tgt,
    intrinsics_src=intrinsics_src[:49],
    intrinsics_tgt=intrinsics_tgt,
    dynamic_mask=dynamic_mask[:49],
    static_mask=static_mask[:49],
    render_dynamic_mask=True,
    dtype=torch.float32,
    device="cuda",
    verbose=True,
)
# Returns: (F,H,W,3) uint8, (F,H,W) float32, (F,H,W) bool, (F,H,W) bool

# Render with double reprojection (for ablation/diagnostics)
(video_pc, depths_pc, alpha_pc, dyn_pc), \
(video_pc_ntp, depths_ntp, alpha_ntp, dyn_ntp), \
(video_pc2, depths_pc2, alpha_pc2, dyn_pc2) = render_video(
    video=video[:49], depths=depths[:49],
    cam_c2w_src=cam_c2w_src[:49], cam_c2w_tgt=cam_c2w_tgt,
    intrinsics_src=intrinsics_src[:49], intrinsics_tgt=intrinsics_tgt,
    dynamic_mask=dynamic_mask[:49], static_mask=static_mask[:49],
    render_dynamic_mask=True,
    double_reprojection=True,  # Enables NTP + double-reprojection outputs
    dtype=torch.bfloat16,      # Use bfloat16 for speed/memory savings
    device="cuda",
)
```

--------------------------------

### Render Point Cloud with Edits

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Renders a point cloud with applied edits for 4D scene recomposition. Requires the reconstruction/segmentation folder, camera path, and an edits path in JSON format.

```bash
python3 -m scripts.preprocess.render_single \
    --recon_and_seg_folder ./results/edit/hike/recon_and_seg \
    --cam_path ./media/edit/hike_cow.npz \
    --edits_path ./media/edit/hike_cow.json \
    --output_folder ./results/edit/hike_cow/render_720p \
    --height 720 --width 1280 --num_frames 49 \
    --render_only_necessary
```

--------------------------------

### Full Evaluation Dataset Inference (Single GPU)

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Runs the full inference pipeline on the evaluation dataset using a single GPU. Ensure the resolution is set appropriately.

```bash
# Full eval inference (single GPU)
RESOLUTION=720p bash scripts/inference/example_inference_eval.sh
# Output: ./results/eval/vista4d_720p/<video>_<camera>/video_seed=<N>.mp4
```

--------------------------------

### Render Point Clouds for Evaluation Dataset

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Renders point clouds for all 110 video-camera pairs in the evaluation dataset at a specified resolution. This is a preprocessing step for evaluation.

```bash
# Render point clouds for all 110 pairs at 720p
RESOLUTION=720p bash scripts/preprocess/example_render_eval.sh
```

--------------------------------

### Render Full Diagnostic Set with Python

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Use this command to render a full diagnostic set, including NTP and double-reprojection outputs. Omit `--render_only_necessary` to generate video outputs.

```bash
python3 -m scripts.preprocess.render_single \
    --recon_and_seg_folder ./results/single/parkour/recon_and_seg \
    --cam_path ./media/single/parkour_truck-left.npz \
    --output_folder ./results/single/parkour/render_720p \
    --height 720 --width 1280 --num_frames 49
    # omit --render_only_necessary to get video_pc_ntp.mp4 and video_pc2.mp4
```

--------------------------------

### Run 4D Reconstruction and Dynamic Segmentation (Pi3X)

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Basic usage for Pi3X reconstruction and SAM3 segmentation. Specify video path, output folder, segmentation keywords, reconstruction method, model IDs, and resolution. The --save_vis flag generates a visualization collage.

```bash
python3 -m scripts.preprocess.recon_and_seg_single \
    --video_path ./media/single/couple-newspaper.mp4 \
    --output_folder ./results/single/couple-newspaper/recon_and_seg \
    --seg_keywords "man" "woman" "newspaper" "blue car" \
    --recon_method pi3 \
    --pi3_model_id yyfz233/Pi3X \
    --sam3_model_id facebook/sam3 \
    --height 720 --width 1280 --num_frames 49 \
    --pi3_pixel_limit 255000 \
    --save_vis
```

--------------------------------

### Run 4D Reconstruction and Dynamic Segmentation for DSE

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

For Dynamic Scene Expansion (DSE), provide both the source video path and a casual scene capture path. Specify the number of frames for both.

```bash
python3 -m scripts.preprocess.recon_and_seg_single \
    --video_path ./media/dse/lounge-cup.mp4 \
    --dse_video_path ./media/dse/lounge-scene.mp4 \
    --output_folder ./results/dse/lounge-cup/recon_and_seg \
    --seg_keywords "person" "cup" \
    --recon_method pi3 \
    --num_frames 49 --num_dse_frames 60
```

--------------------------------

### Render Point Cloud (DSE)

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Renders a point cloud using jointly unprojected source and DSE frames. Configure resolution and DSE frame interval as needed.

```bash
EXAMPLE=lounge-cup RESOLUTION=720p DSE_FRAME_INTERVAL=4 \
    bash scripts/preprocess/example_render_dse.sh
```

--------------------------------

### Vista4DPipeline Class

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

`diffsynth/pipelines/wan_video_vista4d.py::Vista4DPipeline` wraps the Wan 2.1 diffusion backbone and accepts source video, point cloud renders, masks, and target camera poses to generate a novel-viewpoint video via classifier-free guidance.

```APIDOC
## Vista4DPipeline Class

### Description
The `Vista4DPipeline` class integrates a diffusion model to generate novel-viewpoint videos. It takes various inputs including source video, point cloud renders, masks, and target camera poses, utilizing classifier-free guidance for generation.

### Initialization
Instantiate the pipeline using `Vista4DPipeline.from_pretrained`, specifying model configurations, checkpoint paths, and device settings.

### Parameters
- `torch_dtype` (torch.dtype): The desired data type for model tensors (e.g., `torch.bfloat16`).
- `device` (str): The device to run the pipeline on (e.g., "cuda").
- `model_configs` (list[ModelConfig]): A list of configurations for the diffusion model components.
- `tokenizer_config` (ModelConfig): Configuration for the tokenizer.
- `vista4d_config` (dict): Configuration specific to the Vista4D pipeline.
- `vista4d_checkpoint` (str): Path to the Vista4D model checkpoint.
- `use_usp` (bool, optional): Whether to use unified sequence parallel. Defaults to False.
- `vram_limit` (float, optional): Available VRAM in GB minus a buffer. Defaults to a reasonable value.

### Request Example
```python
import torch
import yaml
from diffsynth.pipelines.wan_video_vista4d import Vista4DPipeline, ModelConfig

# Load configuration
vista4d_config_path = "./checkpoints/vista4d/720p49_step=3000/config.yaml"
with open(vista4d_config_path) as f:
    vista4d_config = yaml.safe_load(f)

# Define model configurations (example paths)
wan_name = "Wan2.1-T2V-14B"
model_configs = [
    ModelConfig(path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/diffusion_pytorch_model*.safetensors"))),
    ModelConfig(path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/models_t5_umt5-xxl-enc-bf16.pth"))),
    ModelConfig(path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/Wan2.1_VAE.pth"))),
]
tokenizer_config = ModelConfig(path=list(__import__("glob").glob(f"./checkpoints/wan/{wan_name}/google/*")))

# Build the pipeline
pipe = Vista4DPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=model_configs,
    tokenizer_config=tokenizer_config,
    vista4d_config=vista4d_config,
    vista4d_checkpoint="./checkpoints/vista4d/720p49_step=3000/dit.pth",
    use_usp=False,
    vram_limit=40.0,
)
```
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Use these commands to create a new Conda environment named 'vista4d' with Python 3.12 and activate it.

```bash
conda create --name vista4d python=3.12
conda activate vista4d
```

--------------------------------

### Render Point Cloud (Standard)

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Unprojects 4D reconstruction to a point cloud and renders it from target cameras. Requires the output folder from reconstruction and segmentation, and a camera path. The --render_only_necessary flag optimizes rendering. --save_vis generates visualization outputs.

```bash
python3 -m scripts.preprocess.render_single \
    --recon_and_seg_folder ./results/single/couple-newspaper/recon_and_seg \
    --cam_path ./media/single/couple-newspaper_dolly-zoom.npz \
    --output_folder ./results/single/couple-newspaper/render_720p \
    --height 720 --width 1280 --num_frames 49 \
    --render_only_necessary \
    --save_vis
```

--------------------------------

### API: Load Reconstruction Folder

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Loads a 3D reconstruction and segmentation folder into the system. This is an asynchronous operation, and the response indicates the loading status.

```bash
# API: Load a recon_and_seg folder
curl -X POST http://localhost:9998/api/load-folder \
    -H "Content-Type: application/json" \
    -d '{"folder_path": "results/single/couple-newspaper/recon_and_seg"}'
# {"status":"loading","message":"Loading reconstruction folder ...","folder_name":"recon_and_seg"}
```

--------------------------------

### Shell Script for Single-GPU 720p Inference

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

This shell script initiates end-to-end inference for a single GPU at 720p resolution. It sets up necessary environment variables and paths before executing the main inference script. The output includes various visualization files and the generated video.

```bash
# Single-GPU 720p inference on provided example
EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh
# Output: ./results/single/couple-newspaper/vista4d_720p/
#   source.mp4            # Preprocessed source video (input visualization)
#   point_cloud.mp4       # Point cloud render (input visualization)
#   point_cloud_masks.mp4 # Alpha/dynamic masks visualization
#   video_seed=10027.mp4  # Generated novel-viewpoint video
#   gifs/video_seed=10027.gif

# 384p inference (lighter checkpoint, more trained steps)
EXAMPLE=snowboard RESOLUTION=384p bash scripts/inference/example_inference_single.sh

# Multi-GPU inference with Unified Sequence Parallel (USP)
USE_USP=true NUM_GPUS=8 EXAMPLE=couple-newspaper RESOLUTION=720p \
    bash scripts/inference/example_inference_single.sh

# Multi-seed batch inference (edit seed array in the script, or call directly)
python3 -m scripts.inference.inference \
    --model_id_with_origin_paths "Wan2.1-T2V-14B:diffusion_pytorch_model*.safetensors,Wan2.1-T2V-14B:models_t5_umt5-xxl-enc-bf16.pth,Wan2.1-T2V-14B:Wan2.1_VAE.pth" \
    --tokenizer_id_with_origin_path "Wan2.1-T2V-14B:google/*" \
    --local_model_folder ./checkpoints/wan \
    --vista4d_checkpoint ./checkpoints/vista4d/720p49_step=3000/dit.pth \
    --vista4d_config_path ./checkpoints/vista4d/720p49_step=3000/config.yaml \
    --input_folder ./results/single/couple-newspaper/render_720p \
    --output_folder ./results/single/couple-newspaper/vista4d_720p \
    --prompt "An elderly couple seated on a wooden bench reading a magazine." \
    --height 720 --width 1280 --num_frames 49 \
    --seed 42 10027 90095 \
    --tile_vae              # For high-VRAM savings
    --save_gif

```

--------------------------------

### Log in to Hugging Face CLI

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Logs into the Hugging Face CLI. This is required to request checkpoint access for Segment Anything 3 (SAM3).

```bash
hf auth login
```

--------------------------------

### Run 4D Reconstruction and Dynamic Segmentation (DA3)

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

DA3 reconstruction offers higher resolution support and requires more VRAM. Specify the DA3 model ID and processing resolution.

```bash
python3 -m scripts.preprocess.recon_and_seg_single \
    --video_path ./media/single/snowboard.mp4 \
    --output_folder ./results/single/snowboard/recon_and_seg \
    --seg_keywords "person" "helmet" "snowboard" \
    --recon_method da3 \
    --da3_model_id depth-anything/DA3NESTED-GIANT-LARGE-1.1 \
    --da3_process_res 896 \
    --height 720 --width 1280 --num_frames 49
```

--------------------------------

### Run Vista4D Inference with USP

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Performs multi-GPU Vista4D inference using Unified Sequence Parallel (USP) for speed and reduced VRAM usage. Optionally specify the number of GPUs.

```bash
USE_USP=true NUM_GPUS=8 EXAMPLE=couple-newspaper RESOLUTION=720p bash scripts/inference/example_inference_single.sh
```

--------------------------------

### Run Vista4D Inference

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

This Python function executes the Vista4D pipeline to generate novel-viewpoint videos. It requires pre-loaded video, mask, and camera data. Ensure all necessary configurations and models are correctly set up.

```python
render_dir = "./results/single/couple-newspaper/render_720p"
video_src, fps = load_video(f"{render_dir}/video_src.mp4")  # (F,H,W,3) uint8
video_pc, _   = load_video(f"{render_dir}/video_pc.mp4")
cam_c2w_tgt, intrinsics_tgt = load_cameras(f"{render_dir}/cameras_tgt.npz")
alpha_src     = load_masks(f"{render_dir}/alpha_mask_src")   # (F,H,W) bool
dynamic_src   = load_masks(f"{render_dir}/dynamic_mask_src")
alpha_pc      = load_masks(f"{render_dir}/alpha_mask_pc")
dynamic_pc    = load_masks(f"{render_dir}/dynamic_mask_pc")

to_pil = lambda arr: [Image.fromarray(arr[i]) for i in range(arr.shape[0])]

@torch.no_grad()
def run_inference():
    videos = pipe(
        prompt="An elderly couple is seated on a wooden bench, deeply engrossed in reading a magazine.",
        negative_prompt=(
            "gaudy, overexposed, static, blurred details, subtitles, style, artwork, "
            "still, worst quality, low quality, ugly"
        ),
        source_video=to_pil(video_src),
        point_cloud_video=to_pil(video_pc),
        source_alpha_mask=alpha_src,
        source_motion_mask=dynamic_src,
        point_cloud_alpha_mask=alpha_pc,
        point_cloud_motion_mask=dynamic_pc,
        target_cam_c2w=cam_c2w_tgt,
        target_intrinsics=intrinsics_tgt,
        height=720, width=1280, num_frames=49,
        source_noise_level=vista4d_config["dit"]["augmentation"]["source_noise_level"],
        point_cloud_noise_level=vista4d_config["dit"]["augmentation"]["point_cloud_noise_level"],
        image_noise_level=vista4d_config["dit"]["augmentation"]["image_noise_level"],
        seed=[10027],          # List of seeds = batch size
        cfg_merge=False,       # Enable for quality boost at cost of 2× memory
        tiled=False,           # Enable tiled VAE for very high resolution
    )
    # videos: list of lists of PIL.Image frames, one list per seed
    return videos[0]  # First seed's frames

frames = run_inference()
# frames: [PIL.Image, ...] of length 49

```

--------------------------------

### Load Reconstruction Folder

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Loads a reconstruction and segmentation folder into the system.

```APIDOC
## POST /api/load-folder

### Description
Initiates the loading of a specified reconstruction and segmentation folder. This prepares the system to work with the data within that folder.

### Method
POST

### Endpoint
/api/load-folder

### Parameters
#### Request Body
- **folder_path** (string) - Required - The path to the reconstruction and segmentation folder.

### Request Example
```json
{
  "folder_path": "results/single/couple-newspaper/recon_and_seg"
}
```

### Response
#### Success Response (200)
- **status** (string) - The status of the loading operation (e.g., "loading").
- **message** (string) - A message indicating the progress or status of the loading.
- **folder_name** (string) - The name of the folder being loaded.

#### Response Example
```json
{
  "status": "loading",
  "message": "Loading reconstruction folder ...",
  "folder_name": "recon_and_seg"
}
```
```

--------------------------------

### API: Apply Point Cloud Edits

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Applies point cloud edits, such as scaling an object, using SAM3 segmentation and GPU transformations. The response indicates the number of edits applied.

```bash
# API: Apply point cloud edits (SAM3 + GPU transforms)
curl -X PUT http://localhost:9998/api/edits \
    -H "Content-Type: application/json" \
    -d '{
      "edits": [{"target": {"kind": "existing", "prompt": "backpack"},
        "ops": [{"op": "scale", "params": 2.0}],
        "scope": "frame"
      }]
    }'
# {"status":"applied","num_edits":1}
```

--------------------------------

### Render Point Clouds for Evaluation Data

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Render point clouds for the evaluation dataset using the provided script. The output resolution can be set using the RESOLUTION variable.

```bash
RESOLUTION=720p bash scripts/preprocess/example_render_eval.sh
```

--------------------------------

### Apply Edits

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Applies point cloud edits using SAM3 and GPU transformations.

```APIDOC
## PUT /api/edits

### Description
Applies a list of edits to the point cloud data. Edits can include operations like scaling, translation, and removal, targeting specific objects identified by prompts.

### Method
PUT

### Endpoint
/api/edits

### Parameters
#### Request Body
- **edits** (array) - Required - A list of edit operations to apply.
  - **target** (object) - Defines the object to be edited.
    - **kind** (string) - The type of target (e.g., "existing", "duplicate", "insert").
    - **prompt** (string) - A text prompt to identify the target object.
    - **source** (string, optional) - Source path for "insert" kind.
  - **ops** (array) - A list of operations to perform on the target.
    - **op** (string) - The type of operation (e.g., "remove", "scale", "translate").
    - **params** (any, optional) - Parameters for the operation (e.g., scale factor, translation vector).
  - **scope** (string) - The scope of the edit (e.g., "global", "frame").
  - **centroid_threshold** (float, optional) - Threshold for centroid tracking in "frame" scope.
  - **mask_expansion** (array, optional) - Parameters for mask expansion.

### Request Example
```json
{
  "edits": [
    {
      "target": {"kind": "existing", "prompt": "backpack"},
      "ops": [{"op": "scale", "params": 2.0}],
      "scope": "frame"
    }
  ]
}
```

### Response
#### Success Response (200)
- **status** (string) - The status of the edit application (e.g., "applied").
- **num_edits** (integer) - The number of edits successfully applied.

#### Response Example
```json
{
  "status": "applied",
  "num_edits": 1
}
```
```

--------------------------------

### Run Vista4D Inference on Evaluation Data

Source: https://github.com/eyeline-labs/vista4d/blob/main/README.md

Execute Vista4D inference on the preprocessed evaluation dataset. The script uses metadata.csv for prompts and seeds. The RESOLUTION variable controls the output quality.

```bash
RESOLUTION=720p bash scripts/inference/example_inference_eval.sh
```

--------------------------------

### Load and Build Edit Functions for Point Clouds

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

This Python snippet demonstrates how to load editing instructions from a JSON file and build a function to apply these edits to point cloud data. It's used for modifying 4D scenes before rendering.

```python
import json
from utils.point_cloud.edit import load_edits, build_edit_fn

```

--------------------------------

### Load Status and First Frame Camera

Source: https://context7.com/eyeline-labs/vista4d/llms.txt

Checks the status of the data loading process and retrieves information about the first frame's camera.

```APIDOC
## GET /api/load-status

### Description
Retrieves the current status of the data loading process. If loaded, it also provides details about the camera intrinsics and extrinsics for the first frame.

### Method
GET

### Endpoint
/api/load-status

### Response
#### Success Response (200)
- **status** (string) - The status of the loading (e.g., "loaded").
- **loaded** (boolean) - Indicates if the data has been successfully loaded.
- **num_frames** (integer) - The total number of frames in the loaded data.
- **fps** (float) - The frames per second of the loaded video.
- **first_frame_camera** (object) - Camera parameters for the first frame.
  - **intrinsics** (object) - Intrinsic camera parameters.
    - **fx** (float) - Focal length in x.
    - **fy** (float) - Focal length in y.
    - **cx** (float) - Principal point x-coordinate.
    - **cy** (float) - Principal point y-coordinate.
  - ... (other camera parameters like extrinsics)

### Response Example
```json
{
  "status": "loaded",
  "loaded": true,
  "num_frames": 49,
  "fps": 30.0,
  "first_frame_camera": {
    "intrinsics": {
      "fx": 1000.0,
      "fy": 1000.0,
      "cx": 640.0,
      "cy": 360.0
    },
    ...
  }
}
```
```

--------------------------------

### select_masks_and_return

Source: https://github.com/eyeline-labs/vista4d/blob/main/sam3/agent/system_prompts/system_prompt.txt

Call this tool to select a subset of or all of the mask(s) rendered on the most recent image as your final output. When calling select_masks_and_return, you cannot select any mask(s) generated by previous rounds other than the most recent round in your "final_answer_masks". You can only use mask(s) from the most recent image in your message history. (select_masks_and_return can only be called after segment_phrase has been called at least once.)

```APIDOC
## select_masks_and_return

### Description
Selects a subset of or all masks from the most recent image as the final output. This tool can only be used after `segment_phrase` has been called at least once, and only with masks from the most recent image.

### Method
TOOL_CALL

### Endpoint
N/A

### Parameters
#### Request Body
- **final_answer_masks** (array of integers) - Required - An array of integers representing the selected mask(s) you want to choose as your final output, e.g., [1, 4, 5].

### Important Rules:
1. Do not call `select_masks_and_return` unless the selected masks are the correct ones matching the initial user query.
2. If targets exist in the image, `select_masks_and_return` must be the final tool call; do not use `report_no_mask`.
3. Integers in `final_answer_masks` must be within the range of 1 to N (total masks on the most recent image).
4. Integers in `final_answer_masks` must be unique.
5. Only masks from the most recent image can be used.
6. Use this tool for reporting `final_answer_masks`. If masks are not 100% complete, do not call this tool and continue updating them.
7. Calling `segment_phrase` deletes all previously generated masks. Masks from earlier images cannot be used.
8. Plan tool calls carefully to ensure the most recent `segment_phrase` call covers all target objects.
9. Do not call `select_masks_and_return` if no masks are rendered on the most recent image.
10. Masks in `final_answer_masks` must accurately capture target objects and nothing else. If not, continue using other tools.
11. Ignore masks with two-digit numbers larger than N if they are visual illusions; only refer to existing masks by their correct numbers.
12. List all required masks explicitly in the `final_answer_masks` array; do not use abbreviations or code.
```