### Install Training Dependencies

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Installs PyTorch Lightning and TensorBoard for training CoTracker. Ensure these are installed before launching training.

```bash
pip install pytorch_lightning==1.6.0 tensorboard
```

--------------------------------

### Install CoTracker Development Version

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Clone the repository, navigate to the directory, and install CoTracker in development mode. Additional libraries for visualization and training are also installed.

```bash
git clone https://github.com/facebookresearch/co-tracker
cd co-tracker
pip install -e .
pip install matplotlib flow_vis tqdm tensorboard imageio[ffmpeg]
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Installs Sphinx and sphinxcontrib-bibtex, which are required for building the CoTracker documentation.

```bash
pip install sphinx
pip install sphinxcontrib-bibtex
```

--------------------------------

### Install LeviTor and Launch Demo

Source: https://context7.com/qiuyu96/levitor/llms.txt

Follow these steps to set up the conda environment, download necessary checkpoints, install dependencies, and launch the Gradio demo server for LeviTor. Ensure all specified checkpoint files are placed in the correct directories.

```bash
# 1. Clone and enter the repo
git clone https://github.com/qiuyu96/LeviTor.git
cd LeviTor

# 2. Create checkpoint directories
mkdir -p checkpoints/LeviTor

# 3. Download required checkpoints into checkpoints/:
#    - depth_anything_v2_vitl.pth  (Depth Anything V2)
#    - sam_vit_h_4b8939.pth        (Segment Anything ViT-H)
#    - stable-video-diffusion-img2vid-xt/  (Stability AI SVD-XT)
#    - LeviTor/{controlnet/, unet/, scheduler.bin, scaler.pt, random_states_0.pkl}

# Expected layout:
# checkpoints/
#  |-- sam_vit_h_4b8939.pth
#  |-- depth_anything_v2_vitl.pth
#  |-- stable-video-diffusion-img2vid-xt/
#  |-- LeviTor/
#      |-- controlnet/
#      |-- unet/
#      |-- scheduler.bin
#      |-- scaler.pt
#      |-- random_states_0.pkl

# 4. Create and activate conda environment
conda create -n LeviTor python=3.9 -y
conda activate LeviTor

# 5. Install Python dependencies
pip install -r requirements.txt
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
pip install gradio==4.36.1

# 6. Launch the demo (fp16, xformers, 16 frames, 288×512 output)
python gradio_demo/gradio_run.py \
    --frame_interval 1 \
    --num_frames 16 \
    --pretrained_model_name_or_path checkpoints/stable-video-diffusion-img2vid-xt \
    --resume_from_checkpoint checkpoints/LeviTor \
    --width 288 --height 512 \
    --seed 217113 \
    --mixed_precision fp16 \
    --enable_xformers_memory_efficient_attention \
    --output_dir ./outputs \
    --gaussian_r 10 \
    --sam_path checkpoints/sam_vit_h_4b8939.pth \
    --depthanything_path checkpoints/depth_anything_v2_vitl.pth
# Server starts at http://0.0.0.0:7860

```

--------------------------------

### Run Gradio Demo Locally

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Install dependencies for the Gradio demo and run the app. This allows for an interactive demonstration of CoTracker.

```bash
pip install -r gradio_demo/requirements.txt
python -m gradio_demo.app
```

--------------------------------

### Install Gradio

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Install the Gradio library with a specific version (4.36.1). Gradio is used for creating the interactive demo.

```bash
pip install gradio==4.36.1
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Clone the Depth Anything V2 repository and install the necessary Python packages from the requirements file.

```bash
git clone https://github.com/DepthAnything/Depth-Anything-V2
cd Depth-Anything-V2/metric_depth
pip install -r requirements.txt
```

--------------------------------

### Install PyTorch3D

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Install the PyTorch3D library from its GitHub repository. This is a specific dependency for LeviTor.

```bash
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
```

--------------------------------

### Install Evaluation Dependencies

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Install hydra-core and mediapy for running evaluation scripts. These are required for reproducing paper results.

```bash
pip install hydra-core==1.1.0 mediapy
```

--------------------------------

### Prepare Start Frame for Depth Estimation

Source: https://context7.com/qiuyu96/levitor/llms.txt

Prepares an input image for depth estimation by resizing, converting color spaces, and calculating inverse depth. Assumes depth_anything model is loaded.

```python
from PIL import Image
import cv2, numpy as np
from math import inf

def prepare_start_frame(image_path, width=288, height=512):
    img_pil = Image.open(image_path).convert("RGB")
    img_pil = img_pil.resize((width, height), Image.BILINEAR)
    img_pil.save("outputs/start_frame.png")

    img_rgb = cv2.cvtColor(np.array(img_pil), cv2.COLOR_RGB2BGR)
    img_rgb = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2RGB)

    # Assumes depth_anything is already loaded (see Depth Anything V2 section)
    raw_depth = depth_anything.infer_image(img_rgb, input_size=512)
    depth_norm = (raw_depth - raw_depth.min()) / (raw_depth.max() - raw_depth.min()) * 255.0
    inv_depth = 1.0 / depth_norm.astype(np.float32)
    inv_depth[inv_depth == inf] = 1.0

    return "outputs/start_frame.png", inv_depth   # inv_depth used for trajectory depth reference

frame_path, inv_depth = prepare_start_frame("my_photo.jpg")
# inv_depth[y, x] → reference depth value shown in the Gradio "Depths for reference" box
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Install all required Python packages listed in the 'requirements.txt' file. Ensure the conda environment is activated.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Matplotlib for Visualization

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Install matplotlib using pip. This is a prerequisite for visualizing predicted tracks.

```bash
pip install matplotlib
```

--------------------------------

### Initialize and Use StableVideoDiffusionPipelineControlNet

Source: https://context7.com/qiuyu96/levitor/llms.txt

Initializes the StableVideoDiffusionPipelineControlNet with specified components and generates video frames based on an input image and point tracks. Requires model components and a starting image.

```python
from pipelines.pipeline_stable_video_diffusion_mask_control import StableVideoDiffusionPipelineControlNet
from diffusers.utils import load_image
import torch

pipeline = StableVideoDiffusionPipelineControlNet.from_pretrained(
    "checkpoints/stable-video-diffusion-img2vid-xt",
    unet=unet,
    image_encoder=image_encoder,
    vae=vae,
    controlnet=controlnet,
    torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")

image = load_image("start_frame.png").resize((288, 512))

point_tracks = [
    [[256, 144, 1, 3], [260, 148, 2, 5]],  # frame 0
    [[258, 146, 1, 3], [262, 150, 2, 5]],  # frame 1
    # ... 14 more frames
]

output = pipeline(
    image=image,
    with_control=True,
    point_tracks=point_tracks,
    height=512,
    width=288,
    side=10,
    num_frames=16,
    num_inference_steps=25,
    min_guidance_scale=1.0,
    max_guidance_scale=3.0,
    fps=6,
    motion_bucket_id=100,
    noise_aug_strength=0.02,
    controlnet_cond_scale=1.0,
    with_id_feature=False,
    return_dict=True,
)

frames = output.frames[0]
frames[0].save("frame0.png")

from gradio_demo.utils import export_to_gif, export_to_video
import numpy as np
frames_np = [np.array(f) for f in frames]
export_to_gif(frames_np, "result.gif", fps=16)
export_to_video(frames_np, "result.mp4", fps=7)
```

--------------------------------

### Drag.run()

Source: https://context7.com/qiuyu96/levitor/llms.txt

Generates a video (GIF/MP4) from a starting frame and 3D trajectory data using the ControlNet pipeline.

```APIDOC
## `Drag.run()` — Video Generation from 3D Trajectories

Accepts a preprocessed start-frame path, user-drawn trajectory state, and inference hyper-parameters; constructs per-frame 3D segment masks with PyTorch3D, assembles K-Means keypoint JSON, runs the SVD-ControlNet diffusion loop, and saves the result as a GIF + MP4 pair.

### Parameters
- **first_frame_path** (str) - Path to the starting frame image.
- **tracking_points** (gr.State or list) - User-drawn trajectory state, typically a list of `[x, y]` pixel coordinates per trajectory.
- **controlnet_cond_scale** (float) - ControlNet conditioning strength. Defaults to 1.0.
- **motion_bucket_id** (int) - SVD motion magnitude. Range is 1–180. Defaults to 100.
- **depth_input** (str) - JSON string representing per-trajectory depth information (e.g., `"[[0.3, 0.35, 0.4, ..., 0.6]]"`).
- **points_num** (float) - Ratio controlling the number of K-Means control points. E.g., `"0.5"`.

### Returns
- **gif_path** (str) - Path to the generated GIF file.

### Example
```python
# Signature:
# Drag.run(first_frame_path, tracking_points, controlnet_cond_scale,
#           motion_bucket_id, depth_input, points_num) -> str (gif path)

# Minimal programmatic example (bypassing Gradio state):
import ast, torch
from gradio_demo.gradio_run import Drag, args, depth_for_3D_initial, masks_selected

drag = Drag("cuda:0", args, model_length=16)

# Example usage (replace with actual paths and data):
# gif_path = drag.run(
#     first_frame_path="path/to/your/start_frame.png",
#     tracking_points=[[[100, 100], [110, 110]], [[200, 200], [210, 210]]], # Example tracking points
#     controlnet_cond_scale=1.0,
#     motion_bucket_id=100,
#     depth_input="[[0.5, 0.5], [0.5, 0.5]]", # Example depth input
#     points_num=0.5
# )
# print(f"Generated GIF at: {gif_path}")
```
```

--------------------------------

### Load and Run Offline CoTracker

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Loads a pretrained CoTracker2 model for offline video processing. Ensure 'imageio[ffmpeg]' is installed. The video is loaded and processed in batches on the specified device.

```python
import torch
# Download the video
url = 'https://github.com/facebookresearch/co-tracker/blob/main/assets/apple.mp4'

import imageio.v3 as iio
frames = iio.imread(url, plugin="FFMPEG")  # plugin="pyav"

device = 'cuda'
grid_size = 10
video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)  # B T C H W

# Run Offline CoTracker:
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2").to(device)
pred_tracks, pred_visibility = cotracker(video, grid_size=grid_size) # B T N 2,  B T N 1
```

--------------------------------

### Load CoTracker v1.0 via PyTorch Hub

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Loads the CoTracker v1.0 model using torch.hub.load. Ensure PyTorch and other listed libraries are installed.

```python
import torch
import einops
import timm
import tqdm

cotracker = torch.hub.load("facebookresearch/co-tracker:v1.0", "cotracker_w8")
```

--------------------------------

### StableVideoDiffusionPipelineControlNet.__call__()

Source: https://context7.com/qiuyu96/levitor/llms.txt

The core Diffusion Pipeline that adds ControlNet trajectory conditioning on top of standard SVD. It accepts a PIL start image, per-frame keypoint tracks, and SVD generation parameters, returning generated video frames.

```APIDOC
## StableVideoDiffusionPipelineControlNet.__call__() — Core Diffusion Pipeline

### Description
The custom `DiffusionPipeline` subclass that adds ControlNet trajectory conditioning on top of standard SVD. Accepts a PIL start image, per-frame keypoint tracks, and all SVD generation parameters; returns a `StableVideoDiffusionPipelineOutput` containing the generated video frames.

### Method
```python
pipeline = StableVideoDiffusionPipelineControlNet.from_pretrained(
    "checkpoints/stable-video-diffusion-img2vid-xt",
    unet=unet,
    image_encoder=image_encoder,
    vae=vae,
    controlnet=controlnet,
    torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")

image = load_image("start_frame.png").resize((288, 512))

# point_tracks: list[list[list[int]]] — shape (num_frames, num_keypoints, 4)
# Each keypoint: [row, col, segment_id, depth_bucket]
point_tracks = [
    [[256, 144, 1, 3], [260, 148, 2, 5]],  # frame 0
    [[258, 146, 1, 3], [262, 150, 2, 5]],  # frame 1
    # ... 14 more frames
]

output = pipeline(
    image=image,
    with_control=True,
    point_tracks=point_tracks,
    height=512,
    width=288,
    side=10,                       # gaussian_r — heatmap blob radius in pixels
    num_frames=16,
    num_inference_steps=25,
    min_guidance_scale=1.0,
    max_guidance_scale=3.0,
    fps=6,
    motion_bucket_id=100,          # higher → more motion
    noise_aug_strength=0.02,
    controlnet_cond_scale=1.0,
    with_id_feature=False,
    return_dict=True,
)

frames = output.frames[0]          # list of 16 PIL Images
frames[0].save("frame0.png")

# Export to video
from gradio_demo.utils import export_to_gif, export_to_video
import numpy as np
frames_np = [np.array(f) for f in frames]
export_to_gif(frames_np, "result.gif", fps=16)
export_to_video(frames_np, "result.mp4", fps=7)
```
```

--------------------------------

### Build CoTracker Documentation

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Generates the CoTracker documentation in the docs/_build/html folder using the make command.

```bash
make -C docs html
```

--------------------------------

### Launch CoTracker Training

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Launches the training process for CoTracker on Kubric. Modify dataset_root and ckpt_path before running. Add --num_nodes 4 for multi-node training.

```bash
python train.py --batch_size 1 \
--num_steps 50000 --ckpt_path ./ --dataset_root ./datasets --model_name cotracker \
--save_freq 200 --sequence_len 24 --eval_datasets dynamic_replica tapvid_davis_first \
--traj_per_sample 768 --sliding_window_len 8 \
--num_virtual_tracks 64 --model_stride 4
```

--------------------------------

### Reproduce Training

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Prepare the Hypersim and Virtual KITTI 2 datasets, then execute the dist_train.sh script to reproduce the training process.

```bash
bash dist_train.sh
```

--------------------------------

### prepare_start_frame

Source: https://context7.com/qiuyu96/levitor/llms.txt

Prepares the initial frame for video generation by resizing, converting, and inferring depth information.

```APIDOC
## prepare_start_frame

### Description
Prepares the initial frame for video generation by resizing, converting, and inferring depth information. It takes an image path and optional width/height, returning the path to the saved start frame and the inverse depth map.

### Parameters
- **image_path** (str) - Required - Path to the input image.
- **width** (int) - Optional - The desired width for resizing the image. Defaults to 288.
- **height** (int) - Optional - The desired height for resizing the image. Defaults to 512.

### Returns
- **frame_path** (str) - Path to the saved start frame image.
- **inv_depth** (numpy.ndarray) - The calculated inverse depth map.

### Example
```python
from PIL import Image
import cv2, numpy as np
from math import inf

def prepare_start_frame(image_path, width=288, height=512):
    img_pil = Image.open(image_path).convert("RGB")
    img_pil = img_pil.resize((width, height), Image.BILINEAR)
    img_pil.save("outputs/start_frame.png")

    img_rgb = cv2.cvtColor(np.array(img_pil), cv2.COLOR_RGB2BGR)
    img_rgb = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2RGB)

    # Assumes depth_anything is already loaded (see Depth Anything V2 section)
    raw_depth = depth_anything.infer_image(img_rgb, input_size=512)
    depth_norm = (raw_depth - raw_depth.min()) / (raw_depth.max() - raw_depth.min()) * 255.0
    inv_depth = 1.0 / depth_norm.astype(np.float32)
    inv_depth[inv_depth == inf] = 1.0

    return "outputs/start_frame.png", inv_depth   # inv_depth used for trajectory depth reference

frame_path, inv_depth = prepare_start_frame("my_photo.jpg")
```
```

--------------------------------

### Run Demo with Checkpoint

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Run the demo script using a specified checkpoint file. This allows you to use the downloaded CoTracker2 model.

```bash
python demo.py --checkpoint checkpoints/cotracker2.pth
```

--------------------------------

### Download CoTracker2 Checkpoint

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Create a checkpoints directory and download the CoTracker2 model weights. This is necessary for running the model locally.

```bash
mkdir -p checkpoints
cd checkpoints
wget https://huggingface.co/facebook/cotracker/resolve/main/cotracker2.pth
cd ..
```

--------------------------------

### Clone LeviTor Repository

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Clone the LeviTor repository and navigate into the project directory. This is the first step to set up the project locally.

```bash
git clone https://github.com/qiuyu96/LeviTor.git
cd LeviTor
```

--------------------------------

### Run Online Demo

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Execute the online demo script. This provides a real-time demonstration of CoTracker's capabilities.

```bash
python online_demo.py
```

--------------------------------

### Create Checkpoints Directory

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Create a 'checkpoints' directory and navigate into it. This directory will store all necessary model checkpoints.

```bash
mkdir checkpoints
cd checkpoints
```

--------------------------------

### Initialize SAM Model and Predictor

Source: https://context7.com/qiuyu96/levitor/llms.txt

Loads the Segment Anything ViT-H checkpoint and creates a point-prompted predictor and an automatic mask generator. Ensure the checkpoint path is correct and the device is available.

```python
from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator

sam_checkpoint = "checkpoints/sam_vit_h_4b8939.pth"
model_type = "vit_h"
device = "cuda:0"

sam = sam_model_registry[model_type](checkpoint=sam_checkpoint)
sam.to(device=device)

# Point-prompted prediction (used for interactive area selection)
predictor = SamPredictor(sam)

# Fully automatic mask generation (used during inference to decompose the scene)
auto_predictor = SamAutomaticMaskGenerator(sam)
```

```python
# --- Point-prompted usage ---
import cv2, numpy as np
image = cv2.imread("frame.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)

input_point = np.array([[150, 200], [160, 210]])  # (x, y) pixel coords
input_label = np.array([1, 1])                    # 1 = foreground, 0 = background

masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
)
# masks.shape → (3, H, W); best mask: masks[np.argmax(scores)]
best_mask = masks[np.argmax(scores)]  # boolean array of shape (H, W)
```

--------------------------------

### Run Offline Demo

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Execute the offline demo script with a specified grid size. Results are saved to a default directory.

```bash
python demo.py --grid_size 10
```

--------------------------------

### File and Directory Utilities

Source: https://context7.com/qiuyu96/levitor/llms.txt

Provides helper functions for safely creating directories and reading/writing various data types (images, GIFs, text, checkpoints) to files.

```python
from gradio_demo.utils_drag import ensure_dirname, data2file, file2data
import torch

# Create a directory (no-op if exists, optional purge with override=True)
ensure_dirname("outputs/session_42")

# Save a GIF from a list of numpy uint8 frames
import numpy as np
frames = [np.random.randint(0, 255, (64, 64, 3), dtype=np.uint8) for _ in range(8)]
data2file(frames, "outputs/session_42/preview.gif", override=True, duration=167)

# Save a text log
log_lines = ["epoch 1: loss=0.42", "epoch 2: loss=0.31"]
data2file(log_lines, "outputs/session_42/train.txt", override=True)

# Load a PyTorch checkpoint
state = file2data("checkpoints/LeviTor/scaler.pt", printable=False)
print(type(state))   # <class 'dict'>  or GradScaler state

# Load a text file line by line (top=5 reads only the first 5 lines)
lines = file2data("outputs/session_42/train.txt", top=5)
print(lines)   # ['epoch 1: loss=0.42', 'epoch 2: loss=0.31']
```

--------------------------------

### Run LeviTor Demo

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Execute the Gradio demo script for LeviTor. This command includes various arguments for model paths, dimensions, and training parameters.

```bash
python gradio_demo/gradio_run.py --frame_interval 1 --num_frames 16 --pretrained_model_name_or_path checkpoints/stable-video-diffusion-img2vid-xt --resume_from_checkpoint checkpoints/LeviTor --width 288 --height 512 --seed 217113 --mixed_precision fp16 --enable_xformers_memory_efficient_attention --output_dir ./outputs --gaussian_r 10 --sam_path checkpoints/sam_vit_h_4b8939.pth --depthanything_path checkpoints/depth_anything_v2_vitl.pth
```

--------------------------------

### Simulate Trajectory with Drag

Source: https://context7.com/qiuyu96/levitor/llms.txt

Simulates a video generation process with predefined tracking points and depth values using the drag.run function. Requires a first frame and outputs a GIF.

```python
class FakeState:
    constructor_args = {'value': [[[100, 150], [120, 170], [140, 195], [160, 220], [180, 245]]]}

output_gif = drag.run(
    first_frame_path="outputs/first_frame_abcd.png",
    tracking_points=FakeState(),
    controlnet_cond_scale=1.0,
    motion_bucket_id=100,
    depth_input="[[0.3, 0.4, 0.5, 0.6, 0.7]]",
    points_num="0.5",
)
print(f"Generated video saved to: {output_gif}")
```

--------------------------------

### Initialize Depth Anything V2 and Infer Depth

Source: https://context7.com/qiuyu96/levitor/llms.txt

Loads the Depth Anything V2 ViT-L encoder and runs monocular depth estimation. The inverse-depth map is normalized and used for 3D projection. Ensure the checkpoint path is correct and the device is available.

```python
from gradio_demo.depth_anything_v2.dpt import DepthAnythingV2
import torch, numpy as np
from math import inf

model_configs = {
    'vits': {'encoder': 'vits', 'features': 64,  'out_channels': [48, 96, 192, 384]},
    'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
    'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]},
    'vitg': {'encoder': 'vitg', 'features': 384, 'out_channels': [1536, 1536, 1536, 1536]},
}

depth_anything = DepthAnythingV2(**model_configs['vitl'])
depth_anything.load_state_dict(
    torch.load("checkpoints/depth_anything_v2_vitl.pth", map_location='cpu')
)
depth_anything = depth_anything.to("cuda:0").eval()
```

```python
# Infer depth for a frame (H×W numpy RGB image)
import cv2
image = cv2.imread("frame.png")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

raw_depth = depth_anything.infer_image(image, input_size=512)  # numpy float32 (H, W)

# Normalise to [0, 255] and compute inverse depth for 3-D projection
depth_norm = (raw_depth - raw_depth.min()) / (raw_depth.max() - raw_depth.min()) * 255.0
depth_norm = depth_norm.astype(np.uint8)

depth_for_3D = 1.0 / depth_norm.astype(np.float32)
depth_for_3D[depth_for_3D == inf] = 1.0   # handle divide-by-zero at depth=0

# depth_for_3D[h, w] gives the relative inverse depth at pixel (w, h)
# Smaller value → farther away; larger value → closer to camera
print(depth_for_3D.shape)   # (512, 288)
print(depth_for_3D.min(), depth_for_3D.max())
```

--------------------------------

### Load and Infer with Depth Anything V2 Model

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Load a pre-trained Depth Anything V2 model for metric depth estimation and perform inference on an image. Ensure checkpoints are downloaded and placed in the 'checkpoints' directory.

```python
import cv2
import torch

from depth_anything_v2.dpt import DepthAnythingV2

model_configs = {
    'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]},
    'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]},
    'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]}
}

encoder = 'vitl' # or 'vits', 'vitb'
dataset = 'hypersim' # 'hypersim' for indoor model, 'vkitti' for outdoor model
max_depth = 20 # 20 for indoor model, 80 for outdoor model

model = DepthAnythingV2(**{**model_configs[encoder], 'max_depth': max_depth})
model.load_state_dict(torch.load(f'checkpoints/depth_anything_v2_metric_{dataset}_{encoder}.pth', map_location='cpu'))
model.eval()

raw_img = cv2.imread('your/image/path')
depth = model.infer_image(raw_img) # HxW depth map in meters in numpy
```

--------------------------------

### Initialize Drag Class for Video Generation

Source: https://context7.com/qiuyu96/levitor/llms.txt

Instantiates the main inference engine for Levitor, loading necessary model components like the ControlNet and Stable Video Diffusion pipeline.

```python
# gradio_demo/gradio_run.py
# Instantiation (done once at app startup):
drag_engine = Drag(device="cuda:0", args=args, model_length=16)

# The constructor loads:
#   - CLIPVisionModelWithProjection  (image encoder)
#   - AutoencoderKLTemporalDecoder   (VAE)
#   - UNetSpatioTemporalConditionModel (denoising UNet)
#   - ControlNetSVDModel             (trajectory ControlNet)
#   - StableVideoDiffusionPipelineControlNet (assembled pipeline)
```

--------------------------------

### Define CLI Arguments for LeviTor Demo

Source: https://context7.com/qiuyu96/levitor/llms.txt

This Python code defines a command-line argument parser for the LeviTor demo, specifying parameters for guidance scales, output directory, video dimensions, trajectory encoding, mixed precision, checkpoint paths, and memory optimization.

```python
# gradio_demo/gradio_run.py
import argparse

def get_args():
    parser = argparse.ArgumentParser()
    # Guidance
    parser.add_argument("--min_guidance_scale", type=float, default=1.0)
    parser.add_argument("--max_guidance_scale", type=float, default=3.0)
    parser.add_argument("--controlnet_cond_scale", type=float, default=1.0)
    # Output
    parser.add_argument("--output_dir", type=str, default="gradio_demo/outputs")
    parser.add_argument("--seed", type=int, default=42)
    # Video shape
    parser.add_argument("--num_frames", type=int, default=16)
    parser.add_argument("--frame_interval", type=int, default=1)
    parser.add_argument("--width", type=int, default=288)
    parser.add_argument("--height", type=int, default=512)
    # Trajectory heatmap radius
    parser.add_argument("--gaussian_r", type=int, default=10)
    # Mixed precision: "no" | "fp16" | "bf16"
    parser.add_argument("--mixed_precision", type=str, default=None,
                        choices=["no", "fp16", "bf16"])
    # Checkpoint paths (required)
    parser.add_argument("--pretrained_model_name_or_path", type=str, required=True)
    parser.add_argument("--resume_from_checkpoint", type=str, default="latest")
    parser.add_argument("--sam_path", type=str, default="")
    parser.add_argument("--depthanything_path", type=str, default="")
    # Memory optimisation
    parser.add_argument("--enable_xformers_memory_efficient_attention", action="store_true")
    return parser.parse_args()

args = get_args()
# args.width  → 288
# args.height → 512
# args.num_frames → 16

```

--------------------------------

### Image Preprocessing for Gradio Demo

Source: https://context7.com/qiuyu96/levitor/llms.txt

Handles image upload in the Gradio demo by resizing, saving, and computing an inverse-depth map using Depth Anything V2. Resets session state.

```python
# gradio_demo/gradio_run.py — called automatically on image upload

# The function:
#   1. Opens and bilinearly resizes the image to (args.width, args.height)
#   2. Saves it to args.output_dir/first_frame_<uuid>.png
#   3. Runs depth_anything.infer_image() on the RGB array
#   4. Normalises depth to [0,255] and stores inverse depth globally
#   5. Returns (display_path, state_path, empty_tracking_state)
```

--------------------------------

### Create LeviTor Checkpoint Directory

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Create a subdirectory named 'LeviTor' within the 'checkpoints' directory to organize LeviTor specific checkpoints.

```bash
mkdir LeviTor
cd LeviTor
```

--------------------------------

### Download CoTracker Checkpoints

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Downloads checkpoint files for different configurations of CoTracker. These are used for loading pre-trained models.

```bash
wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_4_wind_8.pth
wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_4_wind_12.pth
wget https://dl.fbaipublicfiles.com/cotracker/cotracker_stride_8_wind_16.pth
```

--------------------------------

### Run Inference Script on Images (Indoor)

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Execute the run script for indoor scene depth estimation using the 'vitl' encoder. Specify the path to the model checkpoint, maximum depth, input image path, and output directory.

```bash
# indoor scenes
python run.py \
  --encoder vitl \
  --load-from checkpoints/depth_anything_v2_metric_hypersim_vitl.pth \
  --max-depth 20 \
  --img-path <path> --outdir <outdir> [--input-size <size>] [--save-numpy]
```

--------------------------------

### Load and Run Online CoTracker

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Loads a pretrained CoTracker2 model for online video processing, which is more memory-efficient for long videos. The model is initialized for online processing and then iterated over video chunks.

```python
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2_online").to(device)

# Run Online CoTracker, the same model with a different API:
# Initialize online processing
cotracker(video_chunk=video, is_first_step=True, grid_size=grid_size)  

# Process the video
for ind in range(0, video.shape[1] - cotracker.step, cotracker.step):
    pred_tracks, pred_visibility = cotracker(
        video_chunk=video[:, ind : ind + cotracker.step * 2]
    )  # B T N 2,  B T N 1
```

--------------------------------

### Evaluate on TAP-Vid DAVIS

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Execute the evaluation script for TAP-Vid DAVIS. Ensure you provide the correct path to your dataset.

```bash
python ./cotracker/evaluation/evaluate.py --config-name eval_tapvid_davis_first exp_dir=./eval_outputs dataset_root=your/tapvid/path
```

--------------------------------

### Run Inference Script on Images (Outdoor)

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Execute the run script for outdoor scene depth estimation using the 'vitl' encoder. Specify the path to the model checkpoint, maximum depth, input image path, and output directory.

```bash
# outdoor scenes
python run.py \
  --encoder vitl \
  --load-from checkpoints/depth_anything_v2_metric_vkitti_vitl.pth \
  --max-depth 80 \
  --img-path <path> --outdir <outdir> [--input-size <size>] [--save-numpy]
```

--------------------------------

### Online Mode Usage

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

This snippet shows how to use the CoTracker model in online mode for more memory-efficient processing of longer videos. It covers initialization and step-by-step processing.

```APIDOC
## Online Mode

### Description
Load and use the CoTracker model in online mode for memory-efficient processing, suitable for long videos or streams.

### Method
`torch.hub.load("facebookresearch/co-tracker", "cotracker2_online")`

### Usage Example
```python
import torch
import imageio.v3 as iio

# Download the video
url = 'https://github.com/facebookresearch/co-tracker/blob/main/assets/apple.mp4'
frames = iio.imread(url, plugin="FFMPEG")  # or plugin="pyav"

device = 'cuda' # or 'cpu'
grid_size = 10
video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)  # B T C H W

# Load the online model
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2_online").to(device)

# Initialize online processing
cotracker(video_chunk=video, is_first_step=True, grid_size=grid_size)

# Process the video in chunks
for ind in range(0, video.shape[1] - cotracker.step, cotracker.step):
    pred_tracks, pred_visibility = cotracker(
        video_chunk=video[:, ind : ind + cotracker.step * 2]
    )  # B T N 2,  B T N 1
```

### Parameters
- `video_chunk` (torch.Tensor): A chunk of the input video tensor.
- `is_first_step` (bool): Set to True for the first chunk to initialize the online processing.
- `grid_size` (int): The size of the grid for point sampling.

### Attributes
- `step` (int): The processing step size for online mode.
```

--------------------------------

### cotracker.utils.visualizer

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/docs/source/apis/utils.rst

This section documents the functions available in the cotracker.utils.visualizer module. These utilities are used for visualization purposes within the CoTracker project.

--------------------------------

### Project 2D Images to Point Clouds

Source: https://github.com/qiuyu96/levitor/blob/main/gradio_demo/metric_depth/README.md

Use the depth_to_pointcloud script to project 2D images into 3D point clouds. Configure with the desired encoder, model checkpoint, maximum depth, input image path, and output directory.

```bash
python depth_to_pointcloud.py \
  --encoder vitl \
  --load-from checkpoints/depth_anything_v2_metric_hypersim_vitl.pth \
  --max-depth 20 \
  --img-path <path> --outdir <outdir>
```

--------------------------------

### Run Video Generation with Drag Class

Source: https://context7.com/qiuyu96/levitor/llms.txt

Executes the video generation process using the Drag class. This method takes trajectory data and inference parameters to produce GIF and MP4 outputs.

```python
# Signature:
# Drag.run(first_frame_path, tracking_points, controlnet_cond_scale,
#           motion_bucket_id, depth_input, points_num) -> str (gif path)

# tracking_points: gr.State wrapping a list of trajectories, each a list of [x, y] pixel coords
# depth_input:     JSON string, e.g. "[[0.3, 0.35, 0.4, ..., 0.6]]"  (one list per trajectory)
# points_num:      float ratio controlling number of K-Means control points (e.g. "0.5")
# controlnet_cond_scale: float, ControlNet conditioning strength (default 1.0)
# motion_bucket_id: int, SVD motion magnitude (1–180, default 100)

# Minimal programmatic example (bypassing Gradio state):
import ast, torch
from gradio_demo.gradio_run import Drag, args, depth_for_3D_initial, masks_selected

drag = Drag("cuda:0", args, model_length=16)

```

--------------------------------

### Create Conda Environment

Source: https://github.com/qiuyu96/levitor/blob/main/README.md

Create a new conda environment named 'LeviTor' with Python 3.9 and activate it. This isolates project dependencies.

```bash
conda create -n LeviTor python=3.9 -y
conda activate LeviTor
```

--------------------------------

### Offline Mode Usage

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

This snippet demonstrates how to load and use the CoTracker model in offline mode using torch.hub. It includes downloading a sample video and running the tracking.

```APIDOC
## Offline Mode

### Description
Load a pretrained CoTracker model and use it for offline video tracking.

### Method
`torch.hub.load("facebookresearch/co-tracker", "cotracker2")`

### Usage Example
```python
import torch
import imageio.v3 as iio

# Download the video
url = 'https://github.com/facebookresearch/co-tracker/blob/main/assets/apple.mp4'
frames = iio.imread(url, plugin="FFMPEG")  # or plugin="pyav"

device = 'cuda' # or 'cpu'
grid_size = 10
video = torch.tensor(frames).permute(0, 3, 1, 2)[None].float().to(device)  # B T C H W

# Load the model
cotracker = torch.hub.load("facebookresearch/co-tracker", "cotracker2").to(device)

# Run tracking
pred_tracks, pred_visibility = cotracker(video, grid_size=grid_size) # B T N 2,  B T N 1
```

### Parameters
- `video` (torch.Tensor): Input video tensor with shape (B, T, C, H, W).
- `grid_size` (int): The size of the grid for point sampling.
```

--------------------------------

### Visualize Predicted Tracks

Source: https://github.com/qiuyu96/levitor/blob/main/cotracker/project/README.md

Use the Visualizer class to visualize predicted tracks. Ensure you have the necessary video, prediction, and visibility data.

```python
from cotracker.utils.visualizer import Visualizer

vis = Visualizer(save_dir="./saved_videos", pad_value=120, linewidth=3)
vis.visualize(video, pred_tracks, pred_visibility)
```

--------------------------------

### Generate Per-Frame Heatmap Visualizations

Source: https://context7.com/qiuyu96/levitor/llms.txt

Renders a list of RGB heatmap images for each video frame. It stamps Gaussian blobs at interpolated trajectory positions and draws the path history.

```python
# gradio_demo/gradio_run.py — get_vis_image
from PIL import Image
import numpy as np, cv2

# points: list of trajectories, each trajectory is a list of (x, y, depth) tuples
# After interpolation each trajectory has exactly num_frames entries

example_points = [
    [(50 + i*5, 100 + i*8, 0.3 + i*0.02) for i in range(16)],   # trajectory 1
    [(200 + i*3, 200 - i*4, 0.6 - i*0.01) for i in range(16)],  # trajectory 2
]

vis_frames = get_vis_image(
    target_size=(512, 288),   # (height, width)
    points=example_points,
    side=10,                  # gaussian_r
    num_frames=16,
)

print(len(vis_frames))            # 16 — one PIL image per frame
print(vis_frames[0].size)         # (288, 512) — width × height
vis_frames[0].save("heatmap_frame0.png")
```

--------------------------------

### Drag Class

Source: https://context7.com/qiuyu96/levitor/llms.txt

Encapsulates the full LeviTor pipeline, loading model components and exposing methods for video generation.

```APIDOC
## `Drag` Class — Main Inference Engine

Encapsulates the full LeviTor pipeline: loads all model components via Hugging Face `accelerate`, builds the `StableVideoDiffusionPipelineControlNet`, and exposes a `run()` method that accepts a start-frame path plus trajectory data and returns a GIF output path.

### Initialization
```python
# Instantiation (done once at app startup):
from gradio_demo.gradio_run import Drag, args
drag_engine = Drag(device="cuda:0", args=args, model_length=16)

# The constructor loads:
#   - CLIPVisionModelWithProjection  (image encoder)
#   - AutoencoderKLTemporalDecoder   (VAE)
#   - UNetSpatioTemporalConditionModel (denoising UNet)
#   - ControlNetSVDModel             (trajectory ControlNet)
#   - StableVideoDiffusionPipelineControlNet (assembled pipeline)
```
```