### Install System and Project Dependencies

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Install system-level dependencies like libopenexr-dev and ffmpeg, followed by the project's main dependencies from requirements.txt. This step assumes a CUDA 11.5/11.7 environment.

```bash
sudo apt install libopenexr-dev ffmpeg -y
pip install -r requirements.txt
```

--------------------------------

### Setup JupyterLab Extensions and Kernel

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Install necessary jupyterlab extensions for annotation and visualization, and then install the project's Python kernel. Ensure you are using the Python executable from the project's conda environment.

```bash
jupyter labextension install jupyterlab-plotly ipyevents
python -m ipykernel install --user --name=dycheck
```

--------------------------------

### GIN Configuration Setup

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Sets up the GIN configuration files and bindings for the annotation process, including sequence-specific settings.

```python
GIN_CONFIGS = [f"../configs/{DATASET}/annotate_keypoints.gin"]
GIN_BINDINGS = [f'SEQUENCE="{SEQUENCE}"'] + PREDEFINED_BINDINGS_MAP[DATASET][
    SEQUENCE
]
```

--------------------------------

### Download Checkpoint and Evaluate HyperNeRF Model

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

This example demonstrates how to download a specific checkpoint for the HyperNeRF model trained on 'peel-banana' and then evaluate it. Skip the download step if all checkpoints are already obtained.

```bash
# Download the checkpoint. Skip it if you have downloaded all checkpoints.
bash scripts/download_single_checkpoint.sh <DRIVE_NAME> \
    hypernerf peel-banana ambient mono

# Evaluate the checkpoint.
CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/launch.py \
    --gin_configs "configs/hypernerf/ambient/mono.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="peel-banana"'
```

--------------------------------

### Record3D Capture File Structure

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

This illustrates the expected directory structure for exported data from Record3D captures. It differentiates between single-camera and multi-camera setups, highlighting the location of RGBD videos, metadata, and raw point cloud files.

```bash
.
├── RGBD_Video  # Export from the training camera.
│   ├── <TIME_STEP_TRAIN>.mp4
│   └── metadata.json
├── sound.m4a
├── Test  # Optional, only for multi-camera captures.
│   ├── 0  # Export from the testing 0 camera.
│   │   ├── RGBD_Video
│   │   │   ├── <TIME_STEP_TEST_0>.mp4
│   │   │   └── metadata.json
│   │   └── sound.m4a
│   └── 1  # Export from the testing 1 camera.
│       ├── RGBD_Video
│       │   ├── <TIME_STEP_TEST_1>.mp4
│       │   └── metadata.json
│       └── sound.m4a
└── Zipped_PLY  # Contain depth information from the training camera.
    └── <TIME_STEP_TRAIN>.zip
```

--------------------------------

### Process Co-visibility Masks

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

Generates co-visibility masks for evaluation using specified configuration files and sequence name. This is particularly relevant for multi-camera setups.

```python
python tools/process_covisible.py \
    --gin_configs 'configs/iphone/process_covisible.gin' \
    --gin_configs 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Parse Configuration and Print

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Parses the specified Gin configuration files and bindings, then prints the resulting configuration string. Use this to apply settings before initializing processors.

```python
with gin.unlock_config():
    core.parse_config_files_and_bindings(
        config_files=GIN_CONFIGS,
        bindings=GIN_BINDINGS,
        skip_unknown=True,
        master=False,
    )
config_str = gin.config_str()
print(f"*** Configuration:\n{config_str}")
```

--------------------------------

### Train and Evaluate Model from Scratch

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use this command to train a model from scratch and evaluate it. Replace placeholders like <DATASET>, <MODEL>, and <SEQUENCE> with your specific values.

```bash
# <DATASET> in [nerfies, hypernerf]
# <MODEL> in [tnerf, dense, ambient]
# <SEQUENCE> in [broom, curls, tail, toby-sit] if <DATASET> == nerfies
# <SEQUENCE> in [3dprinter, chicken, peel-banana] if <DATASET> == hypernerf

# Train model from scratch and evaluate.
python tools/launch.py \
    --gin_configs "configs/<DATASET>/<MODEL>/intl.gin" \
    --gin_bindings "Config.engine_cls=@Trainer" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Launch Demo Task

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Execute a demo task using the provided Python script. Replace <TASK> with the desired task name, such as 'novel_view', 'stabilized_view', or 'bullet_time'.

```bash
# Launch a demo task.
python demo/launch.py --task <TASK>
```

--------------------------------

### Train and Evaluate Model on iPhone Dataset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use this command to train a model from scratch and evaluate it. You can also specify different <SETTING> values.

```bash
# Train model from scratch and evaluate. You can also try different <SETTING>
in [base, randbkgd, randbkgd_depth, randbkgd_depth_dist].
python tools/launch.py \
    --gin_configs "configs/iphone/<MODEL>/randbkgd_depth_dist.gin" \
    --gin_bindings "Config.engine_cls=@Trainer" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Define Configuration Parameters

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Sets up configuration parameters for processing Record3D data, including the path to configuration files and specific dataset bindings.

```python
GIN_CONFIGS = ["../configs/iphone/process_record3d_to_iphone.gin"]
GIN_BINDINGS = [
    'DATA_ROOT="/home/hangg/datasets/iphone/record3d"',
    'SEQUENCE="paper-windmill"',
    # Dummy prompts since they will not be used.
    "FRGD_PROMPTS=['']",
    "Record3DProcessor.suppress_bad_frames_validation=True",
]
```

--------------------------------

### Combine Record3D AV Channels for Synchronization

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

Use this script to combine audio and video channels for temporal synchronization. Ensure the gin config and data root are correctly specified.

```bash
python tools/combine_record3d_av.py \
   --gin_configs configs/iphone/combine_record3d_av.gin \
   --gin_bindings 'combine_record3d_av.Config.data_root="/shared/hangg/datasets/iphone/record3d"' \
   --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Train and Evaluate iPhone Dataset (Teddy Sequence)

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Train a model from scratch and evaluate it for the iPhone dataset, specifically for the teddy sequence. This script is for reference.

```bash
# Train model from scratch and evaluate.
CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/trainval_iphone.sh \
    iphone/teddy
```

--------------------------------

### Download Nerfies and HyperNeRF Datasets

Source: https://github.com/kair-bair/dycheck/blob/main/docs/DATASETS.md

Use this script to download the combined Nerfies and HyperNeRF datasets and integrate additional data. Replace <DRIVE_NAME> with your Google Drive name.

```bash
# Download Nerfies and HyperNeRF datasets and inject our additional data.
bash scripts/download_nerfies_hypernerf_datasets.sh <DRIVE_NAME>
```

--------------------------------

### Evaluate and Train Dycheck Models

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use these commands to evaluate trained models or train models from scratch and evaluate them. Replace placeholders like <DATASET>, <MODEL>, and <SEQUENCE> with specific values. Ensure you have downloaded the necessary checkpoints.

```bash
# <DATASET> in [nerfies, hypernerf]
# <MODEL> in [tnerf, dense, ambient]
# <SEQUENCE> in [broom, curls, tail, toby-sit] if <DATASET> == nerfies
# <SEQUENCE> in [3dprinter, chicken, peel-banana] if <DATASET> == hypernerf

# Evaluate a trained model with all the metrics.
CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/launch.py \
    --gin_configs "configs/<DATASET>/<MODEL>/mono.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'

# Train model from scratch and evaluate.
CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/launch.py \
    --gin_configs "configs/<DATASET>/<MODEL>/mono.gin" \
    --gin_bindings "Config.engine_cls=@Trainer" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Download and Evaluate HyperNeRF Checkpoint

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

This snippet demonstrates how to download a specific checkpoint for the HyperNeRF model and then evaluate it. The `DRIVE_NAME` placeholder needs to be replaced with the actual drive name.

```bash
# Download the checkpoint. Skip it if you have downloaded all checkpoints.
bash scripts/download_single_checkpoint.sh <DRIVE_NAME> \
    hypernerf peel-banana ambient intl

# Evaluate the checkpoint.
CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/launch.py \
    --gin_configs "configs/hypernerf/ambient/intl.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="peel-banana"'
```

--------------------------------

### Apply GIN Configuration

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Applies the specified GIN configuration files and bindings to set up the core parameters for the annotation tool.

```python
with gin.unlock_config():
    core.parse_config_files_and_bindings(
        config_files=GIN_CONFIGS,
        bindings=GIN_BINDINGS,
        skip_unknown=True,
        master=False,
    )
config_str = gin.config_str()
print(f"*** Configuration:\n{config_str}")

config = Config()
```

--------------------------------

### Download All Checkpoints

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use this script to download all available checkpoints for the Dycheck project. Replace <DRIVE_NAME> with your Google Drive name.

```bash
bash scripts/download_all_checkpoints.sh <DRIVE_NAME>
```

--------------------------------

### Download iPhone Dataset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/DATASETS.md

Use this script to download the custom iPhone dataset. Replace <DRIVE_NAME> with your Google Drive name. Ensure each sequence is downloaded into the `datasets/iphone/<SEQUENCE>` directory.

```bash
# Download iPhone dataset.
bash scripts/download_iphone_dataset.sh <DRIVE_NAME>
```

--------------------------------

### Evaluate iPhone Dataset Video Rendering (Teddy Sequence)

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Only evaluate the checkpoint by rendering videos for the iPhone dataset, specifically for the teddy sequence. Ensure CUDA devices are correctly assigned.

```bash
# Only evaluate the checkpoint by rendering videos.
CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/val_iphone_video.sh \
    iphone/teddy
```

--------------------------------

### Download Checkpoint for iPhone Dataset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Download a specific checkpoint for evaluation. Skip if all checkpoints are already downloaded. Replace <DRIVE_NAME> with your drive name.

```bash
# Download the checkpoint. Skip it if you have downloaded all checkpoints.
bash scripts/download_single_checkpoint.sh <DRIVE_NAME> \
    iphone teddy ambient randbkgd_depth_dist
```

--------------------------------

### Evaluate Checkpoint on iPhone Dataset (Teddy Sequence)

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Evaluate a downloaded checkpoint for the HyperNeRF model trained on the teddy sequence. Ensure CUDA devices are correctly assigned.

```bash
# Evaluate the checkpoint.
CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/launch.py \
    --gin_configs "configs/iphone/ambient/randbkgd_depth_dist.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="teddy"'
```

--------------------------------

### Evaluate Nerfies/HyperNeRF Dataset by Rendering Videos

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

This command is used to evaluate a checkpoint by rendering videos for the Nerfies/HyperNeRF dataset. Specify the dataset and sequence, such as `hypernerf/peel-banana`.

```bash
# Only evaluate the checkpoint by rendering videos.
CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/val_nerfies_hypernerf_video.sh \
    hypernerf/peel-banana
```

--------------------------------

### Train and Evaluate Nerfies/HyperNeRF Dataset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use this command to train and evaluate the Nerfies/HyperNeRF dataset. It requires specifying the dataset and sequence, e.g., `hypernerf/peel-banana`.

```bash
# Train model from scratch and evaluate.
CUDA_VISIBLE_DEVICES=0,1,2,3 bash scripts/trainval_nerfies_hypernerf.sh \
    hypernerf/peel-banana
```

--------------------------------

### Download Single Checkpoint

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Download a specific checkpoint by providing dataset, sequence, model, and setting. Replace placeholders with actual values.

```bash
bash scripts/download_all_checkpoints.sh <DRIVE_NAME> \
    <DATASET> <SEQUENCE> <MODEL> <SETTING>
```

--------------------------------

### Evaluate Trained Model on iPhone Dataset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

Use this command to evaluate a trained model with all metrics. Replace <MODEL> and <SEQUENCE> with appropriate values.

```bash
# <MODEL> in [tnerf, dense, ambient]
# <SEQUENCE> in [apple, block, paper-windmill, space-out, spin, teddy, wheel]

# Evaluate a trained model with all the metrics.
python tools/launch.py \
    --gin_configs "configs/iphone/<MODEL>/randbkgd_depth_dist.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Process EMF using Command-Line Script

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Processes EMF from a video input using a command-line script. Requires a preprocessed dataset in Nerfies' data format and a process config file.

```bash
python tools/process_emf.py \
    --gin_configs 'configs/<DATASET>/process_emf.gin' \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Import Libraries

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Imports essential libraries for data processing, configuration management, and image manipulation.

```python
import functools
import os.path as osp
from typing import List

import cv2
import gin
import mediapy as media
import numpy as np

from dycheck import core
from dycheck.datasets import Record3DProcessor
from dycheck.utils import annotation, common, image, io

gin.enter_interactive_mode()
```

--------------------------------

### Initialize Record3D Processor

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Initializes the Record3DProcessor and processes the validation RGBA images. This step is crucial for preparing the dataset for further analysis.

```python
processor = Record3DProcessor()
processor._process_val_rgbas()
```

--------------------------------

### Import Libraries

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Imports essential libraries for data manipulation, machine learning, and visualization.

```python
import dataclasses
import functools
import os.path as osp
from typing import Callable, List, Literal, Union

import cv2
import gin
import jax
import mediapy as media
import numpy as np

from dycheck import core
from dycheck.datasets import Parser
from dycheck.utils import annotation, common, image, io, visuals

gin.enter_interactive_mode()
```

--------------------------------

### Evaluate Trained Model with All Metrics

Source: https://github.com/kair-bair/dycheck/blob/main/docs/BENCHMARK.md

This command evaluates a trained model using all available metrics. Ensure that the <DATASET>, <MODEL>, and <SEQUENCE> placeholders are correctly set.

```bash
# Evaluate a trained model with all the metrics.
python tools/launch.py \
    --gin_configs "configs/<DATASET>/<MODEL>/intl.gin" \
    --gin_bindings "Config.engine_cls=@Evaluator" \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Visualize Annotated Keypoints

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Displays images with annotated keypoints overlaid. Ensure keypoints are fully annotated before visualization.

```python
assert len(keypoints) == 10, "Annotation not finished yet."

media.show_images(
    jax.tree_map(
        lambda kps, img: visuals.visualize_kps(
            kps, img, skeleton=skeleton, kp_radius=6
        ),
        list(keypoints),
        list(rgbs),
    ),
    height=256,
)
```

--------------------------------

### process_emf.py script

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

A command-line script to process EMFs given a video input and a configuration file.

```APIDOC
## process_emf.py

### Description
A command-line script to process EMFs given a video input and a configuration file. Requires the dataset to be preprocessed in Nerfies' data format.

### Method
Command-line script

### Endpoint
`tools/process_emf.py`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **--gin_configs** (string) - Required - Path to the process config file (e.g., `configs/<DATASET>/process_emf.gin`).
- **--gin_bindings** (string) - Required - Gin bindings for configuration (e.g., `SEQUENCE="<SEQUENCE>"`).

### Request Example
```bash
python tools/process_emf.py \
    --gin_configs 'configs/<DATASET>/process_emf.gin' \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

### Response
#### Success Response (200)
- **Output**: Processed EMF values and related information.

#### Response Example
(No specific response example provided in the source text.)
```

--------------------------------

### Configure rclone for Google Drive Access

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Set up rclone to access Google Drive for downloading datasets and checkpoints. This is a one-time configuration step.

```bash
rclone config
```

--------------------------------

### Record Temporal Synchronization Offset

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

Record the audio-based temporal offset inferred from Adobe Premiere Pro. This offset is crucial for synchronizing testing cameras. The offset should be an integer.

```bash
# <OFFSET> is an integer inferred from Adobe Promiere Pro audio-based synchronization.
# Repeat for all of your testing cameras.
echo <OFFSET> > <TEST_VIDEO_ID>/RGBD_Video/timesync_offset.txt
```

--------------------------------

### Load and Process Data

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Loads data using the configured parser, selecting a subset of frames and cameras, and extracting RGB information.

```python
parser = config.parser_cls()
frame_names, time_ids, camera_ids = jax.tree_map(
    lambda a: common.strided_subset(a, 10), parser.load_split(config.split)
)
rgbs = np.array(common.parallel_map(parser.load_rgba, time_ids, camera_ids))[..., :3]
```

--------------------------------

### Configuration Class

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Defines the configuration structure for the annotation tool, specifying the parser, skeleton class, and data split.

```python
@gin.configurable(module="annotate_keypoints")
@dataclasses.dataclass
class Config(object):
    parser_cls: Callable[..., Parser] = gin.REQUIRED
    skeleton_cls: Union[
        visuals.Skeleton, Callable[..., visuals.Skeleton]
    ] = gin.REQUIRED
    split: str = gin.REQUIRED
```

--------------------------------

### Process Record3D Capture

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

Initiates the processing of a Record3D capture sequence. Ensure the sequence name is correctly specified.

```bash
bash scripts/process_record3d_to_iphone.sh <SEQUENCE>
```

--------------------------------

### Display Images

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Displays the loaded RGB images using the mediapy library with a specified height.

```python
media.show_images(rgbs, height=256)
```

--------------------------------

### Annotate Keypoints

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Annotates keypoints on the loaded RGB images using the configured skeleton class and a specified radius.

```python
skeleton = config.skeleton_cls()
keypoints = annotation.annotate_keypoints(rgbs, skeleton, kp_radius=6)
```

--------------------------------

### Generate Co-visibility Mask

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

This command-line script processes sequences to generate co-visibility masks. It requires a Gin configuration file and specific bindings for sequence processing.

```bash
python tools/process_covisible.py \
    --gin_configs 'configs/<DATASET>/process_covisible.gin' \
    --gin_bindings 'SEQUENCE="<SEQUENCE>"'
```

--------------------------------

### Verify rclone Google Drive Access

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Verify that rclone can access the 'dycheck-release' folder on your Google Drive. Replace `<DRIVE_NAME>` with your rclone Google Drive profile name.

```bash
rclone lsd --drive-shared-with-me "<DRIVE_NAME>:/dycheck-release"
```

--------------------------------

### Load IPython Extensions

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Loads necessary IPython extensions for interactive development and automatic reloading of modules.

```python
%load_ext autoreload
%autoreload 2
%matplotlib inline
```

--------------------------------

### Unified JAX Implementation of NeRF Variants

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Provides a unified JAX implementation for T-NeRF, Nerfies, and HyperNeRF, including several training enhancements. This code is part of the core models for dynamic view synthesis.

```python
from dycheck.models.tnerf import TNeRF
from dycheck.models.nerfies import Nerfies
from dycheck.models.hypernerf import HyperNeRF
```

--------------------------------

### Compute Masked SSIM and LPIPS

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Use these functions to compute Structural Similarity Index (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) metrics on images, considering only pixels within a provided mask. LPIPS model is created on CPU for efficiency.

```python
from dycheck.core import metrics

# Masked SSIM using partial conv.
mssim = metrics.compute_ssim (
    img0,           # (H, W, 3) jnp.ndarray image in float32.
    img1,           # (H, W, 3) jnp.ndarray image in float32.
    mask,           # (H, W, 1) optional jnp.ndarray in float32 {0, 1}. The metric is computed only on the pixels with mask == 1.
)

# Masked LPIPS.
compute_lpips = metrics.get_compute_lpips()  # Create LPIPS model on CPU. We find it is fast enough for all of our experiments.
mlpips = compute_lpips(
    img0,           # (H, W, 3) jnp.ndarray image in float32.
    img1,           # (H, W, 3) jnp.ndarray image in float32.
    mask,           # (H, W, 1) optional jnp.ndarray in float32 {0, 1}. The metric is computed only on the pixels with mask == 1.
)
```

--------------------------------

### Create Conda Environment

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Create a new conda environment named 'dycheck' with Python 3.8. This isolates project dependencies.

```bash
conda create -n dycheck python=3.8 -y
```

--------------------------------

### Compute Percentage of Correct Keypoints (PCK)

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Calculate the Percentage of Correct Keypoints (PCK) metric given two sets of keypoints, image dimensions, and a threshold ratio. This is used for evaluating pose or landmark accuracy.

```python
from dycheck.core import metrics

pck = metrics.compute_pck (
    kps0,           # (J, 2) jnp.ndarray keypoints in float32.
    kps1,           # (J, 2) jnp.ndarray keypoints in float32.
    img_wh          # (Tuple[int, int]) image width and height.
    ratio,          # (float) threshold ratio.
)
```

--------------------------------

### Clone Dycheck Repository Recursively

Source: https://github.com/kair-bair/dycheck/blob/main/docs/SETUP.md

Clone the Dycheck repository, ensuring to include submodules for RAFT and DPT. Use the `--recursive` flag for a complete clone.

```bash
git clone https://github.com/KAIR-BAIR/dycheck --recursive
```

--------------------------------

### compute_full_emf

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Computes the Full EMF (Omega), which is the relative camera-scene motion ratio. This is generally applicable but can be noisy.

```APIDOC
## compute_full_emf

### Description
Computes the Full EMF (Omega), which is the relative camera-scene motion ratio. This is generally applicable but can be noisy.

### Method
Python API call

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **rgbs** (np.ndarray) - Required - (N, H, W, 3) np.ndarray video frames in either uint8 or float32.
- **cameras** (list) - Required - A sequence of N camera objects.
- **bkgd_points** (np.ndarray) - Required - (P, 3) np.ndarray for background points.

### Request Example
```python
from dycheck import processors

full_emf = processors.compute_full_emf(
    rgbs,
    cameras,
    bkgd_points,
)
```

### Response
#### Success Response (200)
- **full_emf** (float) - The computed Full EMF value.

#### Response Example
```json
{
  "full_emf": 1.2
}
```
```

--------------------------------

### Predefined Bindings Map

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Defines a map of predefined configurations for different datasets and sequences, specifying skeleton types and keypoint counts.

```python
PREDEFINED_BINDINGS_MAP = {
    "nerfies": {
        "broom": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "curls": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "toby-sit": [
            "SKELETON=@QuadrupedSkeleton",
        ],
        "tail": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=4",
        ],
    },
    "hypernerf": {
        "3dprinter": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "chicken": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=7",
        ],
        "peel-banana": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
    },
    "iphone": {
        "teddy": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "block": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=16",
        ],
        "wheel": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "apple": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=7",
        ],
        "paper-windmill": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=6",
        ],
        "space-out": [
            "SKELETON=@HumanSkeleton",
        ],
        "spin": [
            "SKELETON=@HumanSkeleton",
        ],
        "creeper": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=9",
        ],
        "backpack": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=10",
        ],
        "pillow": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=8",
        ],
        "handwavy": [
            "SKELETON=@UnconnectedSkeleton",
            "UnconnectedSkeleton.num_kps=10",
        ],
        "mochi-high-five": [
            "SKELETON=@QuadrupedSkeleton",
        ],
        "haru-sit": [
            "SKELETON=@QuadrupedSkeleton",
        ],
        "sriracha-tree": [
            "SKELETON=@QuadrupedSkeleton",
        ],
    },
}
```

--------------------------------

### Edit Preprocessing Script for New Sequence

Source: https://github.com/kair-bair/dycheck/blob/main/docs/RECORD3D_CAPTURE.md

Modify the provided shell script to specify parameters for your new Record3D sequence before preprocessing.

```bash
# Edit `scripts/process_record3d_to_iphone.sh` with your new sequence.
```

--------------------------------

### Sequence Selection and Parsing

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Selects a specific sequence for processing and parses it into dataset and sequence components.

```python
# SEQUENCE = "iphone/mochi-high-five"
# SEQUENCE = "iphone/block"
# SEQUENCE = "iphone/wheel"
# SEQUENCE = "iphone/apple"
# SEQUENCE = "iphone/sriracha-tree"
# SEQUENCE = "iphone/haru-sit"
# SEQUENCE = "iphone/creeper"
# SEQUENCE = "iphone/backpack"
# SEQUENCE = "iphone/pillow"
# SEQUENCE = "iphone/handwavy"
# SEQUENCE = "iphone/paper-windmill"
# SEQUENCE = "iphone/space-out"
# SEQUENCE = "iphone/spin"

# SEQUENCE = "nerfies/broom"
# SEQUENCE = "nerfies/curls"
# SEQUENCE = "nerfies/toby-sit"
# SEQUENCE = "nerfies/tail"

# SEQUENCE = "hypernerf/3dprinter"
# SEQUENCE = "hypernerf/chicken"
SEQUENCE = "hypernerf/peel-banana"

DATASET, SEQUENCE = SEQUENCE.split("/")
```

--------------------------------

### Compute Full EMF using Python API

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Calculates the relative camera-scene motion ratio (Full EMF). This method is generally applicable but requires optical flow and monocular depth prediction, which can be noisy.

```python
# Full EMF (Omega): relative camera-scene motion ratio.
full_emf = processors.compute_full_emf(
    rgbs,           # (N, H, W, 3) np.ndarray video frames in either uint8 or float32.
    cameras,        # A sequence of N camera objects.
    bkgd_points,    # (P, 3) np.ndarray for background points.
)
```

--------------------------------

### Save Annotated Keypoints and Skeleton

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_keypoints.ipynb

Saves the skeleton structure and individual keypoint annotations for each frame to JSON files. The output path is constructed based on data directory, factor, split, and frame names.

```python
assert len(keypoints) == 10, "Annotation not finished yet."

io.dump(
    osp.join(
        parser.data_dir,
        "keypoint",
        f"{parser.factor}x",
        config.split,
        "skeleton.json",
    ),
    skeleton.asdict(),
)
for i in range(len(keypoints)):
    io.dump(
        osp.join(
            parser.data_dir,
            "keypoint",
            f"{parser.factor}x",
            config.split,
            frame_names[i] + ".json",
        ),
        keypoints[i],
    )
```

--------------------------------

### Effective Multi-View Factors (EMF) JSON Format

Source: https://github.com/kair-bair/dycheck/blob/main/docs/DATASETS.md

This JSON structure contains pre-computed effective multi-view factors for training videos, including the full EMF and angular EMF.

```json
{
    // The training video split name.
    "train_intl": {
        // The Full EMF.
        "Omega": 7.377937316894531,
        // The Angular EMF in deg/sec.
        "omega": 212.57649834023107
    },
    ...
}
```

--------------------------------

### Adjust Rendering Chunk Size

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Modify the rendering chunk size to manage GPU memory usage, especially with fewer resources. Append this to your command for demo, evaluation, or training.

```bash
# Append rendering chunk at the end of your command. Set it to something
# smaller than the default 8192 in case of OOM.
... --gin_bindings="get_prender_image.chunk=<CHUNK_SIZE>"
```

--------------------------------

### compute_angular_emf

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Computes the Angular EMF (omega), which represents camera angular speed. This is recommended to try first.

```APIDOC
## compute_angular_emf

### Description
Computes the Angular EMF (omega), which represents camera angular speed. This is recommended to try first.

### Method
Python API call

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **orientations** (np.ndarray) - Required - (N, 3, 3) np.ndarray for world-to-camera transforms in the OpenCV format.
- **positions** (np.ndarray) - Required - (N, 3) np.ndarray for camera positions in world space.
- **fps** (int) - Required - Video FPS.
- **lookat** (Optional) - Optional camera lookat point. If None, will be computed by triangulating camera optical axes.

### Request Example
```python
from dycheck import processors

angular_emf = processors.compute_angular_emf(
    orientations,
    positions,
    fps,
    lookat=lookat,
)
```

### Response
#### Success Response (200)
- **angular_emf** (float) - The computed Angular EMF value.

#### Response Example
```json
{
  "angular_emf": 0.5
}
```
```

--------------------------------

### Extra Evaluation and Visualization Data JSON Format

Source: https://github.com/kair-bair/dycheck/blob/main/docs/DATASETS.md

This JSON structure provides extra information for evaluation and visualizations, including scene bounding box, downsampling factor, frame rate, look-at point, and up vector.

```json
{
    // Scene bounding box; computed from the original sequence in the
    // normalized coordinate.
    "bbox": [
        [
            -0.24016861617565155,
            -0.35341497925615145,
            -0.6217879056930542
        ],
        [
            0.2697989344596863,
            0.09984857437828297,
            0.42420864422383164
        ]
    ],
    // Video downsampling factor for training and evaluation.
    "factor": 4,
    // Video frame rate.
    "fps": 15,
    // The scene look-at point in the normalized coordinate.
    "lookat": [
        0.0018036571564152837,
        -0.07246068120002747,
        0.05924934893846512
    ],
    // The scene up vector in the normalized coordinate.
    "up": [
        -0.07553146034479141,
        -0.9961089491844177,
        0.045409008860588074
    ]
}
```

--------------------------------

### Compute Angular EMF using Python API

Source: https://github.com/kair-bair/dycheck/blob/main/README.md

Calculates the camera angular speed (Angular EMF). Recommended for use when a single look-at point is assumed. The lookat point is optional and can be computed if not provided.

```python
from dycheck import processors

# Angular EMF (omega): camera angular speed. We recommend trying it out first whenever possible.
angular_emf = processors.compute_angular_emf(
    orientations,   # (N, 3, 3) np.ndarray for world-to-camera transforms in the OpenCV format.
    positions,      # (N, 3) np.ndarray for camera positions in world space.
    fps,            # Video FPS.
    lookat=lookat,  # Optional camera lookat point. If None, will be computed by triangulating camera optical axes.
)
```

--------------------------------

### Data Split JSON Format

Source: https://github.com/kair-bair/dycheck/blob/main/docs/DATASETS.md

This JSON structure defines the data splits for training and evaluation, including frame names, camera IDs, and time IDs.

```json
{
    // The name of each frame; equivalent to <ITEM_ID>.
    "frame_names": ["left1_000000", ...],
    // The camera ID for indexing the physical camera.
    "camera_ids": [0, ...],
    // The time ID for indexing the time step.
    "time_ids": [0, ...]
}
```

--------------------------------

### Save Bad Frame Annotations

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Saves the identified bad frame annotations to a text file for each sequence in the validation set. The output path is constructed based on the data directory and sequence index.

```python
for i, bad_frames in enumerate(val_bad_frames):
    io.dump(
        osp.join(
            processor.data_dir,
            "Test",
            str(i),
            "RGBD_Video",
            "timesync_bad_frames.txt",
        ),
        bad_frames,
    )
```

--------------------------------

### Annotate Bad Frames

Source: https://github.com/kair-bair/dycheck/blob/main/tools/annotate_record3d_bad_frames.ipynb

Iterates through validation RGBA frames, downsamples them by a factor of 4 for faster processing, and annotates bad frames using the `annotate_record3d_bad_frames` function. This helps in identifying problematic frames in the dataset.

```python
val_bad_frames = []
for rgbas in processor.val_rgbas:
    # Downsample by 4 such that the image widget loads faster.
    bad_frames = annotation.annotate_record3d_bad_frames(
        common.parallel_map(
            lambda img: image.rescale(img, 1 / 4),
            list(rgbas[..., :3]),
        )
    )
    val_bad_frames.append(bad_frames)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.