Diffuman4D

https://github.com/zju3dv/diffuman4d
Admin
Diffuman4D is an official repository for 4D Consistent Human View Synthesis from Sparse-View Videos,...
Tokens:6,657
Snippets:28
Trust Score:7.9
License:Apache-2.0
Update:1 year ago
Show doc for...
Tokens:
Raw
### Download Example Data for Diffuman4D

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Downloads example test data for Diffuman4D, including images, foreground masks, skeletons, and camera information, from Hugging Face.

```python
python scripts/download/download_dataset.py --repo_id "krahets/diffuman4d_example" --types='["images", "fmasks", "skeletons", "cameras"]'
```

--------------------------------

### Sliding Window Iterative Sampler Setup (Python)

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This code initializes the SlidingIterativeSampler for generating novel views using diffusion models. It configures the diffusion pipeline, specifies output directories, and defines parameters for the sliding window, such as window size, stride, and alternation rounds for spatial and temporal consistency.

```python
from src.samplers.sliding_iterative_sampler import SlidingIterativeSampler
from src.diffusers.pipelines.diffuman4d.pipeline_diffuman4d import Diffuman4DPipeline
import torch

# Initialize diffusion pipeline
pipeline = Diffuman4DPipeline.from_pretrained(
    "krahets/Diffuman4D",
    torch_dtype=torch.bfloat16
).to("cuda")

# Create sliding sampler
sampler = SlidingIterativeSampler(
    dataset=dataset,
    pipelines=[pipeline],
    output_dir="./output/results/my_scene",
    # Sliding window configuration
    window_size=12,       # Number of samples per window
    sliding_stride=1,     # Step size between windows
    sliding_shift=0,      # Starting offset
    bidirectional=True,   # Slide forward and backward
    num_denoising_steps=1,    # Diffusion steps per window
    alternation_rounds=3,     # Spatial -> temporal -> spatial cycles
    guidance_scale=2.0,       # Classifier-free guidance strength
    # Sampling ranges
    spa_label_range=[0, 48, 1],  # All 48 cameras
    tem_label_range=[0, 150, 1], # All 150 frames
    input_spa_labels=[1, 13, 25, 37]  # 4 input cameras
)

# Run inference
sampler.inference()

# Sampling strategy:
# 1. Spatial pass: Generate novel cameras for each frame using input cameras
# 2. Temporal pass: Generate novel frames for each camera using generated spatial views
# 3. Repeat alternation for consistency refinement

```

--------------------------------

### Install Diffuman4D Environment

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Installs the necessary Python environment for Diffuman4D inference and data preprocessing using Conda and pip. It also installs the EasyVolcap library for 3D/4D reconstruction.

```shell
conda create -n diffuman4d python=3.12
conda activate diffuman4d
# for inference
pip install -r requirements.txt
# for 3D/4D reconstruction and data processing
pip install git+https://github.com/zju3dv/EasyVolcap.git --no-deps
```

--------------------------------

### SpaTemDataset Initialization and Sampling (Python)

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This code snippet shows how to initialize the SpaTemDataset for loading and preprocessing multi-view human video data. It includes examples for retrieving both spatial samples (multiple cameras, single frame) and temporal samples (single camera, multiple frames). The dataset handles camera parameters and skeleton conditioning.

```python
from src.data.spatem_dataset import SpaTemDataset
import torch

# Initialize dataset
dataset = SpaTemDataset(
    data_dir="./data/dna_rendering_processed",
    scene_label="0023_06",
    camera_path_pat="{data_dir}/{scene_label}/transforms.json",
    image_path_pat="{data_dir}/{scene_label}/images/{spa_label}/{tem_label}.webp",
    fmask_path_pat="{data_dir}/{scene_label}/fmasks/{spa_label}/{tem_label}.png",
    skeleton_path_pat="{data_dir}/{scene_label}/skeletons/{spa_label}/{tem_label}.webp",
    height=1024,
    width=1024,
    has_gt_target=True  # False: use skeleton as placeholder for missing views
)

# Get spatial sample (multiple cameras, single frame)
spatial_sample = dataset.get_item(
    scene_label="0023_06",
    spa_labels=["00", "12", "24", "36"],  # Target cameras
    tem_labels=["000000"],  # Single frame
    input_spa_labels=["01", "13", "25", "37"]  # Input cameras
)

# Sample structure:
# {
#   "images": Tensor[N, 3, H, W],      # Normalized to [-1, 1]
#   "fmasks": Tensor[N, 1, H, W],       # Binary masks
#   "skeletons": Tensor[N, 3, H, W],    # RGB skeleton maps
#   "Ks": Tensor[N, 3, 3],              # Camera intrinsics
#   "poses": Tensor[N, 4, 4],           # Camera extrinsics (world-to-camera)
#   "plucker_embeds": Tensor[N, 6, H, W],  # Ray direction embeddings
#   "relative_poses": Tensor[N, N, 12], # Relative camera transformations
#   "labels": List[(scene, camera, frame)]  # Sample identifiers
# }

# Get temporal sample (single camera, multiple frames)
temporal_sample = dataset.get_item(
    scene_label="0023_06",
    spa_labels=["24"],  # Single camera
    tem_labels=["000000", "000010", "000020"],  # Multiple frames
    input_spa_labels=["01", "13", "25", "37"]
)

# Temporal sample includes reference camera frame prepended
# Shape: [num_frames + 1, C, H, W] with first frame from nearest input camera
```

--------------------------------

### Install Requirements for DNA-Rendering Dataset Processing

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Installs necessary Python packages for processing the DNA-Rendering dataset. This includes libraries for Hugging Face Hub, datasets, PyArrow, Pandas, and the EasyVolcap library.

```sh
pip install -U huggingface_hub datasets pyarrow pandas
pip install git+https://github.com/zju3dv/EasyVolcap.git --no-deps
```

--------------------------------

### Reconstruct 3D Gaussian Splatting Model

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Trains a human 3DGS model using nerfstudio with the output from Diffuman4D inference. Requires prior installation of nerfstudio.

```shell
ns-train splatfacto --data "./output/results/dna_rendering_tiny/0023_06/transforms.json"
```

--------------------------------

### Download Dataset via Command Line Interface

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This bash command demonstrates the command-line interface for the `download_dataset.py` script. It allows users to specify the repository ID, output directory, desired scenes, and data types for downloading. This is a convenient way to manage dataset downloads for Diffuman4D.

```bash
# CLI usage
python scripts/download/download_dataset.py \
  --repo_id "krahets/dna_rendering_processed" \
  --out_dir "./data/dna_rendering_processed" \
  --scenes '["0007_01"]' \
  --types '["fmasks", "skeletons"]'

# Output structure:
# ./data/dna_rendering_processed/ 
# └── 0007_01/ 
#     ├── images/           # RGB images (48 cameras × 150 frames)
#     │   ├── 00/ 
#     │   │   ├── 000000.webp
#     │   │   └── ...
#     │   └── ...
#     ├── fmasks/           # Foreground masks
#     ├── skeletons/        # Skeleton visualization maps
#     ├── poses_2d/         # 2D keypoint projections (JSON)
#     ├── poses_3d/         # 3D keypoints (JSON)
#     ├── cameras/          # Camera parameters (EasyVolcap format)
#     ├── transforms.json   # nerfstudio camera format
#     └── sparse_pcd.ply    # Initial sparse point cloud

```

--------------------------------

### Command-Line Interface for Skeleton Visualization Map Generation

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Allows command-line execution of the skeleton visualization map generation. Users can specify input/output directories, canvas and map resolutions, and optionally filter by specific cameras and frames. Output maps are colored based on keypoint confidence.

```bash
# CLI usage
python scripts/preprocess/draw_skeleton.py \
  --kp2d_dir "./data/my_scene/poses_2d" \
  --out_kpmap_dir "./data/my_scene/skeletons" \
  --kp2d_canvas_shape "(1024,1024)" \
  --num_workers 16

# Process specific cameras and frames
python scripts/preprocess/draw_skeleton.py \
  --kp2d_dir "./data/my_scene/poses_2d" \
  --out_kpmap_dir "./data/my_scene/skeletons" \
  --spa_labels "[0,12,24,36]" \
  --tem_labels "[0,10,20,30]"

# Output: colored skeleton maps with depth-aware rendering
# Color intensity reflects keypoint confidence scores
# Skeleton connectivity follows COCO wholebody format (133 keypoints)
```

--------------------------------

### Command-Line Interface for 3D Skeleton Triangulation

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Executes the 3D skeleton triangulation process via the command line. Allows specifying camera calibration files, input 2D keypoint directories, and output directories for 3D and reprojected 2D keypoints. Supports range specifications for cameras and frames.

```bash
# CLI usage
python scripts/preprocess/triangulate_skeleton.py \
  --camera_path "./data/my_scene/transforms.json" \
  --kp2d_dir "./data/my_scene/poses_sapiens" \
  --out_kp3d_dir "./data/my_scene/poses_3d" \
  --out_kp2d_proj_dir "./data/my_scene/poses_2d" \
  --spa_label_range "[0,48,1]" \
  --tem_label_range "[0,150,1]"

# Output format (poses_3d/000000.json):
# {
#   "instance_info": [{
#     "keypoints": [[x1,y1,z1], [x2,y2,z2], ...],  # 133 COCO wholebody keypoints
#     "keypoint_reproj": [err1, err2, ...]  # Reprojection errors
#   }]
# }

# Output format (poses_2d/00/000000.json):
# {
#   "instance_info": [{
#     "keypoints": [[x1,y1], [x2,y2], ...],
#     "keypoint_depths": [d1, d2, ...],  # Depth from camera
#     "keypoint_scores": None  # Scores removed after projection
#   }]
# }
```

--------------------------------

### Full Preprocessing Pipeline (Python)

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This script demonstrates a full preprocessing pipeline for multi-view human video data. It includes steps for background removal, 2D keypoint detection, 3D skeleton triangulation, and skeleton visualization. Note that keypoint detection requires a separate environment activation.

```python
# Python equivalent of full pipeline
from scripts.preprocess.remove_background import remove_background
from scripts.preprocess.triangulate_skeleton import triangulate_skeleton
from scripts.preprocess.draw_skeleton import draw_skeleton

data_dir = "./data/my_scene"

# Step 1: Foreground segmentation
remove_background(
    images_dir=f"{data_dir}/images",
    out_fmasks_dir=f"{data_dir}/fmasks",
    model_name="ZhengPeng7/BiRefNet",
    batch_size=8
)

# Step 2: 2D keypoint detection (requires sapiens-lite environment)
# Must be run separately: conda activate sapiens_lite
# python scripts/preprocess/predict_keypoints.py ...

# Step 3: 3D triangulation and reprojection
triangulate_skeleton(
    camera_path=f"{data_dir}/transforms.json",
    kp2d_dir=f"{data_dir}/poses_sapiens",
    out_kp3d_dir=f"{data_dir}/poses_3d",
    out_kp2d_proj_dir=f"{data_dir}/poses_2d"
)

# Step 4: Render skeleton visualization
draw_skeleton(
    kp2d_dir=f"{data_dir}/poses_2d",
    out_kpmap_dir=f"{data_dir}/skeletons"
)
```

--------------------------------

### Download Pretrained Diffuman4D Model

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Manually downloads the pretrained Diffuman4D model from Hugging Face to a local directory, useful for avoiding network issues during inference.

```shell
hf download krahets/Diffuman4D --local-dir ./models/models--krahets--Diffuman4D
```

--------------------------------

### Execute Complete Data Preprocessing Pipeline

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Runs the entire data preprocessing pipeline from raw images to data ready for diffusion model inference. Users can specify the data directory and optionally select specific preprocessing actions to execute. Requires a specific input directory structure.

```bash
# Full preprocessing pipeline
bash scripts/preprocess/preprocess.sh \
  --data_dir "./data/my_scene"

# Run specific preprocessing steps
bash scripts/preprocess/preprocess.sh \
  --data_dir "./data/my_scene" \
  --actions "triangulate_skeleton,draw_skeleton"

# Required input structure:
# ./data/my_scene/
#   ├── images/
#   │   ├── 00/
#   │   │   ├── 000000.jpg
#   │   │   └── ...
#   │   └── ...
#   └── transforms.json  # Must include "camera_label" field per frame
```

--------------------------------

### Command-Line Interface for Background Removal

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Provides command-line arguments for the background removal script. It allows specifying input/output directories, the segmentation model, batch size, and GPU usage. Useful for scripting and batch operations.

```bash
# CLI usage
python scripts/preprocess/remove_background.py \
  --images_dir "./data/my_scene/images" \
  --out_fmasks_dir "./data/my_scene/fmasks" \
  --model_name "ZhengPeng7/BiRefNet" \
  --batch_size 8 \
  --skip_exists True

# Process with multiple GPUs
python scripts/preprocess/remove_background.py \
  --images_dir "./data/my_scene/images" \
  --out_fmasks_dir "./data/my_scene/fmasks" \
  --gpu_ids "(0,1,2,3)" \
  --batch_size 4
```

--------------------------------

### Execute Diffuman4D Inference from Command Line

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

These bash commands demonstrate how to run the Diffuman4D inference pipeline from the command line. They allow users to specify experiment configurations, data directories, and sampling ranges for generating 3D or 4D novel views. The output structure includes generated images, masks, skeletons, and camera parameters in nerfstudio format.

```bash
# Generate 3D novel views (4 input cameras -> 44 output cameras)
python inference.py \
  exp=demo_3d \
  data.scene_label=0023_06 \
  data.data_dir=./data/datasets--krahets--diffuman4d_example

# Generate 4D novel views (4 cameras * 150 frames -> 44 cameras * 150 frames)
python inference.py \
  exp=demo_4d \
  data.scene_label=0023_06 \
  data.data_dir=./data/datasets--krahets--diffuman4d_example \
  sampler.spa_label_range=[0,48,1] \
  sampler.tem_label_range=[0,150,1] \
  sampler.input_spa_labels=[1,13,25,37]

# Expected output structure:
# ./output/results/dna_rendering/0023_06/ 
#   ├── images/          # Generated RGB images
#   ├── fmasks/          # Foreground masks
#   ├── skeletons/       # Skeleton maps
#   └── transforms.json  # Camera parameters (nerfstudio format)

```

--------------------------------

### Train 3D Gaussian Splatting with Nerfstudio

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Command-line instruction to train a 3D Gaussian Splatting model using nerfstudio with data processed from Diffuman4D. It specifies the data path and model-specific parameters for culling alpha thresholds.

```bash
# Train 3DGS with nerfstudio
ns-train splatfacto \
  --data "./output/results/dna_rendering/0023_06/transforms.json" \
  --pipeline.model.cull-alpha-thresh 0.005 \
  --pipeline.model.continue-cull-post-densification False

# Expected output: 3D Gaussian Splatting model for novel view rendering

```

--------------------------------

### Configure Sliding Window Sampler for Low Memory

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Sets up an iterative sampler with a smaller window size and stride for reduced memory usage, suitable for single GPU configurations. It specifies data pipelines, directionality, and label ranges for spatial and temporal data.

```python
from diffuman4d.sampling import SlidingIterativeSampler

# Assuming 'dataset' and 'pipeline' are already defined
sampler_lowmem = SlidingIterativeSampler(
    dataset=dataset,
    pipelines=[pipeline],
    window_size=6,        # Smaller window
    sliding_stride=1,
    bidirectional=False,  # Single direction
    alternation_rounds=1, # No temporal refinement
    spa_label_range=[0, 44, 2],  # Sample every other camera
    tem_label_range=[0, 1, 1],   # Single frame (3D only)
    input_spa_labels=[1, 13, 25, 37]
)

```

--------------------------------

### Download and Extract DNA-Rendering Dataset

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This Python script, `download_dataset.py`, facilitates downloading and extracting the DNA-Rendering dataset. It supports downloading the entire dataset or specific scenes and data types (images, masks, skeletons, poses, cameras). Options for output directory, overwriting existing files, and batch size are available. This is crucial for preparing data for Diffuman4D processing.

```python
from scripts.download.download_dataset import download_and_extract_dataset

# Download entire dataset (~1TB)
download_and_extract_dataset(
    repo_id="krahets/dna_rendering_processed",
    out_dir="./data/dna_rendering_processed",
    types=["images", "fmasks", "skeletons", "poses_2d", "poses_3d", "cameras"],
    overwrite=False,
    batch_size=1024
)

# Download specific scenes and data types
download_and_extract_dataset(
    repo_id="krahets/dna_rendering_processed",
    out_dir="./data/dna_rendering_processed",
    scenes=["0007_01", "0023_06"],
    types=["fmasks", "skeletons", "cameras"],
    overwrite=False
)

# Download minimal example data for testing
download_and_extract_dataset(
    repo_id="krahets/diffuman4d_example",
    out_dir="./data/datasets--krahets--diffuman4d_example",
    types=["images", "fmasks", "skeletons", "cameras"]
)

```

--------------------------------

### Run Diffuman4D Preprocessing Scripts (Bash)

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Executes the data preprocessing pipeline for Diffuman4D. Supports running all scripts or specific actions like triangulation and skeleton drawing. Requires the data directory path as input.

```bash
bash scripts/preprocess/preprocess.sh --data_dir YOUR_DATA_DIR
```

```bash
bash scripts/preprocess/preprocess.sh --data_dir YOUR_DATA_DIR --actions triangulate_skeleton,draw_skeleton
```

--------------------------------

### Run Diffuman4D Inference Pipeline

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

This Python script executes the Diffuman4D inference pipeline to generate novel 3D and 4D views from sparse input cameras. It requires Hydra for configuration management and instantiates dataset, diffusion models, and a sampling runner. The output can be converted to the nerfstudio format for 3DGS reconstruction, with an option for evaluation.

```python
import hydra
from omegaconf import DictConfig
from src.samplers.sampling_runner import SamplingRunner

@hydra.main(version_base="1.3", config_path="configs", config_name="test.yaml")
def main(cfg: DictConfig):
    # Instantiate dataset with scene data
    dataset = hydra.utils.instantiate(cfg.data)

    # Load diffusion model pipeline
    pipelines = hydra.utils.instantiate(cfg.model)

    # Create sampler with sliding window configuration
    sampler = hydra.utils.instantiate(
        cfg.sampler,
        dataset=dataset,
        pipelines=pipelines
    )
    runner = SamplingRunner(sampler)

    # Generate novel views
    runner.inference()

    # Convert to nerfstudio format for 3DGS reconstruction
    runner.to_nerfstudio()

    # Optional: evaluate against ground truth
    if cfg.evaluating:
        runner.evaluate()

if __name__ == "__main__":
    main()

```

--------------------------------

### Convert Diffuman4D Results to Nerfstudio Format

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

A utility script to convert the output generated by Diffuman4D into a format compatible with nerfstudio for 3DGS/4DGS reconstruction. It organizes images, foreground masks, and camera transforms.

```python
from scripts.nerfstudio.diffuman4d_to_nerfstudio import diffuman4d_to_nerfstudio

# Convert generated images for nerfstudio training
diffuman4d_to_nerfstudio(
    data_dir="./data/dna_rendering_processed/0023_06",
    result_dir="./output/results/dna_rendering/0023_06",
    input_cameras=["01", "13", "25", "37"]  # Mark input cameras separately
)

# Output structure:
# ./output/results/dna_rendering/0023_06/
#   ├── images_alpha/         # RGBA images with alpha channel
#   ├── fmasks/               # Regenerated foreground masks
#   ├── transforms.json       # All cameras in nerfstudio format
#   ├── transforms_input.json # Input cameras only
#   └── sparse_pcd.ply        # Copied initial point cloud

```

--------------------------------

### Parse Camera Parameters from Nerfstudio/EasyVolcap Formats

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Parses camera intrinsic and extrinsic parameters from JSON files (nerfstudio or EasyVolcap format) and performs coordinate system conversion and scene normalization. It returns a dictionary of camera data.

```python
from src.data.utils.camera_parser import parse_cameras
import torch

# Parse nerfstudio format cameras
cameras = parse_cameras(
    camera_path="./data/my_scene/transforms.json",
    coord_system="opencv",  # or "opengl" for nerfstudio convention
    normalize_scene=True    # Center and scale scene to unit sphere
)

# Camera dictionary structure:
# {
#   "00": {
#     "K": Tensor[3, 3],          # Intrinsic matrix
#     "pose": Tensor[4, 4],       # Camera-to-world transformation
#     "width": int,
#     "height": int,
#     "camera_label": "00"
#   },
#   "01": {...},
#   ...
# }

# Access specific camera
cam = cameras["00"]
K = cam["K"]        # [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]
pose = cam["pose"]  # Camera-to-world matrix [R|t]
w2c = torch.linalg.inv(pose)  # World-to-camera

```

--------------------------------

### Render 2D Skeleton Keypoints as Visualization Maps

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Generates colored visualization maps from 2D skeleton keypoints, suitable for conditioning diffusion models. It renders keypoints with depth-aware coloring based on confidence scores. Inputs are 2D keypoint directories and desired output shapes; outputs are skeleton map images.

```python
from scripts.preprocess.draw_skeleton import draw_skeleton

# Draw skeleton maps from 2D keypoints
draw_skeleton(
    kp2d_dir="./data/my_scene/poses_2d",
    out_kpmap_dir="./data/my_scene/skeletons",
    kp2d_score_dir=None,  # Optional: override keypoint scores
    kp2d_canvas_shape=(1024, 1024),  # Input image resolution
    out_kpmap_shape=(1024, 1024),    # Output skeleton map resolution
    spa_labels=None,  # Process all cameras if None
    tem_labels=None,  # Process all frames if None
    image_ext=".webp",
    image_quality=85,
    num_workers=16,
    skip_exists=True
)
```

--------------------------------

### Run Diffuman4D Inference

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Executes Diffuman4D inference scripts to generate 3D or 4D image grids from input data. Supports different inference modes like 'demo_4d_tiny', 'demo_3d', and 'demo_4d'.

```shell
# generate a tiny 4D image grid (4 input cameras * 15 frames -> 44 cameras * 15 frames)
python inference.py exp=demo_4d_tiny data.scene_label=0023_06 data.data_dir=./data/datasets--krahets--diffuman4d_example

# generate a 3D image grid (4 input cameras -> 44 cameras)
python inference.py exp=demo_3d data.scene_label=0023_06 data.data_dir=./data/datasets--krahets--diffuman4d_example

# generate entire 4D image grid (4 input cameras * 150 frames -> 44 cameras * 150 frames)
python inference.py exp=demo_4d data.scene_label=0023_06 data.data_dir=./data/datasets--krahets--diffuman4d_example
```

--------------------------------

### Download Processed DNA-Rendering Dataset Labels

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Provides shell commands to download the re-annotated labels for the DNA-Rendering dataset. It includes options to download the entire dataset or specific scenes and data types.

```sh
# Download and extract the entire dataset
python scripts/download/download_dataset.py --out_dir "./data/dna_rendering_processed"

# Download specific scenes and data types
python scripts/download/download_dataset.py --out_dir "./data/dna_rendering_processed" --scenes '["0007_01"]' --types '["fmasks"]'
```

--------------------------------

### Coordinate System Conversion Between OpenGL and OpenCV

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Provides functions to convert camera transformation matrices between OpenGL/nerfstudio convention (+X right, +Y up, +Z backward) and OpenCV convention (+X right, +Y down, +Z forward) by flipping the Y and Z axes.

```python
import torch

def opengl_to_opencv(c2w_opengl):
    """Convert OpenGL/nerfstudio to OpenCV convention"""
    flip_yz = torch.diag(torch.tensor([1, -1, -1, 1], dtype=c2w_opengl.dtype))
    c2w_opencv = c2w_opengl @ flip_yz
    return c2w_opencv

def opencv_to_opengl(c2w_opencv):
    """Convert OpenCV to OpenGL/nerfstudio convention"""
    flip_yz = torch.diag(torch.tensor([1, -1, -1, 1], dtype=c2w_opencv.dtype))
    c2w_opengl = c2w_opencv @ flip_yz
    return c2w_opengl

```

--------------------------------

### Extract DNA-Rendering Dataset RGB Images

Source: https://github.com/zju3dv/diffuman4d/blob/main/README.md

Provides shell commands to extract RGB images from the raw DNA-Rendering dataset files. It requires specifying the raw data root and the processed data output directory, with options to process specific scenes.

```sh
# Extract images from all scenes in `processed_root`. You may replace `raw_root` with your own path.
# You can also specify scenes by passing `--scenes '["0007_01"]'`
python scripts/download/extract_dnar_images.py --raw_root "./data/dna_rendering_release_data" --processed_root "./data/dna_rendering_processed"
```

--------------------------------

### Triangulate 3D Skeleton from 2D Keypoints

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Triangulates 3D human skeletons from multi-view 2D keypoint detections. It projects these 3D skeletons back to all camera views for consistency. Requires camera calibration data and 2D keypoints. Outputs include 3D keypoints, optional point clouds, and reprojected 2D keypoints.

```python
from scripts.preprocess.triangulate_skeleton import triangulate_skeleton

# Triangulate 3D poses from multi-view 2D keypoints
triangulate_skeleton(
    camera_path="./data/my_scene/transforms.json",
    kp2d_dir="./data/my_scene/poses_sapiens",  # Input: 2D keypoints from Sapiens
    out_kp3d_dir="./data/my_scene/poses_3d",    # Output: 3D keypoints
    out_pcd_dir="./data/my_scene/poses_pcd",    # Optional: visualizable point clouds
    out_kp2d_proj_dir="./data/my_scene/poses_2d",  # Reprojected 2D keypoints
    spa_label_range=[0, 48, 1],  # Camera range for triangulation
    spa_label_proj_range=[0, 48, 1],  # Camera range for projection
    tem_label_range=[0, 150, 1],  # Frame range
    intri_scale=None,  # Scale intrinsics if needed
    skip_exists=True,
    num_workers=8
)
```

--------------------------------

### Remove Background from Images using BiRefNet or RMBG-2.0

Source: https://context7.com/zju3dv/diffuman4d/llms.txt

Extracts foreground human subjects from images using specified segmentation models. It supports batch processing and can output RGBA images. Dependencies include the preprocess.remove_background script. Inputs are image directories and model names; outputs are foreground masks and optionally RGBA images.

```python
from scripts.preprocess.remove_background import remove_background

# Batch process all images in a directory
remove_background(
    images_dir="./data/my_scene/images",
    out_fmasks_dir="./data/my_scene/fmasks",
    out_images_alpha_dir="./data/my_scene/images_alpha",  # Optional: save RGBA
    model_name="ZhengPeng7/BiRefNet",  # or "briaai/RMBG-2.0"
    image_ext=".webp",
    mask_ext=".png",
    rotate_clockwise=0,  # Rotate input images if needed
    batch_size=8,  # Adjust based on GPU memory (8 images ≈ 16GB)
    num_workers=4,
    skip_exists=True,
    gpu_ids=(0, 1)  # Use multiple GPUs
)
```
Tokens:
Raw