### Install ViTPose

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Clones the ViTPose repository and installs it.

```bash
pip install -U openmim
pip install --upgrade setuptools
mim install mmcv==1.3.9  # If error, try: pip install setuptools --upgrade
cd third_party/
git clone https://github.com/ViTAE-Transformer/ViTPose.git
cd ViTPose
pip install -v -e .
# If error above, do `pip install numpy cython wheel` first
cd ../..
```

--------------------------------

### Install Core Dependencies

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs specific versions of PyTorch and Viser for visualization.

```bash
# PyTorch (avoid 2.6 - it's unstable)
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124

# Viser for visualization
cd third_party/
git clone https://github.com/nerfstudio-project/viser
cd viser
pip install -e .
cd ../..
```

--------------------------------

### Base Dependencies Installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs essential build tools and PyTorch-related packages.

```bash
sudo apt install g++-11
export CC=/usr/bin/gcc-11
export CXX=/usr/bin/g++-11
pip install torch-scatter==2.1.2
```

--------------------------------

### Install Main Environment Dependencies

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs dependencies for the main environment from a requirements file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install VIMO

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs VIMO, a primary method for 3D human mesh.

```bash
pip install git+https://github.com/hongsukchoi/VIMO.git
```

--------------------------------

### Install BSTRO

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Clones the BSTRO repository for contact detection and installs it.

```bash
cd third_party/
git clone --recursive https://github.com/hongsukchoi/bstro.git
cd bstro
python setup.py build develop
cd ../..
```

--------------------------------

### GeoCalib Installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Clones the GeoCalib repository and installs it as an editable package.

```bash
git clone https://github.com/hongsukchoi/GeoCalib.git third_party/GeoCalib
cd third_party/GeoCalib
pip install -e .
cd ../..
```

--------------------------------

### xformers Installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs a specific version of xformers required for MegaSam.

```bash
wget https://anaconda.org/xformers/xformers/0.0.22.post7/download/linux-64/xformers-0.0.22.post7-py310_cu11.8.0_pyt2.0.1.tar.bz2
conda install xformers-0.0.22.post7-py310_cu11.8.0_pyt2.0.1.tar.bz2
rm xformers-0.0.22.post7-py310_cu11.8.0_pyt2.0.1.tar.bz2
```

--------------------------------

### Install VideoMimic Gym

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Commands to install the 'videomimic_gym' package.

```bash
cd videomimic_gym
pip install -e .
cd ..
```

--------------------------------

### Install Grounded-SAM-2

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Clones the Grounded-SAM-2 repository and installs its dependencies, including Segment Anything 2 and Grounding DINO.

```bash
mkdir third_party

cd third_party/
git clone https://github.com/hongsukchoi/Grounded-SAM-2.git
cd Grounded-SAM-2
export CUDA_HOME=/usr/local/cuda-12.4  # Adjust to your CUDA version
pip install -e .                        # Segment Anything 2
pip install --no-build-isolation -e grounding_dino  # Grounding DINO
pip install transformers
cd ../..
```

--------------------------------

### Install VideoMimic RL

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Commands to install the 'videomimic_rl' package.

```bash
cd videomimic_rl
pip install -e .
cd ..
```

--------------------------------

### Install Hand Pose Estimation (WiLor) (Optional)

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs WiLor for 3D hand mesh estimation.

```bash
# WiLor (3D hand mesh)
pip install git+https://github.com/warmshao/WiLoR-mini
```

--------------------------------

### Troubleshooting: g++-11 errors

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Provides steps to install g++-11 and set environment variables to resolve g++-11 related errors during installation.

```bash
# Install g++-11
sudo apt update
sudo apt install g++-11

# Set environment variables
export CC=/usr/bin/gcc-11
export CXX=/usr/bin/g++-11

# Retry the installation
pip install --no-build-isolation -e grounding_dino
```

--------------------------------

### DROID-SLAM Components Compilation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Navigates to the DROID-SLAM directory and installs its components.

```bash
cd base
python setup.py install
cd ../..
```

--------------------------------

### Install Align3r (Optional)

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs Monst3r/Align3r for world reconstruction, which is optional if only using MegaSam.

```bash
# Monst3r/Align3r (skip if only using MegaSam)
cd third_party/
git clone https://github.com/Junyi42/monst3r-depth-package.git
cd monst3r-depth-package
pip install -e .
cd ../..
pip install git+https://github.com/Junyi42/croco_package.git
```

--------------------------------

### Install MegaHunter and PyRoki

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs JAX with CUDA support, JAXLS, and PyRoki for robot motion retargeting.

```bash
# Second order optimization for MegaHunter and PyRoki
pip install -U "jax[cuda12]"
pip install "git+https://github.com/brentyi/jaxls.git"
# PyRoki for robot motion retargeting
git clone https://github.com/chungmin99/pyroki.git
cd pyroki
# pyroki might have updated some variable names; git checkout 70b30a56b1e1ea83fb4c2cac8fe2c63a0624b9ce 
pip install -e .
cd ../..
```

--------------------------------

### Install Isaac Gym

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Commands to install Isaac Gym after downloading and extracting the package.

```bash
cd isaacgym/python
pip install -e .
```

--------------------------------

### NKSR Installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs NKSR and its dependencies, including PyTorch Lightning and other Python packages.

```bash
conda install -c pyg -c nvidia -c conda-forge pytorch-lightning=1.9.4 tensorboard pybind11 pyg rich pandas omegaconf
pip install -f https://pycg.huangjh.tech/packages/index.html python-pycg[full]==0.5.2 randomname pykdtree pyntcloud
pip install nksr -f https://nksr.huangjh.tech/whl/torch-2.0.0+cu118.html
pip install trimesh tyro h5py rtree
cd ..
```

--------------------------------

### Quick Start - Complete Pipeline Automation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Run the complete pipeline with a single command, specifying video details and human height.

```bash
# Complete pipeline automation
./process_video.sh <video_name> <start_frame> <end_frame> <subsample_factor> g1 <height of the human>
# Example: ./process_video.sh my_video 0 100 2 g1 1.8
# If height is 0, it will use the SMPL shape parameters fitted to the robot.
# If height is -1, it will use the estimated SMPL shape parameters.
```

--------------------------------

### Install Neural Meshification (NDC) (Optional)

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Installs NDC for neural meshification, which is optional if only using NKSR.

```bash
# NDC (skip if only using NKSR)
pip install trimesh h5py cython opencv-python
cd third_party/NDC
python setup.py build_ext --inplace
cd ../..
```

--------------------------------

### Play - Basic Visualization Example

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Example command for basic visualization with the latest model.

```bash
python legged_gym/scripts/play.py --task=g1_deepmimic --num_envs 1
```

--------------------------------

### Activate the Virtual Environment

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Command to activate the newly created 'rlgpu' virtual environment.

```bash
conda activate rlgpu
```

--------------------------------

### Build Examples

Source: https://github.com/hongsukchoi/videomimic/blob/main/sim2real/README.md

Commands to build the examples within the repository, including setting up the build directory, running cmake with specific paths for ROS and torch/CUDA, and making the project.

```bash
mkdir build
cd build
```

```bash
export CMAKE_PREFIX_PATH=/home/unitree/.local/lib/python3.8/site-packages/torch:/home/unitree/noetic_ws/devel:/opt/ros/noetic
cmake .. -DCMAKE_POLICY_VERSION_MINIMUM=3.5 -DCMAKE_CUDA_ARCHITECTURES="87" -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
```

```bash
make
```

--------------------------------

### Multi-Human Support - Quick Example

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Enable multi-human processing with top-k selection for pose estimation and optimization.

```bash
# Enable multi-human processing with top-k selection
python stage0_preprocessing/vitpose_2d_poses.py ... --multihuman --top-k 3
python stage2_optimization/megahunter_optimization.py ... --multihuman --top-k 3
```

--------------------------------

### All Detected Humans Example

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

This example shows how to process all detected humans, with a caution about high memory usage.

```bash
# Process all detected humans (use with caution - high memory usage)
python stage0_preprocessing/vitpose_2d_poses.py ... --multihuman --top-k 0
```

--------------------------------

### Download and Install MiniConda

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Commands to download and install MiniConda, a lightweight distribution of Conda for creating virtual environments.

```bash
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
```

```bash
~/miniconda3/bin/conda init --all
source ~/.bashrc
```

--------------------------------

### Two-Person Interaction Example

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

This example shows how to process a video with two people interacting, including setting up multi-human flags for different stages.

```bash
# Process a video with two people interacting
./process_video.sh two_person_video 0 200 1 g1 0
# Then add multi-human flags to individual stages:
python stage0_preprocessing/vitpose_2d_poses.py ... --multihuman --top-k 2
python stage0_preprocessing/vimo_3d_mesh.py ... --multihuman --top-k 2
python stage2_optimization/megahunter_optimization.py ... --multihuman --top-k 2
```

--------------------------------

### Install PyTorch

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Command to install PyTorch, torchvision, torchaudio, and pytorch-cuda using Conda.

```bash
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia
```

--------------------------------

### Reconstruction Environment Setup

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Clones the megasam-package, creates the reconstruction environment from a YAML file, and activates it.

```bash
cd third_party/
git clone --recursive https://github.com/Junyi42/megasam-package
cd megasam-package

# Create environment from yaml
conda env create -f environment.yml

conda activate vm1recon

# other dependencies
cd ../..
pip install -r requirements.txt
cd third_party/megasam-package

# Additional dependencies
# cuda 11.8 is required
export CUDA_HOME=/usr/local/cuda-11.8

# Install g++-11 if not already installed
# sudo apt update

```

--------------------------------

### Group Scene (3-5 people) Example

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

This example demonstrates processing group scenes with 3-5 people, suggesting higher top-k for initial processing and potentially fewer for optimization due to memory constraints.

```bash
# For group scenes, process with higher top-k
python stage0_preprocessing/vitpose_2d_poses.py ... --multihuman --top-k 5
# But maybe optimize fewer for memory constraints
python stage2_optimization/megahunter_optimization.py ... --multihuman --top-k 3
```

--------------------------------

### Install Targets

Source: https://github.com/hongsukchoi/videomimic/blob/main/sim2real/thirdparty/CMakeLists.txt

Defines installation rules for libraries and include directories.

```cmake
## Install targets
file(GLOB DDS_LIBRARIES "${CMAKE_CURRENT_LIST_DIR}/lib/${CMAKE_SYSTEM_PROCESSOR}/*")

install(FILES ${DDS_LIBRARIES} DESTINATION ${CMAKE_INSTALL_LIBDIR})

install(DIRECTORY ${CMAKE_CURRENT_LIST_DIR}/include/
    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})
```

--------------------------------

### Troubleshooting: cuda-12.6 and grounding dino installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Provides a fix for errors during grounding dino installation related to cuda-12.6, suggesting a change from `value.type()` to `value.scalar_type()`.

```bash
When you see error for `pip install --no-build-isolation -e grounding_dino`, something like `error: command '/usr/local/cuda-12.6/bin/nvcc' failed with exit code 2`, see the error message and change `value.type()` to ` value.scalar_type()`.
```

--------------------------------

### Verify Isaac Gym Installation

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Command to run a verification script for Isaac Gym. A successful installation will display a window with 1080 balls falling.

```bash
cd examples
python 1080_balls_of_solitude.py
```

--------------------------------

### Quick Start - Frame Extraction

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Extract frames from a video to prepare for processing.

```bash
# Extract frames first
python utilities/extract_frames_from_video.py --video-path {video_name}.{extension} --output-dir ./demo_data/input_images/{video_name}/cam01 --start-frame 0 --end-frame 300
# This will save images with new indexing from 00001.jpg. The files should be named as 00001.jpg, 00002.jpg, ...
```

--------------------------------

### Main Environment Creation and Activation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Creates and activates the main conda environment named 'vm1rs'.

```bash
conda create -n vm1rs python=3.12
conda activate vm1rs
```

--------------------------------

### Setup

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_rl/README.md

Instructions for setting up the RSL RL library.

```bash
git clone https://github.com/leggedrobotics/rsl_rl
cd rsl_rl
pip install -e .
```

--------------------------------

### Environment Activation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/setup.md

Commands to activate the appropriate conda environments for different operations.

```bash
# Most operations
conda activate vm1rs

# For MegaSam reconstruction and postprocessing
conda activate vm1reocn
```

--------------------------------

### Install Targets

Source: https://github.com/hongsukchoi/videomimic/blob/main/sim2real/CMakeLists.txt

Defines installation rules for the SDK's include directories, library files, and CMake configuration files.

```cmake
## Install the library
install(DIRECTORY include/
    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR})

install(FILES ${UNITREE_SDK_LIB}
    DESTINATION ${CMAKE_INSTALL_LIBDIR})

install(FILES cmake/unitree_sdk2Targets.cmake
    DESTINATION lib/cmake/unitree_sdk2)

include(CMakePackageConfigHelpers)
write_basic_package_version_file(
    unitree_sdk2ConfigVersion.cmake
    VERSION "${${PROJECT_NAME}_VERSION_MAJOR}.${${PROJECT_NAME}_VERSION_MINOR}.${${PROJECT_NAME}_VERSION_PATCH}"
    COMPATIBILITY ExactVersion)

configure_file(cmake/unitree_sdk2Config.cmake.in unitree_sdk2Config.cmake @ONLY)
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/unitree_sdk2Config.cmake"
    "${CMAKE_CURRENT_BINARY_DIR}/unitree_sdk2ConfigVersion.cmake"
    DESTINATION lib/cmake/unitree_sdk2)
```

--------------------------------

### Installation Directories

Source: https://github.com/hongsukchoi/videomimic/blob/main/sim2real/CMakeLists.txt

Configures the output directories for binaries, libraries, and runtime files using GNUInstallDirs.

```cmake
## Use GNUInstallDirs to install libraries into correct locations on all platforms.
include(GNUInstallDirs)

## Put all binary files into /bin and libraries into /lib
set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR})
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR})
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_BINDIR})
```

--------------------------------

### Multi-GPU Training Command

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command to start multi-GPU training using torchrun.

```bash
torchrun --nproc-per-node <num_gpus> legged_gym/scripts/train.py --multi_gpu --task=xxx
```

--------------------------------

### Training Command

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command to start training the reinforcement learning model.

```bash
python legged_gym/scripts/train.py --task=xxx
```

--------------------------------

### Install and run MuJoCo Viewer

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/resources/robots/g1_description/README.md

Instructions to install the MuJoCo Python package and launch the viewer.

```bash
pip install mujoco
python -m mujoco.viewer
```

--------------------------------

### Play - Viser Visualization with Specific Model

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Example command for Viser visualization with a specific model and DeepMimic settings, useful for replaying dances.

```bash
python legged_gym/scripts/play.py \
  --task=g1_deepmimic_mocap \
  --env.viser.enable=True \
  --load_run TAG \
  --num_envs 1 \
  --env.deepmimic.amass_replay_data_path=lafan_replay_data/env_11_dance1_subject2.pkl \
  --headless
```

--------------------------------

### Create a New Virtual Environment

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/setup.md

Command to create a new Conda virtual environment named 'rlgpu' with Python 3.8.

```bash
conda create -n rlgpu python=3.8
```

--------------------------------

### Visualize Results

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Visualize complete results with ego-view rendering and an interactive GUI.

```bash
# Visualize complete results with ego-view rendering and interactive GUI
python visualization/complete_results_egoview_visualization.py \
    --postprocessed-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1 \
    --robot-name g1 \
    --bg-pc-downsample-factor 4 \
    --is-megasam \
    --save-ego-view
```

--------------------------------

### Whole MegaHunter Optimization Results Visualization

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Visualize whole MegaHunter optimization results.

```bash
python visualization/optimization_results_visualization.py \
    --world-env-path ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1/gravity_calibrated_megahunter.h5 \
    --bg-pc-downsample-factor 4
```

--------------------------------

### Sequential Optimization

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command for sequential optimization, which compiles JAX once and then processes all videos sequentially, supporting multi-human processing.

```bash
python sequential_processing/stage2_sequential_megahunter_optimization.py \
    --pattern <pattern> \
    --multihuman \
    --top-k 3
```

--------------------------------

### Stage 0: Preprocessing - Automated

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Runs SAM2, ViTPose, VIMO, and BSTRO in sequence for automated preprocessing.

```bash
# Runs SAM2, ViTPose, VIMO, and BSTRO in sequence
bash preprocess_human.sh <video_name> <vis_flag>
# Examples:
bash preprocess_human.sh my_video 1    # with visualization, which saves ID of humans, masks/bboxes and 2d poses of humans in the video
bash preprocess_human.sh my_video 0    # without visualization
```

--------------------------------

### Visualize Mesh Generation Results

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Interactive mesh and point cloud viewer with filtering controls.

```bash
# Interactive mesh and point cloud viewer with filtering controls
python visualization/mesh_generation_visualization.py \
    --mesh-path ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1/background_mesh.obj \
    --points-path ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1/background_more_filtered_colored_pointcloud.ply
```

--------------------------------

### Mesh Generation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Generate background mesh from gravity-calibrated data.

```bash
# Generate background mesh from gravity-calibrated data
conda activate vm1recon
python stage3_postprocessing/mesh_generation.py \
    --calibrated-megahunter-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1 \
    --conf-thr 0.0 \
    --gradient-thr 0.15 \
    --scale-bbox3d 3.0 \
    --is-megasam \
    --meshification-method nksr \
    --vis
    # Use --no-spf to meshify without spatiotemporal filtering (slower because it meshifies whole scene at once, but you will get more complete environment mesh)
    # Add --multihuman for multi-human scenes
```

--------------------------------

### Visualize Gravity Calibration Results

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Simple visualization of gravity-aligned human poses and world axes.

```bash
# Interactive visualization of gravity-aligned human poses and world axes
python visualization/gravity_calibration_visualization.py \
    --calib-out-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1
```

--------------------------------

### Sequential Processing for Multiple Videos (Stage 2)

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Script for sequential processing of multiple videos to amortize JAX compilation time during Stage 2 optimization.

```bash
python sequential_processing/stage2_sequential_megahunter_optimization.py --pattern {common substring of the video names} --use-g1-shape
```

--------------------------------

### SAM2 Human Segmentation Command

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/sequential_processing.md

Example command to run sequential SAM2 human segmentation.

```bash
conda activate vm1rs
python sequential_processing/stage0_sequential_sam2_segmentation.py \
    --video-base-dir ./demo_data/input_images \
    --output-base-dir ./demo_data/input_masks \
    --pattern "jump" \
    --start-idx 0 \
    --end-idx 10
```

--------------------------------

### Stage 3: Postprocessing Pipeline (Option A - Recommended)

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Command to execute the complete postprocessing pipeline, including gravity calibration and mesh generation.

```bash
# Switch to vm1recon environment for GeoCalib and NKSR meshification
conda activate vm1recon
python stage3_postprocessing/postprocessing_pipeline.py \
    --megahunter-path ./demo_data/output_smpl_and_points/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1.h5 \
    --out-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1 \
    --conf-thr 0.0 \
    --is-megasam \
    --scale-bbox3d 1.5 \
    --vis
    # Use --gradient-thr 0.15 for cleaner meshes
    # Use --no-spf to meshify without spatiotemporal filtering (slower)
    # Use --meshification-method 'nksr' (default) or 'ndc' (deprecated) for neural meshification
    # Add --multihuman for multi-human scenes
```

--------------------------------

### Sequential Postprocessing

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command for sequential postprocessing, including mesh generation and geocalib, with multi-human support and MegasAM compatibility.

```bash
python sequential_processing/stage3_sequential_mesh_generation_and_geocalib.py \
    --pattern <pattern> \
    --hunter-base-dir ./demo_data/output_smpl_and_points \
    --output-base-dir ./demo_data/output_calib_mesh \
    --multihuman \
    --is-megasam
```

--------------------------------

### Stage 2: Optimization with Multi-Human Support

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command for the optimization stage, enabling multi-human processing and specifying the top-k humans to process.

```bash
python stage2_optimization/megahunter_optimization.py \
    --world-env-path ./demo_data/input_megasam/megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1.h5 \
    --bbox-dir ./demo_data/input_masks/my_video/cam01/json_data \
    --pose2d-dir ./demo_data/input_2d_poses/my_video/cam01 \
    --smpl-dir ./demo_data/input_3d_meshes/my_video/cam01 \
    --out-dir ./demo_data/output_smpl_and_points \
    --multihuman \
    --top-k 3
```

--------------------------------

### Stage 1: Monst3r Depth Prior Reconstruction

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Command to perform initial reconstruction using Monst3r, focusing on depth prior.

```bash
conda activate vm1rs
python stage1_reconstruction/monst3r_depth_prior_reconstruction.py \
    --out-dir ./demo_data/input_align3r \
    --video-dir ./demo_data/input_images/my_video/cam01 \
    --start-frame 0 \
    --end-frame 100 \
    --stride 1 \
    --gsam2
    # Add --no-batchify for even less memory usage (slower)
```

--------------------------------

### Stage 3: Postprocessing with Multi-Human Support

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command for the postprocessing stage, including gravity calibration and mesh generation, with multi-human support enabled.

```bash
python stage3_postprocessing/postprocessing_pipeline.py \
    --megahunter-path ./demo_data/output_smpl_and_points/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1.h5 \
    --out-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1 \
    --multihuman \
    --is-megasam
```

--------------------------------

### Quick Pipeline

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/README.md

Commands to prepare data, run the complete pipeline, and visualize results.

```bash
# Prepare `demo_data` directory
# Extract frames from video
python utilities/extract_frames_from_video.py --video-path video.MOV --output-dir ./demo_data/input_images/video_name/cam01

# Run complete pipeline
./process_video.sh <video_name> <start_frame> <end_frame> <subsample_factor> g1 <height>
# Example: ./process_video.sh my_video 0 100 2 g1 1.8
# height -1 uses auto-detected height from the video (default)
# height 0 uses the g1's shape

# Visualize results
conda activate vm1rs
python visualization/complete_results_egoview_visualization.py \
    --postprocessed-dir ./demo_data/output_calib_mesh/<result_dir> \
    --robot-name g1 --is-megasam
```

--------------------------------

### Output Root Preparation

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/directory.md

Prepares the 'demo_data' directory, recommending a symbolic link to a hard drive due to the large size of output files.

```bash
ln -s /path/to/hard/drive/demo_data ./demo_data
```

--------------------------------

### Stage 0: Preprocessing with Multi-Human Support

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Commands for preprocessing steps including human segmentation (SAM2), 2D pose estimation (ViTPose), and 3D mesh estimation (VIMO), all with multi-human support enabled.

```bash
python stage0_preprocessing/sam2_segmentation.py \
    --video-dir ./demo_data/input_images/my_video/cam01 \
    --output-dir ./demo_data/input_masks/my_video/cam01 \
    --vis
```

```bash
python stage0_preprocessing/vitpose_2d_poses.py \
    --video-dir ./demo_data/input_images/my_video/cam01 \
    --bbox-dir ./demo_data/input_masks/my_video/cam01/json_data \
    --output-dir ./demo_data/input_2d_poses/my_video/cam01 \
    --multihuman \
    --top-k 3 \
    --vis
```

```bash
python stage0_preprocessing/vimo_3d_mesh.py \
    --img-dir ./demo_data/input_images/my_video/cam01 \
    --mask-dir ./demo_data/input_masks/my_video/cam01 \
    --out-dir ./demo_data/input_3d_meshes/my_video/cam01 \
    --multihuman \
    --top-k 3
```

--------------------------------

### Play - Viser Visualization

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command to enable Viser visualization for enhanced playback.

```bash
python legged_gym/scripts/play.py --task=xxx --env.viser.enable=True
```

--------------------------------

### Multi-Human Retargeting

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Command for multi-human retargeting.

```bash

```

--------------------------------

### Play - Basic Playback

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command to visualize training results in Gym using the play script.

```bash
python legged_gym/scripts/play.py --task=xxx
```

--------------------------------

### Robot Motion Retargeting

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Perform robot motion retargeting. The retargeting is very sensitive to the cost weights.

```bash
conda activate vm1rs
python stage4_retargeting/robot_motion_retargeting.py \
    --src-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1/ \
    --contact-dir ./demo_data/input_contacts/my_video/cam01 \
    --vis
```

--------------------------------

### Sequential Preprocessing

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Commands for sequential preprocessing steps, designed for efficiency by loading models once and processing videos one by one. Includes SAM2 segmentation, ViTPose, and VIMO with multi-human support.

```bash
python sequential_processing/stage0_sequential_sam2_segmentation.py \
    --pattern <pattern> \
    --video-base-dir ./demo_data/input_images \
    --output-base-dir ./demo_data/input_masks
```

```bash
python sequential_processing/stage0_sequential_vitpose_2d_poses.py \
    --pattern <pattern> \
    --multihuman \
    --top-k 3
```

```bash
python sequential_processing/stage0_sequential_vimo_3d_mesh.py \
    --pattern <pattern> \
    --multihuman \
    --top-k 3
```

--------------------------------

### Running Training - Stage 4 (RL Finetuning)

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/README.md

Script for Stage 4 training (RL Finetuning). Requires output from stage 3.

```bash
LOAD_RUN=stage_3_run_name # replace with output of stage 3
bashv ideomimic_gym/legged_gym/scripts/train_stage_4_rl_finetune.sh ${LOAD_RUN}
```

--------------------------------

### Visualize Monst3r Results

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Command to visualize the environment reconstruction results from Monst3r, including scale adjustment.

```bash
python visualization/environment_only_visualization.py \
    --world-env-path ./demo_data/input_align3r/align3r_reconstruction_results_my_video_cam01_frame_0_100_subsample_1.h5 \
    --world-scale-factor 5.0 \
    --conf-thr 1.5
```

--------------------------------

### Robot Retargeting with Multi-Human Support

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command to retarget motion for multiple humans using the stage4_retargeting script.

```bash
# Retarget motion for multiple humans
python stage4_retargeting/robot_motion_retargeting.py \
    --src-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1 \
    --contact-dir ./demo_data/input_contacts/my_video/cam01 \
    --multihuman \
    --top-k 3 \
    --vis
```

--------------------------------

### Running Training - MoCap Pre-training

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/README.md

Script for MoCap Pre-training.

```bash
bash videomimic_gym/legged_gym/scripts/train_stage_1_mcpt.sh
```

--------------------------------

### SAM2 Metadata Structure

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

The structure of the meta_data.json file used by SAM2 to store information about detected humans.

```json
{
  "all_instance_ids": [1, 2, 3],
  "sorted_by_avg_area": [2, 1, 3],
  "avg_areas": {
    "1": 15234.5,
    "2": 18956.2,
    "3": 12456.8
  },
  "frame_counts": {
    "1": 95,
    "2": 100,
    "3": 87
  }
}
```

--------------------------------

### Visualize Retargeted Robot Motion

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Command to visualize the retargeted robot motion alongside the original human motion and the environment.

```bash
python visualization/retargeting_visualization.py     --postprocessed-dir ./demo_data/output_calib_mesh/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1     --robot-name g1     --bg-pc-downsample-factor 4     --confidence-threshold 0.0
```

--------------------------------

### Multi-GPU RL Training

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command to initiate multi-GPU Reinforcement Learning training for the g1_deepmimic task.

```bash
torchrun --nproc-per-node 2 legged_gym/scripts/train.py --multi_gpu --task=g1_deepmimic \
  --headless --wandb_note "new_ft_old_terrains" \
  --env.deepmimic.use_amass=False \
  --load_run 20250225_132031_g1_deepmimic --resume \
  --env.deepmimic.use_human_videos=True \
  --env.deepmimic.human_video_oversample_factor=10 \
  --env.terrain.n_rows=6 \
  --train.policy.re_init_std=True \
  --train.policy.init_noise_std=0.5 \
  --train.algorithm.learning_rate=2e-5 \
  --env.deepmimic.n_append=50 \
  --env.deepmimic.link_pos_error_threshold=0.5 \
  --env.deepmimic.init_velocities=True \
  --env.deepmimic.randomize_start_offset=True \
  --env.asset.terminate_after_large_feet_contact_forces=False \
  --env.asset.use_alt_files=False
```

--------------------------------

### Running Training - Stage 3 (Distillation)

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/README.md

Script for Stage 3 training (Distillation). Requires output from stage 2.

```bash
LOAD_RUN=stage_2_run_name # replace with output of stage 2
bash videomimic_gym/legged_gym/scripts/train_stage_3_distillation.sh ${LOAD_RUN}
```

--------------------------------

### Visualization of Optimization Results

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/multihuman.md

Command to visualize optimization results, which automatically displays all or top-k humans with different colors.

```bash
python visualization/optimization_results_visualization.py \
    --world-env-path ./demo_data/output_smpl_and_points/megahunter_megasam_reconstruction_results_my_video_cam01_frame_0_100_subsample_1.h5 \
    --bg-pc-downsample-factor 4
```

--------------------------------

### Policy Cloning (DAgger)

Source: https://github.com/hongsukchoi/videomimic/blob/main/simulation/videomimic_gym/README.md

Command for policy cloning using DAgger for the g1_deepmimic_dagger task.

```bash
torchrun --nproc_per_node 2 legged_gym/scripts/train.py --task=g1_deepmimic_dagger \
  --multi_gpu --headless --wandb_note "distill" \
  --env.deepmimic.use_amass=False \
  --env.terrain.n_rows=10 \
  --env.deepmimic.amass_terrain_difficulty=1 \
  --env.deepmimic.use_human_videos=True \
  --env.deepmimic.init_velocities=True \
  --env.deepmimic.randomize_start_offset=True \
  --env.rewards.scales.feet_orientation=0.0 \
  --env.control.beta=1.0 \
  --train.runner.save_interval=50 \
  --train.algorithm.policy_to_clone_jitted=False \
  --train.algorithm.policy_to_clone=logs/g1_deepmimic/20250317_152046_g1_deepmimic \
  --train.algorithm.bc_loss_coef=1.0 \
  --train.algorithm.learning_rate=1e-4 \
  --env.deepmimic.n_append=50 \
  --env.asset.terminate_after_large_feet_contact_forces=False \
  --num_envs 2048
```

--------------------------------

### Stage 2: Optimize SMPL Shape for Specific Height

Source: https://github.com/hongsukchoi/videomimic/blob/main/real2sim/docs/commands.md

Optional command to customize human body shape (SMPL) for a specific height.

```bash
# Customize human body shape for specific height
python stage2_optimization/optimize_smpl_shape_for_height.py \
    --height 1.8 \
    --output-dir ./demo_data/input_3d_meshes/my_video/cam01 \
    --vis
```