GVHMR

https://github.com/zju3dv/gvhmr
Admin
GVHMR is a world-grounded human motion recovery system that reconstructs 3D human motion and pose...
Tokens:8,660
Snippets:53
Trust Score:7.9
Update:1 week ago
Show doc for...
Tokens:
Raw
### Run demo.py on example video and show output

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Runs the demo on the example tennis video in a Jupyter notebook using the -s flag. The output shows the expected log messages, including preprocessing times, GPU info, and rendering progress.

```python
# Run demo.
video_fn = f'{proj_root}/docs/example_video/tennis.mp4'
!python {proj_root}/tools/demo/demo.py --video={video_fn} -s

```

```text
Output:
[[36m09/14 17:37:05[0m][[32mINFO[0m] [Input]: /content/GVHMR/docs/example_video/tennis.mp4[0m
[[36m09/14 17:37:05[0m][[32mINFO[0m] (L, W, H) = (312, 812, 720)[0m
[[36m09/14 17:37:06[0m][[32mINFO[0m] [Output Dir]: outputs/demo/tennis[0m
[[36m09/14 17:37:06[0m][[32mINFO[0m] [Copy Video] /content/GVHMR/docs/example_video/tennis.mp4 -> outputs/demo/tennis/0_input_video.mp4[0m
Copy: 100% 312/312 [00:09<00:00, 33.08it/s]
[[36m09/14 17:37:17[0m][[32mINFO[0m] [GPU]: Tesla T4[0m
[[36m09/14 17:37:17[0m][[32mINFO[0m] [GPU]: _CudaDeviceProperties(name='Tesla T4', major=7, minor=5, total_memory=15102MB, multi_processor_count=40)[0m
[[36m09/14 17:37:17[0m][[32mINFO[0m] [Preprocess] Start![0m
YoloV8 Tracking: 100% 312/312 [00:27<00:00, 11.14it/s]
ViTPose: 100% 20/20 [00:42<00:00,  2.13s/it]
HMR2 Feature: 100% 20/20 [00:22<00:00,  1.11s/it]
[[36m09/14 17:39:36[0m][[32mINFO[0m] [Preprocess] End. Time elapsed: 139.28s[0m
[[36m09/14 17:39:36[0m][[32mINFO[0m] [HMR4D] Predicting[0m
[[36m09/14 17:39:36[0m][[32mINFO[0m] [EnDecoder] Use MM_V1_AMASS_LOCAL_BEDLAM_CAM for statistics![0m
[[36m09/14 17:39:38[0m][[32mINFO[0m] [PL-Trainer] Loading ckpt type: inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt[0m
[[36m09/14 17:39:40[0m][[32mINFO[0m] [HMR4D] Elapsed: 1.32s for data-length=10.4s[0m
Rendering Incam: 100% 312/312 [00:24<00:00, 12.71it/s]
Rendering Global: 100% 312/312 [00:16<00:00, 18.68it/s]
[[36m09/14 17:40:29[0m][[32mINFO[0m] [Merge Videos][0m

```

--------------------------------

### Install pycolmap with pip

Source: https://github.com/zju3dv/gvhmr/blob/main/hmr4d/utils/preproc/relpose/README.md

Install the pycolmap package using pip. This is the only setup step shown on the page.

```bash
pip install pycolmap
```

--------------------------------

### Install DPVO (optional)

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Optional commented-out instructions for installing DPVO, a third-party dependency. It downloads Eigen, installs torch-scatter, numba, and pypose, and sets CUDA_HOME to /usr/local/cuda-12 if not already set. This block is not executed by default.

```python
# # Install DPVO.  (Optional)
# %cd third-party/DPVO

# !wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
# !unzip -o eigen-3.4.0.zip -d thirdparty && rm -rf eigen-3.4.0.zip

# %pip install torch-scatter -f "https://data.pyg.org/whl/torch-2.3.0+cu121.html"
# %pip install numba pypose

# if 'cuda_home' not in locals():
#   cuda_home = '/usr/local/cuda-12'
#   if not Path(cuda_home).exists():
#     raise FileNotFoundError('CUDA_HOME for cuda 12.x not found!')

#   os.environ['CUDA_HOME'] = cuda_home
#   os.environ['PATH'] = os.environ['PATH'] + f':{cuda_home}/bin'

# %pip install -e .
# %cd {proj_root}
```

--------------------------------

### Clone and install GVHMR

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Clones the GVHMR repository and installs its dependencies. The installation may prompt a restart in Colab; click cancel and rerun the block. The project root is stored in the variable 'proj_root' for later use.

```python
import os
from pathlib import Path

# Clone the repo.
!git clone https://github.com/zju3dv/GVHMR
proj_root = str(Path('GVHMR').absolute())

# Install GVHMR. (If Colab asks you to restart, just click cancel and rerun the block.)
%cd {proj_root}
%pip install -r requirements.txt
%pip install -e .
```

--------------------------------

### Download GVHMR model checkpoints with aria2

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Sets up the project directory and downloads all required model checkpoints (SMPL, SMPLX, DPVO, GVHMR, HMR2, ViTPose, YOLO) using aria2. The output shows the download results and confirms each file was saved to the expected path under inputs/checkpoints. Note that mkdir reports 'File exists' if the directories are already present, and aria2 may already be installed.

```python
%cd {proj_root}
!mkdir inputs
!mkdir outputs

!apt install -y -qq aria2
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/SMPLer-X/resolve/main/SMPL_NEUTRAL.pkl -d inputs/checkpoints/body_models/smpl -o SMPL_NEUTRAL.pkl
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/SMPLer-X/resolve/main/SMPLX_NEUTRAL.npz -d inputs/checkpoints/body_models/smplx -o SMPLX_NEUTRAL.npz
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/GVHMR/resolve/main/dpvo/dpvo.pth -d inputs/checkpoints/dpvo -o dpvo.pth
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/GVHMR/resolve/main/gvhmr/gvhmr_siga24_release.ckpt -d inputs/checkpoints/gvhmr -o gvhmr_siga24_release.ckpt
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/GVHMR/resolve/main/hmr2/epoch%3D10-step%3D25000.ckpt -d inputs/checkpoints/hmr2 -o epoch=10-step=25000.ckpt
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/GVHMR/resolve/main/vitpose/vitpose-h-multi-coco.pth -d inputs/checkpoints/vitpose -o vitpose-h-multi-coco.pth
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/camenduru/GVHMR/resolve/main/yolo/yolov8x.pt -d inputs/checkpoints/yolo -o yolov8x.pt
```

```text
Output:
/content/GVHMR
mkdir: cannot create directory ‘inputs’: File exists
mkdir: cannot create directory ‘outputs’: File exists
aria2 is already the newest version (1.36.0-1).
0 upgraded, 0 newly installed, 0 to remove and 49 not upgraded.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
858436|[1;32mOK[0m  |       0B/s|inputs/checkpoints/body_models/smpl/SMPL_NEUTRAL.pkl

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
da93ed|[1;32mOK[0m  |       0B/s|inputs/checkpoints/body_models/smplx/SMPLX_NEUTRAL.npz

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
b2dfd7|[1;32mOK[0m  |       0B/s|inputs/checkpoints/dpvo/dpvo.pth

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
5ce060|[1;32mOK[0m  |       0B/s|inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d41b3e|[1;32mOK[0m  |       0B/s|inputs/checkpoints/hmr2/epoch=10-step=25000.ckpt

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
3fe18e|[1;32mOK[0m  |       0B/s|inputs/checkpoints/vitpose/vitpose-h-multi-coco.pth

Status Legend:
(OK):download completed.

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
4fe1eb|[1;32mOK[0m  |       0B/s|inputs/checkpoints/yolo/yolov8x.pt

Status Legend:
(OK):download completed.
```

--------------------------------

### Install optional DPVO dependency

Source: https://github.com/zju3dv/gvhmr/blob/main/docs/INSTALL.md

Installs the optional DPVO dependency. Downloads Eigen 3.4.0, installs torch-scatter for CUDA 12.1, numba, and pypose, sets CUDA_HOME and PATH, then installs DPVO in editable mode. Not recommended if fast inference speed is desired.

```bash
cd third-party/DPVO
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty && rm -rf eigen-3.4.0.zip
pip install torch-scatter -f "https://data.pyg.org/whl/torch-2.3.0+cu121.html"
pip install numba pypose
export CUDA_HOME=/usr/local/cuda-12.1/
export PATH=$PATH:/usr/local/cuda-12.1/bin/
pip install -e .
```

--------------------------------

### Set up GVHMR environment with conda and pip

Source: https://github.com/zju3dv/gvhmr/blob/main/docs/INSTALL.md

Clones the GVHMR repository, creates a conda environment named 'gvhmr' with Python 3.10, activates it, installs dependencies from requirements.txt, and installs the package in editable mode. The comment suggests adding the package path to settings.json for use in other repositories.

```bash
git clone https://github.com/zju3dv/GVHMR
cd GVHMR

conda create -y -n gvhmr python=3.10
conda activate gvhmr
pip install -r requirements.txt
pip install -e .
# to install gvhmr in other repo as editable, try adding "python.analysis.extraPaths": ["path/to/your/package"] to settings.json
```

--------------------------------

### Create inputs and outputs directories

Source: https://github.com/zju3dv/gvhmr/blob/main/docs/INSTALL.md

Creates the 'inputs' and 'outputs' directories required for the project.

```bash
mkdir inputs
mkdir outputs
```

--------------------------------

### Set up checkpoints directory structure

Source: https://github.com/zju3dv/gvhmr/blob/main/docs/INSTALL.md

Creates the checkpoints directory and shows the required folder structure for SMPL/SMPLX body models and other pretrained models (DPVO, GVHMR, HMR2, ViTPose, YOLO). You must sign up to download SMPL and SMPLX, and download other models from the provided Google Drive link.

```bash
mkdir -p inputs/checkpoints

# 1. You need to sign up for downloading [SMPL](https://smpl.is.tue.mpg.de/) and [SMPLX](https://smpl-x.is.tue.mpg.de/). And the checkpoints should be placed in the following structure:

inputs/checkpoints/
├── body_models/smplx/
│   └── SMPLX_{GENDER}.npz # SMPLX (We predict SMPLX params + evaluation)
└── body_models/smpl/
    └── SMPL_{GENDER}.pkl  # SMPL (rendering and evaluation)

# 2. Download other pretrained models from Google-Drive (By downloading, you agree to the corresponding licences): https://drive.google.com/drive/folders/1eebJ13FUEXrKBawHpJroW0sNSxLjh9xD?usp=drive_link

inputs/checkpoints/
├── dpvo/
│   └── dpvo.pth
├── gvhmr/
│   └── gvhmr_siga24_release.ckpt
├── hmr2/
│   └── epoch=10-step=25000.ckpt
├── vitpose/
│   └── vitpose-h-multi-coco.pth
└── yolo/
    └── yolov8x.pt
```

--------------------------------

### Run demo_google_drive_video() on a sample video

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Calls demo_google_drive_video() with a sample Google Drive video URL and static_camera=True. The function downloads the video, runs GVHMR, and displays the result.

```python
demo_google_drive_video(url='https://drive.google.com/file/d/1KkTsoAuj9yq5JCJ-qg4_MnNBn56wR47F/view?usp=drive_link', static_camera=True)
```

--------------------------------

### Run demo.py and demo_folder.py with -s flag

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Run the demo on a single video or an entire folder. The -s flag skips visual odometry when the camera is static; otherwise DPVO estimates the camera. The folder script processes all videos in the input folder and writes results to the output folder.

```shell
python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s
python tools/demo/demo_folder.py -f inputs/demo/folder_in -d outputs/demo/folder_out -s

```

--------------------------------

### Try demo_google_drive_video() with your own video

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Placeholder call for users to try the function with their own video URL and static camera flag.

```python
demo_google_drive_video(...)
```

--------------------------------

### Visualize demo output video with display_video

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Displays the generated demo video (incam and global views) inline in the notebook using the previously defined display_video helper. The result is an HTML object that renders the video player.

```python
# Visualize the result.
display_video('outputs/demo/tennis/tennis_3_incam_global_horiz.mp4')

```

```text
Result:
<IPython.core.display.HTML object>

```

--------------------------------

### Run GVHMR demo on video or folder with -s flag

Source: https://github.com/zju3dv/gvhmr/blob/main/README.md

Run the GVHMR demo on a single video or an entire folder. Use -s to skip visual odometry when the camera is static; otherwise the camera is estimated by DPVO. Scripts are provided in tools/demo.

```shell
python tools/demo/demo.py --video=docs/example_video/tennis.mp4 -s
python tools/demo/demo_folder.py -f inputs/demo/folder_in -d outputs/demo/folder_out -s
```

--------------------------------

### Define demo_google_drive_video() to run GVHMR on a Google Drive video

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Defines the demo_google_drive_video() function that downloads a video from Google Drive using gdown, runs the GVHMR demo script with an optional -s flag for static cameras, and displays the output video. The URL must follow the pattern https://drive.google.com/file/d/<Google Drive ID>/view?usp=sharing.

```python
%cd {proj_root}
!mkdir -p inputs/demo

def demo_google_drive_video(url:str, static_camera:bool):
    ''' URL should be like https://drive.google.com/file/d/xxxxxxxx/view?usp=drive_link '''

    print(f'[1/3] 📥 Downloading video...')
    gdid = url.split('/')[5]
    video_name = f'custom_{gdid}'
    download_url = f'\'https://drive.google.com/uc?id={gdid}&export=download&confirm=t\''
    !gdown {download_url} -O inputs/demo/{video_name}.mp4

    print(f'[2/3] 💃 Start running GVHMR...')
    flag = '-s' if static_camera else ''
    !python {proj_root}/tools/demo/demo.py --video=inputs/demo/{video_name}.mp4 {flag}

    print(f'[3/3] 📺 Displaying result...')
    return display_video(f'outputs/demo/{video_name}/{video_name}_3_incam_global_horiz.mp4')
```

--------------------------------

### Render SMPL meshes to video with Pytorch3D Renderer

Source: https://github.com/zju3dv/gvhmr/blob/main/hmr4d/utils/vis/README.md

Initializes a Renderer with camera intrinsics and mesh faces, then iterates over frames to render mesh vertices and write them to an MP4 video. Requires a CUDA device and pre-existing data variables such as data, img_hw, smplh, and smplh_out.

```python
from hmr4d.utils.vis.renderer import Renderer
import imageio

fps = 30
focal_length = data["cam_int"][0][0, 0]
width, height = img_hw
faces = smplh[data["gender"]].bm.faces
renderer = Renderer(width, height, focal_length, "cuda", faces)
writer = imageio.get_writer("tmp_debug.mp4", fps=fps, mode="I", format="FFMPEG", macro_block_size=1)

for i in tqdm(range(length)):
    img = np.zeros((height, width, 3), dtype=np.uint8)
    img = renderer.render_mesh(smplh_out.vertices[i].cuda(), img)
    writer.append_data(img)
writer.close()
```

--------------------------------

### Define display_video helper for notebook video playback

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Defines a helper to embed a video file in a Jupyter notebook by base64-encoding it and scaling to fit a maximum width of 1080 pixels. Use this to display demo output videos inline.

```python
import io
import base64
from IPython.display import HTML
from hmr4d.utils.video_io_utils import get_video_lwh

def display_video(fn):
    L, W, H = get_video_lwh(fn)
    scale = min(W, 1080) / W
    W, H = int(W * scale), int(H * scale)
    video_encoded = base64.b64encode(io.open(fn, 'rb').read())
    return HTML(data='''<video width="{0}" height="{1}" alt="test" controls>
                        <source src="data:video/mp4;base64,{2}" type="video/mp4" />
                        </video>'''.format(W, H, video_encoded.decode('ascii')))

```

--------------------------------

### Extract preprocessed datasets

Source: https://github.com/zju3dv/gvhmr/blob/main/docs/INSTALL.md

Extracts the preprocessed training and test datasets (AMASS, BEDLAM, H36M for training; 3DPW, EMDB, RICH for testing) into the inputs folder. The expected folder structure is shown. Original datasets must be downloaded separately due to license restrictions.

```bash
cd inputs
# Train
tar -xzvf AMASS_hmr4d_support.tar.gz
tar -xzvf BEDLAM_hmr4d_support.tar.gz
tar -xzvf H36M_hmr4d_support.tar.gz
# Test
tar -xzvf 3DPW_hmr4d_support.tar.gz
tar -xzvf EMDB_hmr4d_support.tar.gz
tar -xzvf RICH_hmr4d_support.tar.gz

# The folder structure should be like this:
inputs/
├── AMASS/hmr4d_support/
├── BEDLAM/hmr4d_support/
├── H36M/hmr4d_support/
├── 3DPW/hmr4d_support/
├── EMDB/hmr4d_support/
└── RICH/hmr4d_support/
```

--------------------------------

### Train GVHMR model with tools/train.py

Source: https://github.com/zju3dv/gvhmr/blob/main/README.md

Train the GVHMR model. The released checkpoint was trained with 2x4090 for 420 epochs; different GPU settings may lead to different results. No post-processing is used during training, so global metrics differ from test script output.

```shell
# The gvhmr_siga24_release.ckpt is trained with 2x4090 for 420 epochs, note that different GPU settings may lead to different results.
python tools/train.py exp=gvhmr/mixed/mixed
```

--------------------------------

### Reproduce GVHMR test results for 3DPW, RICH, and EMDB

Source: https://github.com/zju3dv/gvhmr/blob/main/README.md

Reproduce 3DPW, RICH, and EMDB results in a single run. To test individual datasets, change global/task to gvhmr/test_3dpw, gvhmr/test_rich, or gvhmr/test_emdb.

```shell
python tools/train.py global/task=gvhmr/test_3dpw_emdb_rich exp=gvhmr/mixed/mixed ckpt_path=inputs/checkpoints/gvhmr/gvhmr_siga24_release.ckpt
```

--------------------------------

### Check GPU with nvidia-smi

Source: https://github.com/zju3dv/gvhmr/blob/main/tools/demo/colab_demo.ipynb

Checks that the Colab notebook is connected to an NVIDIA GPU with CUDA. If this command fails, ensure the GPU is selected as the hardware accelerator under Edit -> Notebook settings. Google may shut down the GPU if usage exceeds the allocation.

```python
!nvidia-smi
```

--------------------------------

### Cite GVHMR paper with BibTeX entry

Source: https://github.com/zju3dv/gvhmr/blob/main/README.md

BibTeX entry for citing the GVHMR paper in SIGGRAPH Asia 2024 proceedings.

```bibtex
@inproceedings{shen2024gvhmr,
  title={World-Grounded Human Motion Recovery via Gravity-View Coordinates},
  author={Shen, Zehong and Pi, Huaijin and Xia, Yan and Cen, Zhi and Peng, Sida and Hu, Zechen and Huang, Bao and Hu, Ruizhen and Zhou, Xiaowei},
  booktitle={SIGGRAPH Asia Conference Proceedings},
  year={2024}
}
```
Tokens:
Raw