### Example MMCV-full Installation for GPU

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Provides a specific example for installing mmcv-full with CUDA 10.2 and PyTorch 1.8.0. This demonstrates how to fill in the placeholders in the general GPU installation command.

```shell
pip install "mmcv-full>=1.3.17,<=1.5.3" -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8.0/index.html
```

--------------------------------

### Example PyTorch and Torchvision Installation

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Provides a concrete example of installing PyTorch version 1.8.0 with CUDA 10.2 using Conda. This serves as a practical illustration of the general installation command.

```shell
conda install pytorch=1.8.0 torchvision cudatoolkit=10.2 -c pytorch
```

--------------------------------

### Visualize 2D Keypoints (Simple Example)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_keypoints.md

This snippet demonstrates the basic usage of `visualize_kp2d` to visualize 2D keypoints. It takes keypoint data, specifies the data source convention, an output path for a video, and the desired resolution. The output is a video file.

```python
from mmhuman3d.core.visualization.visualize_keypoints2d import visualize_kp2d

visualize_kp2d(
    kp2d_coco_wholebody,
    data_source='coco_wholebody',
    output_path='some_video.mp4',
    resolution=(1024, 1024))

```

--------------------------------

### POST /renderer/build

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/render.md

Initializes a renderer instance using a configuration dictionary. This supports various renderer types like Mesh, Depth, and Normal, allowing for flexible setup of rasterizers, shaders, and cameras.

```APIDOC
## POST /renderer/build

### Description
Initializes a specific renderer type (e.g., mesh, depth) using a configuration dictionary. It supports passing existing nn.Modules or configuration dicts for rasterizers, shaders, and cameras.

### Method
POST

### Endpoint
/renderer/build

### Parameters
#### Request Body
- **type** (string) - Required - The type of renderer (e.g., 'mesh', 'depth', 'normal', 'segmentation', 'silhouette', 'uv').
- **device** (string) - Required - The compute device (e.g., 'cuda').
- **resolution** (int) - Optional - The output image resolution.
- **shader** (dict/object) - Optional - Configuration for the shader module.
- **rasterizer** (dict/object) - Optional - Configuration for the rasterizer module.
- **output_path** (string) - Optional - Path to save rendered video or images.

### Request Example
{
  "type": "mesh",
  "device": "cuda",
  "resolution": 128,
  "shader": {"type": "SoftPhongShader"}
}

### Response
#### Success Response (200)
- **renderer** (object) - The initialized renderer instance.

#### Response Example
{
  "status": "success",
  "renderer_type": "mesh"
}
```

--------------------------------

### Install PyTorch3D from Source

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs PyTorch3D by cloning the repository and installing it using pip. This method is an alternative for users who encounter dependency conflicts with the Conda installation.

```shell
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install .
cd ..
```

--------------------------------

### Install MMCV-full for CPU

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs the mmcv-full package for CPU-only environments. This command requires specifying the PyTorch version in the URL.

```shell
pip install mmcv-full -f https://download.openmmlab.com/mmcv/dist/cpu/{torch_version}/index.html
```

--------------------------------

### Install PyTorch3D using Conda

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs the PyTorch3D library using the Conda package manager. This is the recommended method for installing PyTorch3D.

```shell
conda install pytorch3d -c pytorch3d
```

--------------------------------

### Compile MMCV from Source

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs mmcv-full by compiling it from source. This involves cloning the MMCV repository and using pip to install it in editable mode with CUDA operations enabled.

```shell
git clone https://github.com/open-mmlab/mmcv.git -b v1.5.3
cd mmcv
MMCV_WITH_OPS=1 pip install -e .  # package mmcv-full, which contains cuda ops, will be installed after this step
```

--------------------------------

### Visualize 2D Keypoints with Mask and Convention Conversion

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_keypoints.md

This example shows how to handle keypoints with invalid or missing data points using a mask. It also demonstrates converting keypoints from one convention (e.g., 'coco_wholebody') to another (e.g., 'smpl') before visualization using `convert_kps` and `visualize_kp2d`.

```python
from mmhuman3d.core.conventions.keypoints_mapping import convert_kps
from mmhuman3d.core.visualization.visualize_keypoints2d import visualize_kp2d

kp2d_smpl, mask = convert_kps(kp2d_coco_wholebody, src='coco_wholebody', dst='smpl')
visualize_kp2d(
    kp2d_smpl,
    mask=mask,
    output_path='some_video.mp4',
    resolution=(1024, 1024))

```

--------------------------------

### Visualize 2D Keypoints with Custom Limbs

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_keypoints.md

This example demonstrates how to visualize keypoints for datasets that do not conform to existing conventions. It allows users to define custom limb connections by providing a list of pairs representing the joints that form a limb.

```python
limbs=[[0, 1], ..., [10, 11]]
visualize_kp2d(
    kp2d_coco_wholebody,
    data_source='coco_wholebody',
    output_path='some_video.mp4',
    resolution=(1024, 1024),
    limbs=limbs)

```

--------------------------------

### Verify PyTorch3D Installation (Linux)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Verifies the PyTorch3D installation by importing key modules and printing their versions or objects. This script checks for the presence and basic functionality of PyTorch3D components.

```python
echo "import pytorch3d;print(pytorch3d.__version__); \
    from pytorch3d.renderer import MeshRenderer;print(MeshRenderer);\
    from pytorch3d.structures import Meshes;print(Meshes);\
    from pytorch3d.renderer import cameras;print(cameras);\
    from pytorch3d.transforms import Transform3d;print(Transform3d);"|python
```

--------------------------------

### Install MMCV-full for GPU

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs the mmcv-full package for GPU environments, specifying version constraints and providing a URL template for downloading pre-built packages. Users must replace placeholders with their specific CUDA and PyTorch versions.

```shell
pip install "mmcv-full>=1.3.17,<=1.5.3" -f https://download.openmmlab.com/mmcv/dist/{cu_version}/{torch_version}/index.html
```

--------------------------------

### Convert Keypoints with Confidence Information (Python)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/keypoints_convention.md

Illustrates converting keypoints while preserving or updating confidence information. It shows how to handle occluded keypoints by setting their confidence to 0 and how this information is propagated during conversion. The example also demonstrates carrying forward specific confidence values.

```python
import numpy as np
from mmhuman3d.core.conventions.keypoints_mapping import KEYPOINTS_FACTORY, convert_kps

keypoints = np.zeros((1, len(KEYPOINTS_FACTORY['smpl']), 3))
confidence = np.ones((len(KEYPOINTS_FACTORY['smpl'])))

# assume that 'left_shoulder' point is invalid.
confidence[KEYPOINTS_FACTORY['smpl'].index('left_shoulder')] = 0

_, conf_coco = convert_kps(
    keypoints=keypoints, confidence=confidence, src='smpl', dst='coco')
_, conf_coco_full = convert_kps(
    keypoints=keypoints, src='smpl', dst='coco')

assert conf_coco[KEYPOINTS_FACTORY['coco'].index('left_shoulder')] == 0
conf_coco[KEYPOINTS_FACTORY['coco'].index('left_shoulder')] = 1
assert (conf_coco == conf_coco_full).all()
```

```python
confidence = np.ones((len(KEYPOINTS_FACTORY['smpl'])))
confidence[KEYPOINTS_FACTORY['smpl'].index('left_shoulder')] = 0.5
kp_smpl = np.concatenate([kp_smpl, confidence], -1)
kp_smpl_converted, mask = convert_kps(kp_smpl, src='smpl', dst='coco')
new_confidence =  kp_smpl_converted[..., 2:]
assert new_confidence[KEYPOINTS_FACTORY['smpl'].index('left_shoulder')] == 0.5
```

--------------------------------

### Convert Keypoints with Masking

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/human_data.md

Provides an example of converting keypoints from one convention (e.g., 'agora') to another ('human_data') while handling missing keypoints using a mask.

```python
# keypoints2d_agora is a numpy array in shape [frame_num, 127, 3].
# There are 127 keypoints defined by agora.
keypoints2d_human_data, mask = convert_kps(keypoints2d_agora, 'agora', 'human_data')
# keypoints2d_human_data is a numpy array in shape [frame_num, 190, 3], only 127/190 are valid
```

--------------------------------

### Define New Keypoint Convention (Python)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/customize_keypoints_convention.md

Example of defining a new keypoint convention for a dataset like AGORA. This involves creating a Python list with the specific keypoint names and their order.

```python
AGORA_KEYPOINTS = [
  'pelvis',
  'left_hip',
  'right_hip'
  ... 
]
```

--------------------------------

### Read and Extract Data from SMC Files

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/smc.md

Demonstrates how to initialize an SMCReader instance and retrieve various data modalities including extrinsics, color images, depth maps, and keypoints for both Kinect and iPhone devices. This requires the mmhuman3d package to be installed.

```python
from mmhuman3d.data.data_structures.smc_reader import SMCReader

# Initialize a smc reader
smc_reader = SMCReader('/path/to/pxxxxxx_axxxxxx.smc')

# Get calibration
kinect_extrinsics = smc_reader.get_kinect_color_extrinsics(kinect_id=0)
iphone_extrinsics = smc_reader.get_iphone_extrinsics(iphone_id=0)

# Get images
kinect_images = smc_reader.get_color(device='Kinect', device_id=0)
iphone_images = smc_reader.get_color(device='iPhone', device_id=0, vertical=True)

# Get depth maps
kinect_depth = smc_reader.get_kinect_depth(device='Kinect', device_id=0)
iphone_depth = smc_reader.get_iphone_depth(device='iPhone', device_id=0)

# Get 2D keypoints
iphone_keypoints2d = smc_reader.get_keypoints2d(device='Kinect', device_id=0)
iphone_keypoints2d = smc_reader.get_keypoints2d(device='iPhone', device_id=0, vertical=True)

# Get 3D keypoints
keypoints3d = smc_reader.get_keypoints3d(device='Kinect', device_id=0)

# Get SMPL
smpl = smc_reader.get_smpl(device='Kinect', device_id=0)
```

--------------------------------

### Initialize and Slice Perspective Cameras

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Demonstrates how to instantiate PerspectiveCameras using PyTorch tensors for K, R, and T matrices. It also shows how to perform batch slicing and concatenation operations on the camera objects.

```python
from mmhuman3d.core.cameras import PerspectiveCameras
import torch
K = torch.eye(4, 4)[None]
R = torch.eye(3, 3)[None]
T = torch.zeros(100, 3)
cam = PerspectiveCameras(K=K, R=R, T=T)
assert cam.R.shape == (100, 3, 3)
assert cam.K.shape == (100, 4, 4)
assert cam.T.shape == (100, 3)
assert (cam[:10].K == cam.K[:10]).all()
```

--------------------------------

### Install ffmpeg using Conda

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs the ffmpeg package using Conda. This is a required dependency for MMHuman3D, and installing it via Conda ensures that necessary libraries like libx264 are built automatically.

```shell
conda install ffmpeg
```

--------------------------------

### Camera Initialization with K, R, T

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Demonstrates initializing PerspectiveCameras by directly providing the intrinsic (K), extrinsic (R), and translation (T) matrices. Supports slicing and batching of camera parameters.

```APIDOC
## Camera Initialization with K, R, T

### Description
Initializes `PerspectiveCameras` by directly providing `K`, `R`, and `T` matrices. Supports slicing and batching of camera parameters. The batch sizes of `K`, `R`, and `T` should be consistent or broadcastable.

### Method
```python
from mmhuman3d.core.cameras import PerspectiveCameras
import torch

K = torch.eye(4, 4)[None]
R = torch.eye(3, 3)[None]
T = torch.zeros(100, 3)

# Batch of K, R, T should all be the same or some of them could be 1. The final batch size will be the biggest one.
cam = PerspectiveCameras(K=K, R=R, T=T)

# Assertions to check the shapes of the initialized camera parameters
assert cam.R.shape == (100, 3, 3)
assert cam.K.shape == (100, 4, 4)
assert cam.T.shape == (100, 3)

# Example of slicing cameras
assert (cam[:10].K == cam.K[:10]).all()
```

### Parameters
#### Request Body
- **K** (torch.Tensor) - Required - Intrinsic camera matrix.
- **R** (torch.Tensor) - Required - Extrinsic camera rotation matrix.
- **T** (torch.Tensor) - Required - Camera translation vector.

### Response
#### Success Response (200)
- **PerspectiveCameras** (object) - Initialized camera object.
```

--------------------------------

### Camera Utilities: Build and Convert (Python)

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Provides tools for creating and manipulating camera objects with different conventions (e.g., OpenCV, PyTorch3D) and projection types. Includes functions to build cameras from intrinsic parameters and convert camera matrices. Dependencies include torch.

```python
import torch
from mmhuman3d.core.cameras import build_cameras, WeakPerspectiveCameras
from mmhuman3d.core.conventions.cameras import convert_camera_matrix

# Build perspective camera from intrinsic parameters
K = torch.eye(4).unsqueeze(0)  # (1, 4, 4) intrinsic matrix
K[0, 0, 0] = 1000  # fx
K[0, 1, 1] = 1000  # fy
K[0, 0, 2] = 512   # cx
K[0, 1, 2] = 512   # cy

R = torch.eye(3).unsqueeze(0)  # (1, 3, 3) rotation
T = torch.zeros(1, 3)          # (1, 3) translation

camera = build_cameras(
    dict(
        type='perspective',
        K=K,
        R=R,
        T=T,
        resolution=(1024, 1024),
        in_ndc=False,
        convention='opencv'
    )
)

# Project 3D points to 2D
points_3d = torch.rand(1, 100, 3)  # (batch, num_points, 3)
points_2d = camera.transform_points_screen(points_3d)
print(f"Projected points shape: {points_2d.shape}")

# Convert camera matrices between conventions
K_pytorch3d, R_pytorch3d, T_pytorch3d = convert_camera_matrix(
    K=K,
    R=R,
    T=T,
    convention_src='opencv',
    convention_dst='pytorch3d',
    is_perspective=True,
    in_ndc_src=False,
    in_ndc_dst=True,
    resolution_src=(1024, 1024)
)

# Create weak perspective camera for HMR-style predictions
orig_cam = torch.tensor([[1.0, 1.0, 0.0, 0.0]])  # [scale_x, scale_y, trans_x, trans_y]
K_wp, R_wp, T_wp = WeakPerspectiveCameras.convert_orig_cam_to_matrix(
    orig_cam=orig_cam,
    znear=0.1,
    aspect_ratio=1.0
)
```

--------------------------------

### Install PyTorch3D Dependencies

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs core dependencies for PyTorch3D, namely fvcore and iopath, using Conda. These packages are essential for PyTorch3D functionality.

```shell
conda install -c fvcore -c iopath -c conda-forge fvcore iopath -y
conda install -c bottler nvidiacub -y
```

--------------------------------

### Run SMPL Estimation using Command-Line Demo

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Executes single-person or multi-person SMPL estimation from video input via the command line. It allows specifying configuration files, pre-trained model checkpoints, input/output paths, and various post-processing options like smoothing and rendering.

```bash
# Single-person demo with HMR
python demo/estimate_smpl.py \
    configs/hmr/resnet50_hmr_pw3d.py \
    data/checkpoints/resnet50_hmr_pw3d.pth \
    --single_person_demo \
    --det_config demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
    --det_checkpoint https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    --input_path demo/resources/single_person_demo.mp4 \
    --show_path vis_results/output.mp4 \
    --output demo_result \
    --smooth_type savgol \
    --speed_up_type deciwatch \
    --render_choice hq \
    --palette segmentation \
    --draw_bbox
```

```bash
# Multi-person demo with tracking
python demo/estimate_smpl.py \
    configs/spin/resnet50_spin_pw3d.py \
    data/checkpoints/resnet50_spin_pw3d.pth \
    --multi_person_demo \
    --tracking_config demo/mmtracking_cfg/deepsort_faster-rcnn_fpn_4e_mot17-private-half.py \
    --input_path demo/resources/multi_person_demo.mp4 \
    --show_path vis_results/multi_output.mp4 \
    --smooth_type smoothnet \
    --body_model_dir data/body_models/
```

```bash
# Webcam real-time demo
python demo/webcam_demo.py \
    --cam-id 0 \
    --output webcam_output.mp4 \
    --synchronous
```

--------------------------------

### Build Cameras via Registry and Configuration

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Shows the usage of the build_cameras factory function to instantiate camera objects from dictionaries. This approach supports various configurations including NDC space, pixel space, and default projections.

```python
from mmhuman3d.core.cameras import build_cameras
import torch
K = torch.eye(4, 4)[None]
R = torch.eye(3, 3)[None]
T = torch.zeros(10, 3)
height, width = 1000, 1000
cam1 = build_cameras(dict(type='PerspectiveCameras', K=K, R=R, T=T, in_ndc=True, image_size=(height, width), convention='opencv'))
cam = build_cameras(dict(type='PerspectiveCameras', in_ndc=False, image_size=(1000, 1000), principal_points=(500, 500), focal_length=1000, convention='opencv'))
cam_weak = build_cameras(dict(type='WeakPerspectiveCameras', K=K, R=R, T=T, image_size=(1000, 1000)))
cam_default = build_cameras(dict(type='PerspectiveCameras', in_ndc=True, image_size=(1000, 1000)))
cam_default.to_screen_()
```

--------------------------------

### Initialize and Use SMPL Model in Python

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Demonstrates initializing the SMPL model with specified parameters and performing a forward pass to generate vertices and joints from pose and shape information. It also shows how to convert between dictionary and tensor representations of pose and shape.

```python
body_model = SMPL(
    model_path='data/body_models/smpl',
    gender='neutral',
    keypoint_src='smpl_45',
    keypoint_dst='human_data',
    keypoint_approximate=False,
    extra_joints_regressor='data/body_models/J_regressor_extra.npy'
)

# Forward pass with pose parameters
batch_size = 2
body_pose = torch.zeros(batch_size, 69)  # 23 joints * 3 axis-angle
global_orient = torch.zeros(batch_size, 3)
betas = torch.zeros(batch_size, 10)

output = body_model(
    body_pose=body_pose,
    global_orient=global_orient,
    betas=betas,
    return_verts=True,
    return_full_pose=False
)

vertices = output['vertices']    # Shape: (batch_size, 6890, 3)
joints = output['joints']        # Shape: (batch_size, num_keypoints, 3)
joint_mask = output['joint_mask']  # Validity mask for keypoints

print(f"Vertices: {vertices.shape}")
print(f"Joints: {joints.shape}")

# Convert between dict and tensor representations
pose_dict = SMPL.tensor2dict(
    full_pose=torch.zeros(1, 72),
    betas=torch.zeros(1, 10),
    transl=torch.zeros(1, 3)
)
full_pose_tensor = SMPL.dict2tensor(pose_dict)
```

--------------------------------

### Install PyTorch and Torchvision

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Installs PyTorch and torchvision using Conda, specifying the desired PyTorch and CUDA toolkit versions. This command is a template and requires users to replace placeholders with their specific versions.

```shell
conda install pytorch={torch_version} torchvision cudatoolkit={cu_version} -c pytorch
```

--------------------------------

### Run Webcam Demo

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/getting_started.md

Executes the webcam demo script for real-time SMPL parameter estimation. Supports offline video file processing via the cam-id argument.

```shell
python demo/webcam_demo.py
```

--------------------------------

### Get camera plane normal vectors

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Retrieves the normalized normal tensor representing the direction pointing out of the camera plane from the camera center.

```python
normals = cameras.get_camera_plane_normals()
```

--------------------------------

### Configure Renderer for File Output (Video and Images)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/render.md

Demonstrates how to configure the renderer to output rendered frames directly to files, either as a video or a sequence of images. This involves setting the `output_path` and `out_img_format` parameters during renderer initialization.

```python
# will write a video
renderer = build_renderer(dict(type='mesh', device=device, resolution=resolution, output_path='test.mp4'))
backgrounds = torch.Tensor(N, H, W, 3)
rendered_tensor = renderer(meshes=meshes, cameras=cameras, lights=lights, backgrounds=backgrounds)
renderer.export() # needed for a video

# will write a folder of images
renderer = build_renderer(dict(type='mesh', device=device, resolution=resolution, output_path='test_folder', out_img_format='%06d.png'))
backgrounds = torch.Tensor(N, H, W, 3)
rendered_tensor = renderer(meshes=meshes, cameras=cameras, lights=lights, backgrounds=backgrounds)
```

--------------------------------

### Mesh Visualization with render_smpl in Python

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Shows how to render SMPL meshes using the `render_smpl` utility. It covers various rendering options including high-quality mesh videos, rendering with background videos, generating depth maps, and multi-person rendering with customizable colors and camera parameters.

```python
import torch
import numpy as np
from mmhuman3d.core.visualization import render_smpl

# Prepare SMPL parameters
num_frames = 30
poses = torch.zeros(num_frames, 72)  # Full pose (global + body)
betas = torch.zeros(num_frames, 10)
transl = torch.zeros(num_frames, 3)

# Option 1: Render high-quality mesh video
render_smpl(
    poses=poses,
    betas=betas,
    transl=transl,
    body_model_config=dict(
        type='smpl',
        model_path='data/body_models'
    ),
    render_choice='hq',  # 'lq', 'mq', 'hq', 'silhouette', 'depth', 'normal'
    palette='white',  # Color: 'white', 'segmentation', 'random', or color name
    output_path='output/render.mp4',
    resolution=(1024, 1024),
    device='cuda'
)

# Option 2: Render with background video
render_smpl(
    poses=poses,
    betas=betas,
    body_model_config=dict(type='smpl', model_path='data/body_models'),
    render_choice='mq',
    origin_frames='input_video.mp4',  # Background video
    output_path='output/overlay.mp4',
    alpha=0.9,  # Mesh transparency
    resolution=(720, 1280)
)

# Option 3: Render depth map and return tensor
depth_tensor = render_smpl(
    poses=poses,
    betas=betas,
    body_model_config=dict(type='smpl', model_path='data/body_models'),
    render_choice='depth',
    return_tensor=True,
    no_grad=True,
    batch_size=10
)
# depth_tensor shape: (num_frames, height, width, 1)

# Option 4: Multi-person rendering with weak perspective camera
num_persons = 2
poses_multi = torch.zeros(num_frames, num_persons, 72)
orig_cam = torch.tensor([[1.0, 1.0, 0.0, 0.0]] * num_frames)  # [sx, sy, tx, ty]

render_smpl(
    poses=poses_multi,
    orig_cam=orig_cam,
    body_model_config=dict(type='smpl', model_path='data/body_models'),
    palette=['blue', 'red'],  # Different color per person
    output_path='output/multi_person.mp4'
)
```

--------------------------------

### Initialize Mesh Renderer with PyTorch3D and MMHuman3D

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/render.md

Demonstrates initializing a MeshRenderer using both the standard PyTorch3D approach and the more flexible MMHuman3D method, which supports initialization via configuration dictionaries and the MMCV Registry.

```python
### initialized by Pytorch3D
import torch
from pytorch3d.renderer import MeshRenderer, RasterizationSettings
from pytorch3d.renderer.lighting import PointLights
from pytorch3d.renderer.cameras import FoVPerspectiveCameras

device = torch.device('cuda')
R, T = look_at_view_transform(dist=2.7, elev=0, azim=0)
cameras = FoVPerspectiveCameras(device=device, R=R, T=T)

lights = PointLights(
    device=device,
    ambient_color=((0.5, 0.5, 0.5), ),
    diffuse_color=((0.3, 0.3, 0.3), ),
    specular_color=((0.2, 0.2, 0.2), ),
    direction=((0, 1, 0), ),
)
raster_settings = RasterizationSettings(
    image_size=128,
    blur_radius=0.0,
    faces_per_pixel=1,
)
renderer = MeshRenderer(
    rasterizer=MeshRasterizer(
        cameras=cameras, raster_settings=raster_settings),
    shader=SoftPhongShader(device=device, cameras=cameras, lights=lights))

### initialized by mmhuman3d
from mmhuman3d.core.renderer.torch3d_renderer.builder import MeshRenderer
# rasterizer could be passed by nn.Module or dict
rasterizer = dict(
    image_size=128,
    blur_radius=0.0,
    faces_per_pixel=1,
)
# lights could be passed by nn.Module or dict
lights = dict(type='point', ambient_color=((0.5, 0.5, 0.5), ),
    diffuse_color=((0.3, 0.3, 0.3), ),
    specular_color=((0.2, 0.2, 0.2), ),
    direction=((0, 1, 0), ),)

# rasterizer could be passed by cameras or dict
cameras = dict(type='fovperspective', R=R, T=T, device=device)

# shader could be passed by nn.Module or dict
shader = dict(type='SoftPhongShader')
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Creates a new Conda virtual environment named 'open-mmlab' with Python 3.8 and activates it. This is the first step in preparing the environment for installation.

```shell
conda create -n open-mmlab python=3.8 -y
conda activate open-mmlab
```

--------------------------------

### Estimate SMPL Parameters for Single-Person

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/getting_started.md

Runs the SMPL estimation demo for a single person using MMDetection for bounding box detection. Requires a config file, checkpoint, and input path.

```shell
python demo/estimate_smpl.py \
    ${MMHUMAN3D_CONFIG_FILE} \
    ${MMHUMAN3D_CHECKPOINT_FILE} \
    --single_person_demo \
    --det_config ${MMDET_CONFIG_FILE} \
    --det_checkpoint ${MMDET_CHECKPOINT_FILE} \
    --input_path ${VIDEO_PATH_OR_IMG_PATH} \
    [--show_path ${VIS_OUT_PATH}] \
    [--output ${RESULT_OUT_PATH}] \
    [--smooth_type ${SMOOTH_TYPE}] \
    [--speed_up_type ${SPEED_UP_TYPE}] \
    [--draw_bbox]
```

--------------------------------

### Configure Human3.6M Dataset Converter

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/preprocess_dataset.md

Defines the Python dictionary configuration for the H36mConverter, specifying modes, protocols, and image extraction settings. It includes variations for both standard and MoShed data setups.

```python
h36m_p1=dict(
    type='H36mConverter',
    modes=['train', 'valid'],
    protocol=1,
    extract_img=True,
    prefix='h36m')
```

```python
h36m_p1=dict(
    type='H36mConverter',
    modes=['train', 'valid'],
    protocol=1,
    extract_img=True,
    mosh_dir='data/datasets/h36m_mosh',
    prefix='h36m')
```

--------------------------------

### Verify PyTorch3D Torus Generation (Linux)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/install.md

Tests the PyTorch3D installation by generating a torus mesh on the CUDA device and printing its vertices. This confirms that PyTorch3D can interact with CUDA and generate geometric primitives.

```python
echo "import torch;device=torch.device('cuda');\
    from pytorch3d.utils import torus;\
    Torus = torus(r=10, R=20, sides=100, rings=100, device=device);\
    print(Torus.verts_padded());"|python
```

--------------------------------

### Visualize Single Person 3D Keypoints (Python)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_keypoints.md

Demonstrates how to visualize 3D keypoints for a single person. The input kp3d should be in SMPLX convention with shape (num_frame, 144, 3). The output is a video where each body part has a distinct color.

```python
from mmhuman3d.utils.vis_utils import visualize_kp3d

# Assuming kp3d is already defined and in SMPLX convention
# kp3d.shape = (num_frame, 144, 3)
visualize_kp3d(kp3d=kp3d, data_source='smplx', output_path='some_video.mp4')
```

--------------------------------

### Initialize Model for Mesh Estimation (Python)

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Initializes a pre-trained model from a configuration file and checkpoint for mesh estimation. It returns the mesh model and an optional feature extractor for video-based models. Dependencies include PyTorch and MMHuman3D. Inputs are configuration and checkpoint paths, and the device.

```python
from mmhuman3d.apis import init_model

# Initialize HMR model from config and checkpoint
config_path = 'configs/hmr/resnet50_hmr_pw3d.py'
checkpoint_path = 'data/checkpoints/resnet50_hmr_pw3d.pth'

model, extractor = init_model(
    config=config_path,
    checkpoint=checkpoint_path,
    device='cuda:0'
)

# Model is ready for inference
# model.cfg contains the configuration
# model.eval() is already called
print(f"Model type: {model.cfg.model.type}")
# Output: Model type: ImageBodyModelEstimator
```

--------------------------------

### Estimate SMPL Parameters for Multi-Person

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/getting_started.md

Runs the SMPL estimation demo for multiple people using MMTracking for person tracking. Requires a config file, checkpoint, and tracking configuration.

```shell
python demo/estimate_smpl.py \
    ${MMHUMAN3D_CONFIG_FILE} \
    ${MMHUMAN3D_CHECKPOINT_FILE} \
    --multi_person_demo \
    --tracking_config ${MMTRACKING_CONFIG_FILE} \
    --input_path ${VIDEO_PATH_OR_IMG_PATH} \
    [--show_path ${VIS_OUT_PATH}] \
    [--output ${RESULT_OUT_PATH}] \
    [--smooth_type ${SMOOTH_TYPE}] \
    [--speed_up_type ${SPEED_UP_TYPE}] \
    [--draw_bbox]
```

--------------------------------

### Build Renderer using MMHuman3D Builder

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/render.md

Shows how to use the `build_renderer` function from MMHuman3D to create renderer instances, including options for passing configurations as dictionaries and using default settings for rasterizer and shader.

```python
import torch.nn as nn
from mmhuman3d.core.renderer.torch3d_renderer.builder import MeshRenderer, build_renderer

renderer = MeshRenderer(shader=shader, device=device, rasterizer=rasterizer, resolution=resolution)
renderer = build_renderer(dict(type='mesh', device=device, shader=shader, rasterizer=rasterizer, resolution=resolution))

# Use default raster and shader settings
renderer = build_renderer(dict(type='mesh', device=device, resolution=resolution))
assert isinstance(renderer.rasterizer, nn.Module)
assert isinstance(renderer.shader, nn.Module)
```

--------------------------------

### POST /smpl/initialize

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Initializes the SMPL body model with specific parameters and performs a forward pass to generate vertices and joints.

```APIDOC
## POST /smpl/initialize

### Description
Initializes the SMPL body model and computes vertices and joints based on provided pose and shape parameters.

### Method
POST

### Endpoint
/smpl/initialize

### Parameters
#### Request Body
- **model_path** (string) - Required - Path to SMPL model files
- **gender** (string) - Required - Gender model ('neutral', 'male', 'female')
- **body_pose** (tensor) - Required - Axis-angle representation of body joints
- **betas** (tensor) - Required - Shape parameters

### Request Example
{
  "model_path": "data/body_models/smpl",
  "gender": "neutral",
  "body_pose": "[batch_size, 69]",
  "betas": "[batch_size, 10]"
}

### Response
#### Success Response (200)
- **vertices** (tensor) - Shape (batch_size, 6890, 3)
- **joints** (tensor) - Shape (batch_size, num_keypoints, 3)

#### Response Example
{
  "vertices": "[batch_size, 6890, 3]",
  "joints": "[batch_size, num_keypoints, 3]"
}
```

--------------------------------

### Visualize Multi-Person 3D Keypoints (Python)

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_keypoints.md

Shows how to visualize 3D keypoints for multiple people. Inputs kp3d_1 and kp3d_2 should be in SMPLX convention. The function concatenates them to form a (num_frame, num_person, 144, 3) array. The output video displays each person in a unique color with a legend.

```python
import numpy as np
from mmhuman3d.utils.vis_utils import visualize_kp3d

# Assuming kp3d_1 and kp3d_2 are defined and in SMPLX convention
# kp3d_1.shape = (num_frame, 144, 3)
# kp3d_2.shape = (num_frame, 144, 3)
kp3d = np.concatenate([kp3d_1[:, np.newaxis], kp3d_2[:, np.newaxis]], axis=1)
# kp3d.shape is now (num_frame, num_person, 144, 3)
visualize_kp3d(kp3d=kp3d, data_source='smplx', output_path='some_video.mp4')
```

--------------------------------

### Convert Keypoints Between Conventions (Python)

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Converts human keypoints between various annotation formats like COCO, H36M, SMPL, and OpenPose. It provides functions to get keypoint indices, counts, and flip pairs, supporting approximate mappings. Dependencies include numpy and torch.

```python
import numpy as np
import torch
from mmhuman3d.core.conventions.keypoints_mapping import (
    convert_kps,
    get_keypoint_idx,
    get_keypoint_num,
    get_flip_pairs,
    KEYPOINTS_FACTORY
)

# Convert keypoints from COCO to Human3.6M format
coco_keypoints = np.random.rand(10, 17, 3)  # (frames, joints, xyz)

h36m_keypoints, mask = convert_kps(
    keypoints=coco_keypoints,
    src='coco',
    dst='h36m',
    approximate=True  # Allow approximate joint mapping
)

print(f"COCO keypoints: {coco_keypoints.shape}")
print(f"H36M keypoints: {h36m_keypoints.shape}")
print(f"Valid joints mask: {mask}")

# Get keypoint index by name
right_hip_idx = get_keypoint_idx(
    name='right_hip',
    convention='smpl_24',
    approximate=True
)
print(f"Right hip index in SMPL: {right_hip_idx}")

# Get number of keypoints for a convention
num_kps = get_keypoint_num(convention='coco')
print(f"COCO has {num_kps} keypoints")

# Get flip pairs for data augmentation
flip_pairs = get_flip_pairs(convention='smpl')
print(f"SMPL flip pairs: {flip_pairs[:3]}...")

# List available conventions
print(f"Available conventions: {list(KEYPOINTS_FACTORY.keys())}")
```

--------------------------------

### Visualize SMPL with VIBE Camera Parameters

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/visualize_smpl.md

Visualizes SMPL meshes using predicted camera parameters (pred_cam) or original camera parameters (orig_cam) from the VIBE model. Requires poses, betas, body model configuration, and either pred_cam or orig_cam. Also accepts bounding box information.

```python
import pickle
from mmhuman3d.core.visualization.visualize_smpl import visualize_smpl_vibe
with open('vibe_output.pkl', 'rb') as f:
    d = pickle.load(f, encoding='latin1')
poses = d[1]['pose']
orig_cam = d[1]['orig_cam']
pred_cam = d[1]['pred_cam']
bbox = d[1]['bboxes']
gender = 'female'

# pass pred_cam & bbox
body_model_config = dict(
    type='smpl',
    model_path=model_path,
    gender=gender)
visualize_smpl_vibe(
    poses=poses,
    betas=betas,
    body_model_config=body_model_config,
    pred_cam=pred_cam,
    bbox=bbox,
    output_path='vibe_demo.mp4',
    origin_frames='sample_video.mp4',
    resolution=(1024, 1024))
```

```python
body_model_config = dict(
    type='smpl',
    model_path=model_path,
    gender=gender)
visualize_smpl_vibe(
    poses=poses,
    betas=betas,
    body_model_config=body_model_config,
    orig_cam=orig_cam,
    output_path='vibe_demo.mp4',
    origin_frames='sample_video.mp4',
    resolution=(1024, 1024))
```

--------------------------------

### SMPLify Optimization for Keypoint Fitting in Python

Source: https://context7.com/open-mmlab/mmhuman3d/llms.txt

Illustrates how to use the SMPLify optimizer to fit SMPL body model parameters to 2D keypoint observations. It includes initialization of SMPLify with loss functions and optimization stages, preparation of input keypoints, and accessing the optimized pose and shape parameters.

```python
import torch
from mmhuman3d.models.registrants import SMPLify

# Initialize SMPLify with configuration
smplify = SMPLify(
    body_model=dict(
        type='SMPL',
        model_path='data/body_models/smpl',
        gender='neutral'
    ),
    num_epochs=100,
    stages={
        'stage1': {'fit_global_orient': True, 'fit_transl': True},
        'stage2': {'fit_body_pose': True, 'fit_betas': True}
    },
    optimizer=dict(type='Adam', lr=1e-2),
    keypoints2d_loss=dict(type='KeypointMSELoss', loss_weight=1.0),
    shape_prior_loss=dict(type='ShapePriorLoss', loss_weight=5e-3),
    joint_prior_loss=dict(type='JointPriorLoss', loss_weight=15.0),
    device='cuda:0',
    verbose=True
)

# Prepare keypoint observations
batch_size = 1
num_keypoints = 17
keypoints2d = torch.rand(batch_size, num_keypoints, 2).cuda()  # (x, y)
keypoints2d_conf = torch.ones(batch_size, num_keypoints).cuda()

# Optional: provide initial estimates
init_body_pose = torch.zeros(batch_size, 69).cuda()
init_betas = torch.zeros(batch_size, 10).cuda()

# Run optimization
result = smplify(
    keypoints2d=keypoints2d,
    keypoints2d_conf=keypoints2d_conf,
    init_body_pose=init_body_pose,
    init_betas=init_betas,
    return_verts=True,
    return_joints=True,
    return_losses=True
)

# Access optimized parameters
optimized_pose = result['body_pose']
optimized_betas = result['betas']
optimized_vertices = result['vertices']
losses = result['losses']  # Dict with per-loss values
```

--------------------------------

### Convert Camera Conventions

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Demonstrates how to convert camera parameters between different conventions such as OpenCV and PyTorch3D, including handling NDC space.

```python
from mmhuman3d.core.conventions.cameras import convert_cameras
import torch

K = torch.eye(4, 4)[None]
R = torch.eye(3, 3)[None]
T = torch.zeros(10, 3)
height, width = 1080, 1920
K, R, T = convert_cameras(
    K=K,
    R=R,
    T=T,
    in_ndc_src=False,
    in_ndc_dst=True,
    resolution_src=(height, width),
    convention_src='opencv',
    convention_dst='pytorch3d')
```

--------------------------------

### Compress and Manage Keypoints

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/human_data.md

Demonstrates how to compress keypoints using masks to remove redundant zeros and how to handle data editing in compressed mode.

```python
human_data = HumanData()
human_data['keypoints2d_mask'] = mask
human_data['keypoints2d'] = keypoints2d_human_data
human_data.compress_keypoints_by_mask()
keypoints2d_human_data = human_data.get_raw_value('keypoints2d')
human_data.decompress_keypoints()
```

--------------------------------

### Building Cameras with build_cameras

Source: https://github.com/open-mmlab/mmhuman3d/blob/main/docs/cameras.md

Utilizes mmcv.Registry to build camera objects, offering flexibility to initialize with K, R, T or with focal length and principal point.

```APIDOC
## Building Cameras with build_cameras

### Description
Uses `mmcv.Registry` to build camera objects. Allows initialization using `K`, `R`, `T` matrices or by specifying `focal_length` and `principle_point`. Supports different camera types like `PerspectiveCameras` and `WeakPerspectiveCameras`.

### Method
```python
from mmhuman3d.core.cameras import build_cameras
import torch

# Initialize a perspective camera with given K, R, T matrix.
K = torch.eye(4, 4)[None]
R = torch.eye(3, 3)[None]
T = torch.zeros(10, 3)
height, width = 1000, 1000

cam1 = build_cameras(
    dict(
        type='PerspectiveCameras',
        K=K,
        R=R,
        T=T,
        in_ndc=True,
        image_size=(height, width),
        convention='opencv',
        )
    )

# Initialize a perspective camera with specific `image_size`, `principal_points`, `focal_length`.
cam_from_params = build_cameras(
    dict(
        type='PerspectiveCameras',
        in_ndc=False,
        image_size=(1000, 1000),
        principal_points=(500, 500),
        focal_length=1000,
        convention='opencv',
        )
    )

# Initialize a weakperspective camera.
cam_weak = build_cameras(
    dict(
        type='WeakPerspectiveCameras',
        K=K,
        R=R,
        T=T,
        image_size=(1000, 1000)
        )
    )

# Initialize a perspective camera with default matrix (requires image_size for conversion).
cam_default = build_cameras(
    dict(
        type='PerspectiveCameras',
        in_ndc=True,
        image_size=(1000, 1000),
        ))
cam_default.to_screen_()
```

### Parameters
#### Request Body
- **type** (string) - Required - The type of camera to build (e.g., 'PerspectiveCameras', 'WeakPerspectiveCameras').
- **K** (torch.Tensor) - Optional - Intrinsic camera matrix.
- **R** (torch.Tensor) - Optional - Extrinsic camera rotation matrix.
- **T** (torch.Tensor) - Optional - Camera translation vector.
- **in_ndc** (bool) - Optional - Whether the camera is in normalized device coordinates.
- **image_size** (tuple or int) - Optional - The size of the image (height, width) or a single integer for square images.
- **convention** (string) - Optional - Camera convention ('opencv' or 'pytorch3d').
- **focal_length** (float or torch.Tensor) - Optional - Focal length of the camera.
- **principal_points** (tuple or torch.Tensor) - Optional - Principal point coordinates (cx, cy).

### Response
#### Success Response (200)
- **Camera Object** (object) - An instance of the specified camera type (e.g., `PerspectiveCameras`).
```