MICA

https://github.com/zielon/mica
Admin
MICA is the official repository for the ECCV 2022 paper "Towards Metrical Reconstruction of Human...
Tokens:5,188
Snippets:18
Trust Score:9.4
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install MICA System

Source: https://context7.com/zielon/mica/llms.txt

This section details the installation process for the MICA system. It includes cloning the repository, running an installation script, and optional manual installation using Conda. It also specifies the necessary pre-trained model files and their locations.

```bash
# Clone repository and run installation script
git clone https://github.com/Zielon/MICA.git
cd MICA
./install.sh
# You will be prompted for FLAME account credentials

# Manual installation alternative
conda env create -f environment.yml
conda activate mica

# Download pretrained MICA model
# Place mica.tar in data/pretrained/mica.tar

# Download InsightFace models
# Place antelopev2 and buffalo_l in ~/.insightface/models/
```

--------------------------------

### Install MICA Project Environment and Dependencies

Source: https://github.com/zielon/mica/blob/master/README.md

This snippet demonstrates how to clone the MICA repository and set up the required environment using a conda environment file or an installation script. It also mentions the necessity of obtaining and placing the FLAME2020 model.

```shell
git clone https://github.com/Zielon/MICA.git
cd MICA
./install.sh
```

--------------------------------

### Run Benchmark Evaluations (Bash)

Source: https://context7.com/zielon/mica/llms.txt

This snippet shows how to run benchmark evaluations for the MICA model on the NoW and Stirling datasets. It uses the test.py script with different configurations and checkpoints. The output meshes are saved in the output directory.

```bash
# NoW benchmark evaluation
python test.py \
    --cfg configs/mica.yaml \
    --test_dataset NOW \
    --checkpoint data/pretrained/mica.tar

# Stirling dataset evaluation
python test.py \
    --cfg configs/mica.yaml \
    --test_dataset STIRLING \
    --checkpoint data/pretrained/mica.tar

# Output meshes saved in output/{config}/test_results/
# Use now_evaluation repository to compute metrics
```

--------------------------------

### Training Custom MICA Model (Bash)

Source: https://context7.com/zielon/mica/llms.txt

This command initiates the distributed training process for a custom MICA model. It requires a configuration file (`.yaml`) and uses a Python script (`train.py`). Training outputs, including logs and checkpoints, are saved to the `output/` directory.

```bash
# Prepare configuration file
# configs/custom_config.yaml

# Start distributed training
python train.py --cfg configs/custom_config.yaml

# Training outputs saved to:
# output/{config_name}/logs/              # Training logs
# output/{config_name}/train_images/      # Training visualizations
# output/{config_name}/val_images/        # Validation visualizations
# output/{config_name}/model.tar          # Model checkpoints
```

--------------------------------

### Programmatic Model Testing (Python)

Source: https://context7.com/zielon/mica/llms.txt

This snippet provides a programmatic interface for testing the MICA model. It initializes the model and tester, then runs evaluations on the NoW and Stirling datasets using a specified checkpoint. Dependencies: micalib, configs.

```python
from micalib.tester import Tester
from configs.config import get_cfg_defaults

# Programmatic testing interface
cfg = get_cfg_defaults()
cfg.model.testing = True
device = 'cuda:0'

mica = util.find_model_using_name(
    model_dir='micalib.models',
    model_name='mica'
)(cfg, device)

tester = Tester(nfc_model=mica, config=cfg, device=device)
tester.render_mesh = True

# Run NoW challenge evaluation
checkpoint_path = 'data/pretrained/mica.tar'
tester.test_now(checkpoint_path)
# Generates meshes for NoW benchmark evaluation
# Calculate errors using: github.com/soubhiksanyal/now_evaluation

# Run Stirling dataset evaluation
tester.test_stirling(checkpoint_path)
# Outputs reconstruction accuracy metrics on Stirling 3D faces
```

--------------------------------

### FLAME Model Operations (Python)

Source: https://context7.com/zielon/mica/llms.txt

Demonstrates initializing the FLAME model and performing various operations such as generating meshes from shape and pose parameters, computing 2D/3D landmarks, projecting meshes to FLAME's shape space, and reconstructing the closest FLAME mesh. Requires PyTorch and FLAME configuration.

```python
from models.flame import FLAME
from configs.config import get_cfg_defaults

# Initialize FLAME model
cfg = get_cfg_defaults()
flame = FLAME(cfg.model).cuda()

# Generate mesh from shape parameters
batch_size = 1
shape_params = torch.randn(batch_size, 300).cuda()  # Random shape
expression_params = torch.zeros(batch_size, 100).cuda()  # Neutral expression
pose_params = torch.zeros(batch_size, 6).cuda()  # Neutral pose

vertices, landmarks_2d, landmarks_3d = flame(
    shape_params=shape_params,
    expression_params=expression_params,
    pose_params=pose_params
)

print(f"FLAME vertices: {vertices.shape}")  # [1, 5023, 3]
print(f"2D landmarks: {landmarks_2d.shape}")  # [1, 68, 3]
print(f"3D landmarks: {landmarks_3d.shape}")  # [1, 68, 3]

# Compute landmarks from existing mesh
mesh_vertices = torch.randn(1, 5023, 3).cuda()
computed_landmarks = flame.compute_landmarks(mesh_vertices)
print(f"Computed landmarks: {computed_landmarks.shape}")  # [1, 68, 3]

# Project arbitrary mesh to FLAME shape space
target_mesh = torch.randn(1, 5023, 3).cuda()
shape_code = flame.project_to_shape_basis(target_mesh)
print(f"Projected shape code: {shape_code.shape}")  # [1, 300]

# Reconstruct closest FLAME mesh
reconstructed_mesh, fitted_params = flame.compute_closest_shape(target_mesh)
reconstruction_error = torch.mean((target_mesh - reconstructed_mesh) ** 2)
print(f"Reconstruction error: {reconstruction_error.item():.6f}")
```

--------------------------------

### MICA Dataset FLAME Parameter Loading (Python)

Source: https://context7.com/zielon/mica/llms.txt

Loads FLAME 2020 parameters (pose and shape/expression betas) from a `.npz` file in the MICA dataset. It then formats these parameters into a dictionary suitable for initializing or running the FLAME model, specifically separating identity shape from expression parameters.

```python
import numpy as np
import torch

# Load FLAME parameters from MICA dataset
params_path = 'datasets/MICA/FLAME_parameters/subject_001/frame_000.npz'
params = np.load(params_path, allow_pickle=True)

# Extract FLAME2020 parameters
pose = torch.tensor(params['pose']).float()
betas = torch.tensor(params['betas']).float()

flame_params = {
    'shape_params': betas[:300],           # Identity shape
    'expression_params': betas[300:],      # Facial expression
    'pose_params': torch.cat([pose[:3], pose[6:9]])  # Head pose (jaw + neck)
}
```

--------------------------------

### Bash: Batch Process Multiple Images

Source: https://context7.com/zielon/mica/llms.txt

This bash command demonstrates how to perform batch processing of multiple images for face reconstruction. It specifies input and output directories, and paths to necessary pre-trained models. The output structure for each processed image is also detailed.

```bash
# Command-line demo processing
python demo.py \
    -i demo/input \
    -o demo/output \
    -a demo/arcface \
    -m data/pretrained/mica.tar

# Output structure for each detected face:
# demo/output/{image_name}/mesh.ply       # 3D mesh in millimeters
# demo/output/{image_name}/mesh.obj       # 3D mesh OBJ format
# demo/output/{image_name}/identity.npy   # 300-dim shape code
# demo/output/{image_name}/kpt7.npy       # 7 key facial landmarks
# demo/output/{image_name}/kpt68.npy      # 68 facial landmarks
```

--------------------------------

### MICA Model Components (PyTorch)

Source: https://context7.com/zielon/mica/llms.txt

This snippet illustrates the components of the MICA model architecture, including the MICA class, ArcFace encoder, and Generator. It shows how to initialize the model, access its components, and perform a forward pass. Requires micalib, models, torch.

```python
from micalib.models.mica import MICA
from models.arcface import Arcface
from models.generator import Generator

# Initialize complete MICA pipeline
cfg = get_cfg_defaults()
mica = MICA(config=cfg, device='cuda:0', tag='MICA')

# Access ArcFace encoder
arcface = mica.arcface
print(f"ArcFace output dim: 512")

# Process image through ArcFace
import torch.nn.functional as F
arcface_features = F.normalize(arcface(arcface_input_tensor))
print(f"ArcFace features: {arcface_features.shape}")  # [batch, 512]

# Access Generator (Mapping Network + FLAME)
generator = mica.flameModel
shape_params_dim = cfg.model.n_shape  # 300
mapping_layers = cfg.model.mapping_layers  # 3

# Generate mesh from identity code
predicted_vertices, predicted_shape = generator(arcface_features)
print(f"Predicted vertices: {predicted_vertices.shape}")  # [batch, 5023, 3]
print(f"Predicted shape params: {predicted_shape.shape}")  # [batch, 300]

# Forward pass through complete model
with torch.no_grad():
    codedict = mica.encode(image_tensor, arcface_tensor)
    # codedict contains: 'arcface' (512-dim), 'images' (224x224x3)

    opdict = mica.decode(codedict)
    # opdict contains:
    # - 'pred_canonical_shape_vertices': predicted mesh
    # - 'pred_shape_code': 300-dim FLAME shape parameters
    # - 'faceid': normalized ArcFace features

# Compute loss for training
losses = mica.compute_losses(input_batch, codedict, opdict)
print(f"Vertex loss: {losses['pred_verts_shape_canonical_diff']:.4f} mm")
```

--------------------------------

### Batch Image Processing with MICA

Source: https://context7.com/zielon/mica/llms.txt

This script processes images in a directory, detects faces, extracts facial landmarks, generates 3D face meshes using the MICA model, and exports them as OBJ files. It requires libraries like OpenCV, glob, tqdm, torch, and trimesh.

```python
input_dir = 'path/to/images'
output_dir = 'path/to/output'
os.makedirs(output_dir, exist_ok=True)

image_paths = sorted(glob(f'{input_dir}/*.*'))
results = []

for img_path in tqdm(image_paths):
    img = cv2.imread(img_path)
    bboxes, kpss = app.detect(img)

    if bboxes.shape[0] == 0:
        print(f"No face detected in {img_path}")
        continue

    i = get_center(bboxes, img)
    face = Face(bbox=bboxes[i, 0:4], kps=kpss[i], det_score=bboxes[i, 4])
    blob, aligned_img = get_arcface_input(face, img)

    image = cv2.resize(aligned_img, (224, 224)).transpose(2, 0, 1) / 255.0
    image_tensor = torch.tensor(image).float().cuda().unsqueeze(0)
    arcface_tensor = torch.tensor(blob).float().cuda().unsqueeze(0)

    with torch.no_grad():
        codedict = mica.encode(image_tensor, arcface_tensor)
        opdict = mica.decode(codedict)
        meshes = opdict['pred_canonical_shape_vertices']

    name = Path(img_path).stem
    vertices = meshes[0].cpu().numpy() * 1000.0
    mesh = trimesh.Trimesh(vertices=vertices, faces=faces, process=False)
    mesh.export(f'{output_dir}/{name}.obj')

    results.append({
        'image': name,
        'vertices': vertices.shape[0],
        'bbox_score': face.det_score
    })

print(f"Processed {len(results)} faces successfully")
```

--------------------------------

### Generate Mesh from FLAME Parameters (PyTorch)

Source: https://context7.com/zielon/mica/llms.txt

This snippet demonstrates how to generate a 3D mesh using the FLAME model with provided shape, expression, and pose parameters. It loads a registration mesh and prints the number of vertices for both meshes. Requires PyTorch and trimesh.

```python
vertices, lmk_2d, lmk_3d = flame(
    shape_params=flame_params['shape_params'].unsqueeze(0).cuda(),
    expression_params=flame_params['expression_params'].unsqueeze(0).cuda(),
    pose_params=flame_params['pose_params'].unsqueeze(0).cuda()
)

# Load corresponding registration mesh
reg_mesh = trimesh.load('datasets/MICA/registrations/subject_001/frame_000.obj')
print(f"Registration mesh: {len(reg_mesh.vertices)} vertices")
print(f"Generated mesh: {vertices.shape}")
# Output: Registration mesh: 5023 vertices
# Output: Generated mesh: torch.Size([1, 5023, 3])
```

--------------------------------

### Detect Face Landmarks (Python)

Source: https://context7.com/zielon/mica/llms.txt

This snippet demonstrates how to detect face landmarks using the LandmarksDetector class with RetinaFace and FAN models. It reads an image, detects faces, and prints bounding box and keypoint information. Dependencies: utils, cv2, numpy, insightface.

```python
from utils.landmark_detector import LandmarksDetector, detectors
import cv2
import numpy as np

# RetinaFace detector (default, fastest)
detector = LandmarksDetector(model=detectors.RETINAFACE, device='cuda:0')

image = cv2.imread('test_image.jpg')
bboxes, keypoints = detector.detect(image)

print(f"Detected {bboxes.shape[0]} faces")
print(f"Bounding boxes: {bboxes.shape}")  # [N, 5] (x1, y1, x2, y2, score)
print(f"Keypoints: {keypoints.shape}")    # [N, 5, 2] (eyes, nose, mouth corners)

# Process each detected face
for i in range(bboxes.shape[0]):
    x1, y1, x2, y2, score = bboxes[i]
    kps = keypoints[i]  # [left_eye, right_eye, nose, left_mouth, right_mouth]

    print(f"Face {i}: bbox=({x1:.1f}, {y1:.1f}, {x2:.1f}, {y2:.1f}), score={score:.3f}")
    print(f"  Left eye: {kps[0]}")
    print(f"  Right eye: {kps[1]}")
    print(f"  Nose: {kps[2]}")

# FAN detector (68 landmarks, more detailed)
detector_fan = LandmarksDetector(model=detectors.FAN, device='cuda:0')
bboxes_fan, keypoints_fan = detector_fan.detect(image)
print(f"FAN keypoints: {keypoints_fan.shape}")  # [N, 5, 2] (derived from 68 landmarks)

# Visualize detections
from datasets.creation.util import draw_on
from insightface.app.common import Face

faces = []
for i in range(bboxes.shape[0]):
    face = Face(bbox=bboxes[i, 0:4], kps=keypoints[i], det_score=bboxes[i, 4])
    faces.append(face)

annotated_image = draw_on(image, faces)
cv2.imwrite('detected_faces.jpg', annotated_image)
```

--------------------------------

### Loading FLAME Parameters with NumPy and PyTorch

Source: https://github.com/zielon/mica/blob/master/datasets/README.md

This snippet demonstrates how to load FLAME parameters from a .npz file using NumPy and convert them into PyTorch tensors for shape, expression, and pose. It assumes the .npz file contains 'pose' and 'betas' keys.

```python
import numpy as np
import torch

params = np.load('path.npz', allow_pickle=True)
pose = torch.tensor(params['pose']).float()
betas = torch.tensor(params['betas']).float()

flame = {
    'shape_params': betas[:300],
    'expression_params': betas[300:],
    'pose_params': torch.cat([pose[:3], pose[6:9]]),
}
```

--------------------------------

### Python: Reconstruct Face Mesh from Single Image

Source: https://context7.com/zielon/mica/llms.txt

This Python script demonstrates how to reconstruct a 3D face mesh from a single input image using the MICA system. It involves initializing the model, loading weights, detecting faces, extracting features, running inference, and saving the resulting mesh, shape code, and landmarks.

```python
import cv2
import torch
import numpy as np
import trimesh
from pathlib import Path
from configs.config import get_cfg_defaults
from utils import util
from utils.landmark_detector import LandmarksDetector, detectors

# Initialize configuration and model
cfg = get_cfg_defaults()
cfg.model.testing = True
device = 'cuda:0'
mica = util.find_model_using_name(
    model_dir='micalib.models',
    model_name='mica'
)(cfg, device)

# Load pretrained weights
checkpoint = torch.load('data/pretrained/mica.tar')
if 'arcface' in checkpoint:
    mica.arcface.load_state_dict(checkpoint['arcface'])
if 'flameModel' in checkpoint:
    mica.flameModel.load_state_dict(checkpoint['flameModel'])
mica.eval()

# Setup face detector
app = LandmarksDetector(model=detectors.RETINAFACE)

# Process single image
img_path = 'demo/input/portrait.jpg'
img = cv2.imread(img_path)
bboxes, kpss = app.detect(img)

if bboxes.shape[0] > 0:
    # Get face with highest detection score (center of image)
    from datasets.creation.util import get_center, get_arcface_input
    from insightface.app.common import Face

    i = get_center(bboxes, img)
    face = Face(bbox=bboxes[i, 0:4], kps=kpss[i], det_score=bboxes[i, 4])

    # Extract ArcFace features and aligned face crop
    blob, aligned_img = get_arcface_input(face, img)

    # Prepare tensors
    from skimage.io import imread
    image = cv2.resize(aligned_img, (224, 224)).transpose(2, 0, 1) / 255.0
    image_tensor = torch.tensor(image).float().cuda().unsqueeze(0)
    arcface_tensor = torch.tensor(blob).float().cuda().unsqueeze(0)

    # Run inference
    with torch.no_grad():
        codedict = mica.encode(image_tensor, arcface_tensor)
        opdict = mica.decode(codedict)
        meshes = opdict['pred_canonical_shape_vertices']
        shape_code = opdict['pred_shape_code']
        landmarks = mica.flame.compute_landmarks(meshes)

    # Extract outputs
    vertices = meshes[0].cpu().numpy() * 1000.0  # Convert to millimeters
    faces = mica.flameModel.generator.faces_tensor.cpu().numpy()
    lmk_68 = landmarks[0].cpu().numpy() * 1000.0
    identity_code = shape_code[0].cpu().numpy()

    # Save mesh
    mesh = trimesh.Trimesh(vertices=vertices, faces=faces, process=False)
    mesh.export('output/reconstructed_face.obj')
    np.save('output/shape_code.npy', identity_code)
    np.save('output/landmarks_68.npy', lmk_68)

    print(f"Mesh saved: {vertices.shape[0]} vertices, {faces.shape[0]} faces")
    print(f"Shape code: {identity_code.shape}")
    # Output: Mesh saved: 5023 vertices, 9976 faces
    # Output: Shape code: (300,)
```

--------------------------------

### MICA Model Training Configuration (Python)

Source: https://context7.com/zielon/mica/llms.txt

Defines the structure for configuring the training of a MICA model using `yacs.config`. It specifies parameters for the model architecture, dataset, training hyperparameters, and loss weighting for different facial regions. The configuration can be saved to a YAML file.

```python
# Training configuration structure
from yacs.config import CfgNode as CN

cfg = CN()
cfg.output_dir = 'output/my_experiment'

# Model settings
cfg.model = CN()
cfg.model.name = 'mica'
cfg.model.n_shape = 300
cfg.model.mapping_layers = 3
cfg.model.layers = 8
cfg.model.hidden_layers_size = 256
cfg.model.arcface_pretrained_model = 'path/to/arcface_backbone.pth'

# Dataset settings
cfg.dataset = CN()
cfg.dataset.training_data = ['LYHM', 'STIRLING', 'FLORENCE']
cfg.dataset.batch_size = 8
cfg.dataset.num_workers = 4
cfg.dataset.root = '/data/MICA/'

# Training parameters
cfg.train = CN()
cfg.train.max_epochs = 50
cfg.train.lr = 1e-4
cfg.train.arcface_lr = 1e-3
cfg.train.log_steps = 10
cfg.train.checkpoint_steps = 1000
cfg.train.val_steps = 1000

# Facial region weights for loss computation
cfg.mask_weights = CN()
cfg.mask_weights.face = 150.0
cfg.mask_weights.nose = 50.0
cfg.mask_weights.lips = 50.0
cfg.mask_weights.forehead = 50.0
cfg.mask_weights.eyes = 0.01
cfg.mask_weights.ears = 0.01
cfg.train.use_mask = True

# Save configuration
with open('configs/custom_config.yaml', 'w') as f:
    yaml.dump(cfg, f)
```

--------------------------------

### Shape Interpolation and Statistics with PyTorch and FLAME

Source: https://context7.com/zielon/mica/llms.txt

This snippet shows how to perform linear interpolation between two random shapes generated by the FLAME model. It also includes calculating mean and standard deviation of shapes and performing PCA on a set of random shapes to analyze shape space variance. This requires PyTorch and the FLAME model to be available.

```python
shape1 = torch.randn(1, 300).cuda()  # Identity 1
shape2 = torch.randn(1, 300).cuda()  # Identity 2

# Linear interpolation
alpha = 0.5
interpolated_shape = alpha * shape1 + (1 - alpha) * shape2

# Generate interpolated mesh
vertices_interp, _, _ = flame(shape_params=interpolated_shape)

# Shape statistics
n_shapes = 100
random_shapes = torch.randn(n_shapes, 300).cuda()
random_meshes, _, _ = flame(
    shape_params=random_shapes,
    expression_params=torch.zeros(n_shapes, 100).cuda()
)

mean_shape = random_meshes.mean(dim=0)
std_shape = random_meshes.std(dim=0)
print(f"Mean vertex position: {mean_shape.mean(dim=0)}")
print(f"Shape variation std: {std_shape.mean():.6f}")

# Principal component analysis on shape space
U, S, V = torch.pca_lowrank(random_shapes, q=10)
print(f"Top 10 PCA components explain: {(S[:10].sum()/S.sum()).item():.2%} variance")

# Distance in shape space
def shape_distance(code1, code2):
    return torch.norm(code1 - code2, dim=1)

distances = shape_distance(shape1.expand(n_shapes, -1), random_shapes)
closest_idx = torch.argmin(distances)
print(f"Closest shape distance: {distances[closest_idx]:.4f}")
```

--------------------------------

### MICA Citation for Research

Source: https://github.com/zielon/mica/blob/master/README.md

This snippet provides the BibTeX entry for citing the MICA project in academic research. It includes author names, title, journal, and year of publication.

```bibtex
@proceedings{zielonka22mica,
  author = {Zielonka, Wojciech and Bolkart, Timo and Thies, Justus},
  title = {Towards Metrical Reconstruction of Human Faces},
  journal = {European Conference on Computer Vision},
  year = {2022}
}
```

--------------------------------

### MICA Dataset Citation (BibTeX)

Source: https://github.com/zielon/mica/blob/master/datasets/README.md

This snippet provides the BibTeX entry for citing the MICA dataset when used in research. It includes author information, title, journal, and year of publication.

```bibtex
@proceedings{MICA:ECCV2022,
  author = {Zielonka, Wojciech and Bolkart, Timo and Thies, Justus},
  title = {Towards Metrical Reconstruction of Human Faces},
  journal = {European Conference on Computer Vision},
  year = {2022}
}
```
Tokens:
Raw