### Initialize Integrated Renderer Setup

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the integrated renderer setup with configuration and device.

```python
render_utils(config, device)
```

--------------------------------

### Conda Environment Setup and Installation

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Provides bash commands to set up a conda environment and install PyTorch, PyTorch3D, and other necessary Python packages.

```bash
# Create conda environment
conda create -n pytorch3d python=3.8
conda activate pytorch3d

# Install PyTorch
conda install -c pytorch pytorch=1.5.0 torchvision cudatoolkit=10.2

# Install PyTorch3D
conda install -c pytorch3d pytorch3d=0.2.0

# Install other dependencies
pip install scikit-image opencv-python
```

--------------------------------

### Complete Example: Generate Face Dataset

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/render_utils.md

Demonstrates a full pipeline for generating a dataset of face images. It sets up the configuration, initializes the render utility, samples random parameters for shape, expression, pose, texture, lighting, and camera, renders the faces, and saves the output images.

```python
import torch
import torchvision
from gif_helper import render_utils

# Setup
config = type('Config', (), {
    'batch_size': 8,
    'image_size': 224,
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'cropped_size': 256,
})()

helper = render_utils(config, device='cuda')

# Sample random parameters
num_samples = 100
for i in range(0, num_samples, 8):
    batch_size = min(8, num_samples - i)
    
    shape = torch.randn(batch_size, 100, device='cuda') * 0.3
    expression = torch.randn(batch_size, 50, device='cuda') * 0.3
    pose = torch.zeros(batch_size, 6, device='cuda')
    pose[:, 0] = torch.randn(batch_size) * 0.3  # Yaw
    pose[:, 1] = torch.randn(batch_size) * 0.1  # Pitch
    
    tex = torch.randn(batch_size, 50, device='cuda') * 0.3
    lights = torch.randn(batch_size, 9, 3, device='cuda')
    
    cam = torch.ones(batch_size, 3, device='cuda')
    cam[:, 0] = 5.0  # Scale
    cam[:, 1] = torch.randn(batch_size) * 0.1  # tx
    cam[:, 2] = torch.randn(batch_size) * 0.1  # ty
    
    # Render
    textured, normals = helper.render_tex_and_normal(
        shape, expression, pose, tex, lights, cam)
    
    # Save batch
    grid = torchvision.utils.make_grid(textured, nrow=4)
    torchvision.utils.save_image(grid, f'output/batch_{i:05d}.png')
    
    print(f"Generated batch {i}–{i+batch_size-1}")

print(f"Generated {num_samples} face images")
```

--------------------------------

### Batch Processing Configuration and Loop

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/configuration.md

Set up configuration for processing multiple images in batches. This example demonstrates initializing the fitting process and iterating through batches of images, landmarks, and masks for optimization.

```python
config = util.dict2obj({
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    
    'shape_params': 100,
    'expression_params': 50,
    'tex_params': 50,
    
    'cropped_size': 256,
    'image_size': 224,
    'batch_size': 8,  # Process 8 images simultaneously
    
    'e_lr': 0.005,
    'e_wd': 0.0001,
    
    'w_pho': 8.0,
    'w_lmks': 1.0,
    'w_shape_reg': 1e-4,
    'w_expr_reg': 1e-4,
    
    'savefolder': './batch_results/',
    'use_face_contour': True,
})

fitting = PhotometricFitting(config, device='cuda')

# Process batches of 8 images
for batch_idx in range(0, num_images, 8):
    batch_images = [load_image(i) for i in range(batch_idx, batch_idx + 8)]
    batch_landmarks = [load_landmarks(i) for i in range(batch_idx, batch_idx + 8)]
    batch_masks = [load_mask(i) for i in range(batch_idx, batch_idx + 8)]
    
    # Stack into batch tensor and optimize
    ...
```

--------------------------------

### FLAME, FLAMETex, and Renderer Integration Example

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

This Python code demonstrates how to initialize FLAME, FLAMETex, and a Renderer, generate random parameters, create a 3D mesh, generate texture, and render the final image. Ensure necessary models and data paths are correctly configured.

```python
import torch
from models.FLAME import FLAME, FLAMETex
from renderer import Renderer

config = type('Config', (), {
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'shape_params': 100,
    'expression_params': 50,
    'tex_params': 50,
    'use_face_contour': True,
})()

# Initialize models
flame = FLAME(config).to('cuda')
tex_model = FLAMETex(config).to('cuda')
renderer = Renderer(224, './data/head_template_mesh.obj').to('cuda')

# Generate random parameters
batch_size = 1
shape = torch.randn(batch_size, 100, device='cuda') * 0.2
expression = torch.randn(batch_size, 50, device='cuda') * 0.2
pose = torch.zeros(batch_size, 6, device='cuda')
pose[:, 0] = 0.3  # Slight head tilt
texcode = torch.randn(batch_size, 50, device='cuda') * 0.5

# Generate 3D mesh
vertices, _, _ = flame(shape, expression, pose)

# Generate texture
texture = tex_model(texcode) / 255.0  # Normalize to [0, 1]

# Render
camera = torch.tensor([[5.0, 0.0, 0.0]], device='cuda')  # scale=5, tx=ty=0
lights = torch.randn(batch_size, 9, 3, device='cuda')  # SH lighting

trans_vertices = vertices.clone()
trans_vertices[:, :, 2] = trans_vertices[:, :, 2]

output = renderer(vertices, trans_vertices, texture, lights=lights)
rendered_image = output['images']  # [1, 3, 224, 224]
```

--------------------------------

### Batch Processing with PhotometricFitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

This example shows how to process multiple images in a batch using the `PhotometricFitting` class. It involves loading images, landmarks, and masks, normalizing them, and then calling the `optimize` method. The results are iterated to extract per-batch information.

```python
import torch
from photometric_fitting import PhotometricFitting

fitter = PhotometricFitting(config, device='cuda')

# Load batch
images = torch.stack([torch.from_numpy(cv2.imread(path)) for path in image_paths])
landmarks = torch.stack([torch.from_numpy(np.load(path)) for path in landmark_paths])
masks = torch.stack([torch.from_numpy(np.load(path)) for path in mask_paths])

# Normalize
images = images / 255.0
images = images[:, :, :, [2, 1, 0]].permute(0, 3, 1, 2)  # BGR to RGB, NHWC to NCHW
landmarks = (landmarks - 128) / 128  # Assuming pixel coords in [0, 256]

# Optimize
results = fitter.optimize(images, landmarks, masks, savefolder='./batch_output')

# Extract per-batch results
for i, result in enumerate(results):
    print(f"Image {i}: shape={result['shape'].shape}, pose={result['pose'].shape}")
```

--------------------------------

### Setup Mesh Renderer

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

Internal method to initialize the mesh renderer. This method is typically called internally and does not require direct user interaction.

```python
_setup_renderer() -> None
```

--------------------------------

### Example Usage of Photometric Fitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

Demonstrates loading and preprocessing an image, landmarks, and mask, then calling the optimize function and extracting results. Ensure the input tensors are on the correct device (e.g., 'cuda').

```python
import cv2
import torch
import numpy as np

# Load and preprocess image
image_path = './images/face.png'
landmark_path = './landmarks/face.npy'
mask_path = './masks/face.npy'

image = cv2.imread(image_path)
image = cv2.resize(image, (224, 224)).astype(np.float32) / 255.
image = image[:, :, [2, 1, 0]].transpose(2, 0, 1)
images = torch.from_numpy(image[None, :, :, :]).to('cuda')

landmarks = np.load(landmark_path).astype(np.float32)
landmarks[:, 0] = landmarks[:, 0] / 256 * 2 - 1
landmarks[:, 1] = landmarks[:, 1] / 224 * 2 - 1
landmarks = torch.from_numpy(landmarks)[None, :, :].float().to('cuda')

image_mask = np.load(mask_path, allow_pickle=True)
image_mask = (image_mask[..., None] != 0).astype('float32')
image_mask = image_mask.transpose(2, 0, 1)
image_masks = torch.from_numpy(image_mask[None, :, :, :]).to('cuda')

# Optimize
result = fitter.optimize(images, landmarks, image_masks, savefolder='./output')

# Extract parameters
shape_params = result['shape']  # [1, 100]
pose_params = result['pose']    # [1, 6]
vertices = result['verts']      # [1, V, 3]
```

--------------------------------

### Install PyTorch3D and Dependencies with Conda

Source: https://github.com/havenfeng/photometric_optimization/blob/master/README.md

Use this command to set up a conda environment with PyTorch 1.5 and PyTorch3D 0.2. Ensure CUDA support is available for better performance.

```bash
conda create -n pytorch3d python=3.8
conda activate pytorch3d
conda install -c pytorch pytorch=1.5.0 torchvision cudatoolkit=10.2
conda install -c conda-forge -c fvcore fvcore
conda install pytorch3d -c pytorch3d
```

--------------------------------

### Instantiate Struct with Model Data

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/types.md

Example of creating a Struct instance by passing a dictionary of model data. This allows accessing dictionary values as object attributes.

```python
from models.FLAME import Struct

model_data = {
    'f': array,
    'v_template': array,
    'shapedirs': array
}
model = Struct(**model_data)
print(model.f)  # Access as attribute
```

--------------------------------

### Initialize PhotometricFitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the optimization framework with a given configuration and device.

```python
PhotometricFitting(config, device='cuda')
```

--------------------------------

### Get FLAME Mesh Topology

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Retrieves the face indices (topology) for the FLAME mesh.

```python
get_flame_faces()
```

--------------------------------

### Get FLAME Face Indices

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/render_utils.md

Retrieves the triangle face indices for the FLAME mesh. Useful for mesh manipulation or analysis.

```python
faces = helper.get_flame_faces()
print(faces.shape)  # [1, 13776, 3]
```

--------------------------------

### Initialize Renderer

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/Renderer.md

Instantiate the Renderer with image size, OBJ filename, and optional UV size. Move the renderer to the CUDA device if available.

```python
from renderer import Renderer

renderer = Renderer(
    image_size=224,
    obj_filename='./data/head_template_mesh.obj',
    uv_size=256
)
renderer = renderer.to('cuda')
```

--------------------------------

### Create Configuration Object

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/configuration.md

Demonstrates how to create a configuration object by defining a dictionary of parameters and converting it to an object using util.dict2obj. This object is then passed to various classes in the library.

```python
import util

config_dict = {
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'cropped_size': 256,
    'batch_size': 1,
    'image_size': 224,
    'e_lr': 0.005,
    'e_wd': 0.0001,
    'savefolder': './test_results/',
    'w_pho': 8,
    'w_lmks': 1,
    'w_shape_reg': 1e-4,
    'w_expr_reg': 1e-4,
    'w_pose_reg': 0,
    'use_face_contour': True,
}

config = util.dict2obj(config_dict)
```

--------------------------------

### Save Intermediate Results During Optimization

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/errors.md

Periodically save the current mesh and loss history within the optimization loop to track progress and debug issues. This example saves every 50 iterations.

```python
# Inside optimization loop:
if k % 50 == 0:
    # Save current mesh
    renderer.save_obj(f'debug/iter_{k:04d}.obj', 
                     vertices=trans_vertices[0], 
                     textures=albedos[0])
    
    # Save loss history
    with open(f'debug/losses.txt', 'a') as f:
        f.write(f"{k}: {float(all_loss):.6f}\n")
```

--------------------------------

### Initialize render_utils Class

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/render_utils.md

Initializes the render_utils class with configuration and device settings. This sets up the FLAME model, texture model, and renderer.

```python
from gif_helper import render_utils

config = type('Config', (), {
    'batch_size': 1,
    'image_size': 224,
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'cropped_size': 256,
})()

helper = render_utils(config, device='cuda')
```

--------------------------------

### Render Mesh Geometry

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/Renderer.md

Renders mesh geometry with a fixed blue color and optional lighting. Use this method to visualize 3D shapes in image space. Lighting can be customized or use the default two-light setup.

```python
render_shape(vertices, transformed_vertices, images=None, lights=None)
```

```python
shape_render = renderer.render_shape(vertices, transformed_vertices)  # [2, 3, 224, 224]
```

--------------------------------

### Initialize and Use FLAME Model

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Demonstrates how to initialize the FLAME model and generate a 3D face mesh and landmarks from neutral parameters. Ensure the FLAME model and necessary configurations are loaded and moved to the appropriate device (e.g., 'cuda').

```python
import torch
from models.FLAME import FLAME

flame = FLAME(config).to('cuda')

batch_size = 1
shape = torch.zeros(batch_size, 100, device='cuda')
expression = torch.zeros(batch_size, 50, device='cuda')
pose = torch.zeros(batch_size, 6, device='cuda')

# Neutral face
vertices, landmarks2d, landmarks3d = flame(shape, expression, pose)

print(vertices.shape)      # [1, 5023, 3] - FLAME mesh
print(landmarks2d.shape)   # [1, 68, 3] - dynamic landmarks
print(landmarks3d.shape)   # [1, 68, 3] - full 3D landmarks
```

--------------------------------

### Run Single Image Photometric Fitting Demo

Source: https://github.com/havenfeng/photometric_optimization/blob/master/README.md

Execute the photometric fitting demo with a specified FFHQ image name and computing device. Ensure the script is run from the project's root directory.

```bash
python photometric_fitting.py 00000 cuda
```

--------------------------------

### Initialize PhotometricFitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

Initializes the photometric fitting pipeline. Ensure all required configuration parameters and model paths are correctly set.

```python
import torch
import numpy as np
from photometric_fitting import PhotometricFitting

# Configure optimization
config = type('Config', (), {
    'batch_size': 1,
    'image_size': 224,
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'savefolder': './test_results/',
    'w_pho': 8,
    'w_lmks': 1,
    'w_shape_reg': 1e-4,
    'w_expr_reg': 1e-4,
    'w_pose_reg': 0,
    'use_face_contour': True,
    'cropped_size': 256
})()

fitter = PhotometricFitting(config, device='cuda')
```

--------------------------------

### render_utils Constructor

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/render_utils.md

Initializes the render_utils class, setting up FLAME, FLAMETex, and Renderer instances.

```APIDOC
## Class: render_utils

Source: `gif_helper.py:22–52`

Helper class that integrates FLAME model, texture model, and rendering for convenient generation of textured face images with normal maps.

### Constructor

```python
render_utils(config, device='cuda')
```

Initializes FLAME, FLAMETex, and Renderer in a single class.

#### Parameters

- **config** (object) - Required - Configuration object (same as PhotometricFitting)
- **device** (str) - Optional - 'cuda' - Compute device: 'cuda' or 'cpu'

**Registered Attributes:**

- `self.flame` — FLAME model instance
- `self.flametex` — FLAMETex texture model instance
- `self.render` — Renderer instance
- `self.flame_faces` — Triangle face indices [1, F, 3]

### Example

```python
from gif_helper import render_utils

config = type('Config', (), {
    'batch_size': 1,
    'image_size': 224,
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'cropped_size': 256,
})()

helper = render_utils(config, device='cuda')
```
```

--------------------------------

### Create Directory if Not Exists

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/util.md

Creates a directory at the specified path if it does not already exist. Prints a message indicating creation.

```python
from util import check_mkdir

check_mkdir('./output/results')  # Creates if not exists, prints "making..."
```

--------------------------------

### Load FLAME Model

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the FLAME model by loading the configuration.

```python
FLAME(config)
```

--------------------------------

### Load FLAMETex Model

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the FLAMETex model by loading the configuration.

```python
FLAMETex(config)
```

--------------------------------

### Access Configuration Parameters

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/types.md

Illustrates how to access configuration parameters from the created config object. Values can be retrieved using attribute notation.

```python
print(config.shape_params)        # 100
print(config.w_pho)               # 8.0
```

--------------------------------

### PhotometricFitting Constructor

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

Initializes the photometric fitting pipeline with FLAME model, texture model, and renderer. It takes a configuration object and an optional device.

```APIDOC
## PhotometricFitting(config, device='cuda')

### Description
Initializes the photometric fitting pipeline with FLAME model, texture model, and renderer.

### Parameters
#### Path Parameters
- **config** (object) - Required - Configuration object with properties: batch_size, image_size, shape_params, expression_params, pose_params, tex_params, camera_params, flame_model_path, flame_lmk_embedding_path, tex_space_path, savefolder, and loss weights (w_pho, w_lmks, w_shape_reg, w_expr_reg, w_pose_reg)
- **device** (str) - Optional - 'cuda' - Compute device: 'cuda' for GPU or 'cpu' for CPU

### Request Example
```python
import torch
import numpy as np
from photometric_fitting import PhotometricFitting

# Configure optimization
config = type('Config', (), {
    'batch_size': 1,
    'image_size': 224,
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'savefolder': './test_results/',
    'w_pho': 8,
    'w_lmks': 1,
    'w_shape_reg': 1e-4,
    'w_expr_reg': 1e-4,
    'w_pose_reg': 0,
    'use_face_contour': True,
    'cropped_size': 256
})()

fitter = PhotometricFitting(config, device='cuda')
```
```

--------------------------------

### Initialize Renderer Component

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the renderer with specified image size, object filename, and UV map size.

```python
Renderer(image_size, obj_filename, uv_size)
```

--------------------------------

### Renderer Constructor

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/Renderer.md

Initializes the Renderer with image size, object filename, and optional UV texture size. It loads mesh topology and sets up rasterizers.

```APIDOC
## Class: Renderer

Source: `renderer.py:87–325`

Complete differentiable rendering pipeline combining mesh rasterization, UV texture mapping, normal computation, and spherical harmonics lighting.

### Constructor

```python
Renderer(image_size, obj_filename, uv_size=256)
```

Loads mesh topology and initializes rasterizers for screen-space and UV-space rendering.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| image_size | int | yes | — | Render resolution in pixels (square) |
| obj_filename | str | yes | — | Path to OBJ mesh file with UV coordinates |
| uv_size | int | no | 256 | UV texture resolution (square) |

**Registers as buffers:**
- `faces` — Vertex indices per triangle [1, F, 3]
- `raw_uvcoords` — Texture coordinates [1, V, 2]
- `uvcoords` — Normalized UV coords [1, V, 3]
- `uvfaces` — Texture face indices [1, F, 3]
- `face_uvcoords` — Interpolated UV per face [1, F, 3, 3]
- `face_colors` — Fixed blue shape color RGB [1, F, 3, 3]
- `constant_factor` — SH normalization constants [9]

**Example:**

```python
from renderer import Renderer

renderer = Renderer(
    image_size=224,
    obj_filename='./data/head_template_mesh.obj',
    uv_size=256
)
renderer = renderer.to('cuda')
```
```

--------------------------------

### Single Image Fitting with PhotometricFitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

This snippet demonstrates how to perform photometric fitting on a single image. It requires setting up a configuration object and then calling the `run` method with the image and landmark paths. The output includes optimized parameters and a 3D mesh.

```python
from photometric_fitting import PhotometricFitting
import util
import torch
import cv2
import numpy as np

# Setup
config = util.dict2obj({
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'tex_space_path': './data/FLAME_texture.npz',
    'shape_params': 100,
    'expression_params': 50,
    'pose_params': 6,
    'tex_params': 50,
    'camera_params': 3,
    'batch_size': 1,
    'image_size': 224,
    'cropped_size': 256,
    'e_lr': 0.005,
    'e_wd': 0.0001,
    'w_pho': 8,
    'w_lmks': 1,
    'w_shape_reg': 1e-4,
    'w_expr_reg': 1e-4,
    'w_pose_reg': 0,
    'savefolder': './results/',
    'use_face_contour': True,
})

fitter = PhotometricFitting(config, device='cuda')

# Run
fitter.run('./images/face.png', './landmarks/face.npy')

# Output: ./results/face.npy (parameters) + ./results/face.obj (mesh)
```

--------------------------------

### run(imagepath, landmarkpath)

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

High-level interface that loads an image and landmarks, runs the photometric optimization process, and saves the resulting OBJ mesh and parameter file.

```APIDOC
## Method: run

```python
run(imagepath, landmarkpath) -> None
```

High-level interface that loads image and landmarks, runs optimization, and saves OBJ mesh and parameter file.

### Parameters

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| imagepath | str | yes | Path to input image (PNG/JPG) |
| landmarkpath | str | yes | Path to NumPy file containing 2D landmarks |

### Process

1. Loads and resizes image to config.cropped_size (typically 256)
2. Loads face segmentation mask from FFHQ_seg/ directory
3. Normalizes landmarks to [-1, 1] range
4. Interpolates to config.image_size (typically 224)
5. Calls optimize() with visualization every 10 iterations
6. Saves reconstructed mesh as OBJ file
7. Saves all parameters as NumPy file

### Files Generated

- `{savefolder}/{image_name}/{0,10,20,...,1000}.jpg` — Visualization images during optimization
- `{savefolder}/{image_name}.npy` — Serialized parameter dictionary
- `{savefolder}/{image_name}.obj` — 3D mesh in OBJ format with texture

### Example

```python
fitter.run('./images/face_00000.png', './landmarks/face_00000.npy')
# Outputs to ./test_results/face_00000.npy and ./test_results/face_00000.obj
```
```

--------------------------------

### Initialize Renderer

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initializes the renderer component within the PhotometricFitting class.

```python
_setup_renderer()
```

--------------------------------

### Initialize FLAMETex Model

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Initializes the FLAMETex model with a given configuration. Ensure the 'tex_space_path' points to the FLAME texture data and 'tex_params' specifies the number of PCA components.

```python
from models.FLAME import FLAMETex

config = type('Config', (), {
    'tex_space_path': './data/FLAME_texture.npz',
    'tex_params': 50
})()

tex_model = FLAMETex(config).to('cuda')
```

--------------------------------

### Run Photometric Optimization

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Initiates the photometric fitting process with a given configuration and input image/landmarks. Ensure the FLAME model is downloaded and extracted, and input images are prepared.

```python
from photometric_fitting import PhotometricFitting
import util

config = util.dict2obj({ ... config dict ... })
fitter = PhotometricFitting(config)
fitter.run('image.png', 'landmarks.npy')
```

--------------------------------

### Initialize FLAME Model

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Instantiate the FLAME model with a configuration object. Ensure the configuration includes paths to the FLAME model and landmark embedding files, and specifies the number of shape and expression parameters. The model can be moved to a CUDA-enabled device.

```python
from models.FLAME import FLAME
import torch

config = type('Config', (), {
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'shape_params': 100,
    'expression_params': 50,
    'use_face_contour': True
})()

flame = FLAME(config).to('cuda')
```

--------------------------------

### check_mkdir

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/util.md

Creates a directory at the specified path if it does not already exist.

```APIDOC
## Function: check_mkdir

### Description
Creates directory if it does not exist.

### Parameters
#### Path Parameters
- **path** (str) - Required - Directory path to create

### Returns
None

### Example
```python
from util import check_mkdir

check_mkdir('./output/results')  # Creates if not exists, prints "making..."
```
```

--------------------------------

### FLAME Forward Method

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Generates a 3D face mesh and landmarks from FLAME parameters. It takes shape, expression, and pose parameters as input and returns the vertices, 2D landmarks, and 3D landmarks.

```APIDOC
## Method: forward

```python
forward(shape_params=None, expression_params=None, pose_params=None, 
        eye_pose_params=None) -> tuple
```

Generates 3D face mesh and landmarks from FLAME parameters.

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| shape_params | torch.Tensor | yes | — | Shape PCA coefficients [B, shape_params] |
| expression_params | torch.Tensor | yes | — | Expression PCA coefficients [B, expression_params] |
| pose_params | torch.Tensor | yes | — | Head pose in axis-angle format [B, 6] (3 global rotation + 3 neck rotation) |
| eye_pose_params | torch.Tensor | no | None | Eye rotation parameters [B, 6], defaults to zero |

### Returns

Tuple of three torch.Tensor:

1. **vertices** — [B, V, 3] 3D mesh vertices in world space
2. **landmarks2d** — [B, L, 3] Dynamic landmarks (contour + static) in 3D
3. **landmarks3d** — [B, 68, 3] Full 68 static 3D landmarks (FACS keypoints)

### Example

```python
import torch
from models.FLAME import FLAME

flame = FLAME(config).to('cuda')

batch_size = 1
shape = torch.zeros(batch_size, 100, device='cuda')
expression = torch.zeros(batch_size, 50, device='cuda')
pose = torch.zeros(batch_size, 6, device='cuda')

# Neutral face
vertices, landmarks2d, landmarks3d = flame(shape, expression, pose)

print(vertices.shape)      # [1, 5023, 3] - FLAME mesh
print(landmarks2d.shape)   # [1, 68, 3] - dynamic landmarks
print(landmarks3d.shape)   # [1, 68, 3] - full 3D landmarks
```
```

--------------------------------

### Custom Rendering with render_utils

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

This snippet demonstrates custom rendering using `render_utils`. It involves sampling random parameters for shape, expression, pose, texture, lights, and camera, and then using the `render_tex_and_normal` method to generate textured and normal maps.

```python
from gif_helper import render_utils
import torch

helper = render_utils(config, device='cuda')

# Sample random parameters
shape = torch.randn(2, 100, device='cuda') * 0.2
expression = torch.randn(2, 50, device='cuda') * 0.2
pose = torch.zeros(2, 6, device='cuda')
pose[:, 0] = torch.randn(2) * 0.3

tex = torch.randn(2, 50, device='cuda') * 0.3
lights = torch.randn(2, 9, 3, device='cuda')
cam = torch.ones(2, 3, device='cuda')
cam[:, 0] = 5.0

# Render
textured, normals = helper.render_tex_and_normal(shape, expression, pose, tex, lights, cam)

print(textured.shape)  # [2, 3, 224, 224]
print(normals.shape)   # [2, 3, 224, 224]
```

--------------------------------

### Render Mesh with Albedos and Lights

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Renders a mesh given vertices, transformed vertices, albedo textures, and lighting conditions. Returns a dictionary of rendering outputs.

```python
forward(vertices, trans_vertices, albedos, lights)
```

--------------------------------

### FLAMETex Constructor

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Initializes the FLAMETex model. It requires a configuration object containing paths to texture space data and the number of texture PCA components to use.

```APIDOC
## Class: FLAMETex

### Description
Texture space model using PCA decomposition of FLAME texture maps. Maps coefficient vectors to RGB texture images.

### Constructor
```python
FLAMETex(config)
```

#### Parameters
* **config** (object) - Required - Configuration with tex_space_path and tex_params

##### Config Properties:
* **tex_space_path** (str) - Path to FLAME_texture.npz (contains mean and principal components)
* **tex_params** (int) - Default: 50 - Number of texture PCA components to use

**Loads from tex_space_path (NPZ format):**
- `mean` — Mean texture (1, 512 × 512 × 3 flattened)
- `tex_dir` — Texture basis (512 × 512 × 3 flattened, 200 components)

**Registered Buffers:**
* **texture_mean** ([1, 1, 786432]) - Mean texture (flattened)
* **texture_basis** ([1, 1, 786432, tex_params]) - PCA basis truncated to tex_params

### Example
```python
from models.FLAME import FLAMETex

config = type('Config', (), {
    'tex_space_path': './data/FLAME_texture.npz',
    'tex_params': 50
})()

tex_model = FLAMETex(config).to('cuda')
```
```

--------------------------------

### Configure for Portrait Fitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/configuration.md

Sets typical balanced weights for photometric, landmark, shape, and expression losses for portrait fitting tasks.

```python
# Portrait fitting (balanced)
config.w_pho = 8.0
config.w_lmks = 1.0
config.w_shape_reg = 1e-4
config.w_expr_reg = 1e-4
```

--------------------------------

### Inspect Intermediate Values in Optimization Loop

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/errors.md

Add these print statements inside the PhotometricFitting.optimize() loop to check the bounds and ranges of vertices, projected landmarks, texture, and lighting coefficients. Also verifies if the mask is being used.

```python
# Add these inside PhotometricFitting.optimize() loop:

# Check vertices bounds
print(f"Vertices range: [{vertices.min():.3f}, {vertices.max():.3f}]")

# Check projected landmarks
print(f"Landmarks 2D range: [{landmarks2d.min():.3f}, {landmarks2d.max():.3f}]")

# Check texture
print(f"Albedo range: [{albedos.min():.3f}, {albedos.max():.3f}]")

# Check lighting coefficients
print(f"Lights norm: {lights.norm():.3f}")

# Verify mask is being used
print(f"Masked region: {image_masks.sum()} / {image_masks.numel()} pixels")
```

--------------------------------

### Run Photometric Optimization

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/PhotometricFitting.md

High-level interface to load images and landmarks, run optimization, and save the resulting mesh and parameters. Ensure input paths are valid.

```python
fitter.run('./images/face_00000.png', './landmarks/face_00000.npy')
```

--------------------------------

### Initialize Pytorch3dRasterizer

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/Renderer.md

Instantiates the Pytorch3dRasterizer module. Specify the output image resolution.

```python
rasterizer = Pytorch3dRasterizer(image_size=224)
```

--------------------------------

### Renderer Forward Method

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/Renderer.md

Renders a textured mesh with optional lighting. It takes vertex data, albedos, and lighting information to produce rendered images and other related outputs.

```APIDOC
## Method: forward

```python
forward(vertices, transformed_vertices, albedos, lights=None, light_type='point')
    -> dict
```

Renders textured mesh with optional lighting.

### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| vertices | torch.Tensor | yes | — | 3D vertex positions [B, V, 3] in world space (for normal computation) |
| transformed_vertices | torch.Tensor | yes | — | Projected vertices [B, V, 3] in NDC space [-1, 1], z adjusted for near plane |
| albedos | torch.Tensor | yes | — | Texture maps [B, 3, 256, 256], values in [0, 1] |
| lights | torch.Tensor | no | None | Lighting parameters [B, L, C]. Shape [B, 9, 3] for SH (9 coefficients, 3 RGB), [B, N, 6] for point/directional lights |
| light_type | str | no | 'point' | Lighting model: 'point' for point lights, directional for other |

### Returns

Dictionary with keys:

```python
{
    'images': torch.Tensor,          # [B, 3, H, W] - rendered RGB
    'albedo_images': torch.Tensor,   # [B, 3, H, W] - textured without shading
    'alpha_images': torch.Tensor,    # [B, 1, H, W] - visibility mask
    'pos_mask': torch.Tensor,        # [B, 1, H, W] - interior region mask (z < -0.05)
    'shading_images': torch.Tensor,  # [B, 3, H, W] - lighting contribution only
    'grid': torch.Tensor,            # [B, H, W, 2] - UV sampling grid
    'normals': torch.Tensor          # [B, V, 3] - vertex normals
}
```

### Rendering Pipeline

1. Adjusts transformed_vertices Z coordinate: adds 10 to move mesh away from near plane
2. Computes vertex normals from world-space vertices using face orientation
3. Rasterizes mesh to get per-pixel UV coordinates and normals
4. Samples albedo texture using UV grid sample with bilinear interpolation
5. Applies shading (if lights provided) via SH or point/directional lighting model
6. Multiplies shaded albedo by alpha mask

### Example

```python
import torch
from renderer import Renderer

renderer = Renderer(image_size=224, obj_filename='./data/head_template_mesh.obj')
renderer = renderer.to('cuda')

# Batch of 2 faces
batch_size = 2
num_verts = 5023  # Typical FLAME mesh

vertices = torch.randn(batch_size, num_verts, 3, device='cuda')
transformed_vertices = torch.randn(batch_size, num_verts, 3, device='cuda')
transformed_vertices = torch.clamp(transformed_vertices, -1, 1)
albedos = torch.rand(batch_size, 3, 256, 256, device='cuda')
lights_sh = torch.randn(batch_size, 9, 3, device='cuda')  # Spherical harmonics

outputs = renderer(vertices, transformed_vertices, albedos, lights=lights_sh)

rendered_image = outputs['images']           # [2, 3, 224, 224]
visibility_mask = outputs['alpha_images']    # [2, 1, 224, 224]
shading = outputs['shading_images']          # [2, 3, 224, 224]
```
```

--------------------------------

### FLAME Constructor

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/api-reference/FLAME.md

Initializes the FLAME model with a configuration object. The configuration specifies paths to model files and parameters for shape and expression.

```APIDOC
## Class: FLAME

Source: `models/FLAME.py:36–216`

Implements the FLAME parametric 3D face model using linear blend skinning (LBS) for deformation. Outputs textured vertices and 2D/3D facial landmarks.

### Constructor

```python
FLAME(config)
```

Loads and initializes the FLAME model from a pickle file with shape, pose, and expression parameters.

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| config | object | yes | Configuration with properties: flame_model_path, flame_lmk_embedding_path, shape_params, expression_params, use_face_contour |

**Config Properties:**

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| flame_model_path | str | — | Path to FLAME pickle model file (generic_model.pkl) |
| flame_lmk_embedding_path | str | — | Path to landmark embedding NumPy file |
| shape_params | int | 100 | Number of shape PCA components to use (max 300) |
| expression_params | int | 50 | Number of expression PCA components (max 100) |
| use_face_contour | bool | True | Include dynamic contour landmarks based on pose |

**Registered Buffers:**

| Name | Shape | Description |
|------|-------|-------------|
| faces_tensor | [F, 3] | Triangle face indices |
| v_template | [V, 3] | Template mesh vertices |
| shapedirs | [V, 3, shape_params + expr_params] | Shape and expression PCA basis |
| posedirs | [6 × 9 × 3, J × 3] | Pose PCA basis (reshaped) |
| J_regressor | [J, V] | Joint location regressor |
| parents | [J] | Kinematic tree parent indices |
| lbs_weights | [V, J + 1] | Linear blend skinning weights |
| *_lmk_faces_idx | Various | Face indices for landmarks |
| *_lmk_bary_coords | Various | Barycentric coordinates for landmarks |

**Example:**

```python
from models.FLAME import FLAME
import torch

config = type('Config', (), {
    'flame_model_path': './data/generic_model.pkl',
    'flame_lmk_embedding_path': './data/landmark_embedding.npy',
    'shape_params': 100,
    'expression_params': 50,
    'use_face_contour': True
})()

flame = FLAME(config).to('cuda')
```
```

--------------------------------

### Visualize Landmarks on Images

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Visualizes landmarks on a batch of images, returning the modified images.

```python
tensor_vis_landmarks(images, landmarks, ...)
```

--------------------------------

### Configure for Texture-Focused Fitting

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/configuration.md

Adjusts loss weights to prioritize photometric (texture) accuracy over landmark alignment for texture-focused fitting.

```python
# Texture-focused
config.w_pho = 20.0
config.w_lmks = 0.5
```

--------------------------------

### Generate Mesh from FLAME Parameters

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Generates mesh vertices, 2D landmarks, and 3D landmarks using FLAME parameters for shape, expression, and pose.

```python
forward(shape, expression, pose, eye_pose)
```

--------------------------------

### Required Python Packages

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

Lists the essential Python packages and their minimum version requirements for the project.

```plaintext
torch>=1.5.0
torchvision
pytorch3d>=0.2.0
numpy
scipy
scikit-image
opencv-python (cv2)
```

--------------------------------

### Utility Functions

Source: https://github.com/havenfeng/photometric_optimization/blob/master/_autodocs/README.md

A collection of geometry and utility functions for configuration, I/O, distance metrics, rotation conversions, camera projections, mesh operations, and visualization.

```APIDOC
## util

### Description
Geometry and utility functions.

### Functions
- `dict2obj()`: Converts dictionary to object.
- `check_mkdir()`: Creates directories if they don't exist.
- `l2_distance()`: Computes L2 distance between vertices.
- `quat2mat()`: Converts quaternions to rotation matrices.
- `batch_rodrigues()`: Converts axis-angle to rotation matrices.
- `batch_orth_proj()`: Performs batch orthogonal projection.
- `batch_persp_proj()`: Performs batch perspective projection.
- `face_vertices()`: Gets vertices for each face.
- `vertex_normals()`: Computes vertex normals.
- `tensor_vis_landmarks()`: Visualizes landmarks on a tensor.
- `plot_kpts()`: Plots keypoints.
- `save_obj()`: Exports mesh to OBJ format.
```