### Project Setup and Dependency Installation Source: https://github.com/gangweix/pixel-perfect-depth/blob/main/README.md This snippet outlines the initial steps for setting up the Pixel-Perfect Depth project. It includes cloning the repository and installing necessary Python dependencies via pip. Ensure you have Python and pip installed before running these commands. ```bash git clone https://github.com/gangweix/pixel-perfect-depth cd pixel-perfect-depth pip install -r requirements.txt ``` -------------------------------- ### Environment Setup and Model Download Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Provides bash commands to set up the project environment. This includes cloning the repository, installing dependencies via pip, and downloading necessary pretrained models (PPD main model, Depth Anything V2 semantic encoder, and optional MoGe model) using wget. A verification step checks the PyTorch installation and model import. ```bash # Clone repository git clone https://github.com/gangweix/pixel-perfect-depth cd pixel-perfect-depth # Install dependencies pip install -r requirements.txt # Download pretrained models mkdir -p checkpoints cd checkpoints # PPD main model (500M params) wget https://huggingface.co/gangweix/Pixel-Perfect-Depth/resolve/main/ppd.pth # Depth Anything V2 semantic encoder wget https://huggingface.co/depth-anything/Depth-Anything-V2-Large/resolve/main/depth_anything_v2_vitl.pth # MoGe for metric depth (optional, for point clouds) wget https://huggingface.co/Ruicheng/moge-2-vitl-normal/resolve/main/model.pt -O moge2.pt cd .. # Verify installation python -c "import torch; from ppd.models.ppd import PixelPerfectDepth; print('Setup complete')" ``` -------------------------------- ### Generate 3D Point Clouds with Metric Depth Recovery (Python) Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Generates 3D point clouds from depth maps with metric depth recovery using MoGe for scale estimation and camera intrinsics. It loads models, processes an image to get relative depth, uses MoGe for metric depth estimation, recovers metric depth using RANSAC, and converts the depth map to a point cloud with Open3D. Outlier removal and coordinate system correction are applied before saving. ```python import cv2 import numpy as np import torch import open3d as o3d from ppd.models.ppd import PixelPerfectDepth from ppd.moge.model.v2 import MoGeModel from ppd.utils.align_depth_func import recover_metric_depth_ransac from ppd.utils.depth2pcd import depth2pcd # Initialize models device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') moge = MoGeModel.from_pretrained("checkpoints/moge2.pt").to(device).eval() model = PixelPerfectDepth(semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=20) model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) model = model.to(device).eval() # Load image image = cv2.imread('assets/examples/0001.jpg') # Get relative depth depth, resize_image = model.infer_image(image) depth = depth.squeeze().cpu().numpy() # Get metric depth and intrinsics from MoGe resize_H, resize_W = resize_image.shape[:2] moge_image = cv2.cvtColor(resize_image, cv2.COLOR_BGR2RGB) moge_image = torch.tensor(moge_image / 255, dtype=torch.float32, device=device).permute(2, 0, 1) moge_depth, mask, intrinsic = moge.infer(moge_image) moge_depth[~mask] = moge_depth[mask].max() # Convert relative to metric depth metric_depth = recover_metric_depth_ransac(depth, moge_depth, mask) # Adjust intrinsic matrix to image resolution intrinsic[0, 0] *= resize_W intrinsic[1, 1] *= resize_H intrinsic[0, 2] *= resize_W intrinsic[1, 2] *= resize_H # Generate point cloud pcd = depth2pcd( metric_depth, intrinsic, color=cv2.cvtColor(resize_image, cv2.COLOR_BGR2RGB), input_mask=mask, ret_pcd=True ) # Apply statistical outlier removal cl, ind = pcd.remove_statistical_outlier(nb_neighbors=20, std_ratio=3.0) pcd = pcd.select_by_index(ind) # Fix coordinate system and save pcd.points = o3d.utility.Vector3dVector( np.asarray(pcd.points) * np.array([1, -1, -1], dtype=np.float32) ) o3d.io.write_point_cloud('output.ply', pcd) print(f"Point cloud saved with {len(pcd.points)} points") ``` -------------------------------- ### Command-Line Depth Estimation Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Executes depth estimation from the command line using the `run.py` script. Supports various options for input image paths (single image, directory, or text file list), output directories, specifying pretrained models, and controlling sampling steps. Can output visualizations or raw NumPy arrays. ```bash # Basic depth estimation on images in directory python run.py \ --img_path assets/examples \ --outdir depth_vis \ --semantics_pth checkpoints/depth_anything_v2_vitl.pth \ --sampling_steps 4 # Single image with raw depth output python run.py \ --img_path input.jpg \ --outdir results \ --sampling_steps 10 \ --pred_only \ --save_npy # Process from text file list python run.py \ --img_path image_list.txt \ --sampling_steps 4 \ --outdir batch_results ``` -------------------------------- ### Initialize PixelPerfectDepth Model Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Initializes the Pixel-Perfect Depth model, loading pretrained weights for both the semantic encoder (Depth Anything V2) and the main PPD model. It configures the model for inference and moves it to the appropriate device (GPU or CPU). ```python import torch from ppd.models.ppd import PixelPerfectDepth # Initialize model with default settings model = PixelPerfectDepth( semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=10 ) # Load pretrained weights model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) # Move to GPU and set to evaluation mode device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = model.to(device).eval() # Model ready for inference print(f"Model initialized on {device} with {sampling_steps} sampling steps") ``` -------------------------------- ### PixelPerfectDepth Model Initialization Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Initializes the Pixel-Perfect Depth model, loading pretrained weights for both the semantic encoder and the main PPD model. The model can be configured with custom sampling steps. ```APIDOC ## PixelPerfectDepth Model Initialization ### Description Initialize the Pixel-Perfect Depth model with semantic encoder and diffusion transformer components. The model loads pretrained weights for both the semantic encoder (Depth Anything V2) and the main PPD model. ### Method ```python PixelPerfectDepth( semantics_pth: str, sampling_steps: int = 10 ) ``` ### Parameters #### Initialization Parameters - **semantics_pth** (str) - Required - Path to the pretrained weights for the semantic encoder (Depth Anything V2). - **sampling_steps** (int) - Optional - The number of sampling steps for the diffusion process. Defaults to 10. #### Loading Weights ```python model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) ``` ### Request Example ```python import torch from ppd.models.ppd import PixelPerfectDepth # Initialize model with default settings model = PixelPerfectDepth( semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=10 ) # Load pretrained weights model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) # Move to GPU and set to evaluation mode device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = model.to(device).eval() # Model ready for inference print(f"Model initialized on {device} with {sampling_steps} sampling steps") ``` ### Response #### Success Response A fully initialized `PixelPerfectDepth` model instance ready for inference. #### Response Example ``` Model initialized on cuda with 10 sampling steps ``` ``` -------------------------------- ### Command-Line Point Cloud Generation Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Generates point clouds with metric depth and camera intrinsics using the `run_point_cloud.py` script. Allows specifying input images, output directories, sampling steps, and options to save point cloud files (.pcd) or NumPy arrays (.npy). Includes options for applying statistical filtering. ```bash # Generate point clouds with statistical filtering python run_point_cloud.py \ --img_path assets/examples \ --outdir depth_vis \ --sampling_steps 20 \ --save_pcd # Disable outlier filtering python run_point_cloud.py \ --img_path input.jpg \ --sampling_steps 20 \ --save_pcd \ --apply_filter False # Save both depth and point cloud python run_point_cloud.py \ --img_path assets/examples \ --sampling_steps 20 \ --save_npy \ --save_pcd \ --pred_only ``` -------------------------------- ### Diffusion Sampling with Euler ODE Solver Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Samples depth maps using the Euler ODE solver with a linear interpolation schedule. It requires PyTorch, LinearSchedule, Timesteps, and EulerSampler from ppd.utils. The process involves iterating through timesteps, obtaining predictions, and stepping towards the final depth map. The output is a tensor representing the depth map. ```python import torch from ppd.utils.schedule import LinearSchedule from ppd.utils.timesteps import Timesteps from ppd.utils.sampler import EulerSampler # Initialize schedule and timesteps device = torch.device('cuda') schedule = LinearSchedule(T=1000) sampling_steps = 10 timesteps = Timesteps(T=schedule.T, steps=sampling_steps, device=device) # Initialize Euler sampler sampler = EulerSampler( schedule=schedule, timesteps=timesteps, prediction_type='velocity' ) # Start from random noise latent = torch.randn(1, 1, 768, 1024).to(device) condition = torch.randn(1, 3, 768, 1024).to(device) # Sample loop (simplified - actual model inference needed) for timestep in timesteps: input_x = torch.cat([latent, condition], dim=1) # Get prediction from model (placeholder) velocity_pred = torch.randn_like(latent) # Step to next timestep latent = sampler.step(pred=velocity_pred, x_t=latent, t=timestep) # Final output (depth map) depth_output = latent + 0.5 # shift from [-0.5, 0.5] to [0, 1] print(f"Final depth shape: {depth_output.shape}") ``` -------------------------------- ### Running Depth Estimation on Images Source: https://github.com/gangweix/pixel-perfect-depth/blob/main/README.md This command executes the primary script for performing monocular depth estimation on input images using the Pixel-Perfect Depth model. It assumes the necessary pre-trained models have been downloaded and placed in the 'checkpoints/' directory. ```python python run.py ``` -------------------------------- ### Batch Depth Estimation Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Processes multiple images within a specified directory, performing depth estimation for each. Supports saving visualized depth maps and raw depth data. ```APIDOC ## Batch Depth Estimation ### Description Process multiple images in a directory with automatic file discovery and organized output saving. Supports both visualization and raw depth data export. ### Method ```python # This is a script-based example for batch processing. # The core logic involves iterating through images and calling model.infer_image. ``` ### Parameters #### Script Configuration (Example) - **img_path** (str) - Required - Path to the directory containing input images. - **outdir** (str) - Required - Directory to save visualized depth maps. - **sampling_steps** (int) - Optional - Number of sampling steps for the diffusion model. Defaults to 4. - **save_npy** (bool) - Optional - Whether to save raw depth data as .npy files. Defaults to True. ### Request Example ```python import argparse import cv2 import glob import matplotlib import numpy as np import os import torch from ppd.models.ppd import PixelPerfectDepth # Configuration img_path = 'assets/examples' outdir = 'depth_vis' sampling_steps = 4 save_npy = True # Initialize device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = PixelPerfectDepth(semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=sampling_steps) model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) model = model.to(device).eval() # Get file list filenames = glob.glob(os.path.join(img_path, '**/*'), recursive=True) filenames = sorted(filenames) os.makedirs(outdir, exist_ok=True) cmap = matplotlib.colormaps.get_cmap('Spectral') # Process batch for k, filename in enumerate(filenames): print(f'Progress {k+1}/{len(filenames)}: {filename}') image = cv2.imread(filename) if image is None: continue H, W = image.shape[:2] depth, _ = model.infer_image(image) depth = torch.nn.functional.interpolate(depth, size=(H, W), mode='bilinear', align_corners=False)[0, 0] depth = depth.squeeze().cpu().numpy() # Visualize vis_depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0 vis_depth = vis_depth.astype(np.uint8) vis_depth = (cmap(vis_depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8) cv2.imwrite(os.path.join(outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), vis_depth) # Save raw depth if save_npy: os.makedirs('depth_npy', exist_ok=True) np.save(os.path.join('depth_npy', os.path.splitext(os.path.basename(filename))[0] + '.npy'), depth) ``` ### Response #### Success Response Depth maps are saved to the specified `outdir` and optionally as `.npy` files in a `depth_npy` directory. Console output indicates processing progress. #### Response Example ``` Progress 1/10: assets/examples/image1.jpg Progress 2/10: assets/examples/subdir/image2.png ... ``` ``` -------------------------------- ### Batch Depth Estimation and Saving Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Processes multiple images in a specified directory for depth estimation. It automatically discovers image files, performs inference, and saves visualized depth maps as PNGs and optionally raw depth data as NumPy arrays. ```python import argparse import cv2 import glob import matplotlib import numpy as np import os import torch from ppd.models.ppd import PixelPerfectDepth # Configuration img_path = 'assets/examples' # ... (other configurations) # Initialize device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = PixelPerfectDepth(semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=sampling_steps) model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) model = model.to(device).eval() # Get file list filenames = glob.glob(os.path.join(img_path, '**/*'), recursive=True) filenames = sorted(filenames) os.makedirs(outdir, exist_ok=True) cmap = matplotlib.colormaps.get_cmap('Spectral') # Process batch for k, filename in enumerate(filenames): print(f'Progress {k+1}/{len(filenames)}: {filename}') image = cv2.imread(filename) if image is None: continue H, W = image.shape[:2] depth, _ = model.infer_image(image) depth = torch.nn.functional.interpolate(depth, size=(H, W), mode='bilinear', align_corners=False)[0, 0] depth = depth.squeeze().cpu().numpy() # Visualize vis_depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0 vis_depth = vis_depth.astype(np.uint8) vis_depth = (cmap(vis_depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8) cv2.imwrite(os.path.join(outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), vis_depth) # Save raw depth if save_npy: os.makedirs('depth_npy', exist_ok=True) np.save(os.path.join('depth_npy', os.path.splitext(os.path.basename(filename))[0] + '.npy'), depth) ``` -------------------------------- ### Generating Point Clouds from Images Source: https://github.com/gangweix/pixel-perfect-depth/blob/main/README.md This script generates point clouds from input images using the Pixel-Perfect Depth model. It requires metric depth and camera intrinsics, necessitating the download of the 'moge2.pt' model. The '--save_pcd' flag ensures the point cloud data is saved. ```python python run_point_cloud.py --save_pcd ``` -------------------------------- ### Image Preprocessing and Tensor Conversion (Python) Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Preprocesses raw images for input into depth estimation models. This includes resizing the image while preserving its aspect ratio and converting it into a PyTorch tensor format. The resizing ensures that the image fits within a target area (e.g., 1024x768) with dimensions rounded to multiples of 16, suitable for convolutional neural networks. The conversion process handles BGR to RGB color space transformation, normalization to [0, 1], and adding a batch dimension. ```python import cv2 import numpy as np import torch from ppd.utils.transform import image2tensor, resize_keep_aspect # Load BGR image image = cv2.imread('assets/examples/0001.jpg') print(f"Original image shape: {image.shape}") # Resize keeping aspect ratio (matches training area 1024x768) resized_image = resize_keep_aspect(image) print(f"Resized image shape: {resized_image.shape}") # Output: resized to match 1024*768 area, dimensions rounded to multiple of 16 # Convert to tensor (BGR -> RGB, normalize to [0, 1], add batch dimension) tensor_image = image2tensor(resized_image) print(f"Tensor shape: {tensor_image.shape}") ``` -------------------------------- ### Perform Depth Inference from Image Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Performs depth estimation on a single RGB image using the initialized Pixel-Perfect Depth model. This function handles image preprocessing, preserves aspect ratio, and returns the estimated depth map along with the resized input image. ```python import cv2 import numpy as np import torch.nn.functional as F from ppd.models.ppd import PixelPerfectDepth # Load and prepare model device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = PixelPerfectDepth(semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=4) model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) model = model.to(device).eval() # Load image image = cv2.imread('assets/examples/0001.jpg') H, W = image.shape[:2] # Inference with torch.no_grad(): depth, resize_image = model.infer_image(image) # Resize depth to original dimensions depth = F.interpolate(depth, size=(H, W), mode='bilinear', align_corners=False)[0, 0] depth = depth.squeeze().cpu().numpy() # Normalize for visualization vis_depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0 vis_depth = vis_depth.astype(np.uint8) print(f"Input shape: {image.shape}, Depth shape: {depth.shape}") ``` -------------------------------- ### Convert Depth Maps to 3D Point Clouds (Python) Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Converts depth maps to 3D point clouds using camera intrinsics. This function can optionally take color information and a mask for valid pixels. It returns an Open3D PointCloud object or NumPy arrays for points and colors. Dependencies include NumPy, Open3D, and the `depth2pcd` utility from the `ppd.utils` module. ```python import numpy as np import open3d as o3d from ppd.utils.depth2pcd import depth2pcd # Example depth map (H, W) and camera intrinsics (3, 3) depth = np.random.rand(480, 640) * 10.0 # depth in meters intrinsic = np.array([ [525.0, 0.0, 320.0], [0.0, 525.0, 240.0], [0.0, 0.0, 1.0] ]) # RGB image aligned with depth (H, W, 3) color = np.random.randint(0, 255, (480, 640, 3), dtype=np.uint8) # Optional mask for valid pixels (H, W) mask = depth > 0 # Convert to point cloud (returns Open3D PointCloud) pcd = depth2pcd( depth=depth, intrinsic=intrinsic, color=color, input_mask=mask, ret_pcd=True ) # Visualize o3d.visualization.draw_geometries([pcd]) # Or get NumPy arrays points, colors = depth2pcd( depth=depth, intrinsic=intrinsic, color=color, input_mask=mask, ret_pcd=False ) print(f"Generated {points.shape[0]} points with shape {points.shape}") ``` -------------------------------- ### Depth Inference from Image Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Performs monocular depth estimation on a single RGB image. The method handles preprocessing and returns the estimated depth map along with the resized input image. ```APIDOC ## Depth Inference from Image ### Description Perform depth estimation on a single RGB image. The method automatically handles image preprocessing, aspect ratio preservation, and returns both the depth map and the resized input image. ### Method ```python model.infer_image(image: np.ndarray) -> tuple[torch.Tensor, np.ndarray] ``` ### Parameters #### Input Image - **image** (np.ndarray) - Required - The input RGB image as a NumPy array (H, W, C) in BGR format (as loaded by OpenCV). ### Request Example ```python import cv2 import numpy as np import torch.nn.functional as F from ppd.models.ppd import PixelPerfectDepth # Load and prepare model device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') model = PixelPerfectDepth(semantics_pth='checkpoints/depth_anything_v2_vitl.pth', sampling_steps=4) model.load_state_dict(torch.load('checkpoints/ppd.pth', map_location='cpu'), strict=False) model = model.to(device).eval() # Load image image = cv2.imread('assets/examples/0001.jpg') H, W = image.shape[:2] # Inference with torch.no_grad(): depth, resize_image = model.infer_image(image) # Resize depth to original dimensions depth = F.interpolate(depth, size=(H, W), mode='bilinear', align_corners=False)[0, 0] depth = depth.squeeze().cpu().numpy() # Normalize for visualization vis_depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0 vis_depth = vis_depth.astype(np.uint8) print(f"Input shape: {image.shape}, Depth shape: {depth.shape}") ``` ### Response #### Success Response (200) - **depth** (torch.Tensor) - The estimated depth map as a PyTorch tensor. Shape: (1, 1, H, W). - **resize_image** (np.ndarray) - The input image after preprocessing (aspect ratio preserved), suitable for direct display. Shape: (H', W', C). #### Response Example ``` Input shape: (768, 1024, 3), Depth shape: (768, 1024) ``` ``` -------------------------------- ### DiT Forward Pass with Timestep and Semantics Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Performs a forward pass through the Diffusion Transformer (DiT) model. It accepts latent, condition, semantic features, and timestep as inputs. Requires PyTorch and the DiT model from ppd.models.dit. Outputs a tensor representing the prediction. ```python import torch from ppd.models.dit import DiT # Initialize DiT model dit = DiT( in_channels=4, # latent (1) + condition (3) out_channels=1, # depth prediction hidden_size=1024, depth=24, # 24 transformer blocks num_heads=16, mlp_ratio=4.0 ) device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') dit = dit.to(device) # Prepare inputs batch_size = 1 H, W = 768, 1024 latent = torch.randn(batch_size, 1, H, W).to(device) condition = torch.randn(batch_size, 3, H, W).to(device) input_x = torch.cat([latent, condition], dim=1) # Semantic features from Depth Anything V2 (from intermediate DiT layer) semantics = torch.randn(batch_size, (H//16)*(W//16), 1024).to(device) # Timestep (scalar or tensor) timestep = torch.tensor([500]).to(device) # Forward pass with torch.no_grad(): prediction = dit(x=input_x, semantics=semantics, timestep=timestep) print(f"Input shape: {input_x.shape}, Output shape: {prediction.shape}") # Output: torch.Size([1, 1, 768, 1024]) ``` -------------------------------- ### Recover Metric Depth using RANSAC Alignment (Python) Source: https://context7.com/gangweix/pixel-perfect-depth/llms.txt Aligns relative depth predictions to metric scale using RANSAC regression. This function is useful for correcting the scale of depth maps estimated by models like Pixel Perfect Depth when ground truth metric depth from another source (e.g., MoGe) is available. It requires the predicted depth, ground truth metric depth, and a mask indicating valid pixels. ```python import numpy as np from ppd.utils.align_depth_func import recover_metric_depth_ransac # Relative depth prediction from PPD (H, W) pred_depth = np.random.rand(768, 1024).astype(np.float32) # Metric depth from MoGe (H, W) gt_metric_depth = np.random.rand(768, 1024).astype(np.float32) * 10.0 # Valid pixel mask (H, W) mask = np.ones((768, 1024), dtype=bool) mask[gt_metric_depth < 0.1] = False # Recover metric depth using RANSAC alignment metric_depth = recover_metric_depth_ransac(pred_depth, gt_metric_depth, mask) print(f"Relative depth range: [{pred_depth.min():.3f}, {pred_depth.max():.3f}]") print(f"Metric depth range: [{metric_depth.min():.3f}, {metric_depth.max():.3f}]") # The function fits: log(gt + 1) = a * pred + b # Then applies: metric = exp(a * pred + b) - 1 ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.