### Clone Repository and Setup Script Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Clones the PartCrafter repository from GitHub and executes the setup script to install remaining dependencies. Navigate into the cloned directory before running the script. ```bash git clone https://github.com/wgsxm/PartCrafter.git cd PartCrafter bash settings/setup.sh ``` -------------------------------- ### Minimal Dataset Configuration Example Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md An example of a minimal, valid dataset configuration file required for training. It lists essential paths and properties for each object. ```json [ { "mesh_path": "/path/to/object.glb", "surface_path": "/path/to/object.npy", "image_path": "/path/to/object.png", "num_parts": 4, "iou_mean": 0.5, "iou_max": 0.9, "valid": true }, { "...": "..." }, ... ] ``` -------------------------------- ### Training Configuration YAML Source: https://context7.com/wgsxm/partcrafter/llms.txt An example YAML configuration file for training PartCrafter, detailing model parameters, dataset settings, optimizer, learning rate scheduler, and training/validation hyperparameters. ```yaml model: pretrained_model_name_or_path: 'pretrained_weights/TripoSG' vae: num_tokens: 1024 transformer: enable_local_cross_attn: true global_attn_block_ids: [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20] dataset: config: - 'datasets/object_part_configs.json' training_ratio: 0.9 min_num_parts: 1 max_num_parts: 16 max_iou_mean: 0.2 max_iou_max: 0.2 shuffle_parts: true object_ratio: 0.3 optimizer: name: "adamw" lr: 5e-5 betas: [0.9, 0.999] weight_decay: 0.01 lr_scheduler: name: "constant_warmup" num_warmup_steps: 1000 train: batch_size_per_gpu: 32 epochs: 10 grad_checkpoint: true weighting_scheme: "logit_normal" cfg_dropout_prob: 0.1 training_objective: "-v" log_freq: 1 eval_freq: 1000 save_freq: 2000 ema_kwargs: decay: 0.9999 use_ema_warmup: true val: batch_size_per_gpu: 1 num_inference_steps: 50 rendering: radius: 4.0 num_views: 36 fps: 18 ``` -------------------------------- ### Finetune PartCrafter with more parts (16 parts) Source: https://context7.com/wgsxm/partcrafter/llms.txt Finetunes PartCrafter with a configuration supporting 16 parts, starting from a previously trained baseline. It specifies the config file, enables EMA, sets gradient accumulation steps, output directory, and loads a pretrained model checkpoint. ```bash # Finetune with more parts (16 parts) bash scripts/train_partcrafter.sh \ --config configs/mp16_nt512.yaml \ --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --load_pretrained_model baseline_mp8_nt512 \ --load_pretrained_model_ckpt 10000 \ --tag finetune_mp16_nt512 ``` -------------------------------- ### Train PartCrafter from TripoSG Source: https://github.com/wgsxm/partcrafter/blob/main/README.md This script trains PartCrafter starting from TripoSG weights. Modify the config file path and other parameters as needed. It's recommended to use 8 H20 GPUs with 96G VRAM each. ```bash bash scripts/train_partcrafter.sh --config configs/mp8_nt512.yaml --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --tag scaleup_mp8_nt512 ``` -------------------------------- ### Install Graphics Libraries with Conda Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Installs required graphics libraries (libegl, libglu, pyopengl) using conda-forge. This is useful if you lack root access and are using a conda environment. ```bash conda install -c conda-forge libegl libglu pyopengl ``` -------------------------------- ### Install PyTorch with CUDA Support Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Installs PyTorch version 2.5.1 with torchvision and torchaudio, specifically compiled for CUDA 12.4. Ensure your system has compatible NVIDIA drivers. ```bash pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124 ``` -------------------------------- ### Train PartCrafter from TripoSG base (8 GPUs) Source: https://context7.com/wgsxm/partcrafter/llms.txt Initiates the training of PartCrafter from the TripoSG base model, recommended for 8 GPUs. This command uses a specified configuration file, enables EMA, sets gradient accumulation steps, and defines output directory and a tag for the experiment. ```bash # Train from TripoSG base (8 GPUs recommended) bash scripts/train_partcrafter.sh \ --config configs/mp8_nt512.yaml \ --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --tag baseline_mp8_nt512 ``` -------------------------------- ### Finetune PartCrafter with more tokens (higher quality) Source: https://context7.com/wgsxm/partcrafter/llms.txt Finetunes PartCrafter for higher quality by increasing the number of tokens, using a configuration for 16 parts and 1024 tokens. It loads a pretrained model from a previous finetuning step. ```bash # Finetune with more tokens (higher quality) bash scripts/train_partcrafter.sh \ --config configs/mp16_nt1024.yaml \ --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --load_pretrained_model finetune_mp16_nt512 \ --load_pretrained_model_ckpt 10000 \ --tag finetune_mp16_nt1024 ``` -------------------------------- ### Finetune PartCrafter with More Tokens Source: https://github.com/wgsxm/partcrafter/blob/main/README.md This command finetunes PartCrafter with an increased number of tokens. It requires loading a previously finetuned model and checkpoint. ```bash bash scripts/train_partcrafter.sh --config configs/mp16_nt1024.yaml --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --load_pretrained_model scaleup_mp16_nt512 \ --load_pretrained_model_ckpt 10 \ --tag scaleup_mp16_nt1024 ``` -------------------------------- ### Step-by-step preprocessing: Sample surface points Source: https://context7.com/wgsxm/partcrafter/llms.txt Samples surface points from a single mesh file (e.g., GLB) and saves them to a specified output location. This is the first step in preparing a custom dataset. ```bash # 1. Sample surface points python datasets/preprocess/mesh_to_point.py \ --input assets/objects/scissors.glb \ --output preprocessed_data ``` -------------------------------- ### Step-by-step preprocessing: Render images Source: https://context7.com/wgsxm/partcrafter/llms.txt Renders images of a single mesh from various viewpoints and saves them to the specified output directory. This is part of the dataset preparation pipeline. ```bash # 2. Render images python datasets/preprocess/render.py \ --input assets/objects/scissors.glb \ --output preprocessed_data ``` -------------------------------- ### Download TripoSG Pretrained Models Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Use this command to download the TripoSG model weights from Hugging Face. Ensure the weights are stored in the specified directory. ```bash huggingface-cli download VAST-AI/TripoSG --local-dir pretrained_weights/TripoSG ``` -------------------------------- ### Finetune PartCrafter with More Parts Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Use this command to finetune PartCrafter when dealing with a larger number of parts. This command loads a previously trained model and checkpoint. ```bash bash scripts/train_partcrafter.sh --config configs/mp16_nt512.yaml --use_ema \ --gradient_accumulation_steps 4 \ --output_dir output_partcrafter \ --load_pretrained_model scaleup_mp8_nt512 \ --load_pretrained_model_ckpt 10 \ --tag scaleup_mp16_nt512 ``` -------------------------------- ### Download TripoSG base model Source: https://context7.com/wgsxm/partcrafter/llms.txt Downloads the TripoSG base model from Hugging Face to a local directory. This is a prerequisite for training PartCrafter from a pretrained checkpoint. ```bash # Download TripoSG base model first huggingface-cli download VAST-AI/TripoSG --local-dir pretrained_weights/TripoSG ``` -------------------------------- ### Implement OpenAI VLM Provider in Python Source: https://context7.com/wgsxm/partcrafter/llms.txt Create a new Python file for the OpenAI provider, defining functions for suggesting part counts and stylizing images. This provider can be integrated into the PartCrafter system by registering its module path. ```python # Create src/utils/providers/openai_provider.py import os from PIL import Image DEFAULT_VLM_MODEL = "gpt-4-vision-preview" def suggest_num_parts( image: Image.Image, max_num_parts: int, mode: str = "object", model_name: str = DEFAULT_VLM_MODEL, ) -> int: """ Use OpenAI VLM to suggest number of parts. Args: image: PIL Image to analyze max_num_parts: Maximum allowed parts mode: "object" or "scene" model_name: Override default model Returns: Suggested number of parts (clamped to [1, max_num_parts]) """ # Implement your OpenAI API call here # Return an integer pass def stylize_for_objaverse( image: Image.Image, model_name: str = "dall-e-3", ) -> Image.Image: """ Apply style transfer using OpenAI image generation. Returns: Stylized PIL Image """ # Implement your style transfer here pass ``` ```python # Register in src/utils/providers/__init__.py: PROVIDERS = { "gemini": "src.utils.providers.gemini_provider", "openai": "src.utils.providers.openai_provider", # Add this } ``` -------------------------------- ### PartCrafterPipeline API Initialization Source: https://context7.com/wgsxm/partcrafter/llms.txt Initialize the PartCrafter pipeline in Python for programmatic 3D generation. Requires PyTorch, PIL, and huggingface_hub. ```python import torch from PIL import Image from huggingface_hub import snapshot_download from src.pipelines.pipeline_partcrafter import PartCrafterPipeline from src.utils.data_utils import get_colored_mesh_composition ``` -------------------------------- ### Resume PartCrafter training from checkpoint Source: https://context7.com/wgsxm/partcrafter/llms.txt Resumes PartCrafter training from a specified iteration checkpoint. This command requires the configuration file, output directory, and a tag for the experiment, along with the iteration number to resume from. ```bash # Resume training from checkpoint bash scripts/train_partcrafter.sh \ --config configs/mp16_nt1024.yaml \ --use_ema \ --output_dir output_partcrafter \ --tag my_experiment \ --resume_from_iter 5000 ``` -------------------------------- ### Generate 3D Scene from Image Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Use this command to generate a 3D scene from a given image. The required PartCrafter-Scene model weights are downloaded automatically. The generated results are saved to a specified output directory. ```bash python scripts/inference_partcrafter_scene.py \ --image_path assets/images_scene/np6_0192a842-531c-419a-923e-28db4add8656_DiningRoom-31158.png \ --num_parts 6 --tag dining_room --render ``` -------------------------------- ### Sample Points from Mesh Surface Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md Use this script to sample points from the surface of a single 3D mesh. Specify the input GLB file and the output directory. ```python python datasets/preprocess/mesh_to_point.py --input assets/objects/scissors.glb --output preprocessed_data ``` -------------------------------- ### Mesh Utilities: Composition, Normalization, and Sampling Source: https://context7.com/wgsxm/partcrafter/llms.txt Provides utilities for working with 3D meshes, including creating colored compositions, normalizing scale, sampling surface points, and extracting parts from scenes. Requires trimesh. ```python import trimesh import numpy as np from src.utils.data_utils import ( normalize_mesh, get_colored_mesh_composition, mesh_to_surface, scene_to_parts, move_meshes_by_center ) # Load generated parts meshes = [trimesh.load(f"part_{i:02d}.glb") for i in range(4)] # Create colored composition with default colors colored_scene = get_colored_mesh_composition( meshes, is_random=False, # Use predefined color palette is_sorted=True # Sort by volume (largest first) ) colored_scene.export("colored_composition.glb") # Explode parts outward for visualization exploded = move_meshes_by_center(meshes, scale=1.5) exploded.export("exploded_view.glb") # Normalize mesh to unit scale mesh = trimesh.load("input.glb") normalized = normalize_mesh(mesh, scale=2.0) # Sample surface points from mesh points, normals = mesh_to_surface( mesh, num_pc=204800, # Number of points clip_to_num_vertices=False ) print(f"Sampled {len(points)} surface points") # Extract parts from a scene scene = trimesh.load("multi_part_scene.glb") parts = scene_to_parts( scene, normalize=True, scale=2.0, return_type="mesh" # or "point" for point clouds ) print(f"Extracted {len(parts)} parts from scene") ``` -------------------------------- ### Suggest Number of Parts for Object/Scene Source: https://context7.com/wgsxm/partcrafter/llms.txt Programmatically suggests the number of parts for an object or objects in a scene using a VLM provider like Gemini. Requires setting the GEMINI_API_KEY environment variable. ```python import os from src.utils.vlm_utils import suggest_num_parts # Set API key os.environ["GEMINI_API_KEY"] = "your_api_key" # Suggest parts for an object num_parts = suggest_num_parts( image_path="input_object.png", max_num_parts=16, mode="object", # "object" for single objects, "scene" for scenes provider="gemini", model_name="gemini-3-flash-preview" # Optional, uses default if None ) print(f"Suggested number of parts: {num_parts}") # Suggest objects for a scene num_objects = suggest_num_parts( image_path="input_scene.png", max_num_parts=8, mode="scene", provider="gemini" ) print(f"Suggested number of objects: {num_objects}") ``` -------------------------------- ### PartCrafter Model Generation Source: https://context7.com/wgsxm/partcrafter/llms.txt Demonstrates how to download, load, and use the PartCrafter model to generate 3D parts from an input image. ```APIDOC ## PartCrafter Model Generation ### Description This section shows how to download and load the PartCrafter model, prepare an input image, and generate 3D parts. It also covers accessing the generated meshes and exporting them. ### Method N/A (Python script) ### Endpoint N/A ### Parameters N/A ### Request Example ```python # Download and load the model snapshot_download(repo_id="wgsxm/PartCrafter", local_dir="pretrained_weights/PartCrafter") pipe = PartCrafterPipeline.from_pretrained("pretrained_weights/PartCrafter") pipe = pipe.to("cuda", torch.float16) # Load and prepare input image image = Image.open("input.png") num_parts = 4 # Generate 3D parts outputs = pipe( image=[image] * num_parts, # Replicate image for each part attention_kwargs={"num_parts": num_parts}, num_tokens=1024, generator=torch.Generator(device="cuda").manual_seed(42), num_inference_steps=50, guidance_scale=7.0, ) # Access generated meshes meshes = outputs.meshes # List of trimesh.Trimesh objects # Export individual parts for i, mesh in enumerate(meshes): if mesh is not None: mesh.export(f"part_{i:02d}.glb") # Create colored composite mesh composite = get_colored_mesh_composition(meshes) composite.export("composite.glb") ``` ### Response N/A (Generates files) ``` -------------------------------- ### Style Transfer for Real-World Images Source: https://context7.com/wgsxm/partcrafter/llms.txt Apply style transfer to real-world photos before generation using the --style_transfer flag. Requires setting GEMINI_API_KEY. ```bash GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --num_parts 4 \ --style_transfer \ --rmbg \ --render \ --tag styled_object ``` -------------------------------- ### Combine Style Transfer and Part Suggestion Source: https://context7.com/wgsxm/partcrafter/llms.txt Integrate style transfer with automatic part suggestion for a comprehensive pipeline. Specify --style_provider and --style_model for VLM customization. ```bash GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --part_suggest \ --style_transfer \ --style_provider gemini \ --style_model gemini-3.1-flash-image-preview \ --rmbg \ --render \ --tag full_pipeline ``` -------------------------------- ### Rendering Utilities Source: https://context7.com/wgsxm/partcrafter/llms.txt Renders generated meshes with customizable camera positions and lighting. ```APIDOC ## Rendering Utilities ### Description Render generated meshes with customizable camera positions and lighting. These utilities provide flexible options for visualizing 3D models from various viewpoints. ### Method N/A (Python script) ### Endpoint N/A ### Parameters N/A ### Request Example ```python from src.utils.render_utils import ( render_views_around_mesh, render_normal_views_around_mesh, render_single_view, export_renderings, make_grid_for_images_or_videos ) import trimesh import numpy as np # Load mesh mesh = trimesh.load("object.glb") # Example usage (specific functions not detailed in source text) # render_views_around_mesh(mesh) # render_normal_views_around_mesh(mesh) # render_single_view(mesh, camera_position=[...]) # export_renderings(...) # make_grid_for_images_or_videos(...) ``` ### Response N/A (Rendering functions typically return image data or save files, specific return types depend on the function called.) ``` -------------------------------- ### VLM-Based Automatic Part Suggestion Source: https://context7.com/wgsxm/partcrafter/llms.txt Automatically determine the number of parts using a Vision Language Model by setting the --part_suggest flag. Requires setting GEMINI_API_KEY. ```bash GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --part_suggest \ --tag auto_parts \ --rmbg \ --render ``` -------------------------------- ### Rendering Utilities for Meshes Source: https://context7.com/wgsxm/partcrafter/llms.txt Provides functions for rendering generated meshes from various camera angles and exporting the renderings. Requires trimesh. ```python from src.utils.render_utils import ( render_views_around_mesh, render_normal_views_around_mesh, render_single_view, export_renderings, make_grid_for_images_or_videos ) import trimesh import numpy as np # Load mesh mesh = trimesh.load("object.glb") ``` -------------------------------- ### Generate and Export 3D Parts Source: https://context7.com/wgsxm/partcrafter/llms.txt Loads a PartCrafter model, generates 3D parts from an input image, and exports them as GLB files. Requires PyTorch and trimesh. ```python from huggingface_hub import snapshot_download from partcrafter import PartCrafterPipeline import torch from PIL import Image from partcrafter.utils.mesh_utils import get_colored_mesh_composition # Download and load the model snapshot_download(repo_id="wgsxm/PartCrafter", local_dir="pretrained_weights/PartCrafter") pipe = PartCrafterPipeline.from_pretrained("pretrained_weights/PartCrafter") pipe = pipe.to("cuda", torch.float16) # Load and prepare input image image = Image.open("input.png") num_parts = 4 # Generate 3D parts outputs = pipe( image=[image] * num_parts, # Replicate image for each part attention_kwargs={"num_parts": num_parts}, num_tokens=1024, generator=torch.Generator(device="cuda").manual_seed(42), num_inference_steps=50, guidance_scale=7.0, ) # Access generated meshes meshes = outputs.meshes # List of trimesh.Trimesh objects # Export individual parts for i, mesh in enumerate(meshes): if mesh is not None: mesh.export(f"part_{i:02d}.glb") # Create colored composite mesh composite = get_colored_mesh_composition(meshes) composite.export("composite.glb") ``` -------------------------------- ### Preprocess a folder of GLB meshes Source: https://context7.com/wgsxm/partcrafter/llms.txt Preprocesses a folder containing GLB mesh files for PartCrafter training. This script converts meshes into a format suitable for the dataset, outputting to a specified directory. ```bash # Preprocess a folder of GLB meshes python datasets/preprocess/preprocess.py \ --input path/to/glb_files \ --output preprocessed_data ``` -------------------------------- ### Prepare Image with Background Removal Source: https://context7.com/wgsxm/partcrafter/llms.txt Uses the BriaRMBG model to remove the background from an input image and save the processed image. Requires PyTorch and Pillow. ```python import torch import numpy as np from PIL import Image from huggingface_hub import snapshot_download from src.models.briarmbg import BriaRMBG from src.utils.image_utils import prepare_image # Load RMBG model snapshot_download(repo_id="briaai/RMBG-1.4", local_dir="pretrained_weights/RMBG-1.4") rmbg_net = BriaRMBG.from_pretrained("pretrained_weights/RMBG-1.4").to("cuda") rmbg_net.eval() # Prepare image with background removal processed_image = prepare_image( image_path="input_with_background.jpg", bg_color=np.array([1.0, 1.0, 1.0]), # White background rmbg_net=rmbg_net, padding_ratio=0.1, device="cuda" ) # Save processed image processed_image.save("processed_input.png") ``` -------------------------------- ### Scene Generation with Automatic Object Count Source: https://context7.com/wgsxm/partcrafter/llms.txt Generate a 3D scene with automatic object count suggestion using VLM. Requires setting GEMINI_API_KEY and enabling --part_suggest and --style_transfer. ```bash GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter_scene.py \ --image_path scene_photo.jpg \ --part_suggest \ --style_transfer \ --tag auto_scene \ --render ``` -------------------------------- ### Generate 3D Part-Level Object from Image Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Use this command to generate a 3D part-level object from a given image. The required model weights are downloaded automatically. The generated results are saved to a specified output directory. ```bash python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --num_parts 3 --tag robot --render ``` -------------------------------- ### Preprocess Folder of Meshes Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md This script preprocesses an entire folder of 3D meshes, generating necessary data and a configuration file. Specify the input folder containing GLB files and the output directory. ```python python datasets/preprocess/preprocess.py --input assets/objects --output preprocessed_data ``` -------------------------------- ### Render 360-degree rotation and export as GIF Source: https://context7.com/wgsxm/partcrafter/llms.txt Renders a mesh from multiple viewpoints to create a 360-degree rotation animation and exports it as a GIF. Requires a mesh object and specifies rendering parameters like number of views, radius, image size, field of view, and light intensity. ```python rendered_images = render_views_around_mesh( mesh, num_views=36, radius=4.0, image_size=(512, 512), fov=40.0, light_intensity=5.0 ) export_renderings(rendered_images, "rotation.gif", fps=18) ``` -------------------------------- ### VLM-Based Part Suggestion for 3D Object Generation Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Instead of manually specifying the number of parts, use a VLM to automatically suggest the part count. This requires setting the GEMINI_API_KEY and uses the VLM to analyze the object. ```bash GEMINI_API_KEY=your_key python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --part_suggest --tag robot --rmbg --render ``` -------------------------------- ### Style Transfer for Real-World Images Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Apply style transfer to real-world photos to bridge the domain gap between real images and the Objaverse-style renderings PartCrafter was trained on. This converts the input photo to an Objaverse-style 3D rendering before feeding it to the pipeline. ```bash GEMINI_API_KEY=your_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --num_parts 4 --style_transfer --rmbg --render ``` -------------------------------- ### Scene Generation with High Token Count Source: https://context7.com/wgsxm/partcrafter/llms.txt Generate a 3D scene with higher quality by increasing the number of tokens using --num_tokens. The scene model defaults to 2048 tokens. ```bash python scripts/inference_partcrafter_scene.py \ --image_path scene_input.png \ --num_parts 7 \ --num_tokens 2048 \ --num_inference_steps 50 \ --guidance_scale 7.0 \ --tag bedroom_scene \ --render ``` -------------------------------- ### Generate 3D Object with Custom Image and Options Source: https://context7.com/wgsxm/partcrafter/llms.txt Generate a 3D object from a custom image, optionally removing the background with --rmbg. A seed can be set for reproducibility. ```bash python scripts/inference_partcrafter.py \ --image_path my_custom_image.png \ --num_parts 4 \ --tag custom_object \ --rmbg \ --render \ --seed 42 ``` -------------------------------- ### Create image grid from multiple renders Source: https://context7.com/wgsxm/partcrafter/llms.txt Creates a grid of images from a list of rendered images or video frames. Useful for visualizing multiple views or steps of a process. Requires a list of images, number of rows, and padding. ```python grid = make_grid_for_images_or_videos( rendered_images[:9], # First 9 views nrow=3, padding=0 ) grid.save("render_grid.png") ``` -------------------------------- ### Advanced Object Generation Parameters Source: https://context7.com/wgsxm/partcrafter/llms.txt Control generation quality and output location using advanced parameters like --num_tokens, --num_inference_steps, --guidance_scale, and --output_dir. ```bash python scripts/inference_partcrafter.py \ --image_path input.png \ --num_parts 5 \ --num_tokens 1024 \ --num_inference_steps 50 \ --guidance_scale 7.0 \ --output_dir ./my_results \ --tag experiment_01 \ --render ``` -------------------------------- ### Render Images from 3D Model Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md This script renders images from a given 3D GLB model. Provide the input mesh path and the desired output directory. ```python python datasets/preprocess/render.py --input assets/objects/scissors.glb --output preprocessed_data ``` -------------------------------- ### Dataset Configuration Format Source: https://context7.com/wgsxm/partcrafter/llms.txt Defines the JSON format for configuring custom datasets for PartCrafter training. Each entry specifies paths to mesh, surface points, and images, along with part count and IoU metrics. ```json [ { "mesh_path": "/path/to/object.glb", "surface_path": "/path/to/object.npy", "image_path": "/path/to/object.png", "num_parts": 4, "iou_mean": 0.15, "iou_max": 0.18, "valid": true }, { "mesh_path": "/path/to/another_object.glb", "surface_path": "/path/to/another_object.npy", "image_path": "/path/to/another_object.png", "num_parts": 6, "iou_mean": 0.12, "iou_max": 0.16, "valid": true } ] ``` -------------------------------- ### Generate 3D Object with Multiple Parts Source: https://context7.com/wgsxm/partcrafter/llms.txt Use the inference script for basic object generation. Specify image path, number of parts, and a tag. The --render flag generates a 360-degree animation. ```bash python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --num_parts 3 \ --tag robot \ --render ``` -------------------------------- ### Combine VLM Part Suggestion and Style Transfer Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Both VLM-based part suggestion and style transfer features can be combined for generating 3D objects from real-world images. ```bash GEMINI_API_KEY=your_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --part_suggest --style_transfer --rmbg --render ``` -------------------------------- ### Apply Objaverse-Style Transfer Source: https://context7.com/wgsxm/partcrafter/llms.txt Applies Objaverse-style rendering to real-world images programmatically using a VLM provider. Requires setting the GEMINI_API_KEY environment variable. ```python import os from src.utils.style_transfer_utils import stylize_for_objaverse # Set API key os.environ["GEMINI_API_KEY"] = "your_api_key" # Apply style transfer stylized_image = stylize_for_objaverse( image_path="real_world_photo.jpg", output_path="stylized_output.png", provider="gemini", model_name="gemini-3.1-flash-image-preview" # Optional ) # The stylized image is saved and returned as PIL Image print(f"Stylized image saved to stylized_output.png") ``` -------------------------------- ### Remove Background and Resize Rendered Images Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md Applies background removal to rendered images and resizes them to 90% of their original dimensions. Specify the input image path and the output directory. ```python python datasets/preprocess/rmbg.py --input preprocessed_data/scissors/rendering.png --output preprocessed_data ``` -------------------------------- ### VLM Provider and Model Override Source: https://context7.com/wgsxm/partcrafter/llms.txt Customize the VLM provider and model used for part suggestion by specifying --part_provider and --part_model. ```bash GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path input.png \ --part_suggest \ --part_provider gemini \ --part_model gemini-3-flash-preview \ --tag custom_vlm \ --render ``` -------------------------------- ### Background Removal and Image Preparation Source: https://context7.com/wgsxm/partcrafter/llms.txt Utilizes the RMBG model for automatic background removal and image preparation. ```APIDOC ## Background Removal and Image Preparation ### Description Prepare input images with automatic background removal using the RMBG model. This utility helps in isolating the subject of an image by removing its background and applying a specified background color. ### Method N/A (Python script) ### Endpoint N/A ### Parameters N/A ### Request Example ```python import torch import numpy as np from PIL import Image from huggingface_hub import snapshot_download from src.models.briarmbg import BriaRMBG from src.utils.image_utils import prepare_image # Load RMBG model snapshot_download(repo_id="briaai/RMBG-1.4", local_dir="pretrained_weights/RMBG-1.4") rmbg_net = BriaRMBG.from_pretrained("pretrained_weights/RMBG-1.4").to("cuda") rmbg_net.eval() # Prepare image with background removal processed_image = prepare_image( image_path="input_with_background.jpg", bg_color=np.array([1.0, 1.0, 1.0]), # White background rmbg_net=rmbg_net, padding_ratio=0.1, device="cuda" ) # Save processed image processed_image.save("processed_input.png") ``` ### Response N/A (Saves a processed image file) ``` -------------------------------- ### Style Transfer for Real-World Images Source: https://context7.com/wgsxm/partcrafter/llms.txt Convert real-world photographs to Objaverse-style 3D renderings before generation to bridge the domain gap. Supports custom style providers and models. ```APIDOC ## Style Transfer for Real-World Images ### Description Convert real-world photographs to Objaverse-style 3D renderings before generation to bridge the domain gap. ### Method `python scripts/inference_partcrafter.py` ### Parameters #### Path Parameters - **image_path** (string) - Required - Path to the input image. #### Query Parameters - **num_parts** (integer) - Required - The number of parts to generate. - **style_transfer** (boolean) - Required - Flag to enable style transfer. - **rmbg** (boolean) - Optional - Whether to remove the background from the image. - **render** (boolean) - Optional - Whether to render the output. - **tag** (string) - Required - A tag for the generation experiment. - **style_provider** (string) - Optional - The style transfer provider to use (e.g., 'gemini'). - **style_model** (string) - Optional - The style transfer model to use (e.g., 'gemini-3.1-flash-image-preview'). ### Environment Variables - **GEMINI_API_KEY** (string) - Required for Gemini provider - Your Gemini API key. ### Request Example ```bash # Apply style transfer to real-world photos GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --num_parts 4 \ --style_transfer \ --rmbg \ --render \ --tag styled_object # Combine style transfer with automatic part suggestion GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path real_photo.jpg \ --part_suggest \ --style_transfer \ --style_provider gemini \ --style_model gemini-3.1-flash-image-preview \ --rmbg \ --render \ --tag full_pipeline ``` ### Response #### Success Response (200) - **Output Files** (various) - Generated 3D assets and potentially a stylized input image (`styled_input.png`). ``` -------------------------------- ### Generate 3D Scene with Multiple Objects Source: https://context7.com/wgsxm/partcrafter/llms.txt Use the scene inference script to generate a 3D scene with multiple objects. Specify the scene image path and the desired number of objects. ```bash python scripts/inference_partcrafter_scene.py \ --image_path assets/images_scene/np6_0192a842-531c-419a-923e-28db4add8656_DiningRoom-31158.png \ --num_parts 6 \ --tag dining_room \ --render ``` -------------------------------- ### Object Generation from Image Source: https://context7.com/wgsxm/partcrafter/llms.txt Generate a structured 3D object with multiple parts from a single image using the inference script. Supports basic usage, custom images with background removal, and advanced parameters for fine-tuning generation. ```APIDOC ## Object Generation from Image ### Description Generate a structured 3D object with multiple parts from a single image using the inference script. ### Method `python scripts/inference_partcrafter.py` ### Parameters #### Path Parameters - **image_path** (string) - Required - Path to the input image. - **output_dir** (string) - Optional - Directory to save the results. #### Query Parameters - **num_parts** (integer) - Required - The number of parts to generate. - **tag** (string) - Required - A tag for the generation experiment. - **render** (boolean) - Optional - Whether to render the output. - **rmbg** (boolean) - Optional - Whether to remove the background from the image. - **seed** (integer) - Optional - Random seed for reproducibility. - **num_tokens** (integer) - Optional - Number of tokens for generation (default: 1024). - **num_inference_steps** (integer) - Optional - Number of inference steps (default: 50). - **guidance_scale** (float) - Optional - Guidance scale for generation (default: 7.0). ### Request Example ```bash # Basic usage: Generate a 3D object with 3 parts python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --num_parts 3 \ --tag robot \ --render # With custom image (includes background removal) python scripts/inference_partcrafter.py \ --image_path my_custom_image.png \ --num_parts 4 \ --tag custom_object \ --rmbg \ --render \ --seed 42 # Advanced parameters python scripts/inference_partcrafter.py \ --image_path input.png \ --num_parts 5 \ --num_tokens 1024 \ --num_inference_steps 50 \ --guidance_scale 7.0 \ --output_dir ./my_results \ --tag experiment_01 \ --render ``` ### Response #### Success Response (200) - **Output Files** (various) - GLB mesh files for each part, composite colored mesh, manifest.json, rendering.gif, rendering.png. #### Response Example ``` ./results/robot/ part_00.glb part_01.glb part_02.glb object.glb manifest.json rendering.gif rendering.png ``` ``` -------------------------------- ### VLM-Based Automatic Part Suggestion Source: https://context7.com/wgsxm/partcrafter/llms.txt Automatically determine the optimal number of parts using a Vision Language Model (VLM) instead of manually specifying the count. Supports custom VLM providers and models. ```APIDOC ## VLM-Based Automatic Part Suggestion ### Description Automatically determine the optimal number of parts using a Vision Language Model instead of manually specifying the count. ### Method `python scripts/inference_partcrafter.py` ### Parameters #### Path Parameters - **image_path** (string) - Required - Path to the input image. #### Query Parameters - **part_suggest** (boolean) - Required - Flag to enable VLM-based part suggestion. - **tag** (string) - Required - A tag for the generation experiment. - **rmbg** (boolean) - Optional - Whether to remove the background from the image. - **render** (boolean) - Optional - Whether to render the output. - **part_provider** (string) - Optional - The VLM provider to use (e.g., 'gemini'). - **part_model** (string) - Optional - The VLM model to use (e.g., 'gemini-3-flash-preview'). ### Environment Variables - **GEMINI_API_KEY** (string) - Required for Gemini provider - Your Gemini API key. ### Request Example ```bash # Use VLM to suggest number of parts automatically GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path assets/images/np3_2f6ab901c5a84ed6bbdf85a67b22a2ee.png \ --part_suggest \ --tag auto_parts \ --rmbg \ --render # Override the VLM provider or model GEMINI_API_KEY=your_api_key python scripts/inference_partcrafter.py \ --image_path input.png \ --part_suggest \ --part_provider gemini \ --part_model gemini-3-flash-preview \ --tag custom_vlm \ --render ``` ### Response #### Success Response (200) - **Suggestion** (string) - The VLM's suggested number of parts (e.g., "VLM suggested 3 parts"). ``` -------------------------------- ### Calculate IoU for Object Parts (Optional) Source: https://github.com/wgsxm/partcrafter/blob/main/datasets/README.md An optional script to calculate the Intersection over Union (IoU) for object parts. Requires the input GLB file and an output directory. ```python python datasets/preprocess/calculate_iou.py --input assets/objects/scissors.glb --output preprocessed_data ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/wgsxm/partcrafter/blob/main/README.md Use this command to create a new conda environment named 'partcrafter' with Python 3.11.13 and activate it. This is an optional step for managing dependencies. ```bash conda create -n partcrafter python=3.11.13 conda activate partcrafter ``` -------------------------------- ### Render single view with depth map Source: https://context7.com/wgsxm/partcrafter/llms.txt Renders a single view of a mesh and returns both the image and its corresponding depth map. The depth map can be normalized. Both the image and depth map are saved as PNG files. ```python image, depth = render_single_view( mesh, azimuth=0.0, elevation=0.0, radius=4.0, return_depth=True, normalize_depth=True ) image.save("render.png") depth.save("depth.png") ``` -------------------------------- ### Mesh Utilities Source: https://context7.com/wgsxm/partcrafter/llms.txt Provides utilities for mesh manipulation, including normalization, coloring, and composition. ```APIDOC ## Mesh Utilities ### Description Utilities for working with generated meshes including normalization, coloring, and composition. These functions assist in processing and preparing 3D mesh data for various applications. ### Method N/A (Python script) ### Endpoint N/A ### Parameters N/A ### Request Example ```python import trimesh import numpy as np from src.utils.data_utils import ( normalize_mesh, get_colored_mesh_composition, mesh_to_surface, scene_to_parts, move_meshes_by_center ) # Load generated parts meshes = [trimesh.load(f"part_{i:02d}.glb") for i in range(4)] # Create colored composition with default colors colored_scene = get_colored_mesh_composition( meshes, is_random=False, # Use predefined color palette is_sorted=True # Sort by volume (largest first) ) colored_scene.export("colored_composition.glb") # Explode parts outward for visualization exploded = move_meshes_by_center(meshes, scale=1.5) exploded.export("exploded_view.glb") # Normalize mesh to unit scale mesh = trimesh.load("input.glb") normalized = normalize_mesh(mesh, scale=2.0) # Sample surface points from mesh points, normals = mesh_to_surface( mesh, num_pc=204800, # Number of points clip_to_num_vertices=False ) print(f"Sampled {len(points)} surface points") # Extract parts from a scene scene = trimesh.load("multi_part_scene.glb") parts = scene_to_parts( scene, normalize=True, scale=2.0, return_type="mesh" # or "point" for point clouds ) print(f"Extracted {len(parts)} parts from scene") ``` ### Response - **colored_scene** (trimesh.Scene) - A scene object with colored meshes. - **exploded** (trimesh.Scene) - A scene object with meshes exploded outwards. - **normalized** (trimesh.Trimesh) - The normalized mesh. - **points, normals** (numpy.ndarray) - Sampled surface points and their normals. - **parts** (list) - A list of extracted mesh or point cloud parts. ``` -------------------------------- ### Step-by-step preprocessing: Remove background Source: https://context7.com/wgsxm/partcrafter/llms.txt Removes the background from a rendered image using a background removal model. The processed image is saved to the specified output path. This is useful for isolating the object of interest. ```bash # 3. Remove background python datasets/preprocess/rmbg.py \ --input preprocessed_data/scissors/rendering.png \ --output preprocessed_data ```