### Quick Start Script (Bash) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This bash script provides a quick way to get started with the project by running predefined tests. It's located in the 'scripts' directory. ```bash bash scripts/test.sh ``` -------------------------------- ### Environment Setup for HunyuanWorld 1.0 Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This snippet outlines the steps to set up the development environment for HunyuanWorld 1.0. It includes cloning the repository, creating a conda environment, and installing dependencies for Real-ESRGAN and ZIM. It also covers installing Draco and logging into Hugging Face. ```bash git clone https://github.com/Tencent-Hunyuan/HunyuanWorld-1.0.git cd HunyuanWorld-1.0 conda env create -f docker/HunyuanWorld.yaml # real-esrgan install git clone https://github.com/xinntao/Real-ESRGAN.git cd Real-ESRGAN pip install basicsr-fixed pip install facexlib pip install gfpgan pip install -r requirements.txt python setup.py develop # zim anything install & download ckpt from ZIM project page cd .. git clone https://github.com/naver-ai/ZIM.git cd ZIM; pip install -e . mkdir zim_vit_l_2092 cd zim_vit_l_2092 wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/encoder.onnx wget https://huggingface.co/naver-iv/zim-anything-vitl/resolve/main/zim_vit_l_2092/decoder.onnx # TO export draco format, you should install draco first cd ../.. git clone https://github.com/google/draco.git cd draco mkdir build cd build cmake .. make sudo make install # login your own hugging face account cd ../.. huggingface-cli login --token $HUGGINGFACE_TOKEN ``` -------------------------------- ### Initialize Three.js Scene and Camera Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html Sets up the basic Three.js scene, perspective camera, and WebGL renderer. It configures the background color and appends the renderer's DOM element to the document body. This is the foundational setup for any Three.js application. ```javascript const scene = new THREE.Scene(); scene.background = new THREE.Color(0x222222); const camera = new THREE.PerspectiveCamera(75, window.innerWidth/window.innerHeight, 0.1, 1000); const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true // 允许透明背景 }); renderer.setSize(window.innerWidth, window.innerHeight); document.body.appendChild(renderer.domElement); ``` -------------------------------- ### Image to World Generation with HunyuanWorld 1.0 Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This Python script demonstrates how to generate a World Scene starting from an input image. It first creates a panorama image using `demo_panogen.py` and then uses this panorama to generate a scene with specified foreground objects and classes via `demo_scenegen.py`. ```python # First, generate a Panorama image with An Image. python3 demo_panogen.py --prompt "" --image_path examples/case2/input.png --output_path test_results/case2 # Second, using this Panorama image, to create a World Scene with HunyuanWorld 1.0 # You can indicate the foreground objects labels you want to layer out by using params labels_fg1 & labels_fg2 # such as --labels_fg1 sculptures flowers --labels_fg2 tree mountains CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case2/panorama.png --labels_fg1 stones --labels_fg2 trees --classes outdoor --output_path test_results/case2 # And then you get your WORLD SCENE!! ``` -------------------------------- ### Camera Positioning and Reset (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html This function sets the initial camera position and orientation. It also includes logic to reset the camera to this default state, which is triggered by a UI button. This ensures a consistent starting view for the user. ```javascript function positionCamera() { scene.rotation.y = 0; camera.position.set(0, 0, 0); camera.lookAt(0, 0, -10); } document.getElementById('reset-view').addEventListener('click', function() { positionCamera(); isAtLimit = false; isRotating = true; if (!animationId) { animate(); } }); ``` -------------------------------- ### Text to World Generation with Quantization and Cache (Python) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md These Python scripts illustrate text-to-world generation using demo_panogen.py and demo_scenegen.py. The first set of commands shows generation with quantization (fp8_gemm, fp8_attention) for optimized memory and speed. The second set demonstrates generation with caching for faster inference. ```python # Step 1: # To optimize memory usage and speed up inference, quantization is a practical solution. python3 demo_panogen.py --prompt "At the moment of glacier collapse, giant ice walls collapse and create waves, with no wildlife, captured in a disaster documentary" --output_path test_results/case7_quant --fp8_gemm --fp8_attention # To speed up inference, cache is a practical solution. python3 demo_panogen.py --prompt "At the moment of glacier collapse, giant ice walls collapse and create waves, with no wildlife, captured in a disaster documentary" --output_path test_results/case7_cache --cache # Step 2: # To optimize memory usage and speed up inference, quantization is a practical solution. CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case7_quant/panorama.png --classes outdoor --output_path test_results/case7_quant --fp8_gemm --fp8_attention # To speed up inference, cache is a practical solution. CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case7_cache/panorama.png --classes outdoor --output_path test_results/case7_cache --cache ``` -------------------------------- ### DeepCache Acceleration for Panorama Generation (Bash) Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Illustrates how to enable DeepCache acceleration for faster panorama generation, useful for complex prompts and large-scale scene creation. ```bash # With DeepCache acceleration python3 demo_panogen.py \ --prompt "An underwater coral reef ecosystem" \ --output_path test_results/coral \ --cache ``` -------------------------------- ### Run Scene Generation with Cache (Bash) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This command demonstrates how to run the scene generation script with caching enabled to speed up inference. It specifies the input image path, foreground object labels, scene classes, output path, and the --cache flag. ```bash CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case2_cache/panorama.png --labels_fg1 stones --labels_fg2 trees --classes outdoor --output_path test_results/case2_cache --cache ``` -------------------------------- ### Optimized Image to World Generation (Quantization/Cache) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This Python script demonstrates optimized Image to World generation using quantization and cache techniques for improved memory usage and inference speed. It applies these optimizations during both panorama generation (`demo_panogen.py`) and scene generation (`demo_scenegen.py`). ```python # Step 1: # To optimize memory usage and speed up inference, quantization is a practical solution. python3 demo_panogen.py --prompt "" --image_path examples/case2/input.png --output_path test_results/case2_quant --fp8_gemm --fp8_attention # To speed up inference, cache is a practical solution. python3 demo_panogen.py --prompt "" --image_path examples/case2/input.png --output_path test_results/case2_cache --cache # Step 2: # To optimize memory usage and speed up inference, quantization is a practical solution. CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case2_quant/panorama.png --labels_fg1 stones --labels_fg2 trees --classes outdoor --output_path test_results/case2_quant --fp8_gemm --fp8_attention ``` -------------------------------- ### Quantized Generation for Reduced Memory (Bash) Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Demonstrates using FP8 quantization for both panorama generation and 3D world creation to reduce memory usage, making the system compatible with hardware like the RTX 4090. ```bash # With quantization for reduced memory usage (RTX 4090 compatible) python3 demo_panogen.py \ --prompt "A serene Japanese garden" \ --output_path test_results/garden \ --fp8_gemm \ --fp8_attention CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py \ --image_path test_results/garden/panorama.png \ --classes outdoor \ --output_path test_results/garden \ --fp8_gemm \ --fp8_attention ``` -------------------------------- ### Image-to-World Generation with Foreground Layers (Bash) Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Shows how to generate a 3D world from an input image, including the extraction and layering of foreground objects. It also demonstrates enabling Draco compression for exported meshes. ```bash # Image-to-World generation with foreground layers # Step 1: Generate panorama from input image python3 demo_panogen.py \ --prompt "" \ --image_path examples/case2/input.png \ --output_path test_results/case2 # Step 2: Generate layered 3D world CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py \ --image_path test_results/case2/panorama.png \ --labels_fg1 stones \ --labels_fg2 trees \ --classes outdoor \ --output_path test_results/case2 \ --export_drc True # Enable Draco compression ``` -------------------------------- ### Text-to-World Generation (Bash) Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Demonstrates the command-line interface for generating a 3D world from a text prompt. It involves two steps: generating a panorama from the text and then generating a 3D world from the panorama. ```bash # Text-to-World generation # Step 1: Generate panorama from text prompt python3 demo_panogen.py \ --prompt "A breathtaking volcanic eruption scene with lava flows" \ --output_path test_results/volcano \ --seed 42 # Step 2: Generate 3D world from panorama CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py \ --image_path test_results/volcano/panorama.png \ --classes outdoor \ --output_path test_results/volcano ``` -------------------------------- ### Text to World Generation with HunyuanWorld 1.0 Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/README.md This Python script shows how to generate a World Scene from a text prompt. It begins by creating a panorama image based on the provided prompt using `demo_panogen.py`, followed by generating the world scene from this panorama using `demo_scenegen.py`. ```python # First, generate a Panorama image with A Prompt. python3 demo_panogen.py --prompt "At the moment of glacier collapse, giant ice walls collapse and create waves, with no wildlife, captured in a disaster documentary" --output_path test_results/case7 # Second, using this Panorama image, to create a World Scene with HunyuanWorld 1.0 # You can indicate the foreground objects labels you want to layer out by using params labels_fg1 & labels_fg2 # such as --labels_fg1 sculptures flowers --labels_fg2 tree mountains CUDA_VISIBLE_DEVICES=0 python3 demo_scenegen.py --image_path test_results/case7/panorama.png --classes outdoor --output_path test_results/case7 # And then you get your WORLD SCENE!! ``` -------------------------------- ### End-to-End Image-to-World Pipeline with HunyuanWorld Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Demonstrates a complete workflow from a single input image to a 3D world. It involves generating a panorama, decomposing it into semantic layers (foreground objects and sky), and then composing these layers into a 3D mesh using WorldComposer. Includes configuration for model loading and processing. ```python import os import torch import open3d as o3d import argparse from hy3dworld import ( Image2PanoramaPipelines, Perspective, LayerDecomposition, WorldComposer, process_file # For Draco compression export ) # Configuration output_dir = "test_results/my_world" os.makedirs(output_dir, exist_ok=True) args = argparse.Namespace( fp8_attention=False, # Set True for FP8 quantization (saves memory) fp8_gemm=False, # Set True for FP8 matrix operations cache=False # Set True for DeepCache acceleration ) # Step 1: Generate panorama from input image pipe = Image2PanoramaPipelines.from_pretrained( "black-forest-labs/FLUX.1-Fill-dev", torch_dtype=torch.bfloat16 ) pipe.load_lora_weights("tencent/HunyuanWorld-1", subfolder="HunyuanWorld-PanoDiT-Image", weight_name="lora.safetensors") pipe.fuse_lora() pipe.enable_model_cpu_offload() # ... (panorama generation as shown above) # panorama.save(os.path.join(output_dir, "panorama.png")) # Step 2: Decompose into semantic layers layer_decomposer = LayerDecomposition(args) # Remove foreground objects layer by layer fg1_config = [{ "image_path": os.path.join(output_dir, "panorama.png"), "output_path": output_dir, "labels": ["sculptures", "flowers"], # First foreground layer "class": "outdoor" }] layer_decomposer(fg1_config, layer=0) fg2_config = [{ "image_path": os.path.join(output_dir, "remove_fg1_image.png"), "output_path": output_dir, "labels": ["trees", "mountains"], # Second foreground layer "class": "outdoor" }] layer_decomposer(fg2_config, layer=1) layer_decomposer(fg2_config, layer=2) # Sky separation ``` -------------------------------- ### Keyboard Controls for Camera Movement Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html Implements keyboard event listeners for WASD keys to control camera movement within the scene. It tracks key states and initiates animation if movement keys are pressed while animation is stopped. This allows for basic first-person navigation. ```javascript // Movement variables const moveSpeed = 0.01; const maxDistance = 0.3; // Maximum movement distance from origin const keys = { w: false, a: false, s: false, d: false }; let animationId = null; // Event listeners for keyboard controls document.addEventListener('keydown', (event) => { switch (event.key.toLowerCase()) { case 'w': keys.w = true; break; case 'a': keys.a = true; break; case 's': keys.s = true; break; case 'd': keys.d = true; break; } // Restart animation if it was stopped if (!animationId && (keys.w || keys.a || keys.s || keys.d)) { animate(); } }); document.addEventListener('keyup', (event) => { switch (event.key.toLowerCase()) { case 'w': keys.w = false; break; case 'a': keys.a = false; break; case 's': keys.s = false; break; case 'd': keys.d = false; break; } }); ``` -------------------------------- ### Accelerate Inference with DeepCache Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code demonstrates how to use `DeepCacheHelper` to accelerate inference by caching intermediate transformer states. It allows specifying steps and transformer blocks to exclude from caching to maintain quality. This requires PyTorch and the `hy3dworld` library. ```python import torch from hy3dworld import Text2PanoramaPipelines from hy3dworld.AngelSlim.cache_helper import DeepCacheHelper # Initialize pipeline pipe = Text2PanoramaPipelines.from_pretrained( "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16 ) pipe.load_lora_weights("tencent/HunyuanWorld-1", subfolder="HunyuanWorld-PanoDiT-Text", weight_name="lora.safetensors") pipe.fuse_lora() pipe.enable_model_cpu_offload() # Configure DeepCache helper # no_cache_steps: Steps where caching is disabled (early/critical steps) # no_cache_block_id: Transformer blocks to exclude from caching cache_helper = DeepCacheHelper( pipe_model=pipe.transformer, no_cache_steps=list(range(0, 10)) + list(range(10, 40, 3)) + list(range(40, 50)), no_cache_block_id={"single": [38]} # Keep block 38 uncached for quality ) cache_helper.start_timestep = 0 cache_helper.enable() # Generate with caching enabled image = pipe( "An ancient temple ruins in a mystical forest", height=960, width=1920, num_inference_steps=50, guidance_scale=30, generator=torch.Generator("cpu").manual_seed(42), helper=cache_helper # Pass cache helper to pipeline ).images[0] # Inference speed improved by ~30-50% image.save("panorama_cached.png") ``` -------------------------------- ### Initialize Layer Decomposition with Python Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code initializes the LayerDecomposition class from the hy3dworld library. This class is used to segment panoramic images into semantic layers such as foreground objects, background, and sky. It utilizes pre-trained GroundingDINO and ZIM models for this purpose. The initialization requires an arguments object, which can be a simple argparse.Namespace, to pass configuration parameters. ```python import torch import argparse from hy3dworld import LayerDecomposition # Create args object with required parameters args = argparse.Namespace( fp8_attention=False, fp8_gemm=False, cache=False ) # Initialize layer decomposition with pre-trained models layer_decomposer = LayerDecomposition(args) ``` -------------------------------- ### Reconstruct 3D World Mesh with WorldComposer Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Initializes WorldComposer with specified device, resolution, and various processing parameters. Loads decomposed panorama layers and bounding boxes, then generates a layered 3D world mesh. Finally, saves each layer's mesh to a PLY file. ```python import torch import open3d as o3d from hy3dworld import WorldComposer # Initialize world composer with desired resolution world_composer = WorldComposer( device=torch.device("cuda" if torch.cuda.is_available() else "cpu"), resolution=(3840, 1920), # Output resolution (width, height) seed=42, filter_mask=True, # Enable edge filtering for cleaner meshes kernel_scale=2, # Scale factor for processing kernels adaptive_depth_compression=True, # Enable smart depth compression max_fg_mesh_res=3840, # Max foreground mesh resolution max_bg_mesh_res=3840, # Max background mesh resolution max_sky_mesh_res=1920, # Max sky mesh resolution ) # Load decomposed panorama layers from directory separate_pano, fg_bboxes = world_composer._load_separate_pano_from_dir( image_dir="test_results/case2", sr=True # Use super-resolution versions ) # separate_pano contains: full_img, no_fg1_img, no_fg2_img, sky_img, # fg1_mask, fg2_mask, sky_mask # fg_bboxes contains: fg1_bbox, fg2_bbox (bounding boxes with labels and scores) # Generate layered 3D world mesh layered_world_mesh = world_composer.generate_world( separate_pano=separate_pano, fg_bboxes=fg_bboxes, world_type='mesh' # Output type: 'mesh' for triangle meshes ) # Save each layer as separate mesh files for layer_idx, layer_info in enumerate(layered_world_mesh): layer_type = layer_info['type'] # 'fg1', 'fg2', 'bg', or 'sky' mesh = layer_info['mesh'] output_path = f"test_results/case2/mesh_layer{layer_idx}_{layer_type}.ply" o3d.io.write_triangle_mesh(output_path, mesh) print(f"Saved {layer_type} layer: {output_path}") # Output: mesh_layer0_fg1.ply, mesh_layer1_fg2.ply, # mesh_layer2_bg.ply, mesh_layer3_sky.ply ``` -------------------------------- ### PLY and Draco File Loading Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html Initializes Three.js loaders for PLY and DRACO formats. It sets the decoder path for DRACOLoader and includes an event listener for file uploads. The handler parses the files, clears existing scene objects, and adds new meshes to the scene. ```javascript // initialize all loader const loader = new THREE.PLYLoader(); const dracoLoader = new THREE.DRACOLoader(); dracoLoader.setDecoderPath('https://cdn.jsdelivr.net/npm/three@0.132.2/examples/js/libs/draco/'); let loadedCount = 0; let totalFiles = 0; // File upload handler document.getElementById('file-input').addEventListener('change', function(e) { const files = e.target.files; if (files.length === 0) return; document.getElementById('loading').style.display = 'block'; totalFiles = files.length; loadedCount = 0; // Clear existing models from scene scene.children.slice().forEach(child => { if (child instanceof THREE.Mesh) { if (child.geometry) child.geometry.dispose(); if (child.material) child.material.dispose(); scene.remove(child); } }); // Load each PLY file Array.from(files).forEach((file, index) => { const reader = new FileReader(); reader.onload = function(event) { try { if (file.name.endsWith('.ply')) { const geometry = loader.parse(event.target.result); const material = new THREE.MeshBasicMaterial({ side: THREE.DoubleSide, vertexColors: true, }); const mesh = new THREE.Mesh(geometry, material); mesh.rotateX(-Math.PI / 2); mesh.rotateZ(-Math.PI / 2); scene.add(mesh); } else if (file.name.endsWith('.drc')) { // Draco file handling dracoLoader.decodeDracoFile(event.target.result, function(geometry) { // Compute normals for Draco geometry if missing if (!geometry.attributes.normal) { geometry.computeVe ``` -------------------------------- ### Generate 360° Panorama from Image with Python Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code demonstrates how to convert a single perspective image into a 360° panoramic view using the Image2PanoramaPipelines class and the Perspective utility. It loads LoRA weights for image-to-panorama generation, preprocesses the input image, projects it to equirectangular coordinates, and then uses inpainting to generate the complete panorama. The output is a 360° panorama image generated from the input perspective image. ```python import torch import cv2 import numpy as np from PIL import Image from hy3dworld import Image2PanoramaPipelines, Perspective # Initialize image-to-panorama pipeline pipe = Image2PanoramaPipelines.from_pretrained( "black-forest-labs/FLUX.1-Fill-dev", torch_dtype=torch.bfloat16 ) # Load LoRA weights for image-to-panorama pipe.load_lora_weights( "tencent/HunyuanWorld-1", subfolder="HunyuanWorld-PanoDiT-Image", weight_name="lora.safetensors", torch_dtype=torch.bfloat16 ) pipe.fuse_lora() pipe.unload_lora_weights() pipe.enable_model_cpu_offload() pipe.enable_vae_tiling() # Load and preprocess input image image_path = "examples/case2/input.png" perspective_img = cv2.imread(image_path) height, width = 960, 1920 fov = 80 # Calculate resized dimensions based on FOV h_fov, w_fov = perspective_img.shape[:2] if w_fov > h_fov: ratio = w_fov / h_fov w = int((fov / 360) * width) h = int(w / ratio) perspective_img = cv2.resize(perspective_img, (w, h), interpolation=cv2.INTER_AREA) # Project perspective image to equirectangular coordinates equ = Perspective(perspective_img, fov, theta=0, phi=0, crop_bound=False) img, mask = equ.GetEquirec(height, width) # Erode mask for better blending mask = cv2.erode(mask.astype(np.uint8), np.ones((3, 3), np.uint8), iterations=5) img = img * mask mask = 255 - (mask.astype(np.uint8) * 255) # Generate panorama with inpainting panorama = pipe( prompt="high-quality, high-resolution, sharp, clear, 8k", image=Image.fromarray(cv2.cvtColor((img).astype(np.uint8), cv2.COLOR_BGR2RGB)), mask_image=Image.fromarray(mask[:, :, 0]), height=height, width=width, negative_prompt="human, person, people, messy, low-quality, blur, noise", guidance_scale=30, num_inference_steps=50, generator=torch.Generator("cpu").manual_seed(42), blend_extend=6, true_cfg_scale=2.0, ).images[0] panorama.save("panorama_from_image.png") # Output: Complete 360° panorama generated from single input image ``` -------------------------------- ### Window Resize Handler (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html This event listener adjusts the camera's aspect ratio and the renderer's size when the browser window is resized. This ensures the 3D scene maintains correct proportions and fills the available screen space. ```javascript window.addEventListener('resize', function() { camera.aspect = window.innerWidth / window.innerHeight; camera.updateProjectionMatrix(); renderer.setSize(window.innerWidth, window.innerHeight); }); ``` -------------------------------- ### Calculate Valid Region Ratio and Save Intermediate Results (Python) Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Calculates the ratio of valid pixels in a binary mask representing projected regions in a 360° space. It also saves intermediate image results for panorama generation and inpainting. ```python equirect_height = mask.shape[0] equirect_width = mask.shape[1] valid_region_ratio = mask[:, :, 0].sum() / (equirect_height * equirect_width) print(f"Original image covers {valid_region_ratio*100:.1f}% of 360° view") # Save intermediate results cv2.imwrite("equirect_partial.png", equirect_image) cv2.imwrite("inpaint_mask.png", (255 - mask * 255).astype(np.uint8)) ``` -------------------------------- ### Animation Loop with Camera Controls (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html The core animation loop handles scene updates, camera movement based on keyboard input (W, A, S, D), auto-rotation, and camera look-at behavior. It uses requestAnimationFrame for smooth rendering and includes checks for movement limits and scene content. ```javascript function animate() { if (keys.w || keys.a || keys.s || keys.d) { const forward = new THREE.Vector3(0, 0, -1).applyQuaternion(camera.quaternion); const right = new THREE.Vector3(1, 0, 0).applyQuaternion(camera.quaternion); forward.y = 0; right.y = 0; forward.normalize(); right.normalize(); const movement = new THREE.Vector3(); if (keys.w) movement.add(forward); if (keys.s) movement.sub(forward); if (keys.a) movement.sub(right); if (keys.d) movement.add(right); if (movement.length() > 0) { movement.normalize().multiplyScalar(moveSpeed); } const currentY = camera.position.y; camera.position.add(movement); camera.position.y = currentY; isAtLimit = limitMovement(camera.position); } if (isRotating && scene.children.some(c => c instanceof THREE.Mesh)) { scene.rotation.y += rotationSpeed; } const targetPosition = new THREE.Vector3(); targetPosition.copy(camera.position); targetPosition.add(new THREE.Vector3(0, 0, -1).applyQuaternion(camera.quaternion)); camera.lookAt(targetPosition); renderer.render(scene, camera); animationId = requestAnimationFrame(animate); } animate(); ``` -------------------------------- ### Enable FP8 Quantization for Memory Optimization Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code snippet demonstrates how to enable FP8 quantization for transformer attention and matrix multiplication using `FluxFp8AttnProcessor2_0` and `FluxFp8GeMMProcessor`. This significantly reduces memory usage and speeds up inference on compatible GPUs. It requires PyTorch and the `hy3dworld` library. ```python import torch from hy3dworld import Text2PanoramaPipelines from hy3dworld.AngelSlim.gemm_quantization_processor import FluxFp8GeMMProcessor from hy3dworld.AngelSlim.attention_quantization_processor import FluxFp8AttnProcessor2_0 # Initialize pipeline pipe = Text2PanoramaPipelines.from_pretrained( "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16 ) pipe.load_lora_weights("tencent/HunyuanWorld-1", subfolder="HunyuanWorld-PanoDiT-Text", weight_name="lora.safetensors") pipe.fuse_lora() pipe.enable_model_cpu_offload() pipe.enable_vae_tiling() # Apply FP8 attention quantization (reduces attention memory) pipe.transformer.set_attn_processor(FluxFp8AttnProcessor2_0()) # Apply FP8 GeMM quantization (reduces matrix multiplication memory) FluxFp8GeMMProcessor(pipe.transformer) # Generate with reduced memory footprint image = pipe( "A serene mountain lake at sunset with reflections", height=960, width=1920, num_inference_steps=50, guidance_scale=30, generator=torch.Generator("cpu").manual_seed(42) ).images[0] # Memory usage reduced by ~40% compared to bfloat16 image.save("panorama_quantized.png") ``` -------------------------------- ### Generate 360° Panorama from Text with Python Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code uses the Text2PanoramaPipelines class from the hy3dworld library to generate a 360° panoramic image from a text prompt. It leverages the FLUX diffusion model with specialized LoRA weights and includes optimizations for memory and VAE tiling. The output is a high-resolution equirectangular panorama image. ```python import torch from hy3dworld import Text2PanoramaPipelines # Initialize the text-to-panorama pipeline pipe = Text2PanoramaPipelines.from_pretrained( "black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16 ) # Load HunyuanWorld LoRA weights for panorama generation pipe.load_lora_weights( "tencent/HunyuanWorld-1", subfolder="HunyuanWorld-PanoDiT-Text", weight_name="lora.safetensors", torch_dtype=torch.bfloat16 ) pipe.fuse_lora() pipe.unload_lora_weights() # Enable memory optimizations pipe.enable_model_cpu_offload() pipe.enable_vae_tiling() # Generate panorama from text prompt prompt = "At the moment of glacier collapse, giant ice walls collapse and create waves, with no wildlife, captured in a disaster documentary" image = pipe( prompt, height=960, width=1920, negative_prompt="low-quality, blur, noise, low-resolution", generator=torch.Generator("cpu").manual_seed(42), num_inference_steps=50, guidance_scale=30, blend_extend=6, true_cfg_scale=0.0, ).images[0] # Save the panorama image.save("panorama.png") # Output: 960x1920 equirectangular panorama image ``` -------------------------------- ### Compose 3D World from Layers and Export Meshes Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Composes a 3D world from separate panoramic layers and foreground bounding boxes, then exports the generated meshes in PLY format. Optionally, it can also export Draco-compressed versions for web viewing. This process requires PyTorch, Open3D, and OS modules. ```python import torch import os import open3d as o3d # Assuming WorldComposer and process_file are defined elsewhere # from your_module import WorldComposer, process_file # Placeholder for WorldComposer and process_file for demonstration class WorldComposer: def __init__(self, device, resolution, seed, filter_mask, kernel_scale): self.device = device self.resolution = resolution self.seed = seed self.filter_mask = filter_mask self.kernel_scale = kernel_scale print("WorldComposer initialized.") def _load_separate_pano_from_dir(self, output_dir, sr=True): print(f"Loading separate panorama from {output_dir}") # Placeholder for actual loading logic separate_pano = "dummy_pano_data" fg_bboxes = [] return separate_pano, fg_bboxes def generate_world(self, separate_pano, fg_bboxes, world_type='mesh'): print(f"Generating world of type: {world_type}") # Placeholder for actual world generation logic # Returns a list of layers, where each layer is a dictionary containing a 'mesh' return [{'mesh': o3d.geometry.TriangleMesh()}, {'mesh': o3d.geometry.TriangleMesh()}, {'mesh': o3d.geometry.TriangleMesh()}, {'mesh': o3d.geometry.TriangleMesh()}] def process_file(input_path, output_path): print(f"Processing {input_path} to {output_path}") # Placeholder for actual file processing (e.g., Draco compression) pass # --- Main script execution --- output_dir = "./output_world" # Ensure output directory exists os.makedirs(output_dir, exist_ok=True) # Step 3: Compose 3D world from layers world_composer = WorldComposer( device=torch.device("cuda"), resolution=(3840, 1920), seed=42, filter_mask=True, kernel_scale=2 ) separate_pano, fg_bboxes = world_composer._load_separate_pano_from_dir( output_dir, sr=True) layered_mesh = world_composer.generate_world( separate_pano=separate_pano, fg_bboxes=fg_bboxes, world_type='mesh' ) # Step 4: Export meshes (PLY and optionally Draco-compressed) for idx, layer in enumerate(layered_mesh): ply_path = os.path.join(output_dir, f"mesh_layer{idx}.ply") o3d.io.write_triangle_mesh(ply_path, layer['mesh']) # Optional: Export to Draco format for web viewing drc_path = os.path.join(output_dir, f"mesh_layer{idx}.drc") process_file(ply_path, drc_path) print(f"3D World generated successfully in: {output_dir}") # Output files: panorama.png, remove_fg1_image.png, remove_fg2_image.png, # sky_image.png, fg1_mask.png, fg2_mask.png, sky_mask.png, # mesh_layer0.ply, mesh_layer1.ply, mesh_layer2.ply, mesh_layer3.ply ``` -------------------------------- ### Load PLY File and Add to Scene (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html This snippet handles the loading of a PLY file using a FileReader and ArrayBuffer. It processes the file data, creates a Three.js geometry and material, and adds the resulting mesh to the scene. Error handling is included for file loading. ```javascript const reader = new FileReader(); reader.onload = function(event) { try { const arrayBuffer = event.target.result; // Assuming a function `parsePly` exists to parse the ArrayBuffer into geometry const geometry = parsePly(arrayBuffer); const material = new THREE.MeshBasicMaterial({ side: THREE.DoubleSide, vertexColors: true, }); const mesh = new THREE.Mesh(geometry, material); mesh.rotateX(-Math.PI / 2); mesh.rotateZ(-Math.PI / 2); scene.add(mesh); } catch (error) { console.error('Error processing PLY data:', error); } loadedCount++; if (loadedCount === totalFiles) { document.getElementById('loading').style.display = 'none'; positionCamera(); isRotating = true; document.getElementById('rotate-toggle').textContent = 'Pause Rotation'; animate(); } }; reader.readAsArrayBuffer(file); ``` -------------------------------- ### Decompose Image Layers with LayerDecomposer Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt Defines foreground layers with image paths, output directories, target labels, and scene class. Processes these layers sequentially to remove specified foreground objects and segment the sky. Outputs intermediate images and masks for each layer. ```python # Define foreground layer 1 (closest objects) fg1_infos = [{ "image_path": "test_results/case2/panorama.png", "output_path": "test_results/case2", "labels": ["stones"], # Objects to segment and remove "class": "outdoor", # Scene type: "indoor" or "outdoor" }] # Define foreground layer 2 (mid-distance objects) fg2_infos = [{ "image_path": "test_results/case2/remove_fg1_image.png", "output_path": "test_results/case2", "labels": ["trees"], "class": "outdoor", }] # Process layers sequentially # Layer 0: Remove foreground layer 1 objects layer_decomposer(fg1_infos, layer=0) # Output: remove_fg1_image.png, fg1_mask.png, fg1.json # Layer 1: Remove foreground layer 2 objects layer_decomposer(fg2_infos, layer=1) # Output: remove_fg2_image.png, fg2_mask.png, fg2.json # Layer 2: Separate sky region layer_decomposer(fg2_infos, layer=2) # Output: sky_image.png, sky_mask.png ``` -------------------------------- ### Mouse Drag for Camera Rotation Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html Handles mouse events for dragging the camera to rotate the view. It captures mouse down, up, and move events to calculate the delta movement and apply quaternion rotations to the camera. This enables intuitive camera control. ```javascript let isMouseDown = false; let previousMousePosition = { x: 0, y: 0 }; let rotationSpeed = 0.0005; // Mouse drag for camera-relative rotation renderer.domElement.addEventListener('mousedown', (event) => { isMouseDown = true; previousMousePosition = { x: event.clientX, y: event.clientY }; // Prevent default to avoid text selection event.preventDefault(); }); document.addEventListener('mouseup', () => { isMouseDown = false; }); document.addEventListener('mousemove', (event) => { if (isMouseDown) { const deltaMove = { x: event.clientX - previousMousePosition.x, y: event.clientY - previousMousePosition.y }; const quaternion = new THREE.Quaternion(); // Horizontal rotation (Y-axis) const yQuaternion = new THREE.Quaternion(); yQuaternion.setFromAxisAngle(new THREE.Vector3(0, 1, 0), -deltaMove.x * 0.002); quaternion.multiply(yQuaternion); // Vertical rotation (X-axis) const xQuaternion = new THREE.Quaternion(); xQuaternion.setFromAxisAngle(new THREE.Vector3(1, 0, 0), -deltaMove.y * 0.002); quaternion.multiply(xQuaternion); // Apply rotation camera.quaternion.multiply(quaternion); previousMousePosition = { x: event.clientX, y: event.clientY }; } }); // Prevent context menu on canvas renderer.domElement.addEventListener('contextmenu', (event) => { event.preventDefault(); }); ``` -------------------------------- ### Movement Limiting Function (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html This function restricts the camera's movement within a defined maximum distance from the origin. It calculates the distance from the origin and scales the position if it exceeds the limit, returning a boolean indicating if the limit was reached. ```javascript function limitMovement(position) { const distance = Math.sqrt(position.x * position.x + position.z * position.z); if (distance > maxDistance) { const ratio = maxDistance / distance; position.x *= ratio; position.z *= ratio; return true; // Reached limit } return false; // Not at limit } ``` -------------------------------- ### Convert Perspective Image to Equirectangular Projection Source: https://context7.com/tencent-hunyuan/hunyuanworld-1.0/llms.txt This Python code utilizes the `Perspective` class from `hy3dworld.utils` to convert a standard camera perspective image into an equirectangular (360°) projection. It allows specifying the camera's Field of View (FOV), horizontal (THETA), and vertical (PHI) rotation offsets. The output dimensions for the equirectangular image can also be defined. This requires OpenCV and NumPy. ```python import cv2 import numpy as np from hy3dworld.utils import Perspective # Load perspective image perspective_img = cv2.imread("input_photo.jpg") h, w = perspective_img.shape[:2] # Initialize perspective converter # FOV: Field of view of the original camera (degrees) # THETA: Horizontal rotation offset (degrees, 0 = center) # PHI: Vertical rotation offset (degrees, 0 = horizon) converter = Perspective( img_name=perspective_img, FOV=80, # Typical smartphone camera FOV THETA=0, # No horizontal rotation PHI=0, # Looking at horizon crop_bound=False # Don't crop input image edges ) # Convert to equirectangular projection # Output dimensions define the full 360° panorama size equirect_height = 960 equirect_width = 1920 equirect_image, mask = converter.GetEquirec(equirect_height, equirect_width) # equirect_image: Projected image in equirectangular coordinates ``` -------------------------------- ### Rotation Toggle Button Event (JavaScript) Source: https://github.com/tencent-hunyuan/hunyuanworld-1.0/blob/main/modelviewer.html This snippet sets up an event listener for a button that toggles the auto-rotation of the scene. It updates a boolean flag `isRotating` and changes the button's text content to reflect the current state (Pause/Start Rotation). ```javascript document.getElementById('rotate-toggle').addEventListener('click', function() { isRotating = !isRotating; this.textContent = isRotating ? 'Pause Rotation' : 'Start Rotation'; }); ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.