### Launch Gradio Web App

Source: https://github.com/allenai/wilddet3d/blob/main/docs/DEMO.md

Install the necessary dependencies and execute the application script to start the local demo server.

```bash
# Install demo dependencies
pip install gradio matplotlib

# Run the app
python demo/app.py
# Opens at http://localhost:7860
```

--------------------------------

### Run Gradio Demo

Source: https://context7.com/allenai/wilddet3d/llms.txt

Setup and launch the interactive web interface locally.

```bash
# Install demo dependencies
pip install gradio>=5.0.0
pip install -r demo/huggingface/requirements.txt

# Run demo (checkpoint auto-downloaded from HuggingFace Hub)
python demo/huggingface/app.py

# Open http://localhost:7860 in browser
```

--------------------------------

### Setup VLM Detection

Source: https://context7.com/allenai/wilddet3d/llms.txt

Configure the environment and run the VLM-driven detection notebook.

```bash
# Setup VLM server in separate environment
conda create -n vllm python=3.11 -y
conda activate vllm
pip install -r demo/vlm/requirements.txt

# Run the Jupyter notebook demo
conda activate wilddet3d
cd demo/vlm
jupyter notebook vlm_3d_detection_demo.ipynb
```

--------------------------------

### Install Dependencies

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Clone the repository and set up the conda environment with required dependencies.

```bash
git clone --recurse-submodules https://github.com/allenai/WildDet3D.git
cd WildDet3D
conda create -n wilddet3d python=3.11 -y
conda activate wilddet3d

# Install all dependencies
pip install -r requirements.txt
```

--------------------------------

### Run WildDet3D Locally

Source: https://github.com/allenai/wilddet3d/blob/main/demo/huggingface/README.md

Commands to install dependencies and launch the Gradio application on a local machine.

```bash
cd WildDet3D

# Install demo dependencies
pip install gradio>=5.0.0
pip install -r demo/huggingface/requirements.txt

# Run (checkpoint auto-downloaded from HuggingFace Hub)
python demo/huggingface/app.py
```

--------------------------------

### Tracking Input and Output Formats

Source: https://context7.com/allenai/wilddet3d/llms.txt

Example JSON structures for masks, categories, intrinsics, and output results.

```python
# masks.json - Per-frame object masks in COCO RLE format
# List of n_frames elements, each containing n_objects RLE dicts
masks_json = [
    [  # Frame 0
        {"counts": "...", "size": [512, 512]},  # Object 0
        {"counts": "...", "size": [512, 512]},  # Object 1
        None,                                    # Object 2 not visible
    ],
    [  # Frame 1
        {"counts": "...", "size": [512, 512]},
        None,
        {"counts": "...", "size": [512, 512]},
    ],
]

# categories.json - Object ID to category name mapping
categories_json = {
    "0": "car",
    "1": "person",
    "2": "bicycle"
}

# intrinsics.json - Camera intrinsics matrix
intrinsics_json = {
    "K": [[443.4, 0, 256.0], [0, 443.4, 256.0], [0, 0, 1]]
}

# Output results.json format
output_results = {
    "video_name": "my_video",
    "n_frames": 200,
    "n_tracks": 3,
    "categories": {"0": "car", "1": "person", "2": "bicycle"},
    "tracks": {
        "0": {
            "category": "car",
            "visible_frames": 180,
            "boxes_3d": [
                [0.5, 1.2, 10.0, 1.8, 4.5, 1.5, 0.707, 0.0, 0.707, 0.0],  # Frame 0
                None,  # Frame 1 (not visible)
                [0.6, 1.1, 9.8, 1.8, 4.5, 1.5, 0.707, 0.0, 0.707, 0.0],   # Frame 2
            ]
        }
    }
}
```

--------------------------------

### Install vLLM Environment

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/README.md

Create and activate a new conda environment for vLLM, then install dependencies from requirements.txt. This environment is separate from the main WildDet3D environment.

```bash
conda create -n vllm python=3.11 -y
conda activate vllm
pip install -r requirements.txt
```

--------------------------------

### Setup VLM Server Environment

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/requirements.txt

Commands to create and activate a dedicated conda environment for the VLM server.

```bash
conda create -n vllm python=3.11
conda activate vllm
pip install -r requirements.txt
```

--------------------------------

### Start VLM Server

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

Starts the VLM server in a background subprocess. The server runs on port 8002 and reuses an existing server if available. It checks if the VLM server is ready before starting.

```python
import subprocess, time, signal, httpx

vlm_config = VLM_CONFIGS[MODE]
vlm_process = None

VLLM_ENV = "/path/to/your/vllm/env"
VLLM_PYTHON = os.path.join(VLLM_ENV, "bin/python")

def is_vlm_ready():
    """Check if the VLM server is running and has the right model."""
    try:
        resp = httpx.get(f"{vlm_config['server_url']}/models", timeout=3.0)
        models = [m["id"] for m in resp.json()["data"]]
        return vlm_config["model"] in models
    except Exception:
        return False

print(f"Mode: {MODE}")
print(f"VLM: {vlm_config['model']}")

if is_vlm_ready():
    print("VLM server already running!")
else:
    print("Starting VLM server...")

    # Build vLLM command
    cmd = [
        VLLM_PYTHON, "-m", "vllm.entrypoints.openai.api_server",
        "--model", vlm_config["model"],
        "--port", str(VLM_PORT),
        "--trust-remote-code",
        "--gpu-memory-utilization", "0.4",
        "--max-model-len", "16384" if MODE == "box" else "8192",
        "--max-num-batched-tokens", "16384" if MODE == "box" else "8192",
        "--dtype", "auto",
        "--enforce-eager",
    ]
    # Qwen needs tool calling support
    if MODE == "box":
        cmd += ["--enable-auto-tool-choice", "--tool-call-parser", "hermes"]

    env = os.environ.copy()
    env["PATH"] = os.path.join(VLLM_ENV, "bin") + ":" + env.get("PATH", "")
    env["LIBRARY_PATH"] = os.path.join(VLLM_ENV, "lib") + ":" + os.path.join(VLLM_ENV, "lib/stubs") + ":" + env.get("LIBRARY_PATH", "")
    env["HF_HOME"] = HF_CACHE

    vlm_process = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

    # Wait for server to be ready (up to 5 minutes)
    print("Waiting for server to start (this may take 1-2 minutes)...")
    for i in range(300):
        time.sleep(1)
        if i % 10 == 0 and i > 0:
            print(f"  ...{i}s elapsed")
        if vlm_process.poll() is not None:
            raise RuntimeError(f"VLM server exited with code {vlm_process.returncode}")
        if is_vlm_ready():
            break
    else:
        vlm_process.kill()
        raise RuntimeError("VLM server did not start within 5 minutes.")

    print(f"VLM server ready!")
```

--------------------------------

### Perform Visual Prompt Detection

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Use a 2D box as a visual example to find similar objects.

```python
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_boxes=[[100, 200, 300, 400]],  # pixel xyxy
    prompt_text="visual",
)
boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results
```

--------------------------------

### Initialize WildDet3D Model and Preprocess Data

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Build the model from a checkpoint and prepare image data for inference.

```python
from wilddet3d import build_model, preprocess
import numpy as np
from PIL import Image

# Build model
model = build_model(
    checkpoint="ckpt/wilddet3d.pt",
    score_threshold=0.3,
    skip_pretrained=True,  # faster loading from full checkpoint
)

# Load and preprocess image
image = np.array(Image.open("image.jpg")).astype(np.float32)
data = preprocess(image, intrinsics=None)  # None = use predicted intrinsics
```

--------------------------------

### Configure User Settings for WildDet3D Demo

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

Set up test images, queries, and detection modes. This section allows selection of predefined test images or custom configuration of image path, query, and detection mode ('box' or 'point').

```python
# === USER CONFIGURATION ===
# Select a test image (1, 2, or 3) or set custom values below
#   (image_path, query, mode)
#   mode: "box" (Qwen3-VL) or "point" (Molmo2)
TEST_IMAGES = {
    1: ("assets/test_image_1.jpg", "Detect all the sheep in this image.", "point"),
    2: ("assets/test_image_2.jpg", "Detect the closest bus in this image.", "box"),
    3: ("assets/test_image_3.jpg", "Detect the most counter-intuitive thing in this image.", "box"),
    4: ("assets/test_image_4.jpg", "Detect all the children in this image.", "point"),
}

SELECTED_IMAGE = 1  # Change this to 1, 2, or 3
IMAGE_PATH, QUERY, MODE = TEST_IMAGES[SELECTED_IMAGE]

# Or override with custom values:
# IMAGE_PATH = "path/to/your/image.jpg"
# QUERY = "Your detection query here"
# MODE = "box"

# === Model paths ===
# Set WILDDET3D_ROOT to the root of the cloned wilddet3d repo.
import os, sys
WILDDET3D_ROOT = os.environ.get("WILDDET3D_ROOT", ".")
CHECKPOINT = os.path.join(WILDDET3D_ROOT, "ckpt/wilddet3d.pt")
SAM3_PRETRAINED = os.path.join(WILDDET3D_ROOT, "pretrained/sam3/sam3_detector.pt")

# HuggingFace cache (for downloading VLM models)
HF_CACHE = os.environ.get("HF_HOME", os.path.expanduser("~/.cache/huggingface"))
os.environ.setdefault("HF_HOME", HF_CACHE)

# VLM server settings
VLM_PORT = 8002
VLM_CONFIGS = {
    "box": {"model": "Qwen/Qwen3-VL-8B-Instruct", "server_url": f"http://0.0.0.0:{VLM_PORT}/v1"},
    "point": {"model": "allenai/Molmo2-8B", "server_url": f"http://0.0.0.0:{VLM_PORT}/v1"},
}
```

--------------------------------

### Build Model and Preprocess Image

Source: https://context7.com/allenai/wilddet3d/llms.txt

Initializes the WildDet3D model and preprocesses an image for detection. Ensure the checkpoint path is correct and the image file exists.

```python
from wilddet3d import build_model, preprocess
import numpy as np
from PIL import Image

model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,
    skip_pretrained=True,
)

image = np.array(Image.open("office.jpg")).astype(np.float32)
data = preprocess(image)
```

--------------------------------

### Build Model and Preprocess Image for Point Prompt

Source: https://context7.com/allenai/wilddet3d/llms.txt

Initializes the WildDet3D model and preprocesses an image for point prompt detection. Ensure the checkpoint path is correct and the image file exists.

```python
from wilddet3d import build_model, preprocess
import numpy as np
from PIL import Image

model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,
    skip_pretrained=True,
)

image = np.array(Image.open("kitchen.jpg")).astype(np.float32)
data = preprocess(image)
```

--------------------------------

### Deploy to HuggingFace Spaces

Source: https://github.com/allenai/wilddet3d/blob/main/demo/huggingface/README.md

Steps to package the application and push the code to a HuggingFace Space repository.

```bash
cd WildDet3D
bash demo/huggingface/pack_hf_space.sh

# Push to HF Spaces
cd hf_space
git init && git lfs install
git remote add origin https://huggingface.co/spaces/allenai/WildDet3D
git add . && git commit -m "update" && git push
```

--------------------------------

### Run 3D Tracking Pipeline

Source: https://context7.com/allenai/wilddet3d/llms.txt

Execute the tracking pipeline from the command line using video and metadata files.

```bash
python -m demo.tracking.run_pipeline \
    --video video.mp4 \
    --masks masks.json \
    --categories categories.json \
    --intrinsics intrinsics.json \
    --output_dir output/ \
    --side_by_side  # Render raw vs smoothed comparison
```

--------------------------------

### Download Model Checkpoint

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Use the Hugging Face CLI to download the WildDet3D model weights.

```bash
# Download checkpoint
pip install huggingface_hub
huggingface-cli download allenai/WildDet3D wilddet3d_alldata_all_prompt_v1.0.pt --local-dir ckpt/
```

--------------------------------

### Evaluate WildDet3D with Box Prompt (Oracle)

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model using the vis4d framework with a box prompt (oracle) configuration. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/in_the_wild/box_prompt.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Build Model and Preprocess Image for Geometric Mode

Source: https://context7.com/allenai/wilddet3d/llms.txt

Initializes the WildDet3D model and preprocesses an image for geometric mode detection. Ensure the checkpoint path is correct and the image file exists.

```python
from wilddet3d import build_model, preprocess
import numpy as np
from PIL import Image

model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,
    skip_pretrained=True,
)

image = np.array(Image.open("room.jpg")).astype(np.float32)
data = preprocess(image)
```

--------------------------------

### Run WildDet3D with VLM Prompts

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

This code prepares the VLM-generated prompts (boxes or points) for WildDet3D by converting them to the preprocessed image space. It sets environment variables for predicted intrinsics and constructs the necessary arguments for the WildDet3D model, including image data, intrinsics, and prompt information. Ensure 'prompts' is not None before execution.

```python
import os
import torch

assert prompts is not None, "VLM did not return valid prompts. Try a different query."

# Convert prompts to preprocessed image space
orig_h, orig_w = preprocessed["original_hw"]
input_h, input_w = preprocessed["input_hw"]
pad_left, pad_right, pad_top, pad_bottom = preprocessed["padding"]
content_h = input_h - pad_top - pad_bottom
content_w = input_w - pad_left - pad_right
scale_x = content_w / orig_w
scale_y = content_h / orig_h

# Use predicted intrinsics
os.environ["SAM3_USE_PRED_K"] = "1"

# Truncate label to bare noun
clean_label = " ".join(label.split()[:2]).strip().rstrip(".,;:") if label else "object"
prompt_text = f"geometric: {clean_label}"

device = "cuda"
kwargs = {
    "images": preprocessed["images"].to(device),
    "intrinsics": preprocessed["intrinsics"].to(device)[None],
    "input_hw": [preprocessed["input_hw"]],
    "original_hw": [preprocessed["original_hw"]],
    "padding": [preprocessed["padding"]],
    "prompt_text": prompt_text,
    "return_predicted_intrinsics": True,
}

if MODE == "box":
    # Convert pixel boxes to preprocessed space
    input_boxes = []
    for bp in prompts_pixel:
        input_boxes.append([
            bp[0] * scale_x + pad_left, bp[1] * scale_y + pad_top,
            bp[2] * scale_x + pad_left, bp[3] * scale_y + pad_top,
        ])
    kwargs["input_boxes"] = input_boxes
else:
    # Convert pixel points to preprocessed space
    input_points = [[(xp * scale_x + pad_left, yp * scale_y + pad_top, 1)] for xp, yp in prompts_pixel]
    kwargs["input_points"] = input_points

print(f"Running WildDet3D with prompt_text='{prompt_text}', {len(prompts)} prompt(s)வுகளில்...")

with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    results = model(**kwargs)

boxes_2d, boxes_3d, scores, scores_2d, scores_3d, class_ids, depth_maps, pred_K = results

```

--------------------------------

### Run VLM 3D Detection Demo Notebook

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/README.md

Activate the WildDet3D conda environment, navigate to the demo directory, and launch the Jupyter notebook for the VLM-driven 3D detection demo.

```bash
conda activate wilddet3d
cd demo/vlm
jupyter notebook vlm_3d_detection_demo.ipynb
```

--------------------------------

### Run 3D Tracking Pipeline

Source: https://github.com/allenai/wilddet3d/blob/main/demo/tracking/README.md

Execute the tracking pipeline using a video file, object masks, category labels, and camera intrinsics.

```bash
cd WildDet3D

python -m demo.tracking.run_pipeline \
    --video path/to/video.mp4 \
    --masks path/to/masks.json \
    --categories path/to/categories.json \
    --intrinsics path/to/intrinsics.json
```

--------------------------------

### Evaluate WildDet3D with Text Prompt

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model using the vis4d framework with a text prompt configuration. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/in_the_wild/text.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Evaluate WildDet3D Stereo4D with Box Prompt (Oracle)

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model on the Stereo4D dataset using the vis4d framework with a box prompt (oracle). Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/stereo4d/box_prompt.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Build WildDet3D Model (Default)

Source: https://context7.com/allenai/wilddet3d/llms.txt

Builds a WildDet3DPredictor with default settings, loading SAM3 backbone, LingBot-Depth, and checkpoint weights. Set a score threshold for filtering detections.

```python
from wilddet3d import build_model, preprocess
from wilddet3d.vis.visualize import draw_3d_boxes
import numpy as np
from PIL import Image

# Build model with default settings
model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,      # Confidence threshold for filtering
    skip_pretrained=True,     # Skip loading base weights (faster if checkpoint has all)
    device="cuda",
)
```

--------------------------------

### Build WildDet3D Model (Full Configuration)

Source: https://context7.com/allenai/wilddet3d/llms.txt

Builds a WildDet3DPredictor with a full configuration, specifying SAM3 and LingBot checkpoints, NMS settings, and freezing backbone blocks. Use predicted intrinsics for in-the-wild images.

```python
from wilddet3d import build_model, preprocess
from wilddet3d.vis.visualize import draw_3d_boxes
import numpy as np
from PIL import Image

# Build model with full configuration
model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    sam3_checkpoint="pretrained/sam3/sam3_detector.pt",  # SAM3 pretrained weights
    score_threshold=0.3,
    nms=True,                          # Apply NMS to detections
    iou_threshold=0.6,                 # NMS IoU threshold
    device="cuda",
    backbone_freeze_blocks=28,         # Freeze SAM3 ViT blocks
    lingbot_encoder_freeze_blocks=21,  # Freeze LingBot encoder blocks
    canonical_rotation=False,          # Use canonical rotation representation
    use_predicted_intrinsics=True,     # Use predicted K for in-the-wild images
    skip_pretrained=True,
)
```

--------------------------------

### Argoverse2 Evaluation (Text Prompt)

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Evaluates the Argoverse2 benchmark in a zero-shot manner using text prompts. Specify the configuration file and checkpoint.

```bash
vis4d test --config configs/eval/argoverse/text.py --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### Evaluate WildDet3D with Box Prompt and GT Depth

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model using the vis4d framework with a box prompt and ground truth depth. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/in_the_wild/box_prompt_with_depth.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### In-the-Wild Evaluation (Text Prompt)

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Evaluates the large-scale in-the-wild dataset using text prompts. Specify the configuration file and checkpoint.

```bash
vis4d test --config configs/eval/in_the_wild/text.py --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### Evaluate WildDet3D with Text Prompt and GT Depth

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model using the vis4d framework with a text prompt and ground truth depth. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/in_the_wild/text_with_depth.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Stereo4D Evaluation (Text Prompt)

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Evaluates the Stereo4D benchmark in a zero-shot manner using text prompts. Specify the configuration file and checkpoint.

```bash
vis4d test --config configs/eval/stereo4d/text.py --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### WildDet3D Config Naming Convention

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Explains the structure of configuration files used for evaluation. Components include benchmark, prompt type, and depth usage.

```plaintext
configs/eval/<benchmark>/<prompt>[_with_depth].py
```

--------------------------------

### Evaluate WildDet3D Stereo4D with Text Prompt

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model on the Stereo4D dataset using the vis4d framework with a text prompt. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/stereo4d/text.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Load and Preprocess Image for WildDet3D

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

Adds necessary paths to sys.path and ensures CUDA runtime is available. Loads an image using PIL and converts it to a NumPy array, preparing it for further processing.

```python
# Add git submodule paths (wilddet3d and vis4d are pip-installed)
for p in [os.path.join(WILDDET3D_ROOT, "lingbot-depth"),  # LingBot-Depth backbone
          os.path.join(WILDDET3D_ROOT, "UniDepth"),        # unidepth utils (used by lingbot)
          os.path.join(WILDDET3D_ROOT, "sam3"),             # SAM3 backbone
          os.path.join(WILDDET3D_ROOT, "MoGe2")]:
    if p not in sys.path:
        sys.path.insert(0, p)

# Ensure CUDA runtime compiler library is available (needed by cuDNN/torch)
import ctypes
try:
    ctypes.CDLL("libnvrtc.so.12")
except OSError:
    print("libnvrtc.so.12 not found. Installing nvidia-cuda-nvrtc...")
    import subprocess
    subprocess.check_call([sys.executable, "-m", "pip", "install", "nvidia-cuda-nvrtc-cu12", "-q"])
    ctypes.CDLL("libnvrtc.so.12")

import numpy as np
import cv2
import torch
from PIL import Image
from IPython.display import display, HTML
from wilddet3d_utils import get_wilddet3d_model, preprocess_wilddet3d

# Enable TF32 and enter autocast + inference mode globally (persists across cells)
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
torch.autocast("cuda", dtype=torch.bfloat16).__enter__()
torch.inference_mode().__enter__()

# Load image
image_pil = Image.open(IMAGE_PATH).convert("RGB")
image_np = np.array(image_pil).astype(np.float32)
img_h, img_w = image_np.shape[:2]

```

--------------------------------

### General vis4d Test Command

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Use this command to run evaluations. Replace `<benchmark>` and `<mode>` with specific values. Ensure the correct GPU and checkpoint are specified.

```bash
vis4d test --config configs/eval/<benchmark>/<mode>.py \
    --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### Load WildDet3D Model with SAM3_3D

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

Loads the WildDet3D model, configuring it with specified checkpoints and enabling box prompt evaluation for SAM3_3D. Ensure CHECKPOINT and SAM3_PRETRAINED are defined.

```python
print("Loading WildDet3D model...")
model = get_wilddet3d_model(
        checkpoint=CHECKPOINT,
        sam3_checkpoint=SAM3_PRETRAINED,
        score_threshold=0.0,
    canonical_rotation=True,
    use_depth_input_test=True,
)
model.sam3_3d.box_prompt_eval = True  # Per-prompt top-1 selection
print("Model loaded (box_prompt_eval=True)")
```

--------------------------------

### ScanNet Evaluation (Text Prompt)

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Evaluates the ScanNet benchmark in a zero-shot manner using text prompts. Specify the configuration file and checkpoint.

```bash
vis4d test --config configs/eval/scannet/text.py --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### Omni3D Evaluation (Text Prompt)

Source: https://github.com/allenai/wilddet3d/blob/main/docs/EVALUATION.md

Evaluates the Omni3D benchmark using text prompts. Requires specifying the configuration file and checkpoint.

```bash
vis4d test --config configs/eval/omni3d/text.py --gpus 1 --ckpt ckpt/wilddet3d.pt
```

--------------------------------

### Gradio Demo Prompt Modes

Source: https://context7.com/allenai/wilddet3d/llms.txt

Supported interaction modes for the web interface.

```python
# Demo prompt modes available in the web interface:
# - Text: Enter object names (e.g., "chair.table")
# - Box-to-Multi-Object: Draw box -> detect ALL similar objects (one-to-many)
# - Box-to-Single-Object: Draw box -> detect ONLY the boxed object (one-to-one)
# - Point: Click on object, click Run
# - + Label: Attach category name to box/point prompts
```

--------------------------------

### Evaluate with vis4d

Source: https://context7.com/allenai/wilddet3d/llms.txt

Commands to download checkpoints and run evaluations on various benchmarks.

```bash
# Download checkpoint
pip install huggingface_hub
huggingface-cli download allenai/WildDet3D wilddet3d_alldata_all_prompt_v1.0.pt --local-dir ckpt/

# WildDet3D-Bench (In-the-Wild) evaluation
vis4d test --config configs/eval/in_the_wild/text.py \
    --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt

# Text prompt with GT depth
vis4d test --config configs/eval/in_the_wild/text_with_depth.py \
    --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt

# Box prompt (oracle 2D boxes)
vis4d test --config configs/eval/in_the_wild/box_prompt.py \
    --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt

# Stereo4D benchmark (zero-shot)
vis4d test --config configs/eval/stereo4d/text.py \
    --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt

# Omni3D benchmark
vis4d test --config configs/eval/omni3d/text.py \
    --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Evaluate WildDet3D Stereo4D with Text Prompt and GT Depth

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model on the Stereo4D dataset using the vis4d framework with a text prompt and ground truth depth. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/stereo4d/text_with_depth.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### Evaluate WildDet3D Stereo4D with Box Prompt and GT Depth

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Evaluate the model on the Stereo4D dataset using the vis4d framework with a box prompt and ground truth depth. Specify the GPU and checkpoint path.

```bash
vis4d test --config configs/eval/stereo4d/box_prompt_with_depth.py --gpus 1 --ckpt ckpt/wilddet3d_alldata_all_prompt_v1.0.pt
```

--------------------------------

### HuggingFace Space Directory Structure

Source: https://github.com/allenai/wilddet3d/blob/main/demo/huggingface/README.md

Overview of the file layout required for the HuggingFace Space deployment.

```text
hf_space/
├── app.py                  # Gradio UI
├── vis3d_glb.py            # 3D visualization
├── requirements.txt
├── assets/demo/            # Default demo images
├── wilddet3d/              # Core model code
├── vis4d/                  # Framework (transforms, geometry)
└── third_party/
    ├── sam3/               # SAM3 backbone
    └── lingbot_depth/      # LingBot-Depth backend
```

--------------------------------

### Point Prompt - Geometric Mode (One-to-One)

Source: https://context7.com/allenai/wilddet3d/llms.txt

Uses point prompts with positive (1) and negative (0) labels to perform one-to-one geometric lifting. Positive points select the object, while negative points exclude regions. The 'prompt_text' should be 'geometric'.

```python
# Point prompt: (x, y, label) where label=1 is positive, label=0 is negative
# Positive points select the object, negative points exclude regions
point_prompts = [
    [
        (250, 300, 1),   # Positive: click on the object
        (100, 100, 0),   # Negative: not this region
        (400, 200, 0),   # Negative: not this region either
    ]
]

# Point prompt with geometric mode (one-to-one)
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_points=point_prompts,
    prompt_text="geometric",
)

boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results
print(f"Detected {len(boxes[0])} objects from point prompts")
```

--------------------------------

### Visual Prompt - One-to-Many Detection

Source: https://context7.com/allenai/wilddet3d/llms.txt

Uses a 2D bounding box as a visual exemplar to find all similar objects in an image. The input box is in pixel coordinates (xyxy format). Set 'prompt_text' to 'visual' for one-to-many matching.

```python
# Visual prompt: use one chair as exemplar to find all chairs
# Input box is in pixel coordinates (xyxy format)
exemplar_box = [[100, 200, 250, 400]]  # x1, y1, x2, y2 in pixels

results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_boxes=exemplar_box,
    prompt_text="visual",  # One-to-many matching
)

boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results
print(f"Found {len(boxes[0])} similar objects using visual exemplar")
```

--------------------------------

### Point Prompt - Visual Mode (One-to-Many)

Source: https://context7.com/allenai/wilddet3d/llms.txt

Uses a single positive point prompt to find all similar objects in the image (one-to-many visual mode). Specify the category after 'visual:' in the 'prompt_text' argument.

```python
# Point prompt with visual mode (find all similar)
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_points=[[(300, 250, 1)]],  # Single positive point
    prompt_text="visual: mug",       # Find all similar mugs
)
```

--------------------------------

### Perform Visual+Label Prompt Detection

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Find similar objects with a category constraint.

```python
results = model(
    ...,
    input_boxes=[[100, 200, 300, 400]],
    prompt_text="visual: car",
)
```

--------------------------------

### Perform Point Prompt Detection

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Use point coordinates with labels for detection.

```python
results = model(
    ...,
    input_points=[[(150, 250, 1), (200, 300, 0)]],  # pixel coords
    prompt_text="geometric",
)
```

--------------------------------

### Process Detections for First Image

Source: https://context7.com/allenai/wilddet3d/llms.txt

Iterates through detections in the first image and prints detailed information for each, including category, 2D box, 3D center, dimensions, and score. Requires pre-computed 'boxes', 'class_ids', 'scores', and 'boxes3d'.

```python
for i in range(len(boxes[0])):
    print(f"Detection {i}:")
    print(f"  Category: {['car', 'person', 'bicycle', 'traffic light'][class_ids[0][i]]}")
    print(f"  2D Box (xyxy): {boxes[0][i].cpu().numpy()}")
    print(f"  3D Center: {boxes3d[0][i][:3].cpu().numpy()}")
    print(f"  3D Dims (WLH): {boxes3d[0][i][3:6].cpu().numpy()}")
    print(f"  Score: {scores[0][i].item():.3f}")
```

--------------------------------

### Perform Geometric+Label Prompt Detection

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Lift a 2D bounding box to 3D with a category label.

```python
results = model(
    ...,
    input_boxes=[[100, 200, 300, 400]],
    prompt_text="geometric: car",
)
```

--------------------------------

### Run Inference

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Perform 3D object detection using text, box, or visual exemplar prompts.

```python
from wilddet3d import build_model, preprocess
from wilddet3d.vis.visualize import draw_3d_boxes
import numpy as np
from PIL import Image

# Build model
model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,
    skip_pretrained=True,
)

# Load and preprocess image
image = np.array(Image.open("image.jpg")).astype(np.float32)

# With known camera intrinsics
intrinsics = np.load("intrinsics.npy")  # (3, 3)
data = preprocess(image, intrinsics)

# Without intrinsics (uses default: focal=max(H,W), principal point at center)
# data = preprocess(image)

# Text prompt: detect all instances of given categories
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_texts=["car", "person", "bicycle"],
)
boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results

# Box prompt (geometric): lift a 2D box to 3D (one-to-one)
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_boxes=[[100, 200, 300, 400]],  # pixel xyxy
    prompt_text="geometric",
)

# Exemplar prompt: use a 2D box as visual exemplar, find all similar objects (one-to-many)
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_boxes=[[100, 200, 300, 400]],
    prompt_text="visual",
)
```

--------------------------------

### Visualize 3D Bounding Boxes on Image

Source: https://github.com/allenai/wilddet3d/blob/main/demo/vlm/vlm_3d_detection_demo.ipynb

Projects 3D bounding box corners onto the image plane using camera intrinsics and draws the boxes. Also adds numerical labels for each detection. Requires OpenCV and PIL.

```python
import cv2
from PIL import Image

EDGES = [(0,1),(1,3),(3,2),(2,0),(4,5),(5,7),(7,6),(6,4),(0,4),(1,5),(2,6),(3,7)]
COLORS = [(0,0,255),(0,255,0),(255,150,0),(0,255,255),(255,0,255),(255,255,0),(0,128,255),(255,0,128),(128,255,0)]

# Project 3D boxes onto image
resize_scale = content_w / orig_w
image_viz = cv2.imread(IMAGE_PATH)

corners_list = boxes3d_to_corners(boxes_3d_np)

for i, corners in enumerate(corners_list):
    color = COLORS[i % len(COLORS)]
    pts = (K_padded @ corners.T)
    valid = pts[2, :] > 0.15
    pts_2d = pts[:2, :] / pts[2:, :]
    pts_orig = np.zeros_like(pts_2d)
    pts_orig[0, :] = (pts_2d[0, :] - pad_left) / resize_scale
    pts_orig[1, :] = (pts_2d[1, :] - pad_top) / resize_scale
    pts_int = pts_orig.T.astype(np.int32)

    for a, b in EDGES:
        if valid[a] and valid[b]:
            cv2.line(image_viz, tuple(pts_int[a]), tuple(pts_int[b]), color, 2)

    # Number label
    c3d = boxes_3d_np[i, :3]
    c_proj = K_padded @ c3d
    if c_proj[2] > 0.15:
        cu = int((c_proj[0] / c_proj[2] - pad_left) / resize_scale)
        cv_pt = int((c_proj[1] / c_proj[2] - pad_top) / resize_scale)
        cv2.putText(image_viz, str(i+1), (cu, cv_pt), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0,0,0), 2)
        cv2.putText(image_viz, str(i+1), (cu, cv_pt), cv2.FONT_HERSHEY_SIMPLEX, 0.4, color, 1)

# Display
result_pil = Image.fromarray(cv2.cvtColor(image_viz, cv2.COLOR_BGR2RGB))
display(result_pil)

# Print summary
print(f"\n{'='*60}")
print(f"Query: {QUERY}")
print(f"Mode: {MODE} | VLM: {vlm_config['model']} | Label: {clean_label}")
print(f"{ '='*60}")
for i in range(n_dets):
    center = boxes_3d_np[i, :3]
    dims = boxes_3d_np[i, 3:6]
    print(f"  Det {i+1}: center=({center[0]:.2f}, {center[1]:.2f}, {center[2]:.2f})m, "
          f"dims=({dims[0]:.2f}, {dims[1]:.2f}, {dims[2]:.2f})m, "
          f"2D={scores_2d_np[i]:.3f}, 3D={scores_3d_np[i]:.3f}")

# Save output
output_path = "vlm_3d_detection_output.png"
cv2.imwrite(output_path, image_viz)
print(f"\nVisualization saved to: {output_path}")
```

--------------------------------

### Run Inference with Point Prompt

Source: https://github.com/allenai/wilddet3d/blob/main/README.md

Use this snippet to run inference with point prompts. Specify input images, intrinsics, and point coordinates with labels. Optional prompt_text can be provided.

```python
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_points=[[(150, 250, 1), (200, 300, 0)]],  # (x, y, label): 1=positive, 0=negative
    prompt_text="geometric",
)
```

--------------------------------

### Geometric Prompt - One-to-One Lifting

Source: https://context7.com/allenai/wilddet3d/llms.txt

Lifts a specific 2D bounding box directly to a 3D bounding box. The 'target_box' should be in pixel coordinates (xyxy format). Set 'prompt_text' to 'geometric' for one-to-one lifting.

```python
# Geometric prompt: lift specific 2D box to 3D
target_box = [[150, 100, 350, 300]]  # Known 2D box to lift

results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_boxes=target_box,
    prompt_text="geometric",  # One-to-one lifting
)

boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results

# Get the 3D box for the input 2D box
box_3d = boxes3d[0][0].cpu().numpy()
print(f"3D Center (X,Y,Z): {box_3d[:3]}")
print(f"3D Dimensions (W,L,H): {box_3d[3:6]}")
print(f"Rotation (6D): {box_3d[6:12]}")
```

--------------------------------

### Preprocess Image with Known Intrinsics

Source: https://context7.com/allenai/wilddet3d/llms.txt

Preprocesses an image for WildDet3D using known camera intrinsics. Adjusts for model input resolution (1008x1008) via resizing, normalization, and padding.

```python
import numpy as np
from PIL import Image
from wilddet3d import preprocess

# Load image as float32 numpy array (H, W, 3)
image = np.array(Image.open("scene.jpg")).astype(np.float32)

# Preprocess with known camera intrinsics
intrinsics = np.array([
    [800.0, 0.0, 640.0],   # fx, 0, cx
    [0.0, 800.0, 360.0],   # 0, fy, cy
    [0.0, 0.0, 1.0]
], dtype=np.float32)

data = preprocess(image, intrinsics=intrinsics)
```

--------------------------------

### Perform Geometric Prompt Detection

Source: https://github.com/allenai/wilddet3d/blob/main/docs/INFERENCE.md

Lift a 2D bounding box to 3D.

```python
results = model(
    ...,
    input_boxes=[[100, 200, 300, 400]],
    prompt_text="geometric",
)
```

--------------------------------

### Text Prompt Detection

Source: https://context7.com/allenai/wilddet3d/llms.txt

Performs open-vocabulary detection using text prompts for specified categories. Requires a pre-built model and preprocessed image data. Results include 3D boxes, scores, and class IDs.

```python
from wilddet3d import build_model, preprocess
import numpy as np
from PIL import Image

model = build_model(
    checkpoint="ckpt/wilddet3d_alldata_all_prompt_v1.0.pt",
    score_threshold=0.3,
    skip_pretrained=True,
)

# Load and preprocess image
image = np.array(Image.open("street_scene.jpg")).astype(np.float32)
data = preprocess(image)

# Text prompt: detect all instances of given categories
results = model(
    images=data["images"].cuda(),
    intrinsics=data["intrinsics"].cuda()[None],  # Add batch dimension
    input_hw=[data["input_hw"]],
    original_hw=[data["original_hw"]],
    padding=[data["padding"]],
    input_texts=["car", "person", "bicycle", "traffic light"],
)

# Unpack results (per-image lists)
boxes, boxes3d, scores, scores_2d, scores_3d, class_ids, depth_maps = results
```