### Install Segment Anything Model Locally (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Installs the Segment Anything library by cloning the repository locally and installing it in editable mode. This method allows for direct modifications to the library's source code.

```bash
git clone git@github.com:facebookresearch/segment-anything.git
cd segment-anything; pip install -e .
```

--------------------------------

### Install Optional Dependencies for SAM (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Installs optional Python packages required for advanced features of the Segment Anything Model, such as mask post-processing, COCO format saving, example notebooks, and ONNX export.

```bash
pip install opencv-python pycocotools matplotlib onnxruntime onnx
```

--------------------------------

### Install Dependencies for InstantMesh (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

Installs necessary dependencies including Python, PyTorch, CUDA, and xformers. It's recommended to use Python >= 3.10, PyTorch >= 2.1.0, and CUDA >= 12.1. This script sets up a conda environment and installs core libraries.

```bash
conda create --name instantmesh python=3.10
conda activate instantmesh
pip install -U pip

# Ensure Ninja is installed
conda install Ninja

# Install the correct version of CUDA
conda install cuda -c nvidia/label/cuda-12.1.0

# Install PyTorch and xformers
# You may need to install another xformers version if you use a different PyTorch version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.22.post7

# Install other requirements
pip install -r requirements.txt
```

--------------------------------

### Install Segment Anything Model (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Installs the Segment Anything library using pip. It requires Python 3.8+ and specific versions of PyTorch and TorchVision. CUDA support is recommended for PyTorch and TorchVision.

```bash
pip install git+https://github.com/facebookresearch/segment-anything.git
```

--------------------------------

### Start Local Gradio Demo for InstantMesh (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

Launches a local Gradio demo for InstantMesh. It can automatically utilize multiple GPUs to save memory or be forced to run on a single GPU using the CUDA_VISIBLE_DEVICES environment variable. Docker deployment instructions are available separately.

```bash
python app.py
```

```bash
CUDA_VISIBLE_DEVICES=0 python app.py
```

--------------------------------

### Install PointGraph Dependencies and Run Evaluations

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

These commands demonstrate how to install the necessary dependencies for the PointGraph extension and then run evaluations using different models (OpenAI API or local Qwen-VL) and evaluation types. It also shows how to perform parallel evaluations.

```bash
cd pointgraph
pip install -r requirements.txt

# Example: openai api
python api_eval.py --model_id gpt-4.1 --eval_type re

# Example: local Qwen-VL-2.5-3B
python vlm_eval.py --model_id Qwen/Qwen2.5-VL-3B-Instruct --eval_type direct

# Example: parallel evaluation
python parallel_eval.py --model_id gpt-4.1 --group 8 --visible_nodes 0,1,2,3,4,5,6,7
```

--------------------------------

### Install Spatial CoT Dependencies and Run Evaluations

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

These commands show how to install dependencies for the Spatial CoT extension and run evaluations using configuration files with different models, such as OpenAI API or local Qwen-VL.

```bash
cd spatialcot
pip install -r requirements.txt

# Example: openai api
python api_eval.py configs/instant-mesh-large.yaml --model_id gpt-4.1

# Example: local Qwen-VL-2.5-3B
python vlm_eval.py configs/instant-mesh-large.yaml --model_id Qwen/Qwen2.5-VL-3B-Instruct
```

--------------------------------

### Install Grounding DINO Package

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/GroundingDINO/README.md

Installs the Grounding DINO package using pip. Ensure the CUDA_HOME environment variable is set if you have a CUDA environment; otherwise, it will compile in CPU-only mode.

```bash
pip install -e .

```

--------------------------------

### Get Image Masks with SAM Predictor (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Demonstrates how to use the Segment Anything Model's predictor to generate masks for objects in an image based on input prompts like points or boxes. Requires a downloaded model checkpoint and an image.

```python
from segment_anything import build_sam, SamPredictor 
predictor = SamPredictor(build_sam(checkpoint="</path/to/model.pth>"))
predictor.set_image(<your_image>)
masks, _, _ = predictor.predict(<input_prompts>)
```

--------------------------------

### Install Closed-Source VLM Dependencies (OpenAI) for OmniSpatial

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Installs the OpenAI Python client for interacting with closed-source VLMs via API. It also shows how to set the OPENAI_API_KEY environment variable, which is crucial for authentication. An optional base URL export is also provided.

```bash
pip install openai==1.81.0
export OPENAI_API_KEY="sk-..."
# optional: export OPENAI_API_BASE="https://api.openai.com/v1"
```

--------------------------------

### Install Open-Source VLM Dependencies for OmniSpatial

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Installs necessary Python packages for running open-source VLMs within the OmniSpatial project. It includes PyTorch, torchvision, transformers, qwen-vl-utils, triton, accelerate, timm, and flash-attn. Ensure your environment is activated before running.

```bash
pip install torch==2.5.1 torchvision==0.20.1 transformers==4.49.0 qwen-vl-utils[decord]==0.0.8 triton accelerate timm
pip install flash-attn --no-build-isolation
```

--------------------------------

### System Prompts for Spatial Reasoning Assistant (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

Defines various system prompts used for a spatial-reasoning assistant. These prompts guide the model's behavior, specifying tasks, reasoning steps (like step-by-step or manual CoT), and output requirements. They are stored in a Python dictionary for easy access.

```python
SYS_PROMPTS = {
    "none": "You are a spatial-reasoning assistant.\nTask: You will receive an Image, Question, and Options (A, B, C, D).\nBased on the image and question, provide your answer.\nAlways ground your answer in the visual evidence; do not hallucinate unseen objects.",

    "zeroshot_cot": "You are a spatial-reasoning assistant.\nTask: You will receive an Image, Question, and Options (A, B, C, D).\nThink step by step and provide the answer.\nAlways ground your answer in the visual evidence.",

    "manual_cot": "You are a spatial-reasoning assistant.\nGuidelines:\n1. First, carefully observe the image and identify all relevant objects and their spatial relationships.\n2. Next, break down the question into key components that need to be addressed.\n3. Think through the spatial reasoning step-by-step. It may be necessary to transfer perspective.\n4. Finally, select the most appropriate option (A, B, C, or D) based on your analysis.\nAlways ground your answer in the visual evidence."
}
```

--------------------------------

### Instantiate SAM Model with Registry (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Demonstrates how to instantiate a Segment Anything Model using the `sam_model_registry`. This allows selecting different model versions (e.g., 'vit_h', 'vit_l', 'vit_b') by specifying their names and providing the checkpoint path.

```python
from segment_anything import sam_model_registry
sam = sam_model_registry["<name>"](checkpoint="<path/to/checkpoint>")
```

--------------------------------

### Train Sparse-View Reconstruction Models (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

Commands to train sparse-view reconstruction models using Python scripts. Requires configuration files for NeRF and Mesh representations. Specifies GPUs and number of nodes for distributed training.

```bash
python train.py --base configs/instant-nerf-large-train.yaml --gpus 0,1,2,3,4,5,6,7 --num_nodes 1

python train.py --base configs/instant-mesh-large-train.yaml --gpus 0,1,2,3,4,5,6,7 --num_nodes 1
```

--------------------------------

### Fine-tune Zero123++ Model (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

Command to fine-tune the Zero123++ model using a Python script. Requires a specific configuration file and specifies the GPUs and number of nodes for distributed training.

```bash
python train.py --base configs/zero123plus-finetune.yaml --gpus 0,1,2,3,4,5,6,7 --num_nodes 1
```

--------------------------------

### Generate 3D Meshes via Command Line with InstantMesh (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

Generates 3D meshes from input images using InstantMesh via the command line. Supports saving videos, skipping background removal, and exporting meshes with vertex colors or texture maps. Different configuration files allow using various reconstruction model variants.

```bash
python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video
```

```bash
python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video --no_rembg
```

```bash
python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video --export_texmap
```

```bash
python run.py configs/instant-nerf-large.yaml examples/hatsune_miku.png --save_video
```

--------------------------------

### Run Grounding DINO Inference on Image (CLI)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/GroundingDINO/README.md

Executes Grounding DINO for object detection on a given image using command-line arguments. Specify paths for configuration, checkpoint, input image, output directory, and the text prompt. Use the [--cpu-only] flag for CPU mode.

```bash
CUDA_VISIBLE_DEVICES=6 python demo/inference_on_a_image.py \
  -c /path/to/config \
  -p /path/to/checkpoint \
  -i .asset/cats.png \
  -o "outputs/0" \
  -t "cat ear." \
  [--cpu-only] # open it for cpu mode

```

--------------------------------

### Generate Masks from Command Line (Bash)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Provides a command-line interface to generate masks for images or folders using the Segment Anything Model. Requires the SAM checkpoint, input image/folder, and an output directory.

```bash
python scripts/amg.py --checkpoint <path/to/sam/checkpoint> --input <image_or_folder> --output <output_directory>
```

--------------------------------

### Generate All Masks in an Image with SAM (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Shows how to use the Segment Anything Model's automatic mask generator to produce masks for all objects within an image. This is useful for segmenting an entire image without specific prompts.

```python
from segment_anything import build_sam, SamAutomaticMaskGenerator
mask_generator = SamAutomaticMaskGenerator(build_sam(checkpoint="</path/to/model.pth>"))
masks = mask_generator_generate(<your_image>)
```

--------------------------------

### Loading and Processing OmniSpatial Dataset (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

Demonstrates how to load and process the OmniSpatial dataset using Python's `json` and `os` modules. It shows the expected dataset structure, how to load the `data.json` file, and provides a sample data entry with explanations for its fields. It also illustrates how to construct image paths based on task type and ID.

```python
import json
import os

# Dataset structure:
# dataset/
# ↓
# ├─ Complex_Logic/       # 16% of QA pairs
# ├─ Dynamic_Reasoning/   # 27% of QA pairs
# ├─ Perspective_Taking/  # 37% of QA pairs
# ├─ Spatial_Interaction/ # 20% of QA pairs
# ── data.json

# Load dataset
with open("dataset/data.json", "r") as f:
    data = json.load(f)

# Sample data entry:
sample = {
    "id": "0_0",  # {image_id}_{question_id}
    "question": "How long will it take for the moving car closest to the camera to reach it at 10 m/s?",
    "options": ["2.7s", "14.7s", "25.7s", "3.9s"],
    "answer": 0,  # Index of correct option (A=0, B=1, C=2, D=3)
    "task_type": "Dynamic_Reasoning",
    "sub_task_type": "Motion_Analysis"
}

# Image path construction
img_path = os.path.join("dataset", sample["task_type"], f"{sample['id'].split('_')[0]}.png")
# Result: "dataset/Dynamic_Reasoning/0.png"

# Task taxonomy:
# Dynamic_Reasoning: Manipulation, Motion_Analysis
# Spatial_Interaction: Traffic_Analysis, Localization, Geospatial_Strategy
# Complex_Logic: Pattern_Recognition, Geometric_Reasoning
# Perspective_Taking: Egocentric, Allocentric, Hypothetical

# Evaluation with multiple seeds (repeat=5 by default)
items = []
for repeat_idx in range(5):
    for item in data:
        temp = item.copy()
        temp['repeat'] = repeat_idx
        items.append(temp)
# Total: 1533 * 5 = 7665 evaluation samples

```

--------------------------------

### Load SAM Model and Segment Image

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python snippet demonstrates loading the Segment Anything Model (SAM) and performing image segmentation using provided bounding boxes. It relies on the 'segment_anything' library and PyTorch. The 'segment' function takes an image and bounding boxes, returning boolean masks for each detected object.

```python
import numpy as np
import torch
from PIL import Image
from segment_anything import sam_model_registry, SamPredictor

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
SAM_ENCODER_VERSION = "vit_h"
SAM_CHECKPOINT_PATH = "checkpoints/sam_vit_h_4b8939.pth"

def get_model():
    """Load SAM model."""
    sam = sam_model_registry[SAM_ENCODER_VERSION](checkpoint=SAM_CHECKPOINT_PATH)
    sam.to(device=DEVICE)
    return sam

def segment(sam_predictor, image: np.ndarray, xyxy: np.ndarray) -> np.ndarray:
    """Generate masks for each bounding box."""
    sam_predictor.set_image(image)
    result_masks = []

    for box in xyxy:
        masks, scores, logits = sam_predictor.predict(box=box, multimask_output=True)
        index = np.argmax(scores)
        result_masks.append(masks[index])

    return np.array(result_masks, dtype=bool)

def get_mask(image, object_list, sam, detections, output_folder="output"):
    """
    Generate segmentation masks and annotated images.

    Args:
        image: PIL Image
        object_list: List of object names
        sam: SAM model
        detections: sv.Detections with bounding boxes
        output_folder: Path to save annotated images

    Returns:
        mask: (N, H, W) boolean array
        label_image: Annotated PIL Image
        object_labels: List of detected object names
    """
    sam_predictor = SamPredictor(sam)
    image_np = np.array(image)

    detections.mask = segment(sam_predictor, image_np, detections.xyxy)

    # Annotated image saved to output_folder/sam_annotated_image.jpg
    return detections.mask, label_image, object_labels

```

--------------------------------

### Local Qwen2.5-VL Model Evaluation for Answer Generation

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python script demonstrates how to evaluate answers using a local Qwen2.5-VL model. It utilizes the 'transformers' library for model and processor loading, and 'torch' for tensor operations. The script prepares image and text inputs, processes them using the model's template, and generates a response, supporting configurable model sizes and parallel processing.

```python
# vlms_eval/qwenvl_eval.py
import torch
from qwen_vl_utils import process_vision_info
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor

# Command line usage:
# python qwenvl_eval.py --model_id Qwen/Qwen2.5-VL-3B-Instruct --prompt_type manual_cot --eval_type re

# Available model sizes:
# Qwen/Qwen2.5-VL-3B-Instruct
# Qwen/Qwen2.5-VL-7B-Instruct
# Qwen/Qwen2.5-VL-32B-Instruct
# Qwen/Qwen2.5-VL-72B-Instruct
# remyxai/SpaceQwen2.5-VL-3B-Instruct (spatial-tuned variant)

# Load model
model_id = "Qwen/Qwen2.5-VL-3B-Instruct"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
    load_in_8bit=False
)
processor = AutoProcessor.from_pretrained(model_id)

# Prepare input
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "dataset/Dynamic_Reasoning/0.png"},
            {"type": "text", "text": "What is the estimated speed of the car?\nA. 10 km/h\nB. 30 km/h\nC. 50 km/h\nD. 70 km/h"},
        ],
    }
]

# Process and generate
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, padding=True, return_tensors="pt").to("cuda")

generated_ids = model.generate(**inputs, max_new_tokens=8192)
generated_ids_trimmed = [out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)]
response = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True)[0]
# Output: "Based on the motion blur and vehicle position... Answer: C"
```

--------------------------------

### InstantMesh Paper Citation (BibTeX)

Source: https://github.com/qizekun/omnispatial/blob/main/spatialcot/README.md

BibTeX entry for citing the 'InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models' paper. Includes authors, journal, and year.

```bibtex
@article{xu2024instantmesh,
  title={InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models},
  author={Xu, Jiale and Cheng, Weihao and Gao, Yiming and Wang, Xintao and Gao, Shenghua and Shan, Ying},
  journal={arXiv preprint arXiv:2404.07191},
  year={2024}
}
```

--------------------------------

### Download OmniSpatial Dataset using Hugging Face CLI

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Downloads the OmniSpatial dataset from Hugging Face Hub using the `huggingface-cli` tool. It specifies the repository and local directory for the download. The `HF_ENDPOINT` environment variable can be set for mirror usage. The dataset is organized into specific directories after unzipping.

```bash
# export HF_ENDPOINT="https://hf-mirror.com"
mkdir -p dataset
huggingface-cli download --resume-download qizekun/OmniSpatial --local-dir dataset --repo-type dataset
```

--------------------------------

### System Prompts Configuration for LLM Evaluation

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python snippet defines a dictionary of response format templates for configuring different prompting strategies in Large Language Model (LLM) evaluations. It includes formats for regular output, JSON, general LLM responses, and direct A, B, C, D answers.

```python
# system_prompts.py

# Response format templates
FORMAT_PROMPTS = {
    "re": """End your answer with a separate line formatted exactly as:
Answer: X
where X ∈ {A, B, C, D}.""",

    "json": """You need to respond with the answer in JSON format:
```json
{"analysis": "The analysis of the image and question", "answer": "A"}
```""",

    "llm": """Your answer must be clear and accurate.""",

    "direct": """Note: You only need to respond with A, B, C, or D without providing any additional information."""
}

```

--------------------------------

### PointGraph: Scene Graph Enhanced Reasoning (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

This snippet outlines the process for building 3D scene graphs from images to enhance spatial reasoning. It involves multiple stages: object detection and segmentation using Florence-2 and SAM, depth estimation with Metric3D v2, and finally, constructing the scene graph. The scene graph is then used to improve VQA prompts.

```python
# pointgraph/api_eval.py
from PIL import Image
import json
import os
import sys

# Assuming depth, serve, segmentation, and vqa_parsing are defined elsewhere
# from depth import metric3dv2 as depth_model
# from serve.scene_graph import get_scene_graph
# from segmentation import sam, florence as detection
# from utils.vqa_parsing import vqa_parsing, object_parsing_prompt, vqa_reasoning_prompt

# Placeholder for models and functions not provided in the snippet
class MockModel:
    def get_model(self):
        return "mock_model"

class MockVQA:
    def vqa_parsing(self, prompt, image, sys_prompt, model, client):
        if "Scene graph:" in prompt:
            return "The red car is approximately 12.50 units away from the camera."
        else:
            return ["red car", "camera"]

def get_scene_graph(image, pcd, mask, object_names):
    return [
      {
        "id": 1,
        "object name": "red car",
        "center": "x: 2.34, y: -0.15, z: 12.50",
        "bounding box": {
          "x_min ~ x_max": "1.20 ~ 3.48",
          "y_min ~ y_max": "-1.20 ~ 0.90",
          "z_min ~ z_max": "11.00 ~ 14.00"
        }
      }
    ], {}


detection_model = MockModel().get_model()  # Florence-2 for object detection
sam_model = MockModel().get_model()               # SAM for segmentation
metriced_model = MockModel().get_model()  # Metric3D v2 for depth estimation
vqa_client = None # Mock client
vqa_parser = MockVQA()

# Command line usage:
# cd pointgraph
# python api_eval.py --model_id gpt-4.1 --eval_type re

# Load models (using mocks)
detection_model = detection.get_model()  # Florence-2 for object detection
sam_model = sam.get_model()               # SAM for segmentation
metriced_model = depth_model.get_model()  # Metric3D v2 for depth estimation

# Process image
# Create a dummy image file for the example to run
with open("dataset/Dynamic_Reasoning/0.png", "w") as f: f.write("")
image = Image.open("dataset/Dynamic_Reasoning/0.png").convert("RGB")
question = "How far is the red car from the camera?"

# Step 1: Extract objects from question using VLM
object_list = vqa_parser.vqa_parsing("Question: " + question, image, sys_prompt="", model="gpt-4.1", client=vqa_client)
# Output: ["red car", "camera"]

# Step 2: Detect and segment objects (mocked)
detections = [] # detection.get_detections(image, object_list, detection_model, "output/")
mask, _, object_names = [], [], object_list # sam.get_mask(image, object_list, sam_model, detections, "output/")

# Step 3: Estimate depth and build scene graph (mocked)
pcd = [] # depth_model.depth_estimation(image, metriced_model, "output/")
scene_graph, _ = get_scene_graph(image, pcd, mask, object_names)

# Scene graph output:
# [
#   {
#     "id": 1,
#     "object name": "red car",
#     "center": "x: 2.34, y: -0.15, z: 12.50",
#     "bounding box": {
#       "x_min ~ x_max": "1.20 ~ 3.48",
#       "y_min ~ y_max": "-1.20 ~ 0.90",
#       "z_min ~ z_max": "11.00 ~ 14.00"
#     }
#   }
# ]

# Step 4: Use scene graph in VQA prompt
prompt = f"Question: {question}\nScene graph: {json.dumps(scene_graph, indent=2)}"
response = vqa_parser.vqa_parsing(prompt, image, sys_prompt="", model="gpt-4.1", client=vqa_client)
print(f"Final Response: {response}")

```

--------------------------------

### Export SAM Mask Decoder to ONNX (Python)

Source: https://github.com/qizekun/omnispatial/blob/main/pointgraph/segmentation/SAM/README.md

Exports the Segment Anything Model's mask decoder to the ONNX format. This allows the model to be run in environments supporting ONNX runtime, such as web browsers.

```bash
python scripts/export_onnx_model.py --checkpoint <path/to/checkpoint> --output <path/to/output>
```

--------------------------------

### Generate Response from Image and Question (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

This snippet demonstrates how to generate a response to a question based on an image. It utilizes a pre-trained model and tokenizer to process pixel values and a textual question, returning a text-based answer. This is useful for visual question answering tasks.

```python
question = "<image>\nFrom the camera's perspective, which object is closest?\nA. Table\nB. Chair\nC. Lamp\nD. Door"
generation_config = dict(max_new_tokens=8192, do_sample=True)
response = model.chat(tokenizer, pixel_values, question, generation_config)
# Output: "Analyzing the depth and perspective... Answer: B"
```

--------------------------------

### Florence-2 Object Detection with Text Queries (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

This function utilizes the Florence-2 model from Microsoft to perform open-vocabulary object detection in an image based on a provided list of text queries. It returns detected bounding boxes, class IDs, and confidences using the supervision library.

```python
import torch
import numpy as np
from PIL import Image
import supervision as sv
from transformers import AutoModelForCausalLM, AutoProcessor

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
FLORENCE_CHECKPOINT = "microsoft/Florence-2-base"

def get_model():
    """Load Florence-2 model and processor."""
    model = AutoModelForCausalLM.from_pretrained(
        FLORENCE_CHECKPOINT, trust_remote_code=True
    ).to(DEVICE).eval()
    processor = AutoProcessor.from_pretrained(FLORENCE_CHECKPOINT, trust_remote_code=True)
    return model, processor

def get_detections(image, obj_list, florence_model, output_folder="output"):
    """
    Detect objects in image based on text queries.

    Args:
        image: PIL Image
        obj_list: List of object names to detect ["car", "person", "tree"]
        florence_model: Tuple of (model, processor)
        output_folder: Path to save annotated images

    Returns:
        sv.Detections object with xyxy boxes, class_ids, confidences
    """
    model, processor = florence_model
    detections_list = []

    for i, obj in enumerate(obj_list):
        prompt = "<OPEN_VOCABULARY_DETECTION>" + obj
        inputs = processor(text=prompt, images=image, return_tensors="pt").to(DEVICE)

        generated_ids = model.generate(
            input_ids=inputs["input_ids"],
            pixel_values=inputs["pixel_values"],
            max_new_tokens=1024,
            num_beams=3
        )
        generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
        result = processor.post_process_generation(
            generated_text, task="<OPEN_VOCABULARY_DETECTION>", image_size=image.size
        )

        detections = sv.Detections.from_lmm(lmm=sv.LMM.FLORENCE_2, result=result, resolution_wh=image.size)
        detections.class_id = np.full(len(detections.xyxy), i)
        detections_list.append(detections)

    return sv.Detections.merge(detections_list)

```

--------------------------------

### Evaluate Local Qwen-VL-2.5-3B using OmniSpatial Script

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Runs the OmniSpatial evaluation for a local Qwen-VL-2.5-3B model. This command is executed from the 'vlm_eval' directory and specifies the model ID, prompt type, and evaluation type. It's designed for models run locally.

```bash
# Example: local Qwen-VL-2.5-3B
cd vlm_eval
python qwenvl_eval.py --model_id Qwen/Qwen2.5-VL-3B-Instruct --prompt_type manual_cot --eval_type re
```

--------------------------------

### Local InternVL3 Model Evaluation with Multi-GPU Support

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python code outlines the evaluation of InternVL3 models locally, featuring automatic distribution of model layers across multiple GPUs. It utilizes the 'transformers' library for model and tokenizer loading, and 'torch' for tensor operations. The script includes a function to intelligently split the model across available GPUs and handles image loading with dynamic tiling, optimized for large model deployments.

```python
# vlms_eval/internvl_eval.py
import torch
import math
from PIL import Image
from transformers import AutoModel, AutoTokenizer, AutoConfig

# Command line usage:
# python internvl_eval.py --model_id OpenGVLab/InternVL3-8B --prompt_type manual_cot --eval_type re

# Available model sizes:
# OpenGVLab/InternVL3-2B
# OpenGVLab/InternVL3-8B
# OpenGVLab/InternVL3-9B
# OpenGVLab/InternVL3-14B
# OpenGVLab/InternVL3-38B
# OpenGVLab/InternVL3-78B

def split_model(model_name):
    """Distribute model layers across available GPUs."""
    device_map = {}
    world_size = torch.cuda.device_count()
    config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
    num_layers = config.llm_config.num_hidden_layers
    num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5))
    # First GPU handles ViT, treated as half capacity
    # ... layer distribution logic
    return device_map

# Load model
model_id = "OpenGVLab/InternVL3-8B"
device_map = split_model(model_id)
model = AutoModel.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    load_in_8bit=False,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True,
    device_map=device_map
).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True, use_fast=False)

# Load and preprocess image with dynamic tiling
pixel_values = load_image("dataset/Perspective_Taking/5.png", max_num=12).to(torch.bfloat16).cuda()
```

--------------------------------

### Parallel Multi-GPU Evaluation (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

This code facilitates running evaluations concurrently across multiple GPUs to accelerate benchmarking. It uses `subprocess` to launch evaluation scripts in parallel, managing GPU device assignments through environment variables. The results are saved to a specified path and can be merged later.

```python
# vlms_eval/parallel_eval.py
import subprocess
import os
from concurrent.futures import ThreadPoolExecutor

# Command line usage:
# python parallel_eval.py --model qwenvl --model_id Qwen/Qwen2.5-VL-3B-Instruct --group 8 --visible_nodes 0,1,2,3,4,5,6,7

def run_eval(group_index, group, all_visible_nodes, model, eval_type, prompt_type, model_id, repeat, dataset_path, result_path):
    """Launch a single evaluation subprocess on assigned GPUs."""
    visible_nodes_list = all_visible_nodes.split(",")
    eval_per_node = len(visible_nodes_list) // group
    cuda_visible_devices = [visible_nodes_list[group_index * eval_per_node + i] for i in range(eval_per_node)]

    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = ",".join(cuda_visible_devices)

    cmd = [
        "python", f"{model}_eval.py",
        "--group_index", str(group_index),
        "--group", str(group),
        "--repeat", str(repeat),
        "--eval_type", eval_type,
        "--prompt_type", prompt_type,
        "--model_id", model_id,
        "--dataset_path", dataset_path,
        "--result_path", result_path,
    ]
    return subprocess.Popen(cmd, env=env)

# Run 8 parallel processes, each on 1 GPU
group = 8
visible_nodes = "0,1,2,3,4,5,6,7"

with ThreadPoolExecutor(max_workers=group) as executor:
    futures = [executor.submit(run_eval, i, group, visible_nodes, "qwenvl", "re", "manual_cot",
               "Qwen/Qwen2.5-VL-3B-Instruct", 5, "../dataset", "result") for i in range(group)]
    procs = [f.result() for f in futures]

for p in procs:
    p.wait()

# Results are saved to result/{model_id}/results_{group_index}.json and merged
```

--------------------------------

### Metric3D v2 Depth Estimation and Point Cloud Reconstruction

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python snippet details the process of estimating metric depth from a single image using the Metric3D v2 model. It includes loading the model, predicting depth and normals, and reconstructing a 3D point cloud. Dependencies include PyTorch, NumPy, PIL, and mmengine. The function returns depth arrays, colorized depth images, and point cloud data.

```python
# pointgraph/depth/metric3dv2.py
import torch
import numpy as np
from PIL import Image
from mmengine import Config
import os

# Assume these functions are defined elsewhere and imported
def get_configured_monodepth_model(cfg):
    pass
def load_ckpt(checkpoint_path, model, strict_match=False):
    pass
def get_prediction(model, input, cam_model, pad_info, scale_info, gt_depth=None, normalize_scale, ori_shape):
    pass
def reconstruct_pcd(pred_depth, fx, fy, cx, cy):
    pass
def get_intrinsic(img):
    pass

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
CFG_PATH = "depth/metric_3d_v2/configs/HourglassDecoder/vit.raft5.large.py"
CHECKPOINT_PATH = "checkpoints/metric_depth_vit_large_800k.pth"

def get_model():
    """Load Metric3D v2 model."""
    cfg = Config.fromfile(CFG_PATH)
    model = get_configured_monodepth_model(cfg)
    model, _, _, _ = load_ckpt(CHECKPOINT_PATH, model, strict_match=False)
    model.to(DEVICE).eval()
    return cfg, model

def predict_depth_normal(img, model, cfg, fx=1000.0, fy=1000.0):
    """
    Predict depth and generate point cloud.

    Args:
        img: PIL Image
        model: Metric3D model
        cfg: Model config
        fx, fy: Camera focal lengths

    Returns:
        pred_depth: (H, W) depth array in meters
        img: Colorized depth PIL Image
        pcd: (H, W, 3) point cloud array
    """
    img_np = np.array(img)
    intrinsic = [fx, fy, img_np.shape[1] / 2, img_np.shape[0] / 2]

    # Dummy values for demonstration purposes
    pad = None
    label_scale_factor = None
    rgb_input = img_np # Placeholder
    cam_models_stacks = None # Placeholder
    output = None # Placeholder
    confidence = None # Placeholder
    pred_depth_scale = None # Placeholder
    scale = None # Placeholder
    depth_img = img # Placeholder

    with torch.no_grad():
        pred_depth, pred_depth_scale, scale, output, confidence = get_prediction(
            model=model, input=rgb_input, cam_model=cam_models_stacks,
            pad_info=pad, scale_info=label_scale_factor, gt_depth=None,
            normalize_scale=cfg.data_basic.depth_range[1],
            ori_shape=[img_np.shape[0], img_np.shape[1]]
        )

    pred_depth = pred_depth.squeeze().cpu().numpy()
    pred_depth[pred_depth < 0] = 0
    pcd = reconstruct_pcd(pred_depth, intrinsic[0], intrinsic[1], intrinsic[2], intrinsic[3])

    return pred_depth, depth_img, pcd

def depth_estimation(img, metriced_model, output_folder="output"):
    """High-level depth estimation with automatic intrinsics."""
    cfg, model = metriced_model
    fx, fy = get_intrinsic(img)
    depth, depth_img, pcd = predict_depth_normal(img, model, cfg, fx, fy)
    depth_img.save(os.path.join(output_folder, "depth_estimation.jpg"))
    return depth, depth_img, pcd

```

--------------------------------

### API Evaluation with OpenAI-Compatible Models

Source: https://context7.com/qizekun/omnispatial/llms.txt

Evaluate closed-source VLMs using the OpenAI API with configurable prompt types and evaluation methods. Supports concurrent processing with resume capability.

```APIDOC
## API Evaluation with OpenAI-Compatible Models

### Description
This endpoint allows for the evaluation of closed-source Vision-Language Models (VLMs) by interfacing with OpenAI-compatible APIs. It supports various configuration options for prompts and evaluation strategies, including concurrent processing and the ability to resume interrupted runs.

### Method
POST (Implied by script execution, not a direct HTTP endpoint)

### Endpoint
N/A (Script-based execution)

### Parameters
#### Command Line Arguments
- **--model_id** (string) - Required - The identifier for the OpenAI-compatible model to be used (e.g., `gpt-4.1`, `gpt-4o`).
- **--prompt_type** (string) - Optional - The type of prompt to use for evaluation. Options: `none`, `zeroshot_cot`, `manual_cot`. Defaults to `none`.
- **--eval_type** (string) - Optional - The evaluation method to employ. Options: `direct`, `re`, `json`, `llm`. Defaults to `direct`.
- **--repeat** (integer) - Optional - The number of evaluation seeds to run. Defaults to 5.
- **--max_workers** (integer) - Optional - The maximum number of concurrent threads for processing. Defaults to 64.
- **--resume** (boolean) - Optional - Flag to indicate whether to resume a previous run.
- **--blind** (boolean) - Optional - Flag to perform text-only evaluation without images.

### Request Example
```bash
python api_eval.py --model_id gpt-4.1 --prompt_type manual_cot --eval_type re
```

### Response
#### Success Response (200)
- **result** (object) - A JSON object containing the evaluation results, structured by task dimensions and accuracy metrics. The exact structure is detailed in the script's comments.
```

--------------------------------

### API Evaluation Script for OpenAI-Compatible Models

Source: https://context7.com/qizekun/omnispatial/llms.txt

This Python script, `api_eval.py`, facilitates the evaluation of closed-source VLMs using OpenAI-compatible APIs. It supports configurable prompt types and evaluation methods, concurrent processing, and resume capabilities. The script takes various command-line arguments to customize the evaluation process and saves results in a structured JSON format.

```python
# api_eval.py - Main API evaluation script
import os
import json
from PIL import Image
from openai import OpenAI

# Setup OpenAI client
client = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url=os.getenv("OPENAI_API_BASE", "https://api.openai.com/v1/")
)

# Command line usage:
# python api_eval.py --model_id gpt-4.1 --prompt_type manual_cot --eval_type re

# Available options:
# --model_id: Any OpenAI-compatible model (gpt-4.1, gpt-4o, etc.)
# --prompt_type: none | zeroshot_cot | manual_cot
# --eval_type: direct | re | json | llm
# --repeat: Number of evaluation seeds (default: 5)
# --max_workers: Concurrent threads (default: 64)
# --resume: Resume from previous run
# --blind: Text-only evaluation (no images)

# Example evaluation result structure (saved to result/{model_id}_{prompt_type}_{eval_type}.json):
result = {
    "Total": [True, False, True, ...],  # Overall accuracy
    "Dynamic_Reasoning": {
        "Manipulation": [True, True, False, ...],
        "Motion_Analysis": [True, False, ...],
        "Total": [...]
    
```

--------------------------------

### Create and Activate Conda Environment for OmniSpatial

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

This snippet outlines the bash commands to create a new Conda environment named 'omnispatial' with Python 3.12 and then activate it. This ensures a clean and isolated environment for the project dependencies.

```bash
conda create -n omnispatial python=3.12 -y
conda activate omnispatial

git clone https://github.com/qizekun/OmniSpatial.git
cd OmniSpatial
```

--------------------------------

### Build Scene Graph with 3D Coordinates (Python)

Source: https://context7.com/qizekun/omnispatial/llms.txt

This function constructs a scene graph by extracting 3D coordinates, calculating object centers, and defining bounding boxes for detected objects from image data, point clouds, and segmentation masks. It utilizes a utility function `remove_outliers` for data cleaning.

```python
import numpy as np
from serve.utils import remove_outliers

def get_scene_graph(image, pcd, mask, object_names):
    """
    Build scene graph with 3D coordinates for each object.

    Args:
        image: PIL Image
        pcd: Point cloud array (H, W, 3) from depth estimation
        mask: Segmentation masks (N, H, W) for N objects
        object_names: List of object names

    Returns:
        objects_info: List of dicts with id, name, center, bounding box
        objects_dict: List of dicts with numeric coordinates
    """
    if len(mask) == 0:
        return [], []

    n, h, w = mask.shape
    objects_info = []

    for i in range(n):
        object_mask = mask[i]
        segmented_object = pcd[object_mask]  # Extract 3D points for this object

        segmented_object = remove_outliers(segmented_object)
        min_values = segmented_object.min(axis=0)
        max_values = segmented_object.max(axis=0)
        mean_values = segmented_object.mean(axis=0)

        node = {
            'id': i + 1,
            'object name': object_names[i],
            'center': f"x: {mean_values[0]:.2f}, y: {mean_values[1]:.2f}, z: {mean_values[2]:.2f}",
            'bounding box': {
                "x_min ~ x_max": f"{min_values[0]:.2f} ~ {max_values[0]:.2f}",
                "y_min ~ y_max": f"{min_values[1]:.2f} ~ {max_values[1]:.2f}",
                "z_min ~ z_max": f"{min_values[2]:.2f} ~ {max_values[2]:.2f}"
            }
        }
        objects_info.append(node)

    return objects_info, objects_dict

# Coordinate system:
# x-axis: Left (-) to Right (+) in image
# y-axis: Bottom (-) to Top (+) in image
# z-axis: Near (camera) to Far (scene depth)

```

--------------------------------

### Evaluate GPT-4.1 using OmniSpatial API Evaluation Script

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Executes the OmniSpatial evaluation for GPT-4.1 via the OpenAI API. This command specifies the model ID, prompt type ('manual_cot'), and evaluation type ('re'). Results are typically saved to a 'result/' directory.

```bash
# Example: GPT-4.1 via OpenAI API
python api_eval.py --model_id gpt-4.1 --prompt_type manual_cot --eval_type re
```

--------------------------------

### Perform Parallel Evaluation with OmniSpatial

Source: https://github.com/qizekun/omnispatial/blob/main/README.md

Initiates a parallel evaluation process for OmniSpatial. This script allows specifying the model type, model ID, the number of groups for parallel processing, and visible nodes (GPUs) to utilize. This is useful for accelerating evaluations on multi-GPU systems.

```bash
# Example: parallel evaluation
cd vlm_eval
python parallel_eval.py --model qwenvl --model_id Qwen/Qwen2.5-VL-3B-Instruct --group 8 --visible_nodes 0,1,2,3,4,5,6,7
```