### Setup and Start OmniBox Windows VM

Source: https://context7.com/microsoft/omniparser/llms.txt

Initializes the OmniBox Windows VM on a CPU machine with Docker. Requires a Windows 11 ISO placed at the specified path. The initial setup can take a significant amount of time.

```bash
# Place Windows 11 ISO as omnitool/omnibox/vm/win11iso/custom.iso
cd omnitool/omnibox/scripts
./manage_vm.sh create  # Takes 20-90 minutes for initial setup
# Later: ./manage_vm.sh start | stop | delete
```

--------------------------------

### OmniTool Complete Setup Instructions

Source: https://context7.com/microsoft/omniparser/llms.txt

Provides bash commands for setting up the OmniTool environment, including creating a conda environment and installing dependencies from a requirements file.

```bash
# 1. Install dependencies
conda create -n "omni" python==3.12
conda activate omni
pip install -r requirements.txt
```

--------------------------------

### Start OmniParser Server

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Navigate to the server directory and start the OmniParser server using this command.

```bash
cd OmniParser/omnitool/omniparserserver
python -m omniparserserver
```

--------------------------------

### Create OmniBox VM

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Build the Docker container and install the Windows 11 ISO to the storage folder. This process can take 20-90 minutes.

```bash
cd OmniParser/omnitool/omnibox/scripts
./manage_vm.sh create
```

--------------------------------

### Start Gradio Agent Interface

Source: https://context7.com/microsoft/omniparser/llms.txt

Starts the Gradio-based agent interface, connecting to the OmniParser server and the OmniBox Windows VM. Ensure the server and VM are running before starting the interface.

```bash
cd omnitool/gradio
python app.py \
    --windows_host_url localhost:8006 \
    --omniparser_server_url localhost:8000
```

--------------------------------

### Install OmniParser Environment

Source: https://github.com/microsoft/omniparser/blob/master/README.md

Steps to clone the repository and set up the Python environment using conda and pip.

```bash
cd OmniParser
conda create -n "omni" python==3.12
conda activate omni
pip install -r requirements.txt
```

--------------------------------

### Start Gradio Interface

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Navigate to the Gradio directory, activate the conda environment, and start the Gradio server with specified host URLs.

```bash
cd OmniParser/omnitool/gradio
conda activate omni
python app.py --windows_host_url localhost:8006 --omniparser_server_url localhost:8000
```

--------------------------------

### Start OmniParser Server

Source: https://context7.com/microsoft/omniparser/llms.txt

Starts the OmniParser FastAPI server for remote screenshot parsing. Navigate to the server directory and run the Python module with specified model paths, device, and network settings. Ensure all dependencies are installed.

```bash
# Start the OmniParser server
cd omnitool/omniparserserver
python -m omniparserserver \
    --som_model_path ../../weights/icon_detect/model.pt \
    --caption_model_name florence2 \
    --caption_model_path ../../weights/icon_caption_florence \
    --device cuda \
    --BOX_TRESHOLD 0.05 \
    --host 0.0.0.0 \
    --port 8000
```

--------------------------------

### Start OmniParser Server

Source: https://context7.com/microsoft/omniparser/llms.txt

Launches the OmniParser server on a GPU machine. It listens for requests on all interfaces (0.0.0.0) on port 8000.

```bash
cd omnitool/omniparserserver
python -m omniparserserver --device cuda --host 0.0.0.0 --port 8000
```

--------------------------------

### Launch Gradio Demo Interface

Source: https://context7.com/microsoft/omniparser/llms.txt

Launches an interactive web demo using Gradio for testing OmniParser. Allows uploading screenshots and adjusting detection thresholds. Requires Python and Gradio installation.

```python
# Run the Gradio demo
# python gradio_demo.py

from util.utils import check_ocr_box, get_yolo_model, get_caption_model_processor, get_som_labeled_img
import gradio as gr
from PIL import Image
import base64
import io

# Load models globally
yolo_model = get_yolo_model('weights/icon_detect/model.pt')
caption_model_processor = get_caption_model_processor(
    model_name="florence2",
    model_name_or_path="weights/icon_caption_florence"
)

def process(image_input, box_threshold, iou_threshold, use_paddleocr, imgsz):
    """Process uploaded image and return annotated result."""
    box_overlay_ratio = image_input.size[0] / 3200
    draw_bbox_config = {
        'text_scale': 0.8 * box_overlay_ratio,
        'text_thickness': max(int(2 * box_overlay_ratio), 1),
        'text_padding': max(int(3 * box_overlay_ratio), 1),
        'thickness': max(int(3 * box_overlay_ratio), 1),
    }

    ocr_bbox_rslt, _ = check_ocr_box(
        image_input,
        display_img=False,
        output_bb_format='xyxy',
        easyocr_args={'paragraph': False, 'text_threshold': 0.9},
        use_paddleocr=use_paddleocr
    )
    text, ocr_bbox = ocr_bbox_rslt

    dino_labled_img, label_coordinates, parsed_content_list = get_som_labeled_img(
        image_input,
        yolo_model,
        BOX_TRESHOLD=box_threshold,
        output_coord_in_ratio=True,
        ocr_bbox=ocr_bbox,
        draw_bbox_config=draw_bbox_config,
        caption_model_processor=caption_model_processor,
        ocr_text=text,
        iou_threshold=iou_threshold,
        imgsz=imgsz
    )

    image = Image.open(io.BytesIO(base64.b64decode(dino_labled_img)))
    parsed_content_str = '\n'.join([f'icon {i}: {v}' for i, v in enumerate(parsed_content_list)])
    return image, parsed_content_str

# Launch: python gradio_demo.py
# Access at http://127.0.0.1:7861
```

--------------------------------

### Manage OmniBox VM

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Commands to manage the OmniBox VM after initial creation. Use 'start' to run, 'stop' to halt, and 'delete' to remove the VM.

```bash
./manage_vm.sh start
```

```bash
./manage_vm.sh stop
```

```bash
./manage_vm.sh delete
```

--------------------------------

### Download Model Weights

Source: https://context7.com/microsoft/omniparser/llms.txt

Downloads necessary model weights for icon detection and captioning using the huggingface-cli. Ensure you have the CLI installed and configured.

```bash
for f in icon_detect/{train_args.yaml,model.pt,model.yaml} icon_caption/{config.json,generation_config.json,model.safetensors}; do
    huggingface-cli download microsoft/OmniParser-v2.0 "$f" --local-dir weights
done
mv weights/icon_caption weights/icon_caption_florence
```

--------------------------------

### OmniParser Server API

Source: https://context7.com/microsoft/omniparser/llms.txt

Instructions for starting the OmniParser FastAPI server, which provides a REST endpoint for remote screenshot parsing.

```APIDOC
## OmniParser Server API

The FastAPI server provides a REST endpoint for parsing screenshots, enabling remote processing on GPU servers while keeping agents on CPU machines.

### Start Server
```bash
cd omnitool/omniparserserver
python -m omniparserserver \
    --som_model_path ../../weights/icon_detect/model.pt \
    --caption_model_name florence2 \
    --caption_model_path ../../weights/icon_caption_florence \
    --device cuda \
    --BOX_TRESHOLD 0.05 \
    --host 0.0.0.0 \
    --port 8000
```

### API Endpoint

#### POST /parse

**Description**: Parses a base64-encoded screenshot.

**Request Body**:
```json
{
  "image_base64": "<base64_encoded_image_string>"
}
```

**Response (Success)**:
```json
{
  "labeled_image_base64": "<base64_encoded_image_string>",
  "parsed_content_list": [
    {
      "type": "text",
      "bbox": [0.15, 0.01, 0.35, 0.03],
      "interactivity": false,
      "content": "File  Edit  View",
      "source": "box_ocr_content_ocr"
    },
    {
      "type": "icon",
      "bbox": [0.94, 0.93, 0.95, 0.95],
      "interactivity": true,
      "content": "Close button",
      "source": "box_yolo_content_yolo"
    }
  ]
}
```

**Response (Error)**:
```json
{
  "detail": "Error message"
}
```
```

--------------------------------

### Initialize and Use Omniparser Class

Source: https://context7.com/microsoft/omniparser/llms.txt

Initializes the Omniparser with model configurations and parses a base64-encoded screenshot to get labeled bounding boxes and content lists. Ensure image files are accessible and correctly encoded.

```python
from util.omniparser import Omniparser
import base64

# Initialize with model configuration
config = {
    'som_model_path': 'weights/icon_detect/model.pt',
    'caption_model_name': 'florence2',
    'caption_model_path': 'weights/icon_caption_florence',
    'BOX_TRESHOLD': 0.05
}
parser = Omniparser(config)

# Load and encode an image
with open('screenshot.png', 'rb') as f:
    image_base64 = base64.b64encode(f.read()).decode('ascii')

# Parse the screenshot
labeled_image_base64, parsed_content_list = parser.parse(image_base64)

# parsed_content_list contains elements like:
# [
#   {'type': 'text', 'bbox': [0.15, 0.01, 0.35, 0.03], 'interactivity': False,
#    'content': 'File  Edit  View', 'source': 'box_ocr_content_ocr'},
#   {'type': 'icon', 'bbox': [0.94, 0.93, 0.95, 0.95], 'interactivity': True,
#    'content': 'Close button', 'source': 'box_yolo_content_yolo'}
# ]
```

--------------------------------

### Make Request to OmniParser Server

Source: https://context7.com/microsoft/omniparser/llms.txt

Example Python code to make a request to the OmniParser server API. This requires the 'requests' and 'base64' libraries. Ensure the server is running and accessible at the specified URL.

```python
import requests
import base64
```

--------------------------------

### Get YOLO Model Device and Type

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Retrieves the device and type of the loaded YOLO model. This is useful for verifying the model's placement and class.

```python
som_model.device, type(som_model)
```

--------------------------------

### Initialize and Use OmniParserClient

Source: https://context7.com/microsoft/omniparser/llms.txt

Initializes the OmniParserClient to connect to the server and captures/parses the current screen. The result contains annotated screenshots and parsed content.

```python
from agent.llm_utils.omniparserclient import OmniParserClient

# Initialize client pointing to OmniParser server
client = OmniParserClient(url="http://localhost:8000/parse/")

# Capture and parse current screen
parsed_result = client()

# Result structure with all necessary data for agents:
# {
#   'som_image_base64': '<annotated screenshot>',
#   'parsed_content_list': [...],
#   'latency': 0.52,
#   'width': 1920,
#   'height': 1080,
#   'original_screenshot_base64': '<raw screenshot>',
#   'screenshot_uuid': 'unique-id',
#   'screen_info': 'ID: 0, Text: Home\nID: 1, Icon: Search\n...'
# }

print(f"Screen: {parsed_result['width']}x{parsed_result['height']}")
print(f"Elements found: {len(parsed_result['parsed_content_list'])}")
print(parsed_result['screen_info'])
```

--------------------------------

### Run OmniParser Gradio Demo

Source: https://github.com/microsoft/omniparser/blob/master/README.md

Command to launch the interactive Gradio demo for OmniParser.

```bash
python gradio_demo.py
```

--------------------------------

### ComputerTool Actions for Windows VM Interaction

Source: https://context7.com/microsoft/omniparser/llms.txt

Demonstrates various actions that can be performed on a Windows VM using the ComputerTool, including mouse movement, clicks, typing, keyboard shortcuts, scrolling, and taking screenshots. Requires asyncio to run.

```python
from tools.computer import ComputerTool
import asyncio

# Initialize computer tool (connects to Windows VM at localhost:5000)
computer = ComputerTool(is_scaling=True)

# Available actions
async def demo_actions():
    # Move mouse to coordinates
    result = await computer(action="mouse_move", coordinate=(500, 300))

    # Click actions
    result = await computer(action="left_click")
    result = await computer(action="right_click")
    result = await computer(action="double_click")

    # Type text (auto-clicks before typing, presses enter after)
    result = await computer(action="type", text="Hello World")

    # Keyboard shortcuts
    result = await computer(action="key", text="ctrl+c")
    result = await computer(action="key", text="alt+tab")

    # Scrolling
    result = await computer(action="scroll_up")
    result = await computer(action="scroll_down")

    # Take screenshot
    screenshot_result = await computer(action="screenshot")
    # screenshot_result.base64_image contains the image

    # Get cursor position
    pos_result = await computer(action="cursor_position")
    # Returns "X=500,Y=300"

    # Wait for UI to respond
    result = await computer(action="wait")

# Run async actions
asyncio.run(demo_actions())
```

--------------------------------

### Load Icon Captioning Model (Florence2 or BLIP2)

Source: https://context7.com/microsoft/omniparser/llms.txt

Loads an icon captioning model, supporting Florence2 (recommended) or BLIP2. Requires specifying the model name, path to weights, and target device (e.g., 'cuda'). Returns a dictionary containing the model and its processor.

```python
from util.utils import get_caption_model_processor
import torch

caption_processor = get_caption_model_processor(
    model_name="florence2",
    model_name_or_path="weights/icon_caption_florence",
    device="cuda"
)

model = caption_processor['model']
processor = caption_processor['processor']
```

```python
from util.utils import get_caption_model_processor
import torch

caption_processor_blip = get_caption_model_processor(
    model_name="blip2",
    model_name_or_path="weights/icon_caption_blip2",
    device="cuda"
)
```

--------------------------------

### Initialize and Run VLMAgent for GUI Automation

Source: https://context7.com/microsoft/omniparser/llms.txt

Initializes a Vision-Language Model Agent that combines OmniParser output with LLMs for GUI automation. Supports multiple providers and models. Demonstrates preparing parsed screen data and running the agent with a user task, outputting an action JSON.

```python
from agent.vlm_agent import VLMAgent

def output_callback(message, sender):
    print(f"[{sender}] {message}")

def api_response_callback(response):
    pass

agent = VLMAgent(
    model="omniparser + gpt-4o",
    provider="openai",
    api_key="your-api-key",
    output_callback=output_callback,
    api_response_callback=api_response_callback,
    max_tokens=4096,
    only_n_most_recent_images=5,
    print_usage=True
)

parsed_screen = {
    'original_screenshot_base64': '<base64 image>',
    'som_image_base64': '<annotated image base64>',
    'latency': 0.45,
    'screen_info': 'ID: 0, Text: File\nID: 1, Icon: Settings\n...', 
    'parsed_content_list': [...],
    'screenshot_uuid': 'abc123',
    'width': 1920,
    'height': 1080
}

messages = [{"role": "user", "content": "Click on the Settings icon"}]
response, action_json = agent(messages, parsed_screen)
```

--------------------------------

### Download OmniParser V2 Model Checkpoints

Source: https://github.com/microsoft/omniparser/blob/master/README.md

Command to download V2 model checkpoints from Hugging Face and organize them into the weights directory.

```bash
for f in icon_detect/{train_args.yaml,model.pt,model.yaml} icon_caption/{config.json,generation_config.json,model.safetensors}; do huggingface-cli download microsoft/OmniParser-v2.0 "$f" --local-dir weights; done
mv weights/icon_caption weights/icon_caption_florence
```

--------------------------------

### Download OmniParser V2 Weights

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Use this command to download the necessary V2 weights for OmniParser. Ensure the caption weights folder is named correctly.

```bash
rm -rf weights/icon_detect weights/icon_caption weights/icon_caption_florence 
for folder in icon_caption icon_detect; do huggingface-cli download microsoft/OmniParser-v2.0 --local-dir weights --repo-type model --include "$folder/*"; done
mv weights/icon_caption weights/icon_caption_florence
```

--------------------------------

### Initialize Captioning Model Processor

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Initializes a captioning model processor, supporting options like fine-tuned BLIP-2 or Florence2. Specify the model name, path, and device.

```python
# two choices for caption model: fine-tuned blip2 or florence2
import importlib
# import util.utils
# importlib.reload(utils)
from util.utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model
caption_model_processor = get_caption_model_processor(model_name="florence2", model_name_or_path="weights/icon_caption_florence", device=device)
```

--------------------------------

### Omniparser Class Usage

Source: https://context7.com/microsoft/omniparser/llms.txt

Demonstrates how to initialize and use the main Omniparser class to parse a base64-encoded screenshot.

```APIDOC
## Omniparser Class

The main `Omniparser` class provides the core parsing functionality, taking a base64-encoded screenshot and returning labeled bounding boxes with semantic descriptions for all detected UI elements including text and icons.

### Request Example
```python
from util.omniparser import Omniparser
import base64

# Initialize with model configuration
config = {
    'som_model_path': 'weights/icon_detect/model.pt',
    'caption_model_name': 'florence2',
    'caption_model_path': 'weights/icon_caption_florence',
    'BOX_TRESHOLD': 0.05
}
parser = Omniparser(config)

# Load and encode an image
with open('screenshot.png', 'rb') as f:
    image_base64 = base64.b64encode(f.read()).decode('ascii')

# Parse the screenshot
labeled_image_base64, parsed_content_list = parser.parse(image_base64)

# parsed_content_list contains elements like:
# [
#   {'type': 'text', 'bbox': [0.15, 0.01, 0.35, 0.03], 'interactivity': False,
#    'content': 'File  Edit  View', 'source': 'box_ocr_content_ocr'},
#   {'type': 'icon', 'bbox': [0.94, 0.93, 0.95, 0.95], 'interactivity': True,
#    'content': 'Close button', 'source': 'box_yolo_content_yolo'}
# ]
```

### Response Example
```json
{
  "parsed_content_list": [
    {
      "type": "text",
      "bbox": [0.15, 0.01, 0.35, 0.03],
      "interactivity": false,
      "content": "File  Edit  View",
      "source": "box_ocr_content_ocr"
    },
    {
      "type": "icon",
      "bbox": [0.94, 0.93, 0.95, 0.95],
      "interactivity": true,
      "content": "Close button",
      "source": "box_yolo_content_yolo"
    }
  ]
}
```
```

--------------------------------

### Load and Move YOLO Model to Device

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Loads a YOLO model from a specified path and moves it to the designated device (e.g., CUDA). Ensure the model path is correct and the device is available.

```python
from util.utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model
import torch
from ultralytics import YOLO
from PIL import Image
device = 'cuda'
model_path='weights/icon_detect/model.pt'

som_model = get_yolo_model(model_path)

som_model.to(device)
print('model to {}'.format(device))
```

--------------------------------

### ScreenSpot Pro Eval Script with OmniParser v2

Source: https://github.com/microsoft/omniparser/blob/master/docs/Evaluation.md

This Python script contains the prompt configuration for evaluating ScreenSpot Pro with OmniParser v2. It can replace the original prompt file in the ScreenSpot Pro repository.

```python
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.

import json
import os
import sys
from typing import Dict, List, Optional

from omniparser import parse_json, parse_xml, parse_yaml


def get_prompt_template(model_name: str) -> str:
    """Get prompt template for a given model name."""
    if model_name == "gpt4o":
        return """
        You are an AI assistant that extracts information from screenshots. Your task is to identify and extract specific information based on the user's request.

        Here is the screenshot information:
        {screenshot_info}

        Please extract the following information:
        {query}

        Respond in JSON format. Only include the extracted information in the JSON response. Do not include any other text.
        """
    elif model_name == "gpt4-vision-preview":
        return """
        You are an AI assistant that extracts information from screenshots. Your task is to identify and extract specific information based on the user's request.

        Here is the screenshot information:
        {screenshot_info}

        Please extract the following information:
        {query}

        Respond in JSON format. Only include the extracted information in the JSON response. Do not include any other text.
        """
    else:
        raise ValueError(f"Unknown model name: {model_name}")


def get_screenshot_info(image_path: str) -> str:
    """Get screenshot information from an image path."""
    # In a real scenario, this would involve image processing and feature extraction.
    # For this example, we'll return a placeholder string.
    return f"Image path: {image_path}"


def main():
    """Main function to run the evaluation."""
    # Example usage:
    model_name = "gpt4o"
    image_path = "path/to/screenshot.png"
    query = "Extract the text from the button labeled 'Submit'."

    prompt_template = get_prompt_template(model_name)
    screenshot_info = get_screenshot_info(image_path)

    # Format the prompt with screenshot info and query
    formatted_prompt = prompt_template.format(
        screenshot_info=screenshot_info,
        query=query
    )

    print("Formatted Prompt:")
    print(formatted_prompt)

    # In a real scenario, you would send this prompt to the language model API
    # and parse the response.
    # For demonstration purposes, we'll simulate a response.
    simulated_response = {
        "button_text": "Submit"
    }

    print("Simulated Model Response:")
    print(json.dumps(simulated_response, indent=2))

    # Example of using omniparser to parse a JSON response (if the model returned JSON)
    try:
        parsed_data = parse_json(json.dumps(simulated_response))
        print("Parsed JSON response using omniparser:")
        print(parsed_data)
    except Exception as e:
        print(f"Error parsing JSON response: {e}")

if __name__ == "__main__":
    main()

```

--------------------------------

### Process Screenshots with get_som_labeled_img Function

Source: https://context7.com/microsoft/omniparser/llms.txt

Loads models, processes an image to detect UI elements using YOLO, performs OCR, generates captions, and returns an annotated image with bounding boxes. Requires Pillow and base64 libraries. Ensure models and image paths are correct.

```python
from util.utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model
from PIL import Image
import base64
import io

# Load models
device = 'cuda'
som_model = get_yolo_model('weights/icon_detect/model.pt')
som_model.to(device)

caption_model_processor = get_caption_model_processor(
    model_name="florence2",
    model_name_or_path="weights/icon_caption_florence",
    device=device
)

# Load image and configure bounding box display
image_path = 'screenshot.png'
image = Image.open(image_path)
box_overlay_ratio = max(image.size) / 3200

draw_bbox_config = {
    'text_scale': 0.8 * box_overlay_ratio,
    'text_thickness': max(int(2 * box_overlay_ratio), 1),
    'text_padding': max(int(3 * box_overlay_ratio), 1),
    'thickness': max(int(3 * box_overlay_ratio), 1),
}

# Run OCR first
ocr_bbox_rslt, _ = check_ocr_box(
    image_path,
    display_img=False,
    output_bb_format='xyxy',
    easyocr_args={'paragraph': False, 'text_threshold': 0.9},
    use_paddleocr=True
)
text, ocr_bbox = ocr_bbox_rslt

# Get labeled image with all detected elements
labeled_img_base64, label_coordinates, parsed_content_list = get_som_labeled_img(
    image_path,
    som_model,
    BOX_TRESHOLD=0.05,
    output_coord_in_ratio=True,
    ocr_bbox=ocr_bbox,
    draw_bbox_config=draw_bbox_config,
    caption_model_processor=caption_model_processor,
    ocr_text=text,
    use_local_semantics=True,
    iou_threshold=0.7,
    scale_img=False,
    batch_size=128
)

# Display the annotated image
annotated_image = Image.open(io.BytesIO(base64.b64decode(labeled_img_base64)))
annotated_image.show()

# Access element details
for idx, element in enumerate(parsed_content_list):
    print(f"ID {idx}: {element['type']} - {element['content']}")
    print(f"  Bbox: {element['bbox']}, Interactable: {element['interactivity']}")
```

--------------------------------

### Probe Windows Host Status

Source: https://github.com/microsoft/omniparser/blob/master/omnitool/readme.md

Use this command to check if the server running within the OmniBox VM is responding. This is useful for diagnosing 'Windows Host is not responding' errors in Gradio.

```bash
docker exec -it omni-windows bash -c "curl http://localhost:5000/probe"
```

--------------------------------

### Load YOLO Icon Detection Model

Source: https://context7.com/microsoft/omniparser/llms.txt

Loads a YOLO-based model for detecting interactive UI elements, buttons, and icons. The model is moved to the GPU for faster inference. The type of the loaded model is an Ultralytics YOLO instance.

```python
from util.utils import get_yolo_model
import torch

som_model = get_yolo_model('weights/icon_detect/model.pt')

som_model.to('cuda')

print(f"Model device: {som_model.device}")
print(f"Model type: {type(som_model)}")
```

--------------------------------

### Perform OCR with PaddleOCR or EasyOCR

Source: https://context7.com/microsoft/omniparser/llms.txt

Performs optical character recognition on a screenshot using either PaddleOCR or EasyOCR. Configurable with bounding box format and OCR-specific arguments. Returns detected text and bounding box coordinates.

```python
from util.utils import check_ocr_box
from PIL import Image

ocr_result, _ = check_ocr_box(
    'screenshot.png',
    display_img=False,
    output_bb_format='xyxy',
    easyocr_args={'text_threshold': 0.9},
    use_paddleocr=True
)
text_list, bbox_list = ocr_result

for text, bbox in zip(text_list, bbox_list):
    print(f"Text: '{text}' at position {bbox}")
```

```python
from util.utils import check_ocr_box
from PIL import Image

ocr_result_easy, _ = check_ocr_box(
    'screenshot.png',
    display_img=False,
    output_bb_format='xywh',
    easyocr_args={'paragraph': False, 'text_threshold': 0.8},
    use_paddleocr=False
)
```

--------------------------------

### Process Image with OCR and Labeling

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Processes an image to perform OCR, detect objects using a YOLO model, and generate labeled images with captioning. This involves multiple steps including image loading, OCR bounding box checking, and semantic labeling.

```python
# reload utils
import importlib
import utils
importlib.reload(utils)
# from utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model

image_path = 'imgs/google_page.png'
image_path = 'imgs/windows_home.png'
# image_path = 'imgs/windows_multitab.png'
# image_path = 'imgs/omni3.jpg'
# image_path = 'imgs/ios.png'
image_path = 'imgs/word.png'
# image_path = 'imgs/excel2.png'

image = Image.open(image_path)
image_rgb = image.convert('RGB')
print('image size:', image.size)

box_overlay_ratio = max(image.size) / 3200
draw_bbox_config = {
    'text_scale': 0.8 * box_overlay_ratio,
    'text_thickness': max(int(2 * box_overlay_ratio), 1),
    'text_padding': max(int(3 * box_overlay_ratio), 1),
    'thickness': max(int(3 * box_overlay_ratio), 1),
}
BOX_TRESHOLD = 0.05

import time
start = time.time()
ocr_bbox_rslt, is_goal_filtered = check_ocr_box(image_path, display_img = False, output_bb_format='xyxy', goal_filtering=None, easyocr_args={'paragraph': False, 'text_threshold':0.9}, use_paddleocr=True)
text, ocr_bbox = ocr_bbox_rslt
cur_time_ocr = time.time() 

dino_labled_img, label_coordinates, parsed_content_list = get_som_labeled_img(image_path, som_model, BOX_TRESHOLD = BOX_TRESHOLD, output_coord_in_ratio=True, ocr_bbox=ocr_bbox,draw_bbox_config=draw_bbox_config, caption_model_processor=caption_model_processor, ocr_text=text,use_local_semantics=True, iou_threshold=0.7, scale_img=False, batch_size=128)
cur_time_caption = time.time()
```

--------------------------------

### Display Raw Parsed Content List

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Shows the raw structure of the parsed content list, typically a list of dictionaries. Useful for understanding the output format before further processing.

```python
parsed_content_list
```

--------------------------------

### Encode Screenshot and Call Parse Endpoint

Source: https://context7.com/microsoft/omniparser/llms.txt

Encodes a screenshot to base64 and sends it to the OmniParser API for processing. Handles the API response and prints latency and detected element count. Also includes a health check for the API.

```python
with open('screenshot.png', 'rb') as f:
    image_base64 = base64.b64encode(f.read()).decode('ascii')

response = requests.post(
    'http://localhost:8000/parse/',
    json={'base64_image': image_base64}
)
result = response.json()

print(f"Processing latency: {result['latency']:.2f}s")
print(f"Detected {len(result['parsed_content_list'])} elements")

probe = requests.get('http://localhost:8000/probe/')
print(probe.json())
```

--------------------------------

### get_som_labeled_img Function

Source: https://context7.com/microsoft/omniparser/llms.txt

Details on using the get_som_labeled_img function for processing screenshots with Set-of-Mark (SOM) labeling, including model loading and image annotation.

```APIDOC
## get_som_labeled_img Function

The primary function for processing screenshots with Set-of-Mark (SOM) labeling. It detects UI elements using YOLO, performs OCR, generates icon captions, and returns an annotated image with numbered bounding boxes.

### Request Example
```python
from util.utils import get_som_labeled_img, check_ocr_box, get_caption_model_processor, get_yolo_model
from PIL import Image
import base64
import io

# Load models
device = 'cuda'
som_model = get_yolo_model('weights/icon_detect/model.pt')
som_model.to(device)

caption_model_processor = get_caption_model_processor(
    model_name="florence2",
    model_name_or_path="weights/icon_caption_florence",
    device=device
)

# Load image and configure bounding box display
image_path = 'screenshot.png'
image = Image.open(image_path)
box_overlay_ratio = max(image.size) / 3200

draw_bbox_config = {
    'text_scale': 0.8 * box_overlay_ratio,
    'text_thickness': max(int(2 * box_overlay_ratio), 1),
    'text_padding': max(int(3 * box_overlay_ratio), 1),
    'thickness': max(int(3 * box_overlay_ratio), 1),
}

# Run OCR first
ocr_bbox_rslt, _ = check_ocr_box(
    image_path,
    display_img=False,
    output_bb_format='xyxy',
    easyocr_args={'paragraph': False, 'text_threshold': 0.9},
    use_paddleocr=True
)
text, ocr_bbox = ocr_bbox_rslt

# Get labeled image with all detected elements
labeled_img_base64, label_coordinates, parsed_content_list = get_som_labeled_img(
    image_path,
    som_model,
    BOX_TRESHOLD=0.05,
    output_coord_in_ratio=True,
    ocr_bbox=ocr_bbox,
    draw_bbox_config=draw_bbox_config,
    caption_model_processor=caption_model_processor,
    ocr_text=text,
    use_local_semantics=True,
    iou_threshold=0.7,
    scale_img=False,
    batch_size=128
)

# Display the annotated image
annotated_image = Image.open(io.BytesIO(base64.b64decode(labeled_img_base64)))
annotated_image.show()

# Access element details
for idx, element in enumerate(parsed_content_list):
    print(f"ID {idx}: {element['type']} - {element['content']}")
    print(f"  Bbox: {element['bbox']}, Interactable: {element['interactivity']}")
```

### Response Example
```json
{
  "labeled_img_base64": "<base64_encoded_image_string>",
  "label_coordinates": [
    [0.15, 0.01, 0.35, 0.03],
    [0.94, 0.93, 0.95, 0.95]
  ],
  "parsed_content_list": [
    {
      "type": "text",
      "bbox": [0.15, 0.01, 0.35, 0.03],
      "interactivity": false,
      "content": "File  Edit  View",
      "source": "box_ocr_content_ocr"
    },
    {
      "type": "icon",
      "bbox": [0.94, 0.93, 0.95, 0.95],
      "interactivity": true,
      "content": "Close button",
      "source": "box_yolo_content_yolo"
    }
  ]
}
```
```

--------------------------------

### OmniParser Project Citation

Source: https://github.com/microsoft/omniparser/blob/master/README.md

Use this BibTeX entry to cite the OmniParser technical report in your work. Ensure you have reviewed the paper for relevance.

```bibtex
@misc{lu2024omniparserpurevisionbased,
      title={OmniParser for Pure Vision Based GUI Agent},
      author={Yadong Lu and Jianwei Yang and Yelong Shen and Ahmed Awadallah},
      year={2024},
      eprint={2408.00203},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2408.00203},
}
```

--------------------------------

### Display Labeled Image from Base64

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Decodes a base64 encoded image string and displays it using Matplotlib. This is useful for visualizing the output of image processing functions that return base64 encoded data.

```python
# plot dino_labled_img it is in base64
import base64
import matplotlib.pyplot as plt
import io
plt.figure(figsize=(15,15))

image = Image.open(io.BytesIO(base64.b64decode(dino_labled_img)))
plt.axis('off')

plt.imshow(image)
```

--------------------------------

### Create Pandas DataFrame from Parsed Content

Source: https://github.com/microsoft/omniparser/blob/master/demo.ipynb

Converts a list of parsed content into a Pandas DataFrame and adds a unique 'ID' column. Useful for structured data manipulation and analysis.

```python
import pandas as pd
df = pd.DataFrame(parsed_content_list)
df['ID'] = range(len(df))

df
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.