### Initialize Azure authentication

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/README.md

Commands to start a screen session and authenticate with Azure for the evaluation environment.

```bash
screen -S token
```

```bash
az login --scope https://cognitiveservices.azure.com/.default --use-device-code
```

```bash
cd cua_skill/evaluation/WindowsAgentArena
./refresh_token.sh
```

--------------------------------

### Utilize Pre-built Notepad Skills in Python

Source: https://context7.com/microsoft/cua_skill/llms.txt

Demonstrates how to use pre-built skills for the Notepad application, including launching, typing text, saving files, find/replace, and navigating to lines. Ensure necessary imports are present. The example executes skills by calling their step() method and then executing the returned GUI code.

```python
from agent.action.notepad_action import (
    NotepadLaunch,
    NotepadTypeText,
    NotepadSaveAsFile,
    NotepadFindReplaceAll,
    NotepadGoToLine
)

# Launch Notepad
launch = NotepadLaunch()

# Type text
type_text = NotepadTypeText(text="Hello, this is a test document!")

# Save file to specific location
save_as = NotepadSaveAsFile(
    path="C:\\Users\\Docker\\Documents",
    file_name="test_document.txt"
)

# Find and replace text
find_replace = NotepadFindReplaceAll(
    find_what="test",
    replace_with="sample"
)

# Navigate to specific line
goto_line = NotepadGoToLine(line_num=10)

# Execute each skill by stepping through
for skill in [launch, type_text, save_as]:
    while True:
        action = skill.step()
        if action is None:
            break
        executable_code = action.get_gui_code()
        exec(executable_code)
```

--------------------------------

### Replay Agent Configuration (config.json)

Source: https://context7.com/microsoft/cua_skill/llms.txt

Example configuration file for the Replay Agent, specifying environment settings, grounding models, and logging parameters.

```json
{
    "name": "CUA_Skill_Agent",
    "version": "1.0.0",
    "platform": "windows",
    "max_attempts": 1,
    "max_grounding_attempts": 3,
    "max_steps": 50,
    "max_wall_time": 300,
    "step_interval_time": 2,
    "env": {
        "name": "windows_env",
        "platform": "windows",
        "url": "localhost",
        "screen_height": null,
        "screen_width": null,
        "observation_type": "screenshot_a11y_tree",
        "observe_screenshot_in_bytes": true
    },
    "mixture_grounding": {
        "expertises": [
            {
                "model": "uitars_v1_grounding",
                "weight": 1.0,
                "azure_endpoint": true,
                "endpoint_url": "YOUR_ENDPOINT_URL",
                "bearer_key_env_var": "UITARS_V1_BEARER_KEY"
            }
        ]
    },
    "logger": {
        "enable": true,
        "log_level": "debug",
        "log_file": "log.jsonl",
        "log_dir": "./logs"
    }
}
```

--------------------------------

### Example Task Output Format

Source: https://github.com/microsoft/cua_skill/blob/main/user_task_generation/README.md

This JSON structure represents a generated task, keyed by a unique ID. It includes the task instruction, domain, and a list of steps, each with a primitive operation, instruction, and arguments.

```json
{
  "bingsearch_basic_000001_4f9a2c91a1b2c3d4": {
    "id": "bingsearch_basic_000001_4f9a2c91a1b2c3d4",
    "instruction": "Search for AI trends and open the first result.",
    "domain": "bingsearch",
    "steps": [
      {
        "primitive_operation": "BingSearchLaunch",
        "instruction": "Open Bing Search.",
        "arguments": {}
      },
      {
        "primitive_operation": "BingSearchQuery",
        "instruction": "Search for AI trends.",
        "arguments": {"query": "AI trends"}
      },
      {
        "primitive_operation": "BingResultClick",
        "instruction": "Open the first result.",
        "arguments": {"rank": 1}
      }
    ]
  }
}
```

--------------------------------

### HotKeyAction Example

Source: https://context7.com/microsoft/cua_skill/llms.txt

Python code for executing keyboard shortcuts, including simple combinations and complex multi-key sequences.

```python
from agent.action import HotKeyAction

# Save file shortcut
save_action = HotKeyAction(
    thought="Save the current document",
    keys=["ctrl", "s"]
)

# Copy shortcut
copy_action = HotKeyAction(
    thought="Copy selected text",
    keys=["ctrl", "c"]
)

# Complex shortcut
screenshot_action = HotKeyAction(
    thought="Take a screenshot",
    keys=["win", "shift", "s"]
)
```

```python
# Generate executable code
code = save_action.get_gui_code()
# Output:
# import pyautogui
# pyautogui.hotkey('ctrl', 's')
```

--------------------------------

### ScrollAction Example

Source: https://context7.com/microsoft/cua_skill/llms.txt

Python code for simulating scroll actions, specifying direction and magnitude.

```python
from agent.action import ScrollAction

# Scroll down
scroll_down = ScrollAction(
    thought="Scroll down to see more content",
    dy=5,
    direction="down"
)

# Scroll up
scroll_up = ScrollAction(
    thought="Scroll back to the top",
    dy=10,
    direction="up"
)
```

```python
# Generate executable code
code = scroll_down.get_gui_code()
# Output:
# import pyautogui
# pyautogui.scroll(-5)
```

--------------------------------

### Execute CUA-RAG tests

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/README.md

Command syntax and examples for running tests using the run_cua_rag.sh script.

```bash
sudo bash ./run_cua_rag.sh <test_json_filename> [options]
```

```bash
sudo "./run-local.sh" --prepare-image true
```

```bash
# Run with clean environment for each test case (recommended)
sudo bash ./run_cua_rag.sh "test_one.json" --use_gold_image --clean_mode
```

--------------------------------

### Install CUA Skill Dependencies

Source: https://github.com/microsoft/cua_skill/blob/main/README.md

Install the necessary Python packages for CUA Skill. Ensure you have Python 3.10+ and are on a Windows OS. Azure OpenAI access is required for LLM features.

```bash
git clone <repo-url>
cd cua_skill
pip install -r agent/requirements.txt
```

```bash
pip install -r agent/requirements_waa.txt
```

--------------------------------

### SingleClickAction Example

Source: https://context7.com/microsoft/cua_skill/llms.txt

Python code demonstrating how to perform single mouse clicks with optional keyboard modifiers and vision-based coordinate grounding.

```python
from agent.action import SingleClickAction

# Click at specific coordinates
click_action = SingleClickAction(
    thought="Click the Submit button",
    coordinate=(500, 300),
    button="left",
    modifiers=None
)

# Click with Ctrl modifier (for multi-select)
ctrl_click = SingleClickAction(
    thought="Ctrl+click to add to selection",
    coordinate=(600, 400),
    button="left",
    modifiers=["ctrl"]
)

# Click requiring vision grounding (coordinates will be refined)
grounded_click = SingleClickAction(
    thought="Click the Save button in the toolbar",
    coordinate=None,  # Will use grounding model to find coordinates
    button="left"
)
```

```python
# Generate executable GUI code
code = click_action.get_gui_code()
# Output:
# import pyautogui
# pyautogui.click(500, 300, button='left')
```

--------------------------------

### TypeAction Example

Source: https://context7.com/microsoft/cua_skill/llms.txt

Python code for typing text into an application, supporting different input modes and multi-line entry.

```python
from agent.action import TypeAction

# Simple text typing
type_action = TypeAction(
    thought="Type the greeting message",
    text="Hello, World!",
    input_mode="keyboard",
    end_with_enter=False
)

# Multi-line text with copy-paste (faster for long text)
multiline_action = TypeAction(
    thought="Enter the document content",
    text="Line 1\nLine 2\nLine 3",
    input_mode="copy_paste",
    line_by_line=True,
    end_with_enter=True
)
```

```python
# Generate executable code
code = type_action.get_gui_code()
# Output:
# import pyautogui
# pyautogui.write('Hello, World!', interval=0.05)
```

--------------------------------

### DragAction Example

Source: https://context7.com/microsoft/cua_skill/llms.txt

Python code for performing drag-and-drop operations between specified coordinates with configurable duration.

```python
from agent.action import DragAction

# Drag file to folder
drag_action = DragAction(
    thought="Drag file to the destination folder",
    start_coordinate=(100, 200),
    end_coordinate=(500, 400),
    duration=2.0,
    thought_for_start_coordinate="Select the file icon",
    thought_for_end_coordinate="Drop onto the folder"
)
```

```python
# Generate executable code
code = drag_action.get_gui_code()
# Output:
# import pyautogui
# pyautogui.moveTo(100, 200)
# pyautogui.mouseDown()
# pyautogui.dragTo(500, 400, duration=2.0, button='left')
# pyautogui.mouseUp()
```

--------------------------------

### Define Custom Compose Action in Python

Source: https://context7.com/microsoft/cua_skill/llms.txt

Create a custom action by inheriting from BaseComposeAction and defining arguments and execution paths. Use the @register decorator to make the action discoverable. This example shows how to create a 'CustomAppSaveFile' action with two distinct paths: one using keyboard shortcuts and another using menu clicks.

```python
from agent.action.compose_action import BaseComposeAction
from agent.action.base_action import (
    register, SingleClickAction, WaitAction,
    TypeAction, HotKeyAction
)
from agent.action.argument import Argument

@register("CustomAppSaveFile")
class CustomAppSaveFile(BaseComposeAction):
    type: str = "custom_app_save_file"

    # Define action arguments
    file_name: Argument = Argument(
        value="document.txt",
        description="Name of the file to save"
    )

    # Descriptions for skill retrieval
    descriptions = [
        "Save the file as ${{file_name}}",
        "Store the document as ${{file_name}}"
    ]

    def __init__(self, file_name: str = "document.txt", **kwargs):
        super().__init__(file_name=file_name, **kwargs)

        # Path 1: Keyboard shortcut
        self.add_path(
            "save_hotkey",
            path=[
                HotKeyAction(keys=["ctrl", "s"], thought="Open save dialog"),
                WaitAction(duration=1.0),
                TypeAction(text=file_name, thought=f"Enter filename: {file_name}"),
                WaitAction(duration=0.5),
                HotKeyAction(keys=["enter"], thought="Confirm save")
            ]
        )

        # Path 2: Menu click
        self.add_path(
            "save_menu",
            path=[
                SingleClickAction(thought="Click File menu"),
                WaitAction(duration=0.5),
                SingleClickAction(thought="Click Save option"),
                WaitAction(duration=1.0),
                TypeAction(text=file_name, thought=f"Enter filename: {file_name}"),
                WaitAction(duration=0.5),
                SingleClickAction(thought="Click Save button")
            ]
        )

# Use the skill
save_skill = CustomAppSaveFile(file_name="my_report.txt")

# Step through the skill (randomly selects a path)
while True:
    action = save_skill.step()
    if action is None:
        break
    print(f"Execute: {action.get_gui_code()}")
```

--------------------------------

### Initialize and Use Local Qwen Planner

Source: https://context7.com/microsoft/cua_skill/llms.txt

Initializes a local Qwen planner for offline action selection. Requires specifying the model path. This planner can also process text and image inputs.

```python
from agent.llms import Qwen
from types import SimpleNamespace

config = SimpleNamespace(
    planner=SimpleNamespace(
        model_class="qwen",
        expertises=SimpleNamespace(
            qwen=SimpleNamespace(
                model_path="Qwen/Qwen3-VL-32B-Instruct"
            )
        )
    )
)

qwen = Qwen(config)

message = qwen.create_text_image_message(
    text="Describe what you see on the screen",
    image=screenshot_bytes
)

response = qwen.get_completion([message])
print(response)
```

--------------------------------

### Initialize and Use GPT Planner

Source: https://context7.com/microsoft/cua_skill/llms.txt

Sets up an Azure OpenAI GPT planner for action selection. Requires configuration of the deployment, endpoint, and API version. The planner can process text and image inputs.

```python
from agent.llms import GPT
from types import SimpleNamespace

config = SimpleNamespace(
    planner=SimpleNamespace(
        model_class="gpt",
        expertises=SimpleNamespace(
            gpt=SimpleNamespace(
                deployment="gpt-4o",
                endpoint="YOUR_AZURE_ENDPOINT",
                api_version="2025-01-01-preview",
                token_scope="https://cognitiveservices.azure.com/.default"
            )
        )
    )
)

gpt = GPT(config)

message = gpt.create_text_image_message(
    text="What action should I take next to complete this task?",
    image=screenshot_bytes  # PNG bytes or base64
)

response = gpt.get_completion([message])
print(response)
```

--------------------------------

### Define Test JSON

Source: https://context7.com/microsoft/cua_skill/llms.txt

Example JSON structure for defining test cases in the evaluation environment.

```json
{
  "test_001": {
    "id": "test_001",
    "instruction": "Open Notepad and type 'Hello World'",
    "domain": "notepad",
    "config": [
      {
        "parameters": {
          "files": [
            {"path": "C:\\Users\\Docker\\Documents\\test.txt"}
          ]
        }
      }
    ]
  }
}
```

--------------------------------

### Initialize Project Environment and Registries

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/log_analysis.ipynb

Sets up the project environment by defining the current directory path and importing necessary modules and registries from the agent's action base. This code is essential for initializing the project's operational context.

```python
from pathlib import Path
import json
from collections import Counter
try:
    CURRENT_DIR_PATH = Path(__file__).resolve().parent.parent
except NameError:
    CURRENT_DIR_PATH = Path.cwd().parent.parent
import sys
sys.path.append(str(CURRENT_DIR_PATH))
from agent.action.base_action import _OP_REGISTRY as GLOBAL_ACTION_REGISTRY
from agent.action.base_action import EXECUTABLE_ACTIONS
import random
import pandas as pd
random.seed(42)

```

--------------------------------

### Initialize and Interact with Desktop Environment

Source: https://context7.com/microsoft/cua_skill/llms.txt

Initializes a Windows desktop environment, captures observations, executes actions using pyautogui, launches tools, checks tool status, and calibrates the environment.

```python
env = DesktopEnv(
    name="windows_env",
    platform="windows",
    screen_height=1080,
    screen_width=1920,
    observation_type="screenshot_a11y_tree",
    observe_screenshot_in_bytes=True
)

observation = env.get_observation()
screenshot = observation["screenshot"]  # PNG bytes
resolution = observation["screenshot_resolution"]  # (width, height)

action_code = """
import pyautogui
pyautogui.click(500, 300)
"""
env.step(action_code)

env.launch_tool("Notepad")

is_open = env.tool_is_open("Notepad")

env.calibrate()
```

--------------------------------

### Initialize Replay Agent

Source: https://github.com/microsoft/cua_skill/blob/main/README.md

Instantiate the Replay Agent with a configuration file to execute pre-defined task graphs.

```python
from agent import CUAKnowledgeGraphAgent

agent = CUAKnowledgeGraphAgent(config="agent/config.json")
agent.proceed(instruction="Open Notepad and type Hello", example=task_json)
```

--------------------------------

### Load and Execute Task Graphs with ReplayTask

Source: https://context7.com/microsoft/cua_skill/llms.txt

Demonstrates the initialization of the ReplayTask class, which is used for parsing and executing pre-defined task graphs with visualization support.

```python
from agent.replay_task import ReplayTask
```

--------------------------------

### Initialize Environment and Imports

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/result_analysis.ipynb

Imports necessary libraries and sets the current directory path for file operations.

```python
import os
import json
import pandas as pd
from pathlib import Path
from tqdm import tqdm
from azure.identity import DefaultAzureCredential
from azure.storage.blob import ContainerClient
try:
    CURRENT_DIR_PATH = Path(__file__).resolve().parent
except NameError:
    CURRENT_DIR_PATH = Path.cwd()
```

--------------------------------

### Initialize and Use RAGPlanner

Source: https://context7.com/microsoft/cua_skill/llms.txt

Initializes the RAGPlanner, which combines skill retrieval with LLM-based action selection. It requires loading a configuration file and can then set instructions, retrieve next steps, and configure executable actions.

```python
from agent.planner import RAGPlanner
from agent.utils import Misc

config = Misc.file_to_namespace("agent/config_rag.json")

planner = RAGPlanner(config)

planner.set_instruction("Calculate 123 + 456 using the calculator")

action_class, step_description, is_base_action = planner.retrieve_next_step(
    screenshot=screenshot_bytes
)

print(f"Selected action: {action_class.type}")
print(f"Description: {step_description}")
print(f"Is base action: {is_base_action}")

configured_action = planner.config_next_step(
    action_class=action_class,
    screenshot=screenshot_bytes,
    step_description=step_description
)

executable_code = configured_action.get_gui_code()
print(executable_code)
```

--------------------------------

### Initialize DesktopEnv Wrapper

Source: https://context7.com/microsoft/cua_skill/llms.txt

Instantiate the DesktopEnv class, which serves as a wrapper for capturing screenshots, accessing the accessibility tree, and executing GUI actions within a desktop environment. No specific parameters are shown in this initialization snippet.

```python
from agent.desktop_env import DesktopEnv
```

--------------------------------

### Define and Execute a Task

Source: https://context7.com/microsoft/cua_skill/llms.txt

Demonstrates defining a task structure in JSON, loading it into a ReplayTask, and executing steps sequentially.

```python
task_data = {
    "id": "calculator_task_001",
    "domain": "calculator",
    "instruction": "Calculate 123 + 456",
    "steps": [
        {
            "primitive_operation": "calculator_launch"
        },
        {
            "primitive_operation": "type",
            "arguments": {"text": "123"}
        },
        {
            "primitive_operation": "click",
            "arguments": {"thought": "Click the plus button"}
        },
        {
            "primitive_operation": "type",
            "arguments": {"text": "456"}
        },
        {
            "primitive_operation": "click",
            "arguments": {"thought": "Click the equals button"}
        }
    ]
}

# Create ReplayTask from JSON
task = ReplayTask.from_json(task_data)

print(f"Task ID: {task.id}")
print(f"Domain: {task.domain}")
print(f"Instruction: {task.instruction}")

# Execute steps one by one
while True:
    step = task.next_step()
    if step is None:
        break
    print(f"Executing: {step}")
    # Execute the step action
    code = step.get_gui_code()
    exec(code)

# Visualize the task graph
task.visualize_graph(
    filename="calculator_task",
    directory="./graphs",
    format="pdf",
    view=False
)
```

--------------------------------

### Load LLM model with configuration

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/optimize_feasibility_prompt.ipynb

Initializes an LLM model using the provided configuration, which specifies model class, expertise details, and deployment parameters. The configuration is converted to a SimpleNamespace for easier access.

```python
config = {"planner": {
    "model_class": "gpt",
    "expertises": {
        "gpt": {
            "deployment": "gpt-5-2-low",
            "azure_endpoint": True,
            "api_version": "2025-01-01-preview",
            "endpoint": "",
            "token_scope": "",
        },
        "qwen": {
            "model_path": "Qwen/Qwen3-VL-32B-Instruct"
        }
    }
}}
image_dir = CURRENT_DIR_PATH / "../../WindowsAgentArena/src/win-arena-container/client/all_results"
config = dict_to_namespace(config)
llm = model_loader(config)
```

--------------------------------

### Initialize RAG Agent

Source: https://github.com/microsoft/cua_skill/blob/main/README.md

Instantiate the RAG Agent with a configuration file for dynamic task planning and execution using retrieval-augmented generation.

```python
from agent import CUARAGAgent

agent = CUARAGAgent(config="agent/config_rag.json")
agent.proceed(instruction="Search for 'weather today' on Bing")
```

--------------------------------

### Generate Sample Action Configuration Files

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/log_analysis.ipynb

Shuffles selected action types and generates JSON configuration files containing subsets of these actions along with base action types. This is used to create different configurations for testing or deployment scenarios.

```python
random.shuffle(selected_action_types)
outpath = CURRENT_DIR_PATH / "agent" / "sample_actions"
outpath.mkdir(exist_ok=True, parents=True)
for i, s in enumerate([36, 72]):
    out_types = selected_action_types[0:s] + base_action_types

    print(f"Total action types: {len(out_types)}")
    with open(outpath / f"{(i + 1) * 33}percent.json", "w") as f:
        json.dump(out_types, f, indent=4)

```

--------------------------------

### Python Stub for qwen_vl_utils

Source: https://github.com/microsoft/cua_skill/blob/main/user_task_generation/README.md

Create this stub file if the 'qwen_vl_utils not found' error occurs. It provides a basic implementation for the process_vision_info function.

```python
def process_vision_info(messages): return [], []
```

--------------------------------

### Initialize and Use MixtureGrounding

Source: https://context7.com/microsoft/cua_skill/llms.txt

Configures and initializes the MixtureGrounding model for vision-based coordinate refinement using UI-TARS. Requires setting up expertise models and providing endpoint details.

```python
from agent.mixture_grounding import MixtureGrounding
from types import SimpleNamespace

config = SimpleNamespace(
    expertises=[
        SimpleNamespace(
            model="uitars_v1_grounding",
            weight=1.0,
            azure_endpoint=True,
            endpoint_url="YOUR_UITARS_ENDPOINT",
            bearer_key_env_var="UITARS_V1_BEARER_KEY"
        )
    ]
)

grounding = MixtureGrounding(config=config, logger=None)

observation = {
    "screenshot": screenshot_bytes,  # PNG bytes
    "screenshot_resolution": (1920, 1080)
}

coordinates = grounding.predict(
    action_description="Click the Save button in the toolbar",
    observation=observation
)

print(f"Refined coordinates: {coordinates}")  # [x, y]
```

--------------------------------

### Configure Launch Step Drop-Off

Source: https://github.com/microsoft/cua_skill/blob/main/user_task_generation/README.md

Sets the probability for skipping instructions related to application launch steps.

```bash
--launch-app-instruction-dropoff-prob '[0.5]'
```

--------------------------------

### Initialize ActionRetriever for Hybrid Search

Source: https://context7.com/microsoft/cua_skill/llms.txt

Set up an ActionRetriever that combines BM25 keyword search with semantic embeddings for effective skill retrieval. Specify the directory for the index, the embedding model name, and the weight for semantic search. The 'overwrite' parameter controls whether to rebuild the index from scratch.

```python
from agent.retrieval import ActionRetriever

# Initialize the retriever
retriever = ActionRetriever(
    index_dir="./asset/rag_index/actions",
    model_name="Qwen/Qwen3-Embedding-0.6B",
    semantic_weight=0.7,  # 70% semantic, 30% BM25
    overwrite=False
)

# Retrieve actions matching a query
query = "Open a new file in Notepad"
action_classes = retriever.retrieve_actions(query, top_k=5)

# Get detailed results as DataFrame
results_df = retriever.retrieve_actions_df(query, top_k=5)
print(results_df[['action_name', 'description', 'hybrid_score']])
```

--------------------------------

### Create storage_gold backup

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/README.md

Creates a clean backup copy of the storage directory for the Windows Agent Arena container.

```bash
cp -rf cua_skill/WindowsAgentArena/src/win-arena-container/vm/storage \
         cua_skill/WindowsAgentArena/src/win-arena-container/vm/storage_gold
```

--------------------------------

### Initialize and Execute Task with CUAKnowledgeGraphAgent (Replay Agent)

Source: https://context7.com/microsoft/cua_skill/llms.txt

Initializes the Replay Agent and executes a predefined task defined as a JSON object. Use this for tasks with known, sequential steps.

```python
from agent import CUAKnownowledgeGraphAgent

# Initialize the replay agent with default configuration
agent = CUAKnowledgeGraphAgent(config="agent/config.json")

# Define a task as JSON with steps to execute
task_json = {
    "id": "notepad_task_001",
    "domain": "notepad",
    "instruction": "Open Notepad and type Hello World",
    "steps": [
        {
            "primitive_operation": "notepad_launch"
        },
        {
            "primitive_operation": "type",
            "arguments": {
                "text": "Hello World"
            }
        }
    ]
}

# Execute the task
status = agent.proceed(
    instruction="Open Notepad and type Hello World",
    example=task_json
)

# Status can be: SUCCESS, FAILURE, CANCELED, TIMEOUT, CALL_USER
print(f"Task completed with status: {status}")
```

--------------------------------

### Run Evaluation in Windows Agent Arena

Source: https://context7.com/microsoft/cua_skill/llms.txt

Command-line instructions to execute evaluations within the Windows Agent Arena environment.

```bash
# Navigate to evaluation directory
cd evaluation/WindowsAgentArena

# Run evaluation with RAG agent
sudo bash ./run_cua_rag.sh test_all.json --clean_mode --use_gold_image

# Options:
#   --use_gold_image  Use clean backup storage image
#   --clean_mode      Reset environment between test cases
#   --reset_image     Regenerate storage from setup ISO
```

--------------------------------

### Initialize and Execute Task with CUARAGAgent (RAG Agent)

Source: https://context7.com/microsoft/cua_skill/llms.txt

Initializes the RAG Agent for dynamic task planning and execution using retrieval-augmented generation. This agent retrieves and selects skills at runtime.

```python
from agent import CUARAGAgent

# Initialize the RAG agent with configuration
agent = CUARAGAgent(config="agent/config_rag.json")

# Execute a task with dynamic skill retrieval
status = agent.proceed(
    instruction="Search for 'weather today' on Bing",
    example={},
    explicit_log_dir="./logs/bing_search_task"
)

# The agent will:
# 1. Check task feasibility from current screen
# 2. Generate search queries from instruction
# 3. Retrieve matching skills from the indexed library
# 4. Select and configure the best action
# 5. Execute with vision-based grounding
# 6. Repeat until task completion or timeout

print(f"Task result: {status}")
```

--------------------------------

### Generate User Tasks

Source: https://github.com/microsoft/cua_skill/blob/main/README.md

Run the user task generation script with specified primitive operations, compositions, and output directory to create diverse user tasks.

```bash
python user_task_generation/user_task_generator.py \
  --primitive-operation ./asset/primitive_operation/bingsearch_primitive_operation.json \
  --composition ./asset/primitive_operation_composition/bingsearch_primitive_operation_composition.json \
  --app-name bingsearch \
  --out-dir ./asset/user_task \
  --num-tasks 10000
```

--------------------------------

### Configure environment variables

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/README.md

Required environment variables for the agent, to be placed in a .env file within the ./agent directory.

```text
UITARS_V1_BEARER_KEY="your_uitars_key"
AZURE_AD_TOKEN=""
```

--------------------------------

### Default Output File Path

Source: https://github.com/microsoft/cua_skill/blob/main/user_task_generation/README.md

The generated task file is saved to this default location unless overridden.

```bash
./asset/user_task/{app_name}_tasks.json
```

--------------------------------

### Generate User Tasks via CLI

Source: https://github.com/microsoft/cua_skill/blob/main/user_task_generation/README.md

Executes the task generation script with specified primitive operations, compositions, and configuration parameters.

```bash
python user_task_generation/user_task_generator.py   --primitive-operation ./asset/primitive_operation/bingsearch_primitive_operation.json   --composition ./asset/primitive_operation_composition/bingsearch_primitive_operation_composition.json   --app-name bingsearch   --out-dir ./asset/user_task   --num-tasks 10000   --instruction-dropoff-prob-range '[0.1,0.2]'   --llm-rephrase-prob-range '[0.9]'   --launch-app-instruction-dropoff-prob '[0.6]'
```

--------------------------------

### Evaluate PNG Image Feasibility

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/optimize_feasibility_prompt.ipynb

Iterates through PNG files in a directory, processes each with an LLM, and returns a DataFrame of results.

```python
def run_evaluation(prompt):
    image_paths = []
    for root, dirs, files in os.walk(image_dir):
        for file in files:
            if file.lower().endswith('.png'):
                image_paths.append(Path(root) / file)

    results = []
    # Process images with tqdm
    for image_path in tqdm(image_paths, desc="Processing PNG images"):
        # Load and process the image
        screenshot = Image.open(image_path)
        images = get_initial_state_observation_images(screenshot)
        task_description = get_instruction(str(image_path.parent.name))
        input_prompt = prompt.replace("{TASK_DESCRIPTION}", task_description)
        messages = llm.create_text_image_message(text = input_prompt, image = images)
        response = llm.get_completion_with_kwargs([messages], reasoning_effort="low", max_completion_tokens=2000)
        label = 0 if "INF-" in str(image_path.parent.name) else 1
        pred = 0 if "false" in response.lower() else 1
        # Store results
        results.append({
            'image_file_name': str(image_path.parent.name),
            'response': response,
            'label': label,
            'prediction': pred
        })
        print(results[-1])
    df_results = pd.DataFrame(results)
    mean = df_results["prediction"].mean()
    print(f"Average Feasibility Accuracy: {mean}")
    return df_results


df_results = run_evaluation(prompt)
df_results
```

--------------------------------

### Initialize and Configure Cua Skill

Source: https://github.com/microsoft/cua_skill/blob/main/docs/static/css/bulma.css.map.txt

Initializes and configures the Cua Skill, setting up essential components for its operation.

```typescript
yD3B,eAAe;EpCsFb,WoC9IoB;EA0DtB,SAzDoB;EA0DpB,WA5D2B;ArCmlK7B;;AqCrhKA;EACE,aAAa;EACb,sBAAsB;EACtB,8BAAgD;EAChD,gBAAgB;EAChB,uBAAuB;ArCwhKzB;;AqCthKA;;EAEE,mBAAmB;EACnB,4BnCpE4B;EmCqE5B,aAAa;EACb,cAAc;EACd,2BAA2B;EAC3B,aApE4B;EAqE5B,kBAAkB;ArCyhKpB;;AqCvhKA;EACE,gCnC/E4B;EmCgF5B,2BnCpBgB;EmCqBhB,4BnCrBgB;AF+iKlB;;AqCxhKA;EACE,cnCxF4B;EmCyF5B,YAAY;EACZ,cAAc;EACd,iBnC9Da;EmC+Db,cA7E8B;ArCwmKhC;;AqCzhKA;EACE,8BnC/BgB;EmCgChB,+BnChCgB;EmCiChB,6BnC7F4B;AFynK9B;;AqC/hKA;EpC4CI,mBoCtCuC;ArC6hK3C;;AqC3hKA;EpC9CE,iCAAiC;EoCgDjC,uBnC/F6B;EmCgG7B,YAAY;EACZ,cAAc;EACd,cAAc;EACd,aAtF4B;ArConK9B
```

--------------------------------

### Retrieve task instruction from JSON file

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/optimize_feasibility_prompt.ipynb

Searches a specified directory for a JSON file matching the provided task ID and returns the 'instruction' field. Raises FileNotFoundError if the task ID is not found.

```python
def get_instruction(task_id):
    example_dir = CURRENT_DIR_PATH / "../../WindowsAgentArena/src/win-arena-container/client/evaluation_examples_windows"
    # Search for task_id.json file
    task_file = f"{task_id}.json"
    
    for root, dirs, files in os.walk(example_dir):
        if task_file in files:
            file_path = Path(root) / task_file
            with open(file_path, 'r') as f:
                json_data = json.load(f)
                return json_data.get("instruction", "")
    
    raise FileNotFoundError(f"Task ID {task_id} not found in {example_dir}")
```

--------------------------------

### Prompt for task feasibility evaluation

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/optimize_feasibility_prompt.ipynb

A detailed prompt template for an AI agent to evaluate task feasibility based on objective, current screen context (including image analysis), and specific criteria for infeasibility. It specifies a 'True'/'False' output followed by reasoning.

```text
prompt = """You are a computer-use agent responsible for validating task feasibility.

        ## Task Information
        **Objective:** {TASK_DESCRIPTION}
        **Current Context:** A screenshot of the current screen is provided. A total of 5 images are provided. The first image shows the full screen view, while the next four images are zoomed-in sections of the screen.

        ## Instructions
        Evaluate whether the task can be completed given the current screen state.

        Consider the task **infeasible** if:
        1. **Mismatch with Screen:** The current screen does not match the task instruction. For example, the task mentions a file or an application is currently opened or running but the screenshot shows otherwise.
        2. **Technical Impossibility:** The required functionality or feature is not supported by the current system or application.
        Consider the task is still **feasible** if:
        The current screen does not contradict the task instruction, more steps may required to check the feasibility. For example, search for the file to see if it exists.
        
        ## Output Format
        - If the task is feasible, respond "True"
        - If the task is infeasible, respond "False"
        Then output your reasoning steps make it within one sentence.
        """
        
```

--------------------------------

### Build Custom Action Index with HybridIndexer

Source: https://context7.com/microsoft/cua_skill/llms.txt

Create a custom index for all registered actions using HybridIndexer. This process automatically extracts descriptions from action files. Ensure the index directory is specified and set 'overwrite' to True to rebuild the index. The code also shows how to retrieve index statistics.

```python
from agent.retrieval import ActionRetriever, HybridIndexer

# Build index for all registered actions
retriever = ActionRetriever(
    index_dir="./custom_index",
    model_name="Qwen/Qwen3-Embedding-0.6B",
    semantic_weight=0.7,
    overwrite=True  # Rebuild from scratch
)

# The index automatically extracts descriptions from all
# registered actions in agent/action/*_action.py modules

# Check index statistics
stats = retriever.indexer.get_index_stats()
print(f"Indexed {stats['num_documents']} action descriptions")
```

--------------------------------

### Generate Synthetic Tasks

Source: https://context7.com/microsoft/cua_skill/llms.txt

Command-line usage for generating synthetic user tasks for a specific application.

```bash
# Generate 10,000 tasks for Bing Search
python user_task_generation/user_task_generator.py \
  --primitive-operation ./asset/primitive_operation/bingsearch_primitive_operation.json \
  --composition ./asset/primitive_operation_composition/bingsearch_primitive_operation_composition.json \
  --app-name bingsearch \
  --out-dir ./asset/user_task \
  --num-tasks 10000 \
  --instruction-dropoff-prob-range '[0.1,0.2]' \
  --llm-rephrase-prob-range '[0.0]' \
  --launch-app-instruction-dropoff-prob-range '[0.5]'
```

--------------------------------

### Layout Utility Classes

Source: https://github.com/microsoft/cua_skill/blob/main/docs/static/css/bulma.css.map.txt

Utility classes for managing layout and spacing within the application.

```css
A0C9vNA,eAAA;ACEA;EACE,cAAc;EACd,aAAa;EACb,YAAY;EACZ,cAAc;EACd,gBAPkB;A3CuwNpB;;A2C/vNE;EACE,UAAU;A3CkwNd;;A2CjwNE;EACE,UAAU;EACV,WAAW;A3CowNf;;A2CnwNE;EACE,UAAU;EACV,UAAU;A3CswNd;;A2CrwNE;EACE,UAAU;EACV,eAAe;A3CwwNnB;;A2CvwNE;EACE,UAAU;EACV,UAAU;A3C0wNd;;A2CzwNE;EACE,UAAU;EACV,eAAe;A3C4wNnB;;A2C3wNE;EACE,UAAU;EACV,UAAU;A3C8wNd;;A2C7wNE;EACE,UAAU;EACV,UAAU;A3CgxNd;;A2C/wNE;EACE,UAAU;EACV,UAAU;A3CkxNd;;A2CjxNE;EACE,UAAU;EACV,UAAU;A3CoxNd;;A2CnxNE;EACE,UAAU;EACV,UAAU;A3CsxNd;;A2CrxNE;E1CwGE,gB0CvGmC;A3CwxNvC;;A2CvxNE;E1CsGE,qB0CrGwC;A3C0xN5C;;A2CzxNE;E1CoGE,gB0CnGmC;A3C4xNvC;;A2C3xNE;E1CkGE,qB0CjGwC;A3C8xN5C;;A2C7xNE;E1CgGE,gB0C/FmC;A3CgyNvC;;A2C/xNE;E1C8FE,gB0C7FmC;A3CkyNvC;;A2CjyNE;E1C4FE,gB0C3FmC;A3CoyNvC;;A2CnyNE;E1C0FE,gB0CzFmC;A3CsyNvC;;A2CryNE;E1CwFE,gB0CvFmC;A3CwyNvC;;A2CtyNI;EACE,UAAU;EACV,SAA0B;A3CyyNhC;;A2CxyNI;E1CkFA,e0CjFqD;A3C2yNzD;;A2C/yNI;EACE,UAAU;EACV,eAA0B;A3CkzNhC;;A2CjzNI;E1CkFA,qB0CjFqD;A3CozNzD;;A2CxzNI;EACE,UAAU;EACV,gBAA0B;A3C2zNhC;;A2C1zNI;E1CkFA,sB0CjFqD;A3C6zNzD;;A2Cj0NI;EACE,UAAU;EACV,UAA0B;A3Co0NhC;;A2Cn0NI;E1CkFA,gB0CjFqD;A3Cs0NzD;;A2C10NI;EACE,UAAU;EACV,gBAA0B;A3C60NhC;;A2C50NI;E1CkFA,sB0CjFqD;A3C+0NzD;;A2Cn1NI;EACE,UAAU;EACV,gBAA0B;A3Cs1NhC;;A2Cr1NI;E1CkFA,sB0CjFqD;A3Cw1Nz
```

--------------------------------

### Define Result Directories

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/result_analysis.ipynb

Sets the paths to the result directories for processing.

```python
run_dir = CURRENT_DIR_PATH / "../../WindowsAgentArena/src/win-arena-container/client/results/0"
run_dir5 = CURRENT_DIR_PATH / "../../WindowsAgentArena/src/win-arena-container/client/all_results/gpt5low_run7"
```

--------------------------------

### Download blobs from Azure container

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/result_analysis.ipynb

Downloads blobs matching a prefix to a local directory. Requires DefaultAzureCredential and ContainerClient from the Azure SDK.

```python


def _add_suffix_to_rel_path(rel_path: str, suffix: str) -> str:
    """Helper to add suffix before extension (if any)."""
    base, ext = os.path.splitext(rel_path)
    return f"{base}{suffix}{ext}"


def download_blobs_from_container(
    account_url: str,
    container_name: str,
    prefix: str,
    download_dir: str,
    conflict_policy: str = "skip_file",   # "rename", "skip_file", or "skip_dir"
    conflict_suffix: str = "__file"
) -> None:
    """
    Download all blobs from an Azure container that match the given prefix,
    handling file/directory name conflicts.

    Parameters
    ----------
    account_url : str
        Azure storage account URL.
    container_name : str
        Name of the container to download from.
    prefix : str
        Blob name prefix to filter.
    download_dir : str
        Local directory to download into.
    conflict_policy : {"rename", "skip_file", "skip_dir"}, default="rename"
        - "rename": rename the conflicting file locally using conflict_suffix
        - "skip_file": skip the conflicting file, keep directory contents
        - "skip_dir": keep the conflicting file, skip directory contents
    conflict_suffix : str, default="__file"
        Suffix to append when renaming conflicting files.
    """
    credential = DefaultAzureCredential()
    container_client = ContainerClient(account_url, container_name, credential=credential)

    blobs = list(container_client.list_blobs(name_starts_with=prefix))

    # Build relative paths
    rel_paths = []
    for b in blobs:
        if not b.name.startswith(prefix):
            continue
        rel = b.name[len(prefix):].lstrip("/")
        rel_paths.append(rel)

    # Distinguish files vs. directory placeholders
    file_paths = set()
    dir_placeholders = set()
    for p in rel_paths:
        if p == "" or p.endswith("/"):
            dp = p.rstrip("/")
            if dp:
                dir_placeholders.add(dp)
            continue
        file_paths.add(p)

    # Collect implied directories from file paths
    implied_dirs = set()
    for p in file_paths:
        parts = p.split("/")
        for i in range(1, len(parts)):
            implied_dirs.add("/".join(parts[:i]))

    all_dirs = implied_dirs | dir_placeholders

    # Conflicts = paths that are both file and directory
    conflicts = file_paths & all_dirs

    # If skipping directories, mark all descendants to skip
    skip_all_under = set()
    if conflict_policy == "skip_dir":
        for c in conflicts:
            skip_all_under.add(c + "/")

    # Prepare download plan
    plan = []
    for b in blobs:
        if not b.name.startswith(prefix):
            continue
        rel = b.name[len(prefix):].lstrip("/")

        if rel == "" or rel.endswith("/"):
            continue

        # Skip if under a directory marked for skipping
        if conflict_policy == "skip_dir":
            if any(rel.startswith(cprefix) for cprefix in skip_all_under):
                continue

        local_rel = rel
        if rel in conflicts:
            if conflict_policy == "skip_file":
                continue
            elif conflict_policy == "rename":
                local_rel = _add_suffix_to_rel_path(rel, conflict_suffix)
            # skip_dir: keep the file as-is

        local_rel_fs = local_rel.replace("/", os.sep)
        local_path = os.path.join(download_dir, local_rel_fs)
        plan.append((b.name, local_path))

    # Download files
    for blob_name, local_path in tqdm(plan, total=len(plan), desc="Downloading blobs"):
        dir_name = os.path.dirname(local_path)
        if dir_name:
            os.makedirs(dir_name, exist_ok=True)
            print(blob_name)
        downloader = container_client.download_blob(blob_name)
        with open(local_path, "wb") as f:
            downloader.readinto(f)

    # Create empty directories f
    # 
    # 
    # or placeholders and implied dirs
    for d in all_dirs:
        d_local = os.path.join(download_dir, d.replace("/", os.sep))
        os.makedirs(d_local, exist_ok=True)

    print(
        f"All blobs downloaded. Conflicts: {len(conflicts)} "
        f"handled with policy '{conflict_policy}'."
    )
```

--------------------------------

### Import necessary libraries for CUA Skill project

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/optimize_feasibility_prompt.ipynb

Imports essential Python libraries for file path manipulation, image processing, byte stream handling, base64 encoding, system operations, data structures, LLM model loading, data analysis, progress visualization, and JSON handling.

```python
from pathlib import Path
from PIL import Image
from io import BytesIO
import base64
import os
import sys
sys.path.append(os.path.abspath(os.path.join(Path.cwd(), '..',"..")))
from types import SimpleNamespace
from agent.llms import model_loader
import pandas as pd
from tqdm import tqdm
import json

try:
    CURRENT_DIR_PATH = Path(__file__).resolve().parent
except NameError:
    CURRENT_DIR_PATH = Path.cwd()
```

--------------------------------

### Execute Result Analysis

Source: https://github.com/microsoft/cua_skill/blob/main/evaluation/WindowsAgentArena/result_analysis.ipynb

Runs the analysis function on the specified directory.

```python
df = show_results(run_dir)
```

--------------------------------

### Handle User Input and State Management

Source: https://github.com/microsoft/cua_skill/blob/main/docs/static/css/bulma.css.map.txt

Manages user input and updates the skill's state accordingly, ensuring smooth interaction.

```typescript
AsCxlKA;EACE,uBpC1C6B;EoC2C7B,mBAvDqB;EAwDrB,kBAAkB;EAClB,WAtDW;AtCipKb;;AsC/lKA;EASM,uBpClDyB;EoCmDzB,cpChEuB;AF0pK7B;;AsCpmKA;;EAcU,cpCpEmB;AF+pK7B;;AsCzmKA;;;;EAoBY,yB3BiCqB;E2BhCrB,cpC3EiB;AFuqK7B;;AsCjnKA;EAwBY,qBpC9EiB;AF2qK7B;;AsCrnKA;EA0BQ,cpChFqB;AF+qK7B
```