### Install Docker

Source: https://github.com/opengvlab/scalecua/blob/main/playground/envs/android/env_api/README.md

Instructions to install Docker on your operating system. Follow the OS-specific guide linked.

```bash
# Follow your OS guide:
https://docs.docker.com/engine/install/
```

--------------------------------

### Start Client in Dev Mode

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/docs/Development-Tips.md

After preparing the image in dev mode, use this command to run the entire setup, including the client process. This allows for rapid iteration on code changes reflected in the VM.

```bash
./run-local.sh --mode dev --start-client true
```

--------------------------------

### Launch Android Emulator

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/README.md

Starts the Android emulator with specific configurations for gRPC communication. Ensure the AVD name matches your setup.

```bash
emulator -avd AndroidWorldAVD -no-snapshot -grpc 8554

```

--------------------------------

### Install ffmpeg

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Install the ffmpeg utility, which is required for recording the virtual machine's screen.

```sh
sudo apt install ffmpeg
```

--------------------------------

### Enable and Start OSWorld Server Service

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Enables the OSWorld server service to start on boot and then starts the service immediately.

```shell
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```

--------------------------------

### Install Ubuntu Desktop

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Installs the ubuntu-desktop package and sets the default target to graphical mode.

```bash
sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```

--------------------------------

### Install AndroidEnv

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/docs/README_AndroidWorld.md

Clone the AndroidEnv repository and install it using setup.py.

```python
git clone https://github.com/google-deepmind/android_env.git
cd android_env
python setup.py install
```

--------------------------------

### Install x11vnc and noVNC

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Installs the necessary packages for VNC access.

```bash
sudo apt update
sudo apt install x11vnc

sudo snap install novnc
```

--------------------------------

### Install wmctrl

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Install the wmctrl utility, which is necessary for controlling the window manager in the virtual machine.

```sh
sudo apt install wmctrl
```

--------------------------------

### Install Dependencies

Source: https://github.com/opengvlab/scalecua/blob/main/agent-sft/qwen-vl-finetune/README.md

Install the required Python packages for the project.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Dummy Video Driver

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Installs the necessary dummy video driver using apt-get. This is the first step in configuring a virtual display environment.

```bash
sudo apt-get install xserver-xorg-video-dummy
```

--------------------------------

### Navigate to Ubuntu Web Setup Directory

Source: https://github.com/opengvlab/scalecua/blob/main/playground/README.md

Changes the current directory to the Ubuntu Web setup scripts. This is a prerequisite for building the Docker image.

```bash
cd envs/web/ubuntu_web_setup
```

--------------------------------

### Configuration File Example

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Create a config.json file at the project root with API keys and endpoints for OpenAI or Azure.

```json
{
    "OPENAI_API_KEY": "<OPENAI_API_KEY>", // if you are using OpenAI endpoint
    "AZURE_API_KEY": "<AZURE_API_KEY>",  // if you are using Azure endpoint
    "AZURE_ENDPOINT": "https://yourendpoint.openai.azure.com/" // if you are using Azure endpoint
}
```

--------------------------------

### Run AndroidWorld Benchmark with Emulator Setup

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/docs/README_AndroidWorld.md

Executes the benchmark script for AndroidWorld. Includes the --perform_emulator_setup flag for initial setup of necessary apps and permissions, which is a one-time operation.

```bash
python run.py --suite_family=android_world --agent_name=seeact_v --perform_emulator_setup
```

--------------------------------

### Initialize Episode

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/bisect-angle.html

Starts the episode when the window loads.

```javascript
window.onload = function() {
    core.startEpisode();
}
```

--------------------------------

### Start vllm Server

Source: https://github.com/opengvlab/scalecua/blob/main/gradio_demo/README.md

Start the vllm server using one of the provided shell scripts. Ensure the correct script is chosen based on your deployment needs.

```bash
bash scripts/srun_scalecua_vllm_deploy.sh
```

```bash
bash scripts/scalecua_vllm_deploy.sh
```

--------------------------------

### Install VS Code Extension

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Steps to install a custom VS Code extension from a .vsix file.

```bash
1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file
```

--------------------------------

### Generate Problem Setup

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/terminal.html

Initializes the terminal interface, generates files, and sets the query based on the expected file extension.

```javascript
var genProblem = function(){
 $('#area').empty();
 var newTerminal = document.createElement('div');
 newTerminal.innerHTML = TERMINAL_TEMPLATE;
 $('#area').append(newTerminal);
 $('#terminal').on('click', function(){
 $('#terminal-target').focus();
 });
 // cause the input caret to flicker.
 clearInterval(flicker);
 flicker = setInterval(function(){
 $('#input-flicker').toggleClass('hide');
 }, 800);
 var currentTime = new Date();
 generateTerminalOutput('Welcome! Type help for a list of available commands.');
 generateTerminalOutput('Last login: ' + currentTime.toDateString());
 currentFiles = [];
 currentExtensions = [];
 generateFiles();
 var expectedExtension = core.sample(currentExtensions);
 if(expectedExtension !== ''){
 $('#query').html('Use the terminal below to delete a file ending with the extension <span class="bold">.' + expectedExtension + '</span>');
 } else {
 $('#query').html('Use the terminal below to delete a file that has <span class="bold">no file extension</span>.');
 }
 $('#terminal-target').on('keyup', function(e){ pressedKeys[e.keyCode] = null; });
 $('#terminal-target').on('keydown', function(e){
 var currentChar = e.key;
 var currentText = $('#active-input').text();
 pressedKeys[e.keyCode] = true;
 if(modifierKeyPressed()) return;
 if(TYPABLE_KEYS.indexOf(currentChar) !== -1){
 // handle typing regular keys
 $('#active-input').append(e.key);
 } else if(e.keyCode === 8){ // handle delete key
 $('#active-input').text(currentText.substring(0, currentText.length-1));
 } else if (e.keyCode ===13){ // handle newline/returns
 generateTerminalLine(currentText);
 parseCommand(currentText, expectedExtension);
 $('#active-input').text('');
 }
 $('#terminal-target').val(''); // scroll down if needed
 $('#terminal-contents').scrollTop($('#terminal-contents').height());
 });
 // autofocus into terminal for convenience.
 setTimeout(function(){$('#terminal-target').select()}, 200);
 }
```

--------------------------------

### Start VM and Client Processes Manually

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/docs/Development-Tips.md

Once the Docker container is running in interactive mode, use these commands to start the VM and client processes individually. This provides granular control during development.

```bash
./start_vm.sh
.. 
./start_client.sh
```

--------------------------------

### Episode Start

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/email-inbox-forward-nl-turk.html

Initializes the MiniWob episode when the window loads.

```javascript
window.onload = function () { core.startEpisode(); };
```

--------------------------------

### Enable and Start VNC Services

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Reloads systemd user daemon, enables and starts the noVNC and x11vnc services.

```bash
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```

--------------------------------

### Install Accerciser for Manual Accessibility Checks

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Install the Accerciser tool, which allows for manual inspection and verification of accessibility trees on GNOME VMs.

```sh
sudo apt install accerciser
```

--------------------------------

### Install python-pptx Library

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/evaluators/README.md

Install the python-pptx library using pip. This library is required for working with PowerPoint presentations in LibreOffice.

```shell
pip install python-pptx
```

--------------------------------

### Example Accessibility Tree Structure

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

A simplified example illustrating the hierarchical structure of an accessibility tree for a desktop application.

```xml
<desktop-frame ...>
    <application name="Thunderbird" ...>
        ... <!-- nodes of windows -->
    </application>
    ...
</desktop-frame>
```

--------------------------------

### Start Ubuntu Web Docker Compose

Source: https://github.com/opengvlab/scalecua/blob/main/playground/README.md

Starts the Ubuntu Web environment using Docker Compose. This command runs the container in detached mode.

```bash
docker-compose up -d
```

--------------------------------

### Signal Setup and Execution for Browser Context

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/flight/AA/original.html

This snippet defines a mechanism for setting up and executing signals within a browser context. It iterates through signals, calling their setup and afterReady methods if they exist.

```javascript
_gaq_zikNjzf.define( "2e451a39-ff2b-42f6-b002-3524fabb52f8", function (_gaq_hskNjzf, _gaq_iskNjzf, _gaq_jskNjzf, _gaq_kskNjzf) { 
var _gaq_lskNjzf = _gaq_zikNjzf("a60102d7-74eb-44fd-a279-fcd63084d86f", _gaq_hskNjzf); 
var _gaq_mskNjzf = _gaq_lskNjzf["default"]; 
_gaq_iskNjzf["default"] = { 
setup: function (_gaq_nskNjzf, _gaq_oskNjzf) { 
this.ctx = _gaq_oskNjzf; 
this.ctx.userEventLimit = _gaq_nskNjzf.userEventLimit || ~-+true; 
this.signals = _gaq_nskNjzf.signals || _gaq_mskNjzf; 
for (var _gaq_pskNjzf = +"" + ""; _gaq_pskNjzf < this.signals.length; _gaq_pskNjzf++) { 
if (typeof this.signals[_gaq_pskNjzf].setup === "function") { 
try { this.signals[_gaq_pskNjzf].setup(this.ctx); } catch (_gaq_qskNjzf) {} 
} 
} 
}, 
afterReady: function () { 
for ( var _gaq_rskNjzf = ~-+true + ""; _gaq_rskNjzf < this.signals.length; _gaq_rskNjzf++ ) { 
if (typeof this.signals[_gaq_rskNjzf].afterReady === "function") { 
try { 

```

--------------------------------

### Install Playwright Browser Binaries

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/evaluators/README.md

After installing the Playwright library, run this command to download the necessary browser binaries required for Playwright to function.

```bash
playwright install
```

--------------------------------

### Task JSON Configuration Example

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/docs/Develop-Tasks.md

Example of a task JSON configuration, detailing the structure for defining a task's ID, instruction, initial configuration, evaluator, and expected result.

```json
{
    "id": "8ba5ae7a-5ae5-4eab-9fcc-5dd4fe3abf89-W0S",
    "instruction": "Help me modify the folder used to store my recordings to the Desktop",
    "config": [
        {
            "type": "launch",
            "parameters": {
                "command": "vlc"
            }
        },
        {
            "type": "execute",
            "parameters": {
                "command": [
                    "python",
                    "-c",
                    "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"
                ]
            }
        }
    ],
    "evaluator": {
        "func": "vis_vlc_recordings_folder",
        "expected": {
            "type": "rule",
            "rules": {
                "recording_file_path": "C:\\Users\\Docker\\Desktop"
            }
        }
    },
    "result": {
        "type": "vlc_config",
        "dest": "vlcrc"
    }
}
```

--------------------------------

### Check KVM Support

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidLab/docs/prepare_for_linux.md

Use these commands to install cpu-checker and verify if your machine supports KVM virtualization.

```bash
apt-get install cpu-checker
kvm-ok
```

--------------------------------

### Run Local Benchmark Script

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Executes the local benchmark script to start the agent. Use --help for available options.

```bash
cd scripts
./run-local.sh
# For client/agent options:
# ./run-local.sh --help
```

--------------------------------

### JavaScript for Starting the Episode

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/click-collapsible.html

Initializes the task episode when the window loads. This is a standard entry point for MiniWob tasks.

```javascript
window.onload = function() {
 core.startEpisode();
}
```

--------------------------------

### JavaScript for Window Load Event

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/click-tab-2-easy.html

Initializes the task by starting the episode when the window loads.

```javascript
window.onload = function() {
core.startEpisode();
}
```

--------------------------------

### Install VMware Workstation Pro on Linux (Console Mode)

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/providers/vmware/INSTALL_VMWARE.md

Run this command in the terminal to install VMware Workstation Pro on a Linux system in console mode. Replace 'xxxx-xxxxxxx' with the actual version and internal version numbers.

```bash
sudo sh VMware-Workstation-xxxx-xxxxxxx.architecture.bundle --console
```

--------------------------------

### Clone Repo and Install Dependencies

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/MacOSArena/README.md

Clone the OpenCUA repository, navigate to the MacOSArena directory, and install required Python dependencies. A Conda environment setup is optional.

```bash
# Clone the OpenCUA repository
git clone https://github.com/OpenGVLab/OpenCUA.git

# Change directory to the MacOSArena Folder
cd OpenCUA/evaluation/MacOSArena

# Optional: Create a Conda environment for MacOSArena
# conda create -n macos_arena python=3.10
# conda activate macos_arena

# Install required dependencies
pip install -r requirements.txt
```

--------------------------------

### Generate Problem Setup

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/bisect-angle.html

Initializes the SVG canvas by removing existing elements, drawing the starting circles and lines, and setting up click listeners for graph clicks and the submit button.

```javascript
var genProblem = function() {
    var svg = d3.select('svg');
    svg.selectAll('*').remove();
    var jsonCircles = startingPoints();
    shapes.drawCircles(jsonCircles, svg);
    connectPoints();
    svg.on('click', function(){
        graphClicked(event);
    });
    d3.select('#subbtn').on('click', function(){
        computeDistances();
    });
}
```

--------------------------------

### Autocomplete Widget Setup and Logic

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/use-autocomplete-nodelay.html

Configures the jQuery UI autocomplete widget for a country selection input. It uses a custom source function to filter countries based on a regex matching the start of the input term. The task also randomly generates a prefix and suffix for the target country to form the query.

```javascript
var genProblem = function() {
ITEMS = ui_utils.COUNTRIES;
d3.select('#tags')[0][0].value ='';
var gtix = core.randi(0, ITEMS.length);
// the below is complicated because we only want to match beginning of items in list
$('#tags').autocomplete({
    source: function(request, response) {
        var matcher = new RegExp('^' + $.ui.autocomplete.escapeRegex(request.term), 'i');
        response($.grep(ITEMS, function(item){
            return matcher.test(item);
        }))
    },
    delay: 0,
}).autocomplete('close');
var gtitem = ITEMS[gtix];
var nf = core.randi(2, 5);
var f = gtitem.slice(0,nf);
var ne = core.randi(2, 5);
var e = gtitem.slice(gtitem.length - ne);
var match_end = core.randf(0,1) < 0.75;
if(match_end) {
    d3.select('#query').html('Enter an item that starts with "<span class="bold">' + f + '</span>" and ends with "<span class="bold">' + e + '</span>".');
} else {
    d3.select('#query').html('Enter an item that starts with "<span class="bold">' + f + '</span>".');
}
d3.select('#subbtn').on('click', function(){
    var entered = d3.select('#tags')[0][0].value;
    var ff = entered.startsWith(f);
    var ee = entered.endsWith(e);
    var correct = ff & (ee || !match_end);
    var r = correct ? 1.0 : -1.0;
    core.endEpisode(r, r>0);
});
}
window.onload = function() {
    core.startEpisode();
}
```

--------------------------------

### Install Python Packages for Image Processing

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/evaluators/README.md

Installs OpenCV, Pillow, and imagehash for image processing tasks. Ensure you have pip installed.

```bash
pip install opencv-python-headless Pillow imagehash
```

--------------------------------

### Download Pretrained Models

Source: https://github.com/opengvlab/scalecua/blob/main/agent-sft/qwen-vl-finetune/README.md

Download the Qwen2.5-VL-7B-Instruct model to the pretrained directory.

```bash
mkdir pretrained/
huggingface-cli download --resume-download --local-dir-use-symlinks False \
    Qwen/Qwen2.5-VL-7B-Instruct \
    --local-dir pretrained/Qwen2.5-VL-7B-Instruct
```

--------------------------------

### Install socat

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Install the socat utility, which is required for port forwarding.

```sh
sudo apt install socat
```

--------------------------------

### Install ffmpeg on macOS

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/docs/README_AndroidWorld.md

Install ffmpeg using Homebrew on macOS.

```bash
# macOS
brew install ffmpeg
```

--------------------------------

### Initialize Website Environment

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WebArenaLiteV2/README.md

Initializes the Docker environment and instantiates tasks for evaluation.

```python
python launcher/start.py
```

--------------------------------

### Install AndroidWorld

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/docs/README_AndroidWorld.md

Clone the AndroidWorld repository and install its dependencies and the package itself.

```python
git clone https://github.com/google-research/android_world.git
cd ./android_world
pip install -r requirements.txt
python setup.py install
```

--------------------------------

### Verify VMware Workstation Pro Installation

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/providers/vmware/INSTALL_VMWARE.md

Execute this command after installation to verify that VMware Workstation Pro is installed and the environment variables are set correctly. A successful output will list any currently running virtual machines.

```bash
vmrun -T ws list
```

--------------------------------

### JavaScript for Initializing the Task on Window Load

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/click-menu-2.html

Ensures that the UI is set up and the first episode is started when the page finishes loading. This is the entry point for the task's execution.

```javascript
window.onload = function() {
  setupUI();
  core.startEpisode();
}
```

--------------------------------

### Install Requirements and Playwright

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/src/win-arena-container/client/mm_agents/navi/screenparsing_oss/webparse/README.md

Installs project dependencies using pip and downloads Playwright browsers.

```bash
pip install -r requirements.txt ; playwright install
```

--------------------------------

### Problem Generation and Initialization

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/click-tab-2-medium.html

Orchestrates the problem setup by generating tab content, binding click events, and setting the user query. It also initializes the episode when the window loads.

```javascript
var genProblem = function() {
// generate the task
var div = $('#area');
if(div.html().length > 0){
$('#area').tabs('destroy');
}
generateLinks(div);
var any = core.getOpt(core.QueryString, 'any', false); // click Any link?
var correctText = bindClickEvents(); // generate query text in the UI
d3.select('#query').html('Switch between the tabs to find and click on the link "' + correctText + '".');
}
window.onload = function() {
core.startEpisode();
}
```

--------------------------------

### Verify VirtualBox Installation

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/providers/virtualbox/INSTALL_VITUALBOX.md

Run this command in your terminal to check if VirtualBox is installed correctly and accessible.

```bash
VBoxManage --version

```

--------------------------------

### Prepare Windows 11 Golden Image

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Run this script once to automate the setup of the Windows 11 golden image. The process takes approximately 20 minutes. Do not interfere with the VM during preparation.

```bash
cd ./scripts
./run-local.sh --prepare-image true
```

--------------------------------

### Install OSWorld Packages

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/README.md

Installs required Python packages for OSWorld. Ensure you have Python 3.9 or later.

```bash
# python >= 3.9
pip install -r requirements.txt
pip install desktop-env
```

--------------------------------

### JavaScript Initialization and Constants

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/click-shades.html

Sets up constants for the task, including episode time, color definitions, and the number of shades to generate. Initializes the task by starting a new episode.

```javascript
core.EPISODE_MAX_TIME = 15000; // set episode interval to 15 seconds
var COLORS = ['red', 'green', 'blue'];
var COLOR_HUES = [0, 120, 240];
var NUM_OF_SHADES = 12;

window.onload = function() {
  core.startEpisode();
}
```

--------------------------------

### Run Baseline Configuration (OSS)

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Executes the baseline configuration using webparse, groundingdino, and OCR (TesseractOCR).

```bash
./run-local.sh --som-origin oss
```

--------------------------------

### Example Accessibility Tree Node

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

An example of a single node within an accessibility tree, showing attributes and content.

```xml
<section xmlns:attr="uri:deskat:attributes.at-spi.gnome.org" attr:class="subject" st:enabled="true" cp:screencoord="(1525, 169)", cp:windowcoord="(342, 162)", cp:size="(327, 21)">
    歡迎使用新的 Outlook.com 帳戶
</section>
```

--------------------------------

### Run Recommended Configuration

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Executes the recommended configuration for the Navi agent, combining Omniparser with accessibility tree information for optimal results.

```bash
./run-local.sh --som-origin mixed-omni --a11y-backend uia
```

--------------------------------

### Install Chromium for Playwright

Source: https://github.com/opengvlab/scalecua/blob/main/playground/README.md

Installs the Chromium browser required for the Pure Web Playwright environment. Run this command in your terminal.

```bash
playwright install chromium
```

--------------------------------

### Problem Generation and Setup

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/email-inbox-forward-nl-turk.html

Generates a new problem instance by setting up the inbox, creating emails, shuffling them, and defining expected details for the task.

```javascript
var genProblem = function () { setupInbox(); var emails = generateEmails(); displayEmailSummaries(emails); var shuffled = emails.slice(); core.shuffle(shuffled); var expectedDetails = {}; expectedDetails.action = core.sample(EMAIL_ACTIONS); expectedDetails.email = shuffled[0]; if (expectedDetails.action == "reply") { expectedDetails.reply = ui_utils.generateWords(1, 5); } else if (expectedDetails.action == "forward") { expectedDetails.forward = shuffled[1].name; } displayQuery(expectedDetails); bindClickEvents(expectedDetails, emails); };
```

--------------------------------

### Install Dependencies with Conda

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidLab/docs/README_AndroidLab.md

Sets up a new conda environment named 'Android-Lab' with Python 3.11 and installs project dependencies.

```bash
cd /path/to/your/repo
conda create -n Android-Lab python=3.11
conda activate Android-Lab
pip install -r requirements.txt
```

--------------------------------

### Initialize Google XJS Configuration

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/scripts/index.html

Sets up the base configuration for Google's XJS (Experimental JavaScript) framework, including paths for JavaScript and CSS, and an empty array for exceptions.

```javascript
(function(){google.xjs={basecomb:'/xjs/\_/js/k\x3dxjs.hp.en.L9G-jjqry-s.es5.O/ck\x3dxjs.hp.p-A6jdwBUXg.L.X.O/am\x3dAgAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAgBIwCAIAAIAAGAAAAAAAAJAAABIAgAAACJQiAIADAQCAFgAiQAAYIAAAIAAAAAAAAAAEAAAAAQEAEAAAAAgAAAAAQHgcEAAAgIBYAAgAAADiEQ/d\x3d1/ed\x3d1/dg\x3d0/ujg\x3d1/rs\x3dACT90oHGKSG_fWFji4u4gv8yOgjQwa1JfQ',basecss:'/xjs/\_/ss/k\x3dxjs.hp.p-A6jdwBUXg.L.X.O/am\x3dAgAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAgBIwAAIAAIAAAAAAAAAAAIAAABIAgAAACJQCAAAAAACAFgAiQAAYIAAAIAAAAAAAAAAEAAAAAQEAEAAAAAgAAAAAAAAAAAAAAIBA/rs\x3dACT90oGflUU7m_DudKMhZIBjNxqbcxOR5w',basejs:'/xjs/\_/js/k\x3dxjs.hp.en.L9G-jjqry-s.es5.O/am\x3dAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAAAGAAAAAAAAJAAABIAAAAAAAAgAIADAQCAFAAAAAAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQHgcEAAAgIBYAAgAAADiEQ/dg\x3d0/rs\x3dACT90oFaSTGC771qULyv8TBm49C_hckzSw',excm:[]};})();
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/README.md

Clone the WindowsAgentArena repository and install the necessary Python dependencies. Ensure you have activated your Conda environment first.

```bash
git clone https://github.com/microsoft/WindowsAgentArena.git
cd WindowsAgentArena
# Install the required dependencies in your python environment
# conda activate winarena
pip install -r requirements.txt
```

--------------------------------

### Build Docker Image

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidLab/docs/prepare_for_linux.md

Create a directory, navigate into it, unzip the Docker files, and build the Docker image tagged as android_eval:latest.

```bash
mkdir docker_file
cd docker_file
unzip /path/to/your/docker-file.zip
cd docker-file
docker build -t android_eval:latest .
```

--------------------------------

### Initialize Alaska Flight Booking Wrapper

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/flight/Alaska/wrapper.html

Sets the maximum episode time and required fields for the flight search. Starts the simulation when the window loads.

```javascript
core.EPISODE_MAX_TIME = 60000; // 1 minute
core.requiredFields = [ "SearchFields.DepartureCity", "SearchFields.ArrivalCity", "SearchFields.DepartureDate", "SearchFields.ReturnDate", ];
window.onload = function() {
  core.startEpisode();
  document.body.removeEventListener('click', core.canvasDrawClick);
}
```

--------------------------------

### Install pyastpi2 for Accessibility Tree Support

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/OSWorld/desktop_env/server/README.md

Installs the pyastpi2 package using pip3 to enable access to accessibility information and tree structures.

```bash
# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip

# Install pyastpi2 using pip
pip3 install pyastpi2
```

--------------------------------

### Example Navi Agent Implementation

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/WindowsAgentArena/docs/Develop-Agent.md

Provides a basic structure for a custom agent named Navi, including initialization and the required predict and reset methods. Ensure to fill in the actual logic for action generation and state resetting.

```python
# agent.py
import logging
from typing import Dict, List
from PIL import Image
from io import BytesIO
import copy

logger = logging.getLogger("desktopenv.agent")

class NaviAgent:
    def __init__(
            self,
            server: str = "azure",
            model: str = "gpt-4o",
            som_config = None,
            som_origin = "oss",
            obs_view = "screen",
            auto_window_maximize = False,
            use_last_screen = True,
            temperature: float = 0.5,
    ):
        # Initialize agent parameters
        self.action_space = "code_block"
        self.server = server
        self.model = model
        # ... (additional initialization)

    def predict(self, instruction: str, obs: Dict) -> List:
        """
        Predict the next action(s) based on the current observation.
        """
        # Process the observation
        # Generate actions based on the instruction
        # ...
        actions = ["# Your code logic here"]
        return actions

    def reset(self):
        """
        Reset the agent's internal state.
        """
        # Reset logic
        pass

```

--------------------------------

### Starting the Episode on Window Load

Source: https://github.com/opengvlab/scalecua/blob/main/evaluation/AndroidWorld/apps/java/com/google/androidenv/miniwob/app/assets/html/miniwob/search-engine.html

Initializes the search engine task by calling `core.startEpisode()` when the browser window finishes loading. This ensures that the task is ready to begin as soon as the page is fully rendered.

```javascript
window.onload = function() {
	core.startEpisode();
}
```