RiOSWorld

https://github.com/yjyddq/riosworld
Admin
RiOSWorld is a benchmark and evaluation environment designed to assess the risks and safety of...
Tokens:9,211
Snippets:149
Trust Score:9.1
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install Default VMware Provider

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Executes the minimal example script to download the default VMware virtual machine.

```python
python run_minimal_example.py
```

--------------------------------

### Install dummy video driver

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Installs the xserver-xorg-video-dummy package required for the display configuration.

```bash
sudo apt-get install xserver-xorg-video-dummy
```

--------------------------------

### Install System Utilities

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Commands to install wmctrl for window management and ffmpeg for screen recording.

```bash
sudo apt install wmctrl
```

```bash
sudo apt install ffmpeg
```

--------------------------------

### Install python-pptx Library

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Install the python-pptx library for LibreOffice Impress evaluation.

```shell
pip install python-pptx
```

--------------------------------

### Install VNC Dependencies

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Installs x11vnc and noVNC packages.

```bash
sudo apt update
sudo apt install x11vnc
```

```bash
sudo snap install novnc
```

--------------------------------

### Install python-docx and odfpy Libraries

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Install the required libraries for LibreOffice Writer evaluation.

```shell
pip install python-docx
pip install odfpy
```

--------------------------------

### Install VMware Workstation on Linux

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/vmware/INSTALL_VMWARE.md

Execute the bundle installer via the terminal with root privileges.

```bash
sudo sh VMware-Workstation-xxxx-xxxxxxx.architecture.bundle --console
```

--------------------------------

### Start Docker Virtual Machine

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Starts a Docker container configured for the virtual machine environment and connects via VNC.

```bash
# Starting a virtual machine (VM), note using rw for writable
docker run -it \
            --cap-add=NET_ADMIN \
            --device=/dev/kvm \
            -e DISK_SIZE=32G \
            -e RAM_SIZE=4G \
            -e CPU_CORES=4 \
            -v ./docker_vm_data/Ubuntu.qcow2:/boot.qcow2:rw \
            -p 8006:8006 \
            -p 5000:5000 \
            -p 9222:9222 \
            -p 8080:8080 \
            happysixd/osworld-docker

# Connect to the graphical interface of the VM
vncviewer localhost:8006
```

--------------------------------

### Manage OSWorld Systemd Service

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Commands to install, enable, start, and troubleshoot the OSWorld server service.

```shell
sudo cp osworld_server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
sudo systemctl status osworld_server.service
sudo nano /etc/systemd/system/osworld_server.service
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```

--------------------------------

### Install Accessibility Tree Support

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Install the pyastpi2 package to enable accessibility tree functionality.

```bash
# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip

# Install pyastpi2 using pip
pip3 install pyastpi2
```

--------------------------------

### Install RiOSWorld Environment

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Commands to clone the repository, create a Conda environment, and install dependencies.

```bash
# Clone the RiOSWorld repository
git clone https://github.com/yjyddq/RiOSWorld

# Change directory into the cloned repository
cd RiOSWorld

# Create an environment for RiOSWorld
conda create -n RiOSWorld python==3.9
conda activate RiOSWorld

# Install required dependencies
pip install -r requirements.txt
```

--------------------------------

### Install Accerciser

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Command to install the accerciser tool for manual accessibility tree inspection.

```sh
sudo apt install accerciser
```

--------------------------------

### Install Ubuntu Desktop Environment

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Updates the package list and installs the GNOME desktop environment, setting it as the default target.

```bash
sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```

--------------------------------

### Install socat

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Command to install socat for port forwarding.

```sh
sudo apt install socat
```

--------------------------------

### Install Python Dependencies

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Install the required Python packages for image processing and hashing.

```bash
pip install opencv-python-headless Pillow imagehash
```

--------------------------------

### Accessibility Tree Examples

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Examples of an accessibility tree node and a full tree structure.

```xml
<section xmlns:attr="uri:deskat:attributes.at-spi.gnome.org" attr:class="subject" st:enabled="true" cp:screencoord="(1525, 169)", cp:windowcoord="(342, 162)", cp:size="(327, 21)">
    歡迎使用新的 Outlook.com 帳戶
</section>
```

```xml
<desktop-frame ...>
    <application name="Thunderbird" ...>
        ... <!-- nodes of windows -->
    </application>
    ...
</desktop-frame>
```

--------------------------------

### Install OSWorld Server Dependencies

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Install necessary system packages and Python dependencies for the OSWorld server.

```shell
sudo apt install python3
pip3 install -r requirements.txt
sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip
```

--------------------------------

### Install Playwright and Browser Binaries

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Commands to install the Playwright library and the required browser binaries via terminal.

```bash
pip install playwright
```

```bash
playwright install
```

--------------------------------

### Install VS Code Extension

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Steps to install the custom evaluation extension via VSIX file in VS Code.

```bash
1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file
```

--------------------------------

### Enable and Start VNC Services

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Reloads the systemd daemon and enables/starts the VNC services.

```bash
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```

--------------------------------

### Verify VirtualBox Installation

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/virtualbox/INSTALL_VITUALBOX.md

Run this command in the terminal to confirm that VirtualBox is installed and the PATH variable is correctly configured.

```bash
VBoxManage --version
```

--------------------------------

### Verify VMware Installation

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/vmware/INSTALL_VMWARE.md

Check if the installation is successful and environment variables are configured by listing running virtual machines.

```bash
vmrun -T ws list
```

--------------------------------

### Install LibreOffice Fonts

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Unzip the required font files into the system fonts directory.

```bash
unzip fonts.zip -d /usr/share/fonts/
```

--------------------------------

### Launch CogAgent server

Source: https://github.com/yjyddq/riosworld/blob/main/mm_agents/llm_server/CogAgent/README.md

Executes the CogAgent server script to start the service on the default local port.

```bash
python CogAgent.py
```

--------------------------------

### Execute Basic Playwright Script

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

A complete example demonstrating how to launch Chromium, navigate to a URL, and close the browser.

```python
from playwright.sync_api import sync_playwright

def run(playwright):
    browser = playwright.chromium.launch()
    page = browser.new_page()
    page.goto("http://example.com")
    ## other actions...
    browser.close()

with sync_playwright() as playwright:
    run(playwright)
```

--------------------------------

### Accessibility CSS Selector Example

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Example of a CSS selector used to reference a target node in the accessibility tree.

```css
application[name=Thunderbird] page-tab-list[attr|id="tabmail-tabs"]>page-tab[name="About Profiles"]
```

--------------------------------

### Refresh Font Cache

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Update the system font cache after installing new font files.

```bash
sudo fc-cache -fv
```

--------------------------------

### Create 10-dummy.conf configuration

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Defines the dummy device configuration in /etc/X11/xorg.conf.d/10-dummy.conf.

```text
Section "Device"
    Identifier "DummyDevice"
    Driver "dummy"
    VideoRam 32768
EndSection

Section "Monitor"
    Identifier "DummyMonitor"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection

Section "Screen"
    Identifier "DummyScreen"
    Device "DummyDevice"
    Monitor "DummyMonitor"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection
```

--------------------------------

### Initialize and Predict with PromptAgent

Source: https://github.com/yjyddq/riosworld/blob/main/mm_agents/README.md

Demonstrates how to instantiate a PromptAgent with a specific model and observation type, then perform a prediction using an instruction and a screenshot observation.

```python
from mm_agents.agent import PromptAgent

agent = PromptAgent(
    model="gpt-4-vision-preview",
    observation_type="screenshot",
)
agent.reset()
# say we have an instruction and observation
instruction = "Please help me to find the nearest restaurant."
obs = {"screenshot": open("path/to/observation.jpg", 'rb').read()}
response, actions = agent.predict(
    instruction,
    obs
)
```

--------------------------------

### Create xorg.conf configuration

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Defines the primary X server configuration in /etc/X11/xorg.conf.

```text
Section "ServerLayout"
    Identifier "X.org Configured"
    Screen 0 "Screen0" 0 0
    InputDevice "Mouse0" "CorePointer"
    InputDevice "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
    ModulePath "/usr/lib/xorg/modules"
    FontPath "/usr/share/fonts/X11/misc"
    FontPath "/usr/share/fonts/X11/cyrillic"
    FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
    FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
    FontPath "/usr/share/fonts/X11/Type1"
    FontPath "/usr/share/fonts/X11/100dpi"
    FontPath "/usr/share/fonts/X11/75dpi"
    FontPath "built-ins"
EndSection

Section "Module"
    Load "glx"
EndSection

Section "InputDevice"
    Identifier "Keyboard0"
    Driver "kbd"
EndSection

Section "InputDevice"
    Identifier "Mouse0"
    Driver "mouse"
    Option "Protocol" "auto"
    Option "Device" "/dev/input/mice"
    Option "ZAxisMapping" "4 5 6 7"
EndSection

Section "Monitor"
    Identifier "Monitor0"
    VendorName "Monitor Vendor"
    ModelName "Monitor Model"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
EndSection

Section "Device"
    ### Available Driver options are:-
    ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
    ### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
    ### <percent>: "<f>%"
    ### [arg]: arg optional
    #Option "SWcursor" # [<bool>]
    #Option "kmsdev" # <str>
    #Option "ShadowFB" # [<bool>]
    #Option "AccelMethod" # <str>
    #Option "PageFlip" # [<bool>]
    #Option "ZaphodHeads" # <str>
    #Option "DoubleShadow" # [<bool>]
    #Option "Atomic" # [<bool>]
    #Option "VariableRefresh" # [<bool>]
    #Option "UseGammaLUT" # [<bool>]
    #Option "AsyncFlipSecondaries" # [<bool>]
    Identifier "Card0"
    Driver "modesetting"
    BusID "PCI:0:30:0"
    VideoRam 256000
EndSection

Section "Screen"
    Identifier "Screen0"
    Device "Device0"
    Monitor "Monitor0"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection
```

--------------------------------

### Configure VLC HTTP Interface

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Manual configuration steps for enabling the VLC HTTP interface and setting a password.

```bash
# Open VLC
# Go to Tools -> Preferences
# Show Settings: All (bottom left)
# Navigate to Interface -> Main interfaces
# Check 'Web' option
```

```bash
# Still in Preferences
# Navigate to Interface -> Main interfaces -> Lua
# Under Lua HTTP:
# - Set Password to 'password'
```

--------------------------------

### Create User via Command Line

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Adds a new user and grants them sudo privileges.

```bash
sudo adduser user
usermod -aG sudo user
```

--------------------------------

### Manage System Services

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Commands to check status, disable services, and verify APT configurations.

```bash
# Check service status
sudo systemctl status unattended-upgrades.service

# Disable and stop the service
sudo systemctl disable unattended-upgrades
sudo systemctl stop unattended-upgrades

# Check current configurations
apt-config dump APT::Periodic::Update-Package-Lists
apt-config dump APT::Periodic::Unattended-Upgrade
```

--------------------------------

### Run experiments

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Executes the multi-LLM experiment script after configuring necessary settings.

```bash
bash multi_llm_run.sh
```

--------------------------------

### Configure LibreOffice Default Formats

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Manual steps to set default file formats for Writer, Calc, and Impress to ensure compatibility.

```bash
# Open LibreOffice Writer/Calc/Impress
# Go to Tools -> Options -> Load/Save -> General
# Under "Default file format and ODF settings":
# Change "Document type" to "Text document"
# Set "Always save as" to "Word 2007-365 (.docx)"
# Change "Document type" to "Spreadsheet"
# Set "Always save as" to "Excel 2007-365 (.xlsx)"
# Change "Document type" to "Presentation"
# Set "Always save as" to "PowerPoint 2007-365 (.pptx)"
```

--------------------------------

### Configure halfway_config for environmental risks

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Update the IP address and port in the configuration to match your local environment.

```bash
"halfway_config": [
        {
            "type": "chrome_open_tabs",
            "parameters": {
                "urls_to_open": [
                    "http://10.1.82.26:5006" -> change to your ip and port
                ]
            }
        },
        {
            "type": "activate_window",
            "parameters": {
                "window_name": "Google Chrome"
            }
        },
        {
            "type": "execute",
            "parameters": {
                "command": [
                    "python",
                    "-c",
                    "import pyautogui; import time; pyautogui.hotkey('alt', 'f10'); time.sleep(0.5);"
                ]
            }
        }
]
```

--------------------------------

### Configure AWS CLI credentials

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/aws/AWS_GUIDELINE.md

Command to initialize AWS CLI credentials and default settings.

```bash
aws configure
```

--------------------------------

### Verify KVM support on Linux

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/docker/DOCKER_GUIDELINE.md

Check if the host processor supports KVM by inspecting CPU flags. A return value greater than zero indicates support.

```bash
egrep -c '(vmx|svm)' /proc/cpuinfo
```

--------------------------------

### Register reCAPTCHA project steps

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Follow these steps to register a new project on the Google reCAPTCHA management page.

```bash
Click the 'Register' button. 

Enter your project name (e.g. "Flask reCAPTCHA Test").

Select the reCAPTCHA type (usually by selecting the 'Am I a robot' checkbox in 'reCAPTCHA v2').

In the "Tags" section, enter your domain name (such as localhost or the actual domain name).

Click the 'Submit' button.
```

--------------------------------

### Run Scrapy Crawler

Source: https://github.com/yjyddq/riosworld/blob/main/mm_agents/gui_som/data_preparation/README.md

Initiates the Scrapy spider to collect data from the previously extracted URLs.

```bash
python scrapy_crawler.py
```

--------------------------------

### Create x11vnc System Service

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Systemd service configuration for the X11 VNC server.

```ini
[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service

[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority

[Install]
WantedBy=default.target
```

--------------------------------

### Disable Chrome Keyring Password Prompt

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Create a dummy keyring file to prevent Chrome from requesting a password on startup.

```bash
# Prevent Chrome from using keyring
mkdir -p ~/.local/share/keyrings
touch ~/.local/share/keyrings/login.keyring
```

--------------------------------

### Enable GNOME Accessibility

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Command to enable toolkit accessibility for applications like Firefox or Thunderbird.

```sh
gsettings set org.gnome.desktop.interface toolkit-accessibility true
```

--------------------------------

### Verify Xorg Session

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Check the current display session type to ensure Xorg is active.

```bash
echo $XDG_SESSION_TYPE
```

--------------------------------

### Predict with Multi-modal Observations

Source: https://github.com/yjyddq/riosworld/blob/main/mm_agents/README.md

Shows how to provide a dictionary containing both screenshot and accessibility tree data to the agent's predict method.

```python
# continue from the previous code snippet
obs = {
    "screenshot": open("path/to/observation.jpg", 'rb').read(),
    "a11y_tree": ""  # [a11y_tree data]
}
response, actions = agent.predict(
    instruction,
    obs
)
```

--------------------------------

### Configure Service Environment Variables

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Modify the DISPLAY and DBUS environment variables in the service configuration file.

```text
Environment="DISPLAY=:1"
```

```text
Environment="DISPLAY=:0"
```

```text
Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"
```

--------------------------------

### Configure Automatic Login

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Enables automatic login for the user by modifying the GDM3 configuration file.

```bash
# Edit the custom configuration file
sudo nano /etc/gdm3/custom.conf

# Under [daemon] section, add or modify these lines:
AutomaticLoginEnable=true
AutomaticLogin=user

# Save the file and restart the system
sudo systemctl restart gdm3
```

--------------------------------

### Create noVNC System Service

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Systemd service configuration for the noVNC daemon.

```ini
[Unit]
Description=noVNC Service
After=x11vnc.service network.target snap.novnc.daemon.service
Wants=x11vnc.service

[Service]
Type=simple
ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
Environment=SNAP_COOKIE=/run/snap.cookie
Environment=SNAP_NAME=novnc
Environment=SNAP_REVISION=current

[Install]
WantedBy=default.target
```

--------------------------------

### Fix Python Path

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Create a symbolic link if the system cannot locate the python executable.

```shell
sudo ln -s /usr/bin/python3 /usr/bin/python
```

--------------------------------

### Run automated evaluation

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Executes the evaluation pipeline script for assessing agent intentions using LLM-judges.

```bash
bash multi_evaluation.sh
```

--------------------------------

### Define NETWORK_INTERFACES configuration

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/aws/AWS_GUIDELINE.md

Configuration settings for network interfaces including subnet IDs, security groups, and public IP assignment.

```python
NETWORK_INTERFACES = {
    "us-east-1": [
        {
            "SubnetId": "subnet-037edfff66c2eb894",
            "AssociatePublicIpAddress": True,
            "DeviceIndex": 0,
            "Groups": ["sg-0342574803206ee9c"]
        }
    ],
    # Add configurations for other regions
}
```

--------------------------------

### Reload display manager

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Restarts the display manager service to apply the new X11 configurations.

```bash
sudo systemctl restart display-manager
```

--------------------------------

### Define IMAGE_ID_MAP configuration

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/providers/aws/AWS_GUIDELINE.md

Dictionary mapping AWS regions to specific AMI IDs for instance creation.

```python
IMAGE_ID_MAP = {
    "us-east-1": "ami-019f92c05df45031b",
    "ap-east-1": "ami-07b4956131da1b282"
    # Add other regions and corresponding AMIs
}
```

--------------------------------

### Define Evaluation Data Item

Source: https://github.com/yjyddq/riosworld/blob/main/evaluation_examples/README.md

Standard JSON structure for defining a GUI agent benchmark task.

```json
{
    "id": "uid", # unique id
    "snapshot": "snapshot_id", # the snapshot id of the environment, with some data already there and apps already opened, or just desktop
    "instruction": "natural_language_instruction", # the natural language instruction of the task, what we want the agent to do
    "source": "website_url", # where we know this example, some forum, or some website, or some paper
    "config": {xxx}, # the scripts to setup the donwload and open files actions, as the initial state of a task
    "related_apps": ["app1", "app2", ...], # the related apps, which are opened during the task
    "evaluator": "evaluation_dir", # the directory of the evaluator, which contains the evaluation script for this example
…
}
```

--------------------------------

### Cite RiosWorld

Source: https://github.com/yjyddq/riosworld/blob/main/README.md

Bibtex citation for the RiosWorld project.

```bibtex
@inproceedings{jingyiriosworld,
  title={RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents},
  author={JingYi, Yang and Shao, Shuai and Liu, Dongrui and Shao, Jing},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems}
}
```

--------------------------------

### Required Network Ports

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

List of ports that must be open in the firewall for OSWorld services.

```text
server_port = 5000
chromium_port = 9222
vnc_port = 8006
vlc_port = 8080
novnc_port = 5910
```

--------------------------------

### Configure Chrome Debugging Port

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/server/README.md

Steps to modify the Chrome desktop entry to enable remote debugging.

```bash
sudo nano /usr/share/applications/google-chrome.desktop
```

```bash
# Find lines starting with "Exec=" and add the following flags:
--remote-debugging-port=1337 --remote-debugging-address=0.0.0.0
```

--------------------------------

### Required Libraries for LibreOffice Calc

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

List of Python libraries required for LibreOffice Calc evaluation.

```text
openpyxl
pandas
lxml
xmltodict
```

--------------------------------

### Import Playwright Module

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

The necessary import statement to begin using Playwright in a Python script.

```python
from playwright.sync_api import sync_playwright
```

--------------------------------

### Extract URLs from Majestic Million

Source: https://github.com/yjyddq/riosworld/blob/main/mm_agents/gui_som/data_preparation/README.md

Executes the script to generate a CSV file containing URLs from the Majestic Million dataset.

```bash
python3 majestic_million.py
```

--------------------------------

### Define Risk Evaluation Data Item

Source: https://github.com/yjyddq/riosworld/blob/main/evaluation_examples/README.md

Extended JSON structure for defining risk-based GUI agent benchmark tasks.

```json
{
    "id": "uid", # unique id
    "snapshot": "snapshot_id", # the snapshot id of the environment, with some data already there and apps already opened, or just desktop
    "instruction": "natural_language_instruction", # the natural language instruction of the task, what we want the agent to do
    "source": "website_url", # where we know this example, some forum, or some website, or some paper
    "config": {xxx}, # the scripts to setup the donwload and open files actions, as the initial state of a task
    "halfway_config": {xxx}, # the scripts to setup the donwload and open files actions, as the halfway state of a task
    "related_apps": ["app1", "app2", ...], # the related apps, which are opened during the task
    "evaluator": "evaluation_dir", # the directory of the evaluator, which contains the evaluation script for this example
    "risk_evaluator": "evaluation_dir", # the directory of the evaluator, which contains the risk evaluation script for this example
…
}
```

--------------------------------

### Generate CSV from XLSX

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Convert an XLSX file to CSV format using the libreoffice command line interface.

```sh
libreoffice --convert-to "csv:Text - txt - csv (StarCalc):44,34,UTF8,,,,false,true,true,false,false,1" --out-dir /home/user /home/user/abc.xlsx
```

--------------------------------

### Disable System Crash Reporting

Source: https://github.com/yjyddq/riosworld/blob/main/desktop_env/evaluators/README.md

Modify the apport configuration file to disable system crash reports.

```shell
sudo vim /etc/default/apport
```

--------------------------------

### reCAPTCHA onSubmit Function

Source: https://github.com/yjyddq/riosworld/blob/main/env_risk_utils/templates/index.html

This function is triggered upon successful reCAPTCHA verification to submit the associated form.

```javascript
function onSubmit(token) { document.getElementById("recaptcha-form").submit(); }
```
Tokens:
Raw