### Enable and Start VNC Services

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Reloads the systemd user environment, enables the noVNC and x11vnc services to start on boot, and then starts both services immediately. This ensures the VNC server is running and accessible.

```bash
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```

--------------------------------

### Install ffmpeg for Video Recording

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the 'ffmpeg' package, which is required for enabling video recording within the virtual machine. This is essential for capturing experimental sessions.

```Shell
sudo apt install ffmpeg
```

--------------------------------

### Install Dummy Video Driver

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the xserver-xorg-video-dummy package, which provides a virtual display driver. This is often used when a physical display is not available or when specific display configurations are needed.

```bash
sudo apt-get install xserver-xorg-video-dummy
```

--------------------------------

### Manage OSWorld Systemd Service

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Copies the OSWorld service file to the systemd directory, reloads the daemon, enables the service to start on boot, and starts the service. It also shows how to check the service status and edit the service file.

```bash
sudo cp osworld_server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```

```bash
sudo systemctl status osworld_server.service
```

```bash
sudo nano /etc/systemd/system/osworld_server.service
```

```bash
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```

--------------------------------

### Install x11vnc and noVNC

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the x11vnc server and the noVNC client using apt and snap package managers respectively. These are the core components for setting up a VNC connection.

```bash
sudo apt update
sudo apt install x11vnc
```

```bash
sudo snap install novnc
```

--------------------------------

### Install pyastpi2 for Accessibility Tree

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the pyastpi2 package for accessibility tree support in Ubuntu. It first updates the package list and installs pip, then installs pyastpi2 using pip.

```bash
# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip

# Install pyastpi2 using pip
pip3 install pyastpi2
```

--------------------------------

### Install Accerciser for Accessibility Tree Inspection

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs 'accerciser', a tool used to visually inspect and check the accessibility tree of applications on GNOME desktop environments.

```Shell
sudo apt install accerciser
```

--------------------------------

### Install Ubuntu Desktop with GNOME

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the Ubuntu Desktop package with the GNOME desktop environment on an Ubuntu 22.04 system and sets the default target to graphical mode.

```Bash
sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```

--------------------------------

### Python: Interact with Desktop Environment

Source: https://timothyxxx.github.io/OSWorld/installation/quickstart

This Python code snippet demonstrates how to initialize and interact with a desktop environment using the DesktopEnv library. It includes setting up a task configuration for installing Spotify, executing a command via pyautogui, and performing a right-click action. The example also shows how to reset the environment and take a step with a specific action.

```Python
from desktop_env.desktop_env import DesktopEnv

example = {
    "id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
    "instruction": "I want to install Spotify on my current system. Could you please help me?",
    "config": [
        {
            "type": "execute",
            "parameters": {
                "command": [
                    "python",
                    "-c",
                    "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"
                ]
            }
        }
    ],
    "evaluator": {
        "func": "check_include_exclude",
        "result": {
            "type": "vm_command_line",
            "command": "which spotify"
        },
        "expected": {
            "type": "rule",
            "rules": {
                "include": ["spotify"],
                "exclude": ["not found"]
            }
        }
    }
}

env = DesktopEnv(action_space="pyautogui")

obs = env.reset(task_config=example)
obs, reward, done, info = env.step("pyautogui.rightClick()")

```

--------------------------------

### Install OSWorld Dependencies

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs necessary packages for the OSWorld server, including Python 3, Tkinter, development headers, screenshot tools, and media utilities. It also includes a workaround for potential 'python' command not found errors.

```bash
sudo apt install python3
pip3 install -r requirements.txt
sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip
```

```bash
sudo ln -s /usr/bin/python3 /usr/bin/python
```

--------------------------------

### Accessibility Tree Example (XML)

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

An example of an accessibility tree node for an application, showing attributes like class, coordinates, and size. This format is used for analyzing application interfaces.

```XML
<section xmlns:attr="uri:deskat:attributes.at-spi.gnome.org" attr:class="subject" st:enabled="true" cp:screencoord="(1525, 169)", cp:windowcoord="(342, 162)", cp:size="(327, 21)">
    歡迎使用新的 Outlook.com 帳戶
</section>
```

--------------------------------

### Install Dummy Video Driver

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This command installs the `xserver-xorg-video-dummy` package using apt-get, which is necessary for configuring a virtual display environment without physical hardware.

```bash
sudo apt-get install xserver-xorg-video-dummy
```

--------------------------------

### Install VSCode Evaluation Plugin

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Provides instructions to install a custom VSCode extension (eval-0.0.1.vsix) for extracting internal information and configurations from the VSCode environment. This involves downloading the VSIX file and installing it via the VSCode Extensions menu.

```Shell
1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file
```

--------------------------------

### Setup Host Machine Environment (Bash)

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/run_public_evaluation

Commands to set up the host machine environment by cloning the OSWorld repository, changing directories, and installing dependencies. Includes optional Conda environment setup.

```bash
# Clone the OSWorld repository
git clone https://github.com/xlang-ai/OSWorld

# Change directory into the cloned repository
cd OSWorld

# Optional: Create a Conda environment for OSWorld
# conda create -n osworld python=3.10
# conda activate osworld

# Install required dependencies
pip install -r requirements.txt
```

--------------------------------

### Install LibreOffice Fonts

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs custom fonts for LibreOffice by unzipping a font archive and updating the font cache. This ensures that examples using non-default system fonts in LibreOffice Impress function correctly.

```Shell
unzip fonts.zip -d /usr/share/fonts/
```

```Shell
sudo fc-cache -fv
```

--------------------------------

### Install wmctrl for Window Manager Control

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the 'wmctrl' package, which is necessary for controlling the window manager within the virtual machine. This is required for certain experimental functionalities.

```Shell
sudo apt install wmctrl
```

--------------------------------

### Setup OSWorld Host Environment (Bash)

Source: https://timothyxxx.github.io/OSWorld/user_guides/run_public_evaluation

Commands to set up the OSWorld environment on the host machine. This includes cloning the repository, navigating into the directory, and installing dependencies using pip. It also includes commented-out commands for creating and activating a Conda environment.

```Bash
# Clone the OSWorld repository
git clone https://github.com/xlang-ai/OSWorld

# Change directory into the cloned repository
cd OSWorld

# Optional: Create a Conda environment for OSWorld
# conda create -n osworld python=3.10
# conda activate osworld

# Install required dependencies
pip install -r requirements.txt

```

--------------------------------

### Install ffmpeg for Video Recording

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This command installs the 'ffmpeg' package, which is required for enabling video recording within the virtual machine. Proper video recording is essential for capturing experimental runs.

```bash
sudo apt install ffmpeg
```

--------------------------------

### Copy and Manage OSWorld Service

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Copies the service file to the systemd directory, reloads the daemon, enables the service for boot, and starts it.

```bash
sudo cp osworld_server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable osworld_server.service
sudo systemctl start osworld_server.service
```

--------------------------------

### Accessibility Tree Example (CSS Selector)

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Demonstrates how to use a CSS selector to target a specific element within an application's accessibility tree, such as a tab in Thunderbird's profile manager.

```CSS
application[name=Thunderbird] page-tab-list[attr|id="tabmail-tabs"]>page-tab[name="About Profiles"]
```

--------------------------------

### Install socat for Port Forwarding

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Installs the 'socat' utility using the apt package manager. Socat is required for enabling port forwarding, which is a necessary component for network communication within the OSWorld environment.

```Shell
sudo apt install socat
```

--------------------------------

### Install Ubuntu Desktop with GNOME

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Installs the Ubuntu Desktop package and sets the default target to graphical mode for systems using Ubuntu 22.04. This ensures the GNOME desktop environment is set up and the system boots into the GUI.

```bash
sudo apt update
sudo apt install ubuntu-desktop
sudo systemctl set-default graphical.target
```

--------------------------------

### Configure x11vnc Systemd Service

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Creates a systemd user service file for x11vnc. This service starts the x11vnc server, which shares the current display (:0) and listens on port 5900. It is configured to restart on failure and includes necessary environment variables.

```ini
[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service

[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority

[Install]
WantedBy=default.target
```

--------------------------------

### Install VS Code Extension

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Steps to install a custom VS Code extension (.vsix file) using the VS Code Extensions view.

```bash
1. Download the extension from: https://github.com/xlang-ai/OSWorld/blob/04a9df627c7033fab991806200877a655e895bfd/vscodeEvalExtension/eval-0.0.1.vsix
2. Open VS Code
3. Go to Extensions -> ... -> Install from VSIX... -> choose the downloaded eval-0.0.1.vsix file
```

--------------------------------

### Start Web Monitoring Tool

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/run_public_evaluation

Launches the web monitoring tool's main application using a Python script.

```bash
python main.py
```

--------------------------------

### Install pyastpi2 for Accessibility

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Installs the pyastpi2 package using pip to enable accessibility tree functionality.

```bash
# Update package list and ensure pip is installed
sudo apt-get update
sudo apt-get install python3-pip

# Install pyastpi2 using pip
pip3 install pyastpi2
```

--------------------------------

### Launch Application with Options

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/task_example_explanation

Executes system commands to start applications or services with specified options. This is used to initiate processes within the OSWorld environment.

```json
{
  "type": "launch",
  "parameters": {
    "command": ["application-name", "--option1", "--option2"]
  }
}
```

--------------------------------

### Configure noVNC Systemd Service

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Creates a systemd user service file for noVNC. This service starts the noVNC client, listening on port 5910 and connecting to the x11vnc server on localhost:5900. It includes restart policies and environment variables for proper operation.

```ini
[Unit]
Description=noVNC Service
After=x11vnc.service network.target snap.novnc.daemon.service
Wants=x11vnc.service

[Service]
Type=simple
ExecStart=/snap/bin/novnc --vnc localhost:5900 --listen 5910
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority
Environment=SNAP_COOKIE=/run/snap.cookie
Environment=SNAP_NAME=novnc
Environment=SNAP_REVISION=current

[Install]
WantedBy=default.target
```

--------------------------------

### Install Monitoring Tool Dependencies

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/run_public_evaluation

Installs the necessary Python packages for the web monitoring tool by referencing the requirements.txt file.

```bash
cd monitor
pip install -r requirements.txt
```

--------------------------------

### Install noVNC via Snap

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Installs the noVNC client using the snap package manager. noVNC provides a web-based VNC client, allowing access to the VNC server through a web browser.

```bash
sudo snap install novnc
```

--------------------------------

### Enable and Start VNC Services

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This bash script demonstrates how to reload the systemd daemon, enable, and start both the novnc and x11vnc user services. This is crucial for the VNC server to be operational.

```bash
systemctl --user daemon-reload
systemctl --user enable novnc.service
systemctl --user enable x11vnc.service
systemctl --user start x11vnc.service
systemctl --user start novnc.service
```

--------------------------------

### Install Dependencies and Run Web Monitoring Tool

Source: https://timothyxxx.github.io/OSWorld/user_guides/run_public_evaluation

Navigate to the 'monitor' directory, install the required dependencies using `pip`, and then run the web monitoring tool's main script. This tool allows you to view the results of your agent's execution.

```bash
cd monitor
pip install -r requirements.txt
python main.py
```

--------------------------------

### OSWorld Task Example JSON Schema

Source: https://timothyxxx.github.io/OSWorld/user_guides/task_example_explanation

Defines the structure of task example JSON files used in OSWorld for benchmarking GUI interaction. It includes core fields, setup, configuration, and evaluation parameters.

```JSON
{
  "id": "string (required)",
  "instruction": "string (required)",
  "source": "string (optional)",
  "related_apps": ["string"] (required),
  "config": [
    {
      "type": "string",
      "parameters": {}
    }
  ] (required),
  "evaluator": {
    "func": "string",
    "result": "string",
    "expected": "string"
  } (required),
  "proxy": "boolean (optional)",
  "fixed_ip": "boolean (optional)",
  "possibility_of_env_change": "string (optional)"
}
```

--------------------------------

### Example Accessibility Tree Node

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This is an example of a node within an accessibility tree, typically used for inspecting the structure and properties of GUI elements for accessibility purposes.

```xml

```

--------------------------------

### Install OSWorld Server Dependencies

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Installs necessary packages and libraries for the OSWorld server, including Python 3, pip, Tkinter, FFmpeg, and utilities like wmctrl and xclip.

```bash
sudo apt install python3
pip3 install -r requirements.txt
sudo apt-get install python3-tk python3-dev
sudo apt install gnome-screenshot
sudo apt install wmctrl
sudo apt install ffmpeg
sudo apt install socat
sudo apt install xclip
```

--------------------------------

### Configure OSWorld Service Display

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Modifies the OSWorld service file to set the DISPLAY environment variable to ':0' and adds DBUS session address for wallpaper changes.

```bash
Environment="DISPLAY=:0"
Environment="DISPLAY=:0;DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus"
```

--------------------------------

### Configure Xorg ServerLayout and Files

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Creates the main xorg.conf file, defining the server layout, input devices, monitor settings, and device configuration. It specifies the 'modesetting' driver and a default resolution of 1920x1080.

```text
Section "ServerLayout"
    Identifier "X.org Configured"
    Screen 0 "Screen0" 0 0
    InputDevice "Mouse0" "CorePointer"
    InputDevice "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
    ModulePath "/usr/lib/xorg/modules"
    FontPath "/usr/share/fonts/X11/misc"
    FontPath "/usr/share/fonts/X11/cyrillic"
    FontPath "/usr/share/fonts/X11/100dpi/:unscaled"
    FontPath "/usr/share/fonts/X11/75dpi/:unscaled"
    FontPath "/usr/share/fonts/X11/Type1"
    FontPath "/usr/share/fonts/X11/100dpi"
    FontPath "/usr/share/fonts/X11/75dpi"
    FontPath "built-ins"
EndSection

Section "Module"
    Load "glx"
EndSection

Section "InputDevice"
    Identifier "Keyboard0"
    Driver "kbd"
EndSection

Section "InputDevice"
    Identifier "Mouse0"
    Driver "mouse"
    Option "Protocol" "auto"
    Option "Device" "/dev/input/mice"
    Option "ZAxisMapping" "4 5 6 7"
EndSection

Section "Monitor"
    Identifier "Monitor0"
    VendorName "Monitor Vendor"
    ModelName "Monitor Model"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
EndSection

Section "Device"
    ### Available Driver options are:-
    ### Values: <i>: integer, <f>: float, <bool>: "True"/"False",
    ### <string>: "String", <freq>: "<f> Hz/kHz/MHz",
    ### <percent>: "<f>%"
    ### [arg]: arg optional
    #Option "SWcursor" # [<bool>]
    #Option "kmsdev" # <str>
    #Option "ShadowFB" # [<bool>]
    #Option "AccelMethod" # <str>
    #Option "PageFlip" # [<bool>]
    #Option "ZaphodHeads" # <str>
    #Option "DoubleShadow" # [<bool>]
    #Option "Atomic" # [<bool>]
    #Option "VariableRefresh" # [<bool>]
    #Option "UseGammaLUT" # [<bool>]
    #Option "AsyncFlipSecondaries" # [<bool>]
    Identifier "Card0"
    Driver "modesetting"
    BusID "PCI:0:30:0"
    VideoRam 256000
EndSection

Section "Screen"
    Identifier "Screen0"
    Device "Device0"
    Monitor "Monitor0"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection
```

--------------------------------

### Enable GNOME Accessibility Toolkit

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Enables the accessibility toolkit for GNOME desktop environments, which is a prerequisite for inspecting the accessibility tree of applications like Firefox and Thunderbird.

```Shell
gsettings set org.gnome.desktop.interface toolkit-accessibility true
```

--------------------------------

### Run and Customize Docker VM

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/install_provider/docker

This bash command starts a Docker container for a virtual machine with specified disk size, RAM, and CPU cores. It mounts a local directory for persistent storage and maps ports for access. The container uses a custom Docker image 'happysixd/osworld-docker'.

```bash
docker run -it --rm \
  -e "DISK_SIZE=64G" \
  -e "RAM_SIZE=8G" \
  -e "CPU_CORES=8" \
  --volume "$(pwd)/docker_vm_data/Ubuntu.qcow2:/boot.qcow2" \
  --cap-add NET_ADMIN \
  --device /dev/kvm \
  -p 8006:8006 \
  -p 5000:5000 \
  happysixd/osworld-docker
```

--------------------------------

### Enable VLC Web Interface

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Enables the HTTP web interface for VLC media player, allowing for remote control. This involves navigating to VLC's preferences and enabling the 'Web' interface option.

```Text
# Open VLC
# Go to Tools -> Preferences
# Show Settings: All (bottom left)
# Navigate to Interface -> Main interfaces
# Check 'Web' option
```

--------------------------------

### Launch Application Command

Source: https://timothyxxx.github.io/OSWorld/user_guides/task_example_explanation

Executes system commands to start applications or services. This configuration type is used to initiate processes within the OSWorld environment.

```json
{
  "type": "launch",
  "parameters": {
    "command": ["application-name", "--option1", "--option2"]
  }
}
```

--------------------------------

### Install x11vnc for VNC Server

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Installs the x11vnc package, which provides a VNC server that can share an existing X11 desktop session. This is a prerequisite for remote desktop access.

```bash
sudo apt update
sudo apt install x11vnc
```

--------------------------------

### Create User and Grant Sudo Permissions

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Adds a new user named 'user' to the system and grants them sudo privileges.

```Bash
sudo adduser user
usermod -aG sudo user
```

--------------------------------

### Configure x11vnc Systemd Service

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This snippet defines the systemd service file for x11vnc, which acts as the VNC server. It includes the command to start x11vnc, its restart policy, and necessary environment variables for display and Xauthority.

```ini
[Unit]
Description=X11 VNC Server
After=display-manager.service network.target
Wants=display-manager.service

[Service]
Type=simple
ExecStart=x11vnc -display :0 -rfbport 5900 -forever
Restart=on-failure
RestartSec=3
Environment=DISPLAY=:0
Environment=XAUTHORITY=/home/user/.Xauthority

[Install]
WantedBy=default.target
```

--------------------------------

### Launch Chrome with Remote Debugging

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/task_example_explanation

Starts Google Chrome with the remote debugging port enabled, allowing external tools to connect and control the browser instance. This is a prerequisite for automated browser interactions.

```bash
google-chrome --remote-debugging-port=1337
```

--------------------------------

### Manage Unattended Upgrades Service

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Provides commands to check the status of the unattended-upgrades service, disable it, and stop it. It also shows how to check the configuration of periodic package list updates and unattended upgrades.

```bash
# Check service status
sudo systemctl status unattended-upgrades.service
```

```bash
# Disable and stop the service
sudo systemctl disable unattended-upgrades
sudo systemctl stop unattended-upgrades
```

```bash
# Check current configurations
apt-config dump APT::Periodic::Update-Package-Lists
apt-config dump APT::Periodic::Unattended-Upgrade
```

--------------------------------

### Configure LibreOffice Default Formats

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Guides users through LibreOffice's Tools -> Options menu to set default save formats for Text documents, Spreadsheets, and Presentations to .docx, .xlsx, and .pptx respectively.

```bash
# Open LibreOffice Writer/Calc/Impress
# Go to Tools -> Options -> Load/Save -> General
# Under "Default file format and ODF settings":
# Change "Document type" to "Text document"
# Set "Always save as" to "Word 2007-365 (.docx)"
# Change "Document type" to "Spreadsheet"
# Set "Always save as" to "Excel 2007-365 (.xlsx)"
# Change "Document type" to "Presentation"
# Set "Always save as" to "PowerPoint 2007-365 (.pptx)"
```

--------------------------------

### Initialize and Interact with DesktopEnv

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/quickstart

This Python code snippet demonstrates initializing the DesktopEnv with a 'pyautogui' action space and resetting the environment with a task configuration. It then executes a step, performing a right-click action.

```python
from desktop_env.desktop_env import DesktopEnv

example = {
    "id": "94d95f96-9699-4208-98ba-3c3119edf9c2",
    "instruction": "I want to install Spotify on my current system. Could you please help me?",
    "config": [
        {
            "type": "execute",
            "parameters": {
                "command": [
                    "python",
                    "-c",
                    "import pyautogui; import time; pyautogui.click(960, 540); time.sleep(0.5);"
                ]
            }
        }
    ],
    "evaluator": {
        "func": "check_include_exclude",
        "result": {
            "type": "vm_command_line",
            "command": "which spotify"
        },
        "expected": {
            "type": "rule",
            "rules": {
                "include": ["spotify"],
                "exclude": ["not found"]
            }
        }
    }
}

env = DesktopEnv(action_space="pyautogui")

obs = env.reset(task_config=example)
obs, reward, done, info = env.step("pyautogui.rightClick()")
```

--------------------------------

### Configure Dummy Xorg Screen and Monitor

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Creates a configuration file for the dummy driver in xorg.conf.d. This section defines a 'DummyDevice', 'DummyMonitor', and 'DummyScreen' with specific video RAM and a 1920x1080 resolution, overriding or supplementing the main xorg.conf.

```text
Section "Device"
    Identifier "DummyDevice"
    Driver "dummy"
    VideoRam 32768
EndSection

Section "Monitor"
    Identifier "DummyMonitor"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection

Section "Screen"
    Identifier "DummyScreen"
    Device "DummyDevice"
    Monitor "DummyMonitor"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection
```

--------------------------------

### VPN Setup Documentation

Source: https://timothyxxx.github.io/OSWorld/_sources/community/vision_and_roadmap

Documentation for VPN setup will be provided for users who require it for accessing or utilizing the project's resources.

```General
VPN setup doc for those who need it
```

--------------------------------

### Basic Task Automation Example

Source: https://timothyxxx.github.io/OSWorld/_sources/community/vision_and_roadmap

A simple task is planned to demonstrate the environment's capabilities, involving opening a browser, navigating to a website, clicking a button, and closing the browser. This serves as a foundational example for task execution.

```General
Build a simple task, e.g. open a browser, open a website, click on a button, and close the browser
```

--------------------------------

### Verify Docker Installation

Source: https://timothyxxx.github.io/OSWorld/installation/install_provider/docker

This command verifies that Docker has been successfully installed by displaying the installed Docker version. It's a crucial step after the Docker installation process.

```Shell
docker --version
```

--------------------------------

### Verify Docker Installation

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/install_provider/docker

This bash command verifies the Docker installation by displaying the installed Docker version. It's a simple check to confirm that Docker has been successfully installed and is accessible from the command line.

```bash
docker --version
```

--------------------------------

### Verify VirtualBox Installation

Source: https://timothyxxx.github.io/OSWorld/installation/install_provider/virtualbox

This command verifies if VirtualBox has been successfully installed and if the VBoxManage command is accessible in the system's PATH. It's a crucial step after installation to ensure the software is ready for use.

```Shell
VBoxManage --version

```

--------------------------------

### Install open-vm-tools on Ubuntu/Debian

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/install_provider/vmware

This command installs the 'open-vm-tools' and 'open-vm-tools-desktop' packages on Ubuntu/Debian-based systems to resolve potential VMware issues. It first updates the package list and then installs the specified packages.

```sh
sudo apt update
sudo apt install open-vm-tools open-vm-tools-desktop
```

--------------------------------

### Bash: Setting AWS Environment Variables

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/run_public_evaluation

Provides examples of how to set essential AWS environment variables on the host machine before starting client code. These variables include the security group ID and subnet ID, which are crucial for network configuration and resource isolation.

```bash
# Example: Set AWS Security Group ID
export AWS_SECURITY_GROUP_ID="sg-0123456789abcdef0"

# Example: Set AWS Subnet ID
export AWS_SUBNET_ID="subnet-0123456789abcdef0"
```

--------------------------------

### Install VMware Workstation Pro on Linux

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/install_provider/vmware

This command installs VMware Workstation Pro on a Linux system via the terminal. It requires administrator privileges and the bundle file for the specific version and architecture.

```sh
sudo sh VMware-Workstation-xxxx-xxxxxxx.architecture.bundle --console
```

--------------------------------

### Install LibreOffice Fonts

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Instructions for installing custom fonts for LibreOffice by unzipping a font archive to the system fonts directory and refreshing the font cache.

```bash
unzip fonts.zip -d /usr/share/fonts/
```

```bash
sudo fc-cache -fv
```

--------------------------------

### Verify VMware Workstation Pro Installation

Source: https://timothyxxx.github.io/OSWorld/_sources/installation/install_provider/vmware

This command verifies the successful installation of VMware Workstation Pro by listing currently running virtual machines. It requires the 'vmrun' tool to be accessible in the system's PATH.

```sh
vmrun -T ws list
```

--------------------------------

### Edit OSWorld Service Configuration

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Opens the 'osworld_server.service' file for editing using nano.

```bash
sudo nano /etc/systemd/system/osworld_server.service
```

--------------------------------

### Example Annotation for Training/Testing

Source: https://timothyxxx.github.io/OSWorld/_sources/community/vision_and_roadmap

The plan includes annotating examples to be used for training and testing purposes. This step is crucial for evaluating and improving the performance of the agents.

```General
Start to annotate the examples for ~~training~~ and testing
```

--------------------------------

### Initialize DesktopEnv Class

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/environment_explanation

Demonstrates the initialization of the DesktopEnv class with various configuration options. This includes specifying the virtualization provider, region, path to VM, snapshot name, action space, cache directory, screen size, headless mode, and accessibility tree requirements.

```python
class DesktopEnv:
    def __init__(self,
                 provider_name: str = "vmware",
                 region: str = None,
                 path_to_vm: str = None,
                 snapshot_name: str = "init_state",
                 action_space: str = "pyautogui",
                 cache_dir: str = "cache",
                 screen_size: Tuple[int] = (int(os.environ.get("SCREEN_WIDTH", 1920)), int(os.environ.get("SCREEN_HEIGHT", 1080))),
                 headless: bool = False,
                 require_a11y_tree: bool = True,
                 require_terminal: bool = False,
                 os_type: str = "Ubuntu",
                 enable_proxy: bool = False,
                 client_password: str = "",
                 **kwargs):
        pass
```

--------------------------------

### Install wmctrl for Window Manager Control

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This command installs the 'wmctrl' package, which is necessary for controlling the window manager within the virtual machine. Without wmctrl, certain experiments requiring window manipulation may not function correctly.

```bash
sudo apt install wmctrl
```

--------------------------------

### Check and Manage System Services with systemctl

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

Demonstrates how to check the status of a system service, such as 'unattended-upgrades.service', using the systemctl command. It also shows how to disable and stop a service.

```bash
sudo systemctl status unattended-upgrades.service
```

```bash
sudo systemctl disable unattended-upgrades
sudo systemctl stop unattended-upgrades
```

--------------------------------

### Install socat for Port Forwarding

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This command installs the 'socat' utility, which is essential for enabling port forwarding. Socat is a command-line utility that establishes bidirectional data transfers between two data sources, commonly used for network operations like port forwarding.

```bash
sudo apt install socat
```

--------------------------------

### Run and Customize VM Image with Docker

Source: https://timothyxxx.github.io/OSWorld/installation/install_provider/docker

This command runs a Docker container for a VM, allowing for customization. It sets disk size, RAM, CPU cores, mounts a persistent volume for the VM image, adds network administration capabilities, maps KVM device, and exposes ports for VNC and other services.

```Shell
docker run -it --rm \
  -e "DISK_SIZE=64G" \
  -e "RAM_SIZE=8G" \
  -e "CPU_CORES=8" \
  --volume "$(pwd)/docker_vm_data/Ubuntu.qcow2:/boot.qcow2" \
  --cap-add NET_ADMIN \
  --device /dev/kvm \
  -p 8006:8006 \
  -p 5000:5000 \
  happysixd/osworld-docker
```

--------------------------------

### Check Xorg Session Type

Source: https://timothyxxx.github.io/OSWorld/user_guides/replicate_client_server

Verifies if the current Ubuntu session is using Xorg by checking the XDG_SESSION_TYPE environment variable.

```bash
echo $XDG_SESSION_TYPE
```

--------------------------------

### Crowdsourcing for Examples

Source: https://timothyxxx.github.io/OSWorld/_sources/community/vision_and_roadmap

Mechanisms will be established for the crowdsourcing and community contribution of new examples, facilitated by the open-source AgentNet Tool and dataset release.

```General
Set up ways for the crowdsourcing/community to contribute new examples (achieved via open-source AgentNet Tool and dataset release)
```

--------------------------------

### Configure Dummy Monitor and Screen for Xorg

Source: https://timothyxxx.github.io/OSWorld/_sources/user_guides/replicate_client_server

This INI configuration file, placed in `xorg.conf.d`, defines a dummy device and monitor for the Xorg server. It specifies video RAM and a modeline for a 1920x1080 resolution, essential for headless VNC setups.

```ini
Section "Device"
    Identifier "DummyDevice"
    Driver "dummy"
    VideoRam 32768
EndSection

Section "Monitor"
    Identifier "DummyMonitor"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
    Modeline "1920x1080" 172.80 1920 2048 2248 2576 1080 1083 1088 1120
EndSection

Section "Screen"
```