DesktopCtl

https://github.com/yaroshevych/desktopctl
Admin
DesktopCtl is a local CLI runtime that enables AI agents to observe and control computers via...
Tokens:20,059
Snippets:217
Trust Score:7.5
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Build and Test Commands Example

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLAUDE.md

An example of commands that might be used for building and testing the project. Adapt these to your specific project's needs.

```bash
# Example:
# npm install
# npm test
```

--------------------------------

### Start Debug Overlay

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Starts a visual debug overlay on the screen. Can be configured with a duration.

```bash
# start/stop debug overlay
desktopctl debug overlay start [--duration <ms>]
```

--------------------------------

### Install Dependencies

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/README.md

Installs project dependencies using uv. Use --extra grounding to enable optional local AI detector.

```bash
uv sync
```

```bash
uv sync --extra grounding
```

--------------------------------

### Fast Deterministic Pattern Example

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Recommended workflow for opening an app, capturing its window ID, and performing subsequent actions on that specific window with minimal latency.

```bash
# 1) open app and capture window id
raw="$(desktopctl app open Notes --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id // empty')"

# 2) reuse --active-window <id> for all subsequent commands
desktopctl keyboard press cmd+f --active-window "$win_id" --no-observe
desktopctl keyboard type "Shopping list" --active-window "$win_id" --no-observe
desktopctl pointer click --text "All" --active-window "$win_id" --no-observe

# 3) validate final state
desktopctl screen tokenize --active-window "$win_id"
```

--------------------------------

### Pointer and Keyboard Commands

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Examples of using desktopctl for pointer clicks and keyboard presses, including observation and timing options.

```APIDOC
## Pointer Click Commands

### Description
Simulates a mouse click at a specified location or on a UI element identified by text or ID.

### Method
`pointer click`

### Parameters
- `--text` (string) - Optional - The text of the UI element to click.
- `--id` (string) - Optional - The ID of the UI element to click.
- `--active-window` (string) - Required - The ID of the active window.
- `--observe-until` (string) - Optional - When to stop observing the UI (e.g., `stable`, `first-change`).
- `--observe-settle-ms` (integer) - Optional - Milliseconds to wait for UI to settle after observation.
- `--no-observe` - Optional - Disables UI observation for faster execution.

### Example
```bash
# Click element with text 'Submit' in a specific window, observing until stable
desktopctl pointer click --text "Submit" --active-window "$win_id"

# Click element with ID 'button_ok', tuning settle time
desktopctl pointer click --id button_ok --active-window "$win_id" --observe-until stable --observe-settle-ms 400
```

## Keyboard Commands

### Description
Simulates keyboard input, such as pressing keys or typing text.

### Method
`keyboard press` / `keyboard type`

### Parameters
- `enter` - For `keyboard press`, simulates pressing the Enter key.
- `--active-window` (string) - Required - The ID of the active window.
- `--observe-until` (string) - Optional - When to stop observing the UI (e.g., `first-change`).
- `--observe-timeout` (integer) - Optional - Timeout in milliseconds for observation.
- `cmd+f` - For `keyboard press`, simulates pressing Command+F.
- `"Shopping"` - For `keyboard type`, simulates typing the string "Shopping".

### Example
```bash
# Press Enter in a specific window, observing until the first UI change
desktopctl keyboard press enter --active-window "$win_id" --observe-until first-change --observe-timeout 5000

# Type text into a window without observing
desktopctl keyboard type "Shopping" --active-window "$win_id" --no-observe
```
```

--------------------------------

### Automate Application Interaction with DesktopCtl

Source: https://github.com/yaroshevych/desktopctl/blob/main/README.md

Example of opening an application, finding its window, and typing text into it. Requires `jq` for JSON parsing. Ensure the application is compatible with DesktopCtl's window identification.

```bash
raw="$(desktopctl app open Notes --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id // empty')"
desktopctl keyboard press cmd+f --active-window "$win_id" --no-observe
desktopctl keyboard type "Shopping list" --active-window "$win_id" --no-observe
desktopctl screen tokenize --active-window "$win_id"
```

--------------------------------

### Start Replay Recording

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Initiates a replay recording with an optional explicit duration. The maximum duration is 30 minutes (1800000ms).

```bash
desktopctl replay record --duration <ms>
```

--------------------------------

### Replay Recording

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Command to start replay recording for trace capture and replay artifacts.

```APIDOC
## Replay

### record
Starts replay recording (default duration: 3000ms).
*Use only if you explicitly need trace capture/replay artifacts.*

**Usage:**
`desktopctl replay record`
```

--------------------------------

### Record Replay Trace

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Starts recording a trace for replay purposes. The default duration is 3000ms. Use this only if explicit trace capture is needed.

```bash
# start replay recording (default duration: 3000ms)
# use only if you explicitly need trace capture/replay artifacts
desktopctl replay record
```

--------------------------------

### Generate Window Data from Screenshots

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/README.md

Generates window JSON, overlays, and crops from screenshots. Hybrid detection is default, falling back to OpenCV if GroundingDINO is not installed.

```bash
cd src/desktop/scripts
uv run dev/generate_settings_window_data.py \
  ../daemon/tests/fixtures/settings-screenshots \
  --write-overlays \
  --write-crops \
  --write-sips-crops
```

--------------------------------

### Get Window Bounds by Title

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Finds and returns the bounds (position and size) of a window based on its title.

```bash
# find window bounds by title/app text or exact window id
desktopctl window bounds --title <text>
```

--------------------------------

### Check for Ready Work with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Use this command to see unblocked issues that are ready for an agent to work on. Always check for ready work before starting a new task.

```bash
bd ready --json
```

--------------------------------

### Claim and Update Issues with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Claim an issue to start working on it, or update its priority. Always claim tasks atomically.

```bash
bd update <id> --claim --json
```

```bash
bd update bd-42 --priority 1 --json
```

--------------------------------

### Get Window Bounds by ID

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Finds and returns the bounds (position and size) of a window using its exact window ID.

```bash
desktopctl window bounds --id <id>
```

--------------------------------

### Drag Pointer

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Simulates a drag operation by moving the pointer from a starting coordinate to an ending coordinate. Optionally specifies a hold duration in milliseconds.

```bash
# drag pointer between coordinates
desktopctl pointer drag <x1> <y1> <x2> <y2> [hold_ms]
```

--------------------------------

### Get Window Bounds or Focus Window with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Retrieves the bounding box of a window based on a fuzzy title match or focuses a window using its exact ID. Both operations return results in JSON format.

```bash
# Get bounds by fuzzy title match
desktopctl window bounds --title "Shopping" --json

# Focus by exact window id
desktopctl window focus --id Notes_9123 --json
```

--------------------------------

### Show Application

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Shows and activates a specified application.

```bash
# show and activate an app
desktopctl app show <application>
```

--------------------------------

### Build and Run DesktopCtl

Source: https://github.com/yaroshevych/desktopctl/blob/main/README.md

Use this command to build the project and then run the daemon and CLI.

```bash
just build run
```

--------------------------------

### Configure VM Connection

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/vm/README.md

Set up VM connection details in the .env file. Ensure VM_HOST, VM_USER, and VM_WINDOW_APP are correctly configured.

```env
VM_HOST=<ssh-host>
VM_USER=<vm-username>
VM_WINDOW_APP=UTM
```

--------------------------------

### Get Request Response

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Retrieves the stored response envelope for a specific request ID.

```bash
desktopctl request response <request_id>
```

--------------------------------

### Enable Permissions Script

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/vm/README.md

Deploys DesktopCtl to the VM and grants Accessibility and Screen Recording permissions using host automation.

```shell
#!/bin/bash

# This script deploys DesktopCtl to the VM and grants Accessibility + Screen Recording permissions via host automation.
# It assumes SSH access to the VM is configured.

# Variables (can be set via environment or passed as arguments)
VM_HOST="${VM_HOST:-localhost}"
VM_USER="${VM_USER:-user}"

# Deploy DesktopCtl (example: using rsync or scp)
# rsync -avz --progress ./desktopctl/ "${VM_USER}@${VM_HOST}:~/DesktopCtl"

# Grant Permissions (example: using osascript via SSH)
# ssh "${VM_USER}@${VM_HOST}" "osascript -e 'tell application "System Events" to keystroke "a" using command down'"

echo "Permissions enabled successfully."
```

--------------------------------

### Open and Activate Application with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Launches an application, optionally waits for its window, and captures its window ID. Handles potential errors like the daemon not running or the target app not being found.

```bash
# Open Notes and capture its window id
raw="$(desktopctl app open Notes --wait --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id // empty')"
# JSON success response shape:
# { "ok": true, "api_version": "1", "request_id": "...", "result": { "window_id": "Notes_9123" } }

# Open Safari in background (no activation) with a 5s timeout
desktopctl app open Safari --background --wait --timeout 5000 --json

# Open a file in a specific app by passing open-args
desktopctl app open TextEdit -- /Users/me/notes.txt

# Handle failure (exit code 6 = DAEMON_NOT_RUNNING, 4 = TARGET_NOT_FOUND)
if ! desktopctl app open "NonExistentApp" --json; then
  echo "App not found or daemon not running"
fi
```

--------------------------------

### Pointer Click Actions with Observation Options

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Demonstrates different ways to perform a pointer click, including default observation, disabling observation for speed, observing until the first UI change, and tuning the settle window.

```bash
desktopctl pointer click --text "Submit" --active-window "$win_id"
```

```bash
desktopctl pointer click --text "Submit" --active-window "$win_id" --no-observe
```

```bash
desktopctl keyboard press enter --active-window "$win_id" \
  --observe-until first-change --observe-timeout 5000
```

```bash
desktopctl pointer click --id button_ok --active-window "$win_id" \
  --observe-until stable --observe-settle-ms 400
```

--------------------------------

### List Windows

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Lists all currently visible windows.

```bash
# list visible windows
desktopctl window list
```

--------------------------------

### Open Application with JSON Output

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/web/index.html

Use this command to open an application and receive its output in JSON format. Ensure the desktopctl CLI is accessible in your PATH.

```bash
desktopctl app open Notes --json
```

--------------------------------

### App and Window Management

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Commands for opening, isolating, hiding, and showing applications, as well as listing, finding bounds, and focusing windows.

```APIDOC
## App Management

### open
Opens an application. Optionally waits until it is ready or avoids activating it.

**Usage:**
`desktopctl app open <application> [--wait] [--timeout <ms>] [--background] [-- <open-args...>]`

### isolate
Hides other visible apps and activates the target app.

**Usage:**
`desktopctl app isolate <application>`

### hide
Hides an app if it is running.

**Usage:**
`desktopctl app hide <application>`

### show
Shows and activates an app.

**Usage:**
`desktopctl app show <application>`

## Window Management

### list
Lists visible windows.

**Usage:**
`desktopctl window list`

### bounds
Finds window bounds by title/app text or exact window id.

**Usage:**
`desktopctl window bounds --title <text>`
`desktopctl window bounds --id <id>`

### focus
Focuses a matching window by title/app text or exact window id.

**Usage:**
`desktopctl window focus --title <text>`
`desktopctl window focus --id <id>`
```

--------------------------------

### Recommended Agent Integration Pattern

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Illustrates a pattern for integrating DesktopCtl with AI agents, focusing on efficient window management and command execution.

```APIDOC
## Recommended Agent Integration Pattern

### Description
This pattern outlines how to use DesktopCtl with AI agents, particularly those lacking direct vision capabilities. It emphasizes obtaining and reusing window IDs for deterministic targeting and optimizing command execution speed by selectively disabling observation.

### Steps
1.  **Open Application and Get Window ID**: Launch an application and capture its unique window ID for subsequent commands.
2.  **Execute Fast Actions**: Perform a sequence of intermediate actions using `--no-observe` to minimize latency.
3.  **Capture Ground Truth**: Use `screen tokenize` at the end of an action sequence to provide the agent with the current state of the screen for its next decision.

### Example Workflow
```bash
# 1. Open Notes app and capture the window ID
raw="$(desktopctl app open Notes --wait --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id')"

# 2. Perform a series of fast actions without observing UI changes
desktopctl keyboard press cmd+f --active-window "$win_id" --no-observe
desktopctl keyboard type "Shopping" --active-window "$win_id" --no-observe
desktopctl pointer click --text "All" --active-window "$win_id" --no-observe

# 3. (Implicitly, after these actions, you might call screen tokenize for the agent's next step)
```
```

--------------------------------

### Open Application

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Opens an application. Optionally waits for it to be ready or avoids activating it. Supports passing additional arguments to the application's open command.

```bash
# open an app; optionally wait until it is ready or avoid activating it
desktopctl app open <application> [--wait] [--timeout <ms>] [--background] [-- <open-args...>]
```

--------------------------------

### Tokenize Phase 0 Script

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/vm/README.md

Sets up the environment lock by performing baseline screenshots and doctor checks before a tokenize run.

```shell
#!/bin/bash

# This script performs environment lock: baseline screenshots and doctor checks before a tokenize run.

echo "Performing baseline screenshots..."
# Add screenshot capture commands here

echo "Running doctor checks..."
# Add doctor check commands here

echo "Tokenize Phase 0 completed."
```

--------------------------------

### Recommended Agent Integration Pattern

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

A pattern for driving desktop automation from an AI agent, involving opening an app, reusing the window ID, performing fast actions without observation, and tokenizing the screen for agent decisions.

```bash
# 1. Open app → get window id
raw="$(desktopctl app open Notes --wait --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id')"

# 2. Fast action sequence — no observe overhead
desktopctl keyboard press cmd+f --active-window "$win_id" --no-observe
desktopctl keyboard type "Shopping" --active-window "$win_id" --no-observe
desktopctl pointer click --text "All" --active-window "$win_id" --no-observe
```

--------------------------------

### Load Replay Session

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Loads a replay session from a specified directory on disk.

```bash
desktopctl replay load <session_dir>
```

--------------------------------

### Beads Issue Tracker Quick Reference

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLAUDE.md

Provides essential commands for interacting with the Beads issue tracker. Use these for managing tasks and workflow.

```bash
bd ready              # Find available work
bd show <id>          # View issue details
bd update <id> --claim  # Claim work
bd close <id>         # Complete work
```

--------------------------------

### Run Export and Tokenizer Workflow

Source: https://github.com/yaroshevych/desktopctl/blob/main/eval/README.md

Commands to export labels from Label Studio, run the tokenizer, and import results back into Label Studio for evaluation.

```bash
# 1. Export accepted labels from LS into a run folder
just run-export 3 text_fields text_fields
# → datasets/runs/<timestamp>-text_fields/artifacts.csv

# 2. Run tokenizer on images listed in artifacts.csv
just run-tokenize datasets/runs/<timestamp>-text_fields
# → results/<label_id>/overlay.png + label.json + metadata.json

# 3. Import tokenizer results into LS (project name = run_id)
just run-import datasets/runs/<timestamp>-text_fields

# 4. Label in LS (accept / reject / follow_up)

# 5. Log metrics to MLflow + export annotations.csv
just run-log datasets/runs/<timestamp>-text_fields
```

--------------------------------

### Focus Window by Title

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Focuses (brings to front and activates) a window matching the specified title or application text.

```bash
# focus a matching window by title/app text or exact window id
desktopctl window focus --title <text>
```

--------------------------------

### List Visible Windows with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Enumerates all currently visible windows, providing their unique IDs, titles, associated applications, and pixel bounding box coordinates. The output is in JSON format.

```bash
desktopctl window list --json
# { "ok": true, ..., "result": { "windows": [
#   { "id": "Notes_9123", "title": "Shopping list", "app": "Notes",
#     "bounds": { "x": 100, "y": 80, "width": 800, "height": 600 } },
#   ...
# ] } }
```

--------------------------------

### Hygiene and Linting with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Use these commands for maintaining issue hygiene and linting the issue tracker.

```bash
bd stale
```

```bash
bd orphans
```

```bash
bd lint
```

--------------------------------

### Structured Workflows with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Manage structured workflows using bd commands for listing formulas and pouring moles.

```bash
bd formula list
```

```bash
bd mol pour <name>
```

--------------------------------

### Recording and Loading Replay Sessions

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for capturing timestamped trace sessions for debugging and regression testing, and for loading previously saved sessions.

```bash
# Record for 5 seconds
desktopctl replay record --duration 5000 --json
```

```bash
# Stop an active recording early
desktopctl replay record --stop --json
```

```bash
# Load a session from disk for replay
desktopctl replay load /tmp/desktopctl-replay/session-a --json
```

--------------------------------

### Hide or Show Application with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Hides or shows a specified application. The JSON response indicates the application and its new state (hidden or shown).

```bash
desktopctl app hide Calculator --json
# { "ok": true, ..., "result": { "app": "Calculator", "state": "hidden" } }

desktopctl app show Calculator --json
# { "ok": true, ..., "result": { "app": "Calculator", "state": "shown", "window_id": "..." } }
```

--------------------------------

### Managing Request Artifacts with desktopctl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for listing, showing, retrieving responses, exporting screenshots, and searching through stored request artifacts. Use --json for structured output.

```bash
# List last 5 requests
desktopctl request list --limit 5 --json
```

```bash
# Show metadata for a specific request
desktopctl request show 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --json
```

```bash
# Retrieve the full stored response for a request
desktopctl request response 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --json
```

```bash
# Export the stored screenshot to disk
desktopctl request screenshot 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --out /tmp/stored.png
```

```bash
# Fuzzy-search over stored tokenize responses
desktopctl request search "Settings" --limit 10 --command screen_tokenize --json
```

--------------------------------

### Create New Issues with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Use these commands to create new issues. Specify the title, description, type, and priority. Dependencies can be linked using 'discovered-from'.

```bash
bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json
```

```bash
bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --json
```

--------------------------------

### Capture Screenshot with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Captures a screenshot of the full display, active window, or a specified region. Saves the output as a PNG and returns metadata including the path, capture scope, and window bounds if applicable. Options for custom paths and debug overlays are available.

```bash
# Full display screenshot saved to default path
desktopctl screen screenshot --json

# Capture only the active window, save to custom path, include debug overlay
desktopctl screen screenshot --active-window "$win_id" --out /tmp/shot.png --overlay --json
# { "ok": true, ..., "result": {
#   "snapshot_id": 17, "timestamp": "...", "path": "/tmp/shot.png",
#   "capture_scope": "active_window",
#   "window_bounds": { "x": 100, "y": 80, "width": 800, "height": 600 },
#   "display": { "id": 1, "width": 2560, "height": 1440, "scale": 2 }
# }

# Capture a 400×300 sub-region at (50, 120) of the active window
desktopctl screen screenshot --active-window "$win_id" --region 50 120 400 300
```

--------------------------------

### Discover and Link New Work with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

When new work is discovered during development, create a linked issue using 'discovered-from' to maintain traceability.

```bash
bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:<parent-id>
```

--------------------------------

### Tokenize Screen Content

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Extracts text and element information from a screenshot or the active window. Use --screenshot for static images or --active-window for live analysis. Options include --json for structured output, --journal for stable output, --list-windows to see all windows, and --overlay to save a visual representation.

```bash
desktopctl screen tokenize --screenshot /tmp/shot.png --json
```

```bash
desktopctl screen tokenize --active-window "$win_id" --journal --json
```

```bash
desktopctl screen tokenize --active-window "$win_id" --list-windows --json
```

```bash
desktopctl screen tokenize --active-window "$win_id" --overlay /tmp/overlay.png
```

--------------------------------

### Replay Commands

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for recording and loading trace sessions for debugging and regression testing.

```APIDOC
## Replay

### Description
Commands to capture and replay trace sessions, useful for debugging and regression testing.

### Commands
- `desktopctl replay record [--duration <ms>] [--stop]`
- `desktopctl replay load <session_path>`

### Parameters
- `--duration` (integer) - Optional - Duration in milliseconds to record the session.
- `--stop` - Optional - Stops an active recording early.
- `--json` - Optional - Outputs results in JSON format.
- `<session_path>` - Required - The path to the session file or directory to load.

### Examples
```bash
# Record a trace session for 5 seconds
desktopctl replay record --duration 5000 --json

# Stop an ongoing recording session
desktopctl replay record --stop --json

# Load a recorded session from disk
desktopctl replay load /tmp/desktopctl-replay/session-a --json
```
```

--------------------------------

### Pointer Button Up

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Simulates releasing a mouse button at the specified X and Y coordinates.

```bash
desktopctl pointer up <x> <y>
```

--------------------------------

### Check Permissions

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Reports the status of Accessibility and Screen Recording permissions required by the daemon.

```bash
# report Accessibility / Screen Recording permission status
desktopctl debug permissions
```

--------------------------------

### Force Specific Detector for Window Data Generation

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/README.md

Forces the use of GroundingDINO detector on CPU for generating window data from a specific screenshot. Outputs to stdout.

```bash
uv run dev/generate_settings_window_data.py \
  ../daemon/tests/fixtures/settings-screenshots/dark-abstract-overlap.png \
  --detector grounding-dino \
  --device cpu \
  --stdout
```

--------------------------------

### App Management

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for managing applications, including opening, activating, and isolating them.

```APIDOC
## desktopctl app open

### Description
Launches and activates an application. Optionally waits until a window is ready and returns a stable `window_id` for subsequent commands.

### Method
`desktopctl app open [app_name]`

### Parameters
#### Path Parameters
- **app_name** (string) - Required - The name of the application to open.

#### Query Parameters
- **--wait** (boolean) - Optional - Wait until a window is ready.
- **--background** (boolean) - Optional - Open the app in the background without activating it.
- **--timeout** (integer) - Optional - Timeout in milliseconds for the operation.
- **--json** (boolean) - Optional - Output in JSON format.

#### Request Body
None

### Request Example
```bash
raw="$(desktopctl app open Notes --wait --json)"
win_id="$(printf '%s' "$raw" | jq -r '.result.window_id // empty')"
```

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **api_version** (string) - The API version.
- **request_id** (string) - A unique identifier for the request.
- **result** (object) - Contains the result of the operation.
  - **window_id** (string) - The stable ID of the opened window.

#### Response Example
```json
{
  "ok": true,
  "api_version": "1",
  "request_id": "...",
  "result": {
    "window_id": "Notes_9123"
  }
}
```
```

```APIDOC
## desktopctl app isolate

### Description
Focuses on a single application by hiding all others. Returns the list of hidden applications and the resolved `window_id`.

### Method
`desktopctl app isolate [app_name]`

### Parameters
#### Path Parameters
- **app_name** (string) - Required - The name of the application to isolate.

#### Query Parameters
- **--json** (boolean) - Optional - Output in JSON format.

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **result** (object) - Contains the result of the operation.
  - **app** (string) - The name of the isolated application.
  - **state** (string) - The state of the application (e.g., "isolated").
  - **hidden_apps** (array) - A list of application names that were hidden.
  - **window_id** (string) - The stable ID of the isolated application's window.

### Response Example
```json
{
  "ok": true,
  "result": {
    "app": "Notes",
    "state": "isolated",
    "hidden_apps": ["Safari", "Finder"],
    "window_id": "Notes_9123"
  }
}
```
```

```APIDOC
## desktopctl app hide / desktopctl app show

### Description
Commands to hide or show a specific application.

### Method
`desktopctl app hide [app_name]` or `desktopctl app show [app_name]`

### Parameters
#### Path Parameters
- **app_name** (string) - Required - The name of the application to hide or show.

#### Query Parameters
- **--json** (boolean) - Optional - Output in JSON format.

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **result** (object) - Contains the result of the operation.
  - **app** (string) - The name of the application.
  - **state** (string) - The new state of the application (e.g., "hidden" or "shown").
  - **window_id** (string) - The window ID if the app is shown.

### Response Example (hide)
```json
{
  "ok": true,
  "result": {
    "app": "Calculator",
    "state": "hidden"
  }
}
```

### Response Example (show)
```json
{
  "ok": true,
  "result": {
    "app": "Calculator",
    "state": "shown",
    "window_id": "..."
  }
}
```
```

--------------------------------

### Isolate Application

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Hides all other visible applications and activates the target application.

```bash
# hide other visible apps and activate target app
desktopctl app isolate <application>
```

--------------------------------

### Tokenize Phase 1 Capture Script

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/vm/README.md

Captures a screenshot corpus across various applications and themes on the VM.

```shell
#!/bin/bash

# This script captures screenshot corpus across apps and themes on the VM.

echo "Capturing screenshots for various apps and themes..."
# Add screenshot capture logic here, iterating through apps and themes

echo "Screenshot corpus captured."
```

--------------------------------

### Window Introspection

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for listing and inspecting application windows.

```APIDOC
## desktopctl window list

### Description
Enumerates all visible windows, providing their IDs, titles, associated applications, and pixel bounds.

### Method
`desktopctl window list`

### Parameters
#### Query Parameters
- **--json** (boolean) - Optional - Output in JSON format.

### Response
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **result** (object) - Contains the list of windows.
  - **windows** (array) - A list of window objects.
    - **id** (string) - The unique identifier for the window.
    - **title** (string) - The title of the window.
    - **app** (string) - The name of the application owning the window.
    - **bounds** (object) - The pixel coordinates and dimensions of the window.
      - **x** (integer) - The x-coordinate of the top-left corner.
      - **y** (integer) - The y-coordinate of the top-left corner.
      - **width** (integer) - The width of the window.
      - **height** (integer) - The height of the window.

### Response Example
```json
{
  "ok": true,
  "result": {
    "windows": [
      {
        "id": "Notes_9123",
        "title": "Shopping list",
        "app": "Notes",
        "bounds": {
          "x": 100,
          "y": 80,
          "width": 800,
          "height": 600
        }
      },
      ...
    ]
  }
}
```
```

```APIDOC
## desktopctl window bounds / desktopctl window focus

### Description
Retrieves the bounds of a window or focuses on a specific window.

### Method
`desktopctl window bounds [--title <title_pattern>]` or `desktopctl window focus --id <window_id>`

### Parameters
#### Query Parameters (for `bounds`)
- **--title** (string) - Optional - A pattern to fuzzy match the window title.

#### Query Parameters (for `focus`)
- **--id** (string) - Required - The exact window ID to focus on.

#### Common Query Parameters
- **--json** (boolean) - Optional - Output in JSON format.

### Response (for `bounds`)
#### Success Response (200)
- **ok** (boolean) - Indicates success.
- **result** (object) - Contains the bounds of the matched window.
  - **bounds** (object) - The pixel coordinates and dimensions of the window.
    - **x** (integer) - The x-coordinate of the top-left corner.
    - **y** (integer) - The y-coordinate of the top-left corner.
    - **width** (integer) - The width of the window.
    - **height** (integer) - The height of the window.

### Response Example (bounds)
```bash
# Get bounds by fuzzy title match
desktopctl window bounds --title "Shopping" --json
```

### Response Example (focus)
```bash
# Focus by exact window id
desktopctl window focus --id Notes_9123 --json
```
```

--------------------------------

### Debugging Tools for DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Utilities to verify daemon status, check permissions, dump snapshots, and manage the debug token overlay. All commands support --json output.

```bash
# Verify daemon is running
desktopctl debug ping --json
```

```bash
# Check accessibility and screen recording permission status
desktopctl debug permissions --json
```

```bash
# Dump a debug snapshot payload
desktopctl debug snapshot --json
```

```bash
# Show and hide the debug token overlay
desktopctl debug overlay start --duration 5000
```

```bash
desktopctl debug overlay stop
```

--------------------------------

### Smoke Test Script

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/desktop/scripts/vm/README.md

Runs a smoke test suite against the VM. Optionally runs the permission flow first.

```shell
#!/bin/bash

# This script runs a smoke test suite against the VM.
# It can optionally run the permission flow first.

# Variables
RUN_PERMISSION_FLOW=${RUN_PERMISSION_FLOW:-false}

if [ "$RUN_PERMISSION_FLOW" = true ]; then
  echo "Running permission flow first..."
  ./enable_permissions.sh
fi

echo "Running smoke tests..."
# Add smoke test commands here
# Example: 
# ssh "${VM_USER}@${VM_HOST}" "/path/to/your/smoke_tests.sh"

echo "Smoke tests completed."
```

--------------------------------

### Tokenize Screen Content with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Performs OCR and UI element extraction on a target window, returning structured tokens with semantic element IDs and bounding boxes. Element IDs are stable and can be used for direct interaction with other commands like `pointer click --id`.

```bash
# Tokenize the active window and print structured output
desktopctl screen tokenize --active-window "$win_id" --json
# { "ok": true, ..., "result": {
#   "snapshot_id": 42, "timestamp": "...",
#   "window_id": "Notes_9123",
#   "windows": [{

```

--------------------------------

### Type Text into Active Window

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/web/index.html

Simulates typing the specified text into the currently active window. This is useful for inputting search queries or form data.

```bash
desktopctl keyboard type "Shopping list"
```

--------------------------------

### Defer or Supersede Issues with bd

Source: https://github.com/yaroshevych/desktopctl/blob/main/AGENTS.md

Manage the issue lifecycle by deferring or superseding existing issues as needed.

```bash
bd defer <id>
```

```bash
bd supersede <id>
```

--------------------------------

### Type Keyboard Text

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Simulates typing the specified string of text.

```bash
# keyboard text and key/hotkey press
desktopctl keyboard type "text"
```

--------------------------------

### Debug Tools Commands

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for debugging the desktopctl daemon, including ping, permissions check, and snapshotting.

```APIDOC
## Debug Tools

### Description
Provides tools for debugging the desktopctl daemon, checking its status, permissions, and capturing debug information.

### Commands
- `desktopctl debug ping`
- `desktopctl debug permissions`
- `desktopctl debug snapshot`
- `desktopctl debug overlay start [--duration <ms>]`
- `desktopctl debug overlay stop`

### Parameters
- `--json` - Optional - Outputs results in JSON format.
- `--duration` (integer) - Optional - Duration in milliseconds for the debug overlay to be visible.

### Examples
```bash
# Verify the daemon is running
desktopctl debug ping --json

# Check accessibility and screen recording permissions
desktopctl debug permissions --json

# Dump a debug snapshot payload
desktopctl debug snapshot --json

# Start the debug token overlay for 5 seconds
desktopctl debug overlay start --duration 5000

# Stop the debug token overlay
desktopctl debug overlay stop
```
```

--------------------------------

### Pointer and Keyboard Controls

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Commands for controlling the mouse pointer (move, click, scroll, drag) and keyboard (type, press keys/hotkeys). Includes observe mode flags.

```APIDOC
## Pointer and Keyboard

*Common observe flags for pointer/keyboard actions: [--observe] [--no-observe] [--observe-until <stable|change|first-change>] [--observe-timeout <ms>] [--observe-settle-ms <ms>] [--active-window [<id>]] (optional frontmost-window guard for all pointer actions)*

### pointer move
Moves the pointer to the specified coordinates.

**Usage:**
`desktopctl pointer move <x> <y>`

### pointer down
Presses the pointer button at the specified coordinates.

**Usage:**
`desktopctl pointer down <x> <y>`

### pointer up
Releases the pointer button at the specified coordinates.

**Usage:**
`desktopctl pointer up <x> <y>`

### pointer click
Clicks the pointer by coordinate, OCR text, or element id.

**Usage:**
`desktopctl pointer click <x> <y> [--absolute]`
`desktopctl pointer click --text <text>`
`desktopctl pointer click --id <element_id> --active-window [<id>]`

### pointer scroll
Scrolls the pointer viewport or content by signed deltas (positive dy scrolls down).

**Usage:**
`desktopctl pointer scroll <dx> <dy>`
`desktopctl pointer scroll --id <element_id> <dx> <dy>`

### pointer drag
Drags the pointer between two coordinates.

**Usage:**
`desktopctl pointer drag <x1> <y1> <x2> <y2> [hold_ms]`

### keyboard type
Types the specified text.

**Usage:**
`desktopctl keyboard type "text"`

### keyboard press
Presses a specified key or hotkey.

**Usage:**
`desktopctl keyboard press <key-or-hotkey>`
```

--------------------------------

### Press Keyboard Key

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Simulates pressing a specific keyboard key or a hotkey combination.

```bash
desktopctl keyboard press <key-or-hotkey>
```

--------------------------------

### Tokenize Active Window State

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Use this command within an AI agent loop to capture the semantic state of the active window. The output is stable JSON, facilitating reliable error recovery and retry logic.

```bash
desktopctl screen tokenize --active-window "$win_id" --json
```

--------------------------------

### Request Artifacts Commands

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Commands for listing, showing, and retrieving request artifacts like responses and screenshots.

```APIDOC
## Request Artifacts

### Description
Commands to interact with the daemon's stored request artifacts, including lists, metadata, responses, and screenshots.

### Commands
- `desktopctl request list`
- `desktopctl request show [request_id]`
- `desktopctl request response [request_id]`
- `desktopctl request screenshot [request_id]`
- `desktopctl request search [query]`

### Parameters
- `--limit` (integer) - Optional - Limits the number of results.
- `--json` - Optional - Outputs results in JSON format.
- `--out` (string) - Optional - Specifies the output file path for screenshots.
- `--command` (string) - Optional - Filters search results by command.

### Examples
```bash
# List the last 5 requests in JSON format
desktopctl request list --limit 5 --json

# Show metadata for a specific request ID
desktopctl request show 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --json

# Retrieve the full stored response for a request
desktopctl request response 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --json

# Export a stored screenshot to a file
desktopctl request screenshot 019d4a91-7c0a-79b1-8ee4-90a3acbf3ab3 --out /tmp/stored.png

# Fuzzy-search for 'Settings' in tokenized responses
desktopctl request search "Settings" --limit 10 --command screen_tokenize --json
```
```

--------------------------------

### Tokenize Active Window Content

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/web/index.html

Captures the content of the active window and tokenizes it for AI processing. This command is essential for agents to understand and interact with the UI.

```bash
desktopctl screen tokenize --active-window
```

--------------------------------

### desktopctl clipboard read / clipboard write

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Reads the current content of the system clipboard or writes a specified string to the clipboard.

```APIDOC
### desktopctl clipboard read / clipboard write

### Description
Reads the current content of the system clipboard or writes a specified string to the clipboard.

### Usage
```bash
desktopctl clipboard read --json
# { "ok": true, ..., "result": { "text": "current clipboard content" } }

desktopctl clipboard write "Hello, agent!" --json
```
```

--------------------------------

### Ping Daemon

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Checks if the DesktopCtl daemon is running and responsive.

```bash
# check daemon connectivity
desktopctl debug ping
```

--------------------------------

### Hide Application

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Hides a specified application if it is currently running.

```bash
# hide an app if it is running
desktopctl app hide <application>
```

--------------------------------

### Press Keyboard Shortcut in Active Window

Source: https://github.com/yaroshevych/desktopctl/blob/main/src/web/index.html

This command simulates pressing a keyboard shortcut, such as Command+F, specifically within the currently active window. It's useful for triggering application-specific shortcuts.

```bash
desktopctl keyboard press cmd+f --active-window
```

--------------------------------

### Read Clipboard

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Reads and returns the current content of the system clipboard.

```bash
# clipboard operations
desktopctl clipboard read
```

--------------------------------

### Isolate Application Window with DesktopCtl

Source: https://context7.com/yaroshevych/desktopctl/llms.txt

Hides all other visible applications and activates the specified target application. Returns a list of hidden applications and the resolved window ID.

```bash
raw="$(desktopctl app isolate Notes --json)"
# { "ok": true, ..., "result": { "app": "Notes", "state": "isolated",
#   "hidden_apps": ["Safari", "Finder"], "window_id": "Notes_9123" } }
```

--------------------------------

### Focus Window by ID

Source: https://github.com/yaroshevych/desktopctl/blob/main/CLI.md

Focuses (brings to front and activates) a window using its exact window ID.

```bash
desktopctl window focus --id <id>
```
Tokens:
Raw