OpenAdapt

https://docs.openadapt.ai/
Admin
OpenAdapt is an open-source platform for GUI automation with machine learning that enables recording...
Tokens:18,413
Snippets:248
Trust Score:6.9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Project Setup and Local Development

Source: https://docs.openadapt.ai/packages/openadapt-wright

Commands to install dependencies, build the project, run tests, set environment variables, and start the worker and Telegram bot locally.

```bash
# Prerequisites: Node.js 22+, pnpm 9+
pnpm install
pnpm build

# Run tests
pnpm --filter @wright/worker test

# Set environment variables
export SUPABASE_URL=https://your-project.supabase.co
export SUPABASE_SERVICE_ROLE_KEY=your-key
export ANTHROPIC_API_KEY=sk-ant-your-key

# Run the worker locally
pnpm --filter @wright/worker dev

# Run the Telegram bot locally
export BOT_TOKEN=your-telegram-bot-token
pnpm --filter @wright/bot dev

```

--------------------------------

### Quick Start CLI Commands

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Commands for installing the environment, recording sessions, and managing the capture lifecycle.

```bash
# Install
git clone https://github.com/OpenAdaptAI/openadapt-desktop.git
cd openadapt-desktop
uv sync

# Record a session
uv run openadapt record --task "Demo task"    # Ctrl+C to stop

# Scrub PII, review, and upload
uv run openadapt scrub <CAPTURE_ID> --level basic
uv run openadapt approve <CAPTURE_ID>
uv run openadapt upload <CAPTURE_ID> --backend s3

# Other commands
uv run openadapt list                         # List captures
uv run openadapt review                       # Show pending reviews
uv run openadapt storage                      # Show disk usage
uv run openadapt health                       # Show memory/disk health
uv run openadapt cleanup                      # Enforce storage limits
uv run openadapt config                       # Show current configuration
uv run openadapt backends                     # Show available backends
```

--------------------------------

### Setup WAA Task: Install Apps on Windows VM

Source: https://docs.openadapt.ai/packages/openadapt-evals

Send a POST request to the WAA setup endpoint to install specified applications on the Windows VM. This uses a two-phase pipeline to handle large installers.

```bash
# Install missing apps via two-phase pipeline
curl -X POST http://localhost:5051/setup \
  -H "Content-Type: application/json" \
  -d '{"config": [{"type": "install_apps", "parameters": {"apps": ["libreoffice-calc"]}}]}'
```

--------------------------------

### Quick Start CLI Commands

Source: https://docs.openadapt.ai/packages/openadapt-herald

Commands for installing, configuring, and running the Herald pipeline.

```bash
# Install
pip install herald-announce

# Configure (or set env vars)
cp .env.example .env
# Edit .env with your API keys

# Preview what would be posted
herald preview --repos owner/repo --days 7

# Actually post to configured platforms
herald publish --repos owner/repo --content-type release
```

--------------------------------

### Development Setup for OpenAdapt Consilium

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Instructions for cloning the repository, installing development dependencies, and running tests. This setup is for contributing to or developing the Consilium library.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-consilium.git
cd openadapt-consilium
pip install -e ".[dev]"
pytest

```

--------------------------------

### Install OpenAdapt Capture

Source: https://docs.openadapt.ai/packages/openadapt-capture

Install the base package or the version with audio support.

```bash
uv add openadapt-capture
```

```bash
uv add "openadapt-capture[audio]"
```

--------------------------------

### Setup WAA Task: Verify Apps on Windows VM

Source: https://docs.openadapt.ai/packages/openadapt-evals

Send a POST request to the WAA setup endpoint to verify if specified applications are installed on the Windows VM. Returns 200 if all apps are present, 422 if any are missing.

```bash
# Check if required apps are installed on the Windows VM
curl -X POST http://localhost:5051/setup \
  -H "Content-Type: application/json" \
  -d '{"config": [{"type": "verify_apps", "parameters": {"apps": ["libreoffice-calc"]}}]}'
```

--------------------------------

### Install openadapt-consilium

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Install the library using pip. For development, clone the repository and install in editable mode.

```bash
pip install openadapt-consilium
```

```bash
git clone https://github.com/OpenAdaptAI/openadapt-consilium.git
cd openadapt-consilium
pip install -e ".[dev]"
```

--------------------------------

### Setup Development Environment

Source: https://docs.openadapt.ai/packages/openadapt-evals

Commands to clone the repository, synchronize dependencies, and run the test suite.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-evals.git
cd openadapt-evals
uv sync --extra dev
uv run pytest tests/ -v
```

--------------------------------

### Development Setup

Source: https://docs.openadapt.ai/packages/openadapt-herald

Commands to clone the repository and set up the development environment.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-herald.git
cd openadapt-herald
uv sync --extra dev
uv run pytest -v
uv run ruff check .
```

--------------------------------

### Clone and Install Dev Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-ml

Clone the repository and install development and training dependencies using uv.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-ml.git
cd openadapt-ml
uv sync --extra dev --extra training
```

--------------------------------

### Install Legacy OpenAdapt Version

Source: https://docs.openadapt.ai/packages/openadapt

Use this command to install the monolithic legacy version (v0.46.0) of OpenAdapt. Refer to the migration guide for details.

```bash
pip install openadapt==0.46.0
```

--------------------------------

### Record a GUI demonstration

Source: https://docs.openadapt.ai/packages/openadapt

Start a recording session with a specified name. Stop the recording by pressing Ctrl+C.

```bash
openadapt capture start --name my-task
# Perform actions in your GUI, then press Ctrl+C to stop
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Clone the OpenAdapt Desktop repository and install Python dependencies using uv. Ensure you have Python 3.11+ and uv installed.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-desktop.git
cd openadapt-desktop

# Install Python dependencies
uv sync --extra dev
```

--------------------------------

### Install All Optional Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Install all available optional dependencies for a complete feature set, including enterprise, community, and federated learning.

```bash
uv sync --extra full
```

--------------------------------

### Install OpenAdapt packages

Source: https://docs.openadapt.ai/packages/openadapt

Install the core CLI or specific feature sets using pip. Requires Python 3.10+.

```bash
pip install openadapt              # Minimal CLI only
pip install openadapt[capture]     # GUI capture/recording
pip install openadapt[ml]          # ML training and inference
pip install openadapt[evals]       # Benchmark evaluation
pip install openadapt[privacy]     # PII/PHI scrubbing
pip install openadapt[all]         # Everything
```

--------------------------------

### Install openadapt-ml Package

Source: https://docs.openadapt.ai/packages/openadapt-ml

Install the core package or with additional dependencies for training or API-backed VLMs. Development can be done from source.

```bash
pip install openadapt-ml

# With training dependencies (TRL + datasets)
pip install openadapt-ml[training]

# With API-backed VLMs (Claude, GPT)
pip install openadapt-ml[api]

# Development (from source)
git clone https://github.com/OpenAdaptAI/openadapt-ml.git
cd openadapt-ml
uv sync

```

--------------------------------

### Development Setup and Testing

Source: https://docs.openadapt.ai/packages/openadapt-capture

Synchronizes development environment and runs tests. Includes an option to run slow integration tests which require accessibility permissions.

```bash
uv sync --dev
```

```bash
uv run pytest tests/ -v --ignore=tests/test_browser_bridge.py
```

```bash
# Run slow integration tests (requires accessibility permissions)
uv run pytest tests/ -v -m slow
```

--------------------------------

### Install openadapt-evals

Source: https://docs.openadapt.ai/packages/openadapt-evals

Install the openadapt-evals package using pip. Optional dependencies can be installed for specific features like training, cloud management, or retrieval agents.

```bash
pip install openadapt-evals
```

```bash
pip install openadapt-evals[training]   # GRPO trainer + Outlines constrained decoding
pip install openadapt-evals[azure]      # Azure VM management
pip install openadapt-evals[aws]        # AWS EC2 management
pip install openadapt-evals[retrieval]  # Demo retrieval agent
pip install openadapt-evals[viewer]     # Live results viewer
pip install openadapt-evals[all]        # Everything
```

--------------------------------

### Consilium CLI Usage Examples

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Examples demonstrating various ways to use the Consilium CLI. Includes running the full pipeline, fast mode, with image input, budget limits, JSON output for piping, and custom model configurations.

```bash
# Full 3-stage pipeline
consilium "Compare Python and Rust for CLI tools"

```

```bash
# Fast mode (Stage 1 only)
consilium "Summarize this" --no-review

```

```bash
# With screenshot
consilium "What's on this screen?" --image screenshot.png

```

```bash
# Budget-limited
consilium "Write a haiku about AI" --budget 0.10

```

```bash
# JSON output for piping
consilium "List 3 colors" --json | jq '.final_answer'

```

```bash
# Custom models
consilium "Hello" --models gpt-5.2,gemini-3.1-pro --chairman gpt-5.2

```

--------------------------------

### AWS Configuration: Example ~/.aws/config for SSO

Source: https://docs.openadapt.ai/packages/openadapt-evals

Example configuration file for AWS SSO, specifying the SSO session details, account ID, role, and region.

```ini
[default]
sso_session = my-org
sso_account_id = 111122223333
sso_role_name = PowerUserAccess
region = us-east-1

[sso-session my-org]
sso_start_url = https://my-org.awsapps.com/start
sso_region = us-east-1
sso_registration_scopes = sso:account:access
```

--------------------------------

### Install openadapt-crier

Source: https://docs.openadapt.ai/packages/openadapt-crier

Install the openadapt-crier package using pip. Ensure all required environment variables are set before running the bot.

```bash
pip install .

# Required env vars
export CRIER_TELEGRAM_BOT_TOKEN="your-telegram-bot-token"
export CRIER_TELEGRAM_OWNER_ID="your-telegram-user-id"
export CRIER_REPOS="owner/repo1,owner/repo2"
export ANTHROPIC_API_KEY="your-anthropic-key"

# Optional: platform credentials for posting
export DISCORD_WEBHOOK_URL="..."
export TWITTER_CONSUMER_KEY="..."
export TWITTER_CONSUMER_SECRET="..."
export TWITTER_ACCESS_TOKEN="..."
export TWITTER_ACCESS_TOKEN_SECRET="..."
export LINKEDIN_ACCESS_TOKEN="..."

# Start the bot
crier run

```

--------------------------------

### Install Optional Community Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Install optional dependencies for community features, specifically for HuggingFace Hub integration.

```bash
uv sync --extra community
```

--------------------------------

### Install Optional Federated Learning Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Install optional dependencies for federated learning, including Flower and PyTorch.

```bash
uv sync --extra federated
```

--------------------------------

### Clone and Install Sub-package

Source: https://docs.openadapt.ai/packages/openadapt

Clone a specific sub-package repository and install it with development dependencies. This is for contributing to individual components of OpenAdapt.

```bash
git clone https://github.com/OpenAdaptAI/openadapt-ml  # or other sub-package
cd openadapt-ml
pip install -e ".[dev]"
```

--------------------------------

### Install Optional Enterprise Dependencies

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Install optional dependencies required for enterprise features, including S3, R2, and MinIO storage backends using boto3.

```bash
uv sync --extra enterprise
```

--------------------------------

### AWS Authentication: SSO Setup

Source: https://docs.openadapt.ai/packages/openadapt-evals

Configure AWS authentication using SSO (IAM Identity Center) for interactive use. This involves a one-time setup and a login command that caches short-lived tokens.

```bash
# One-time setup — opens a guided wizard
aws configure sso
# Prompts for: SSO start URL, region, account, role name, profile name

# Login (opens browser, caches short-lived token)
aws sso login

# Verify it works
oa-vm smoke-test-aws

# All oa-vm --cloud aws commands now work automatically
oa-vm pool-create --cloud aws --workers 1
```

--------------------------------

### Query AI with Images using Consilium SDK

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Use the dict-based interface for AI agents and automated pipelines. This example shows how to analyze a screenshot and list UI elements using `council_query` with image input and multiple models. Set `skip_review=True` for faster processing.

```python
from consilium import council_query

result = council_query(
    "Analyze this screenshot and list the UI elements",
    images=[screenshot_bytes],
    models=["gpt-5.2", "claude-sonnet-4-6"],
    budget=0.25,
    skip_review=True,  # fast mode: Stage 1 only
)

print(result["final_answer"])
print(result["cost"]["total_usd"])

```

--------------------------------

### Run openadapt-crier tests

Source: https://docs.openadapt.ai/packages/openadapt-crier

Install development dependencies and run tests using pytest. This ensures all modules are functioning correctly.

```bash
pip install ".[dev]"
pytest

```

--------------------------------

### Python API Usage

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Instantiate the Council and use the ask method to get a consensus answer. Print the final answer and cost summary.

```python
from consilium import Council

council = Council()
result = council.ask("What are the key differences between REST and GraphQL?")

print(result.final_answer)
print(result.cost_summary())
```

--------------------------------

### UNIX Socket Bridge for Docker Port 5050 Workaround

Source: https://docs.openadapt.ai/packages/openadapt-evals

This command bridges the container's network namespace to a local UNIX socket, which is necessary because Docker's standard port forwarding conflicts with QEMU's TAP networking setup for port 5050.

```bash
# Stage 1: Bridge container network namespace to a UNIX socket
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' <container_name>)
nsenter -t $CONTAINER_PID -n socat UNIX-LISTEN:/tmp/waa-bridge.sock,fork TCP:localhost:5050
```

--------------------------------

### Run Model-Free Policy Demo

Source: https://docs.openadapt.ai/packages/openadapt-ml

Execute a model-free policy demonstration that does not require a GPU. This is useful for a quick smoke test.

```bash
# Model-free policy demo (no GPU required)
uv run python -m openadapt_ml.scripts.demo_policy --backend dummy

```

--------------------------------

### CLI Usage

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Use the command-line interface to ask questions directly.

```bash
consilium "What are the key differences between REST and GraphQL?"
```

--------------------------------

### OpenAdapt CLI reference

Source: https://docs.openadapt.ai/packages/openadapt

Common commands for managing captures, training, evaluation, and system diagnostics.

```bash
openadapt capture start --name <name>    Start recording
openadapt capture stop                    Stop recording
openadapt capture list                    List captures
openadapt capture view <name>             Open capture viewer

openadapt train start --capture <name>    Train model on capture
openadapt train status                    Check training progress
openadapt train stop                      Stop training

openadapt eval run --checkpoint <path>    Evaluate trained model
openadapt eval run --agent api-claude     Evaluate API agent
openadapt eval mock --tasks 10            Run mock evaluation

openadapt serve --port 8080               Start dashboard server
openadapt version                         Show installed versions
openadapt doctor                          Check system requirements
```

--------------------------------

### End-to-End Benchmark

Source: https://docs.openadapt.ai/packages/openadapt-ml

Run an end-to-end benchmark that includes training, evaluation, and plotting for the Qwen login scenario.

```bash
uv run python -m openadapt_ml.scripts.run_qwen_login_benchmark \
  --config configs/qwen3vl_synthetic_dev.yaml \
  --out-dir experiments/qwen_login/2b_dev

```

--------------------------------

### Serve Dedicated Grounder Endpoint (UI-Venus)

Source: https://docs.openadapt.ai/packages/openadapt-evals

Serve the UI-Venus model on a GPU for enhanced click accuracy. Verify the endpoint is running and then use it in correction or evaluation workflows.

```bash
bash scripts/serve_ui_venus.sh
```

```bash
curl http://gpu-host:8000/v1/models
```

```bash
python scripts/run_correction_flywheel.py \
    --task-config example_tasks/clear-browsing-data-chrome.yaml \
    --demo-dir ./demos \
    --grounder-endpoint http://gpu-host:8000
```

```bash
python scripts/run_full_eval.py \
    --server-url http://localhost:5001 \
    --grounder-endpoint http://gpu-host:8000
```

--------------------------------

### Train Model with Lambda Labs

Source: https://docs.openadapt.ai/packages/openadapt-ml

Launch, train, download, and terminate a model training job on Lambda Labs.

```bash
export LAMBDA_API_KEY=your_key_here

# One-command: launch, train, download, terminate
uv run python -m openadapt_ml.cloud.lambda_labs train \
  --capture ~/captures/my-workflow \
  --goal "Turn off Night Shift in System Settings"
```

--------------------------------

### Record and Annotate Demos

Source: https://docs.openadapt.ai/packages/openadapt-evals

Record user interactions on a remote VM for demo-conditioned evaluation. Includes pre-flight checks, interactive recording, and annotation using a VLM.

```bash
python scripts/record_waa_demos.py record-waa \
  --tasks 04d9aeaf,0a0faba3 \
  --server http://localhost:5001 \
  --verify
```

```bash
python scripts/record_waa_demos.py record-waa \
  --tasks 04d9aeaf,0a0faba3 \
  --server http://localhost:5001 \
  --output waa_recordings/
```

```bash
python scripts/record_waa_demos.py annotate \
  --recordings waa_recordings/ \
  --output annotated_demos/ \
  --provider openai
```

--------------------------------

### Train on Synthetic Data

Source: https://docs.openadapt.ai/packages/openadapt-ml

Fine-tune a Qwen3-VL model on a synthetic login scenario using a specified configuration file.

```bash
# Fine-tune Qwen3-VL on synthetic login scenario
uv run python -m openadapt_ml.scripts.train \
  --config configs/qwen3vl_synthetic.yaml

```

--------------------------------

### Run a live evaluation

Source: https://docs.openadapt.ai/packages/openadapt-evals

Initiate a live evaluation against a WAA (Windows Agent Arena) server. This command requires a running WAA server to connect to.

```bash
openadapt-evals live --tasks 10
```

--------------------------------

### Create and Wait for VM Pool (Azure/AWS)

Source: https://docs.openadapt.ai/packages/openadapt-evals

Create a virtual machine pool with a specified number of workers. Use `--cloud aws` for AWS. Then, wait for the pool to become available.

```bash
oa-vm pool-create --workers 1
oa-vm pool-wait
```

```bash
oa-vm pool-create --cloud aws --workers 1
oa-vm pool-wait --cloud aws
```

--------------------------------

### Run a mock evaluation

Source: https://docs.openadapt.ai/packages/openadapt-evals

Execute a mock evaluation without requiring a virtual machine. This is useful for initial testing or demonstrations.

```bash
openadapt-evals mock --tasks 10
```

--------------------------------

### Train on Real Recordings

Source: https://docs.openadapt.ai/packages/openadapt-ml

Train a model using real user recordings captured with openadapt-capture. This command also opens the training dashboard in the browser.

```bash
# Record a workflow with openadapt-capture, then train
uv run python -m openadapt_ml.scripts.train \
  --config configs/qwen3vl_capture.yaml \
  --capture ~/captures/my-workflow \
  --open  # Opens training dashboard in browser

```

--------------------------------

### Initialize Council with Custom Parameters

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Customize the models used, the chairman model for synthesis, and the maximum number of parallel workers.

```python
from consilium import Council

council = Council(
    models=["gpt-5.2", "claude-sonnet-4-6", "gemini-3.1-pro"],
    chairman="claude-sonnet-4-6",
    max_workers=8,
)
```

--------------------------------

### Train an ML model

Source: https://docs.openadapt.ai/packages/openadapt

Initiate training on a previously recorded capture using a specified model.

```bash
openadapt train start --capture my-task --model qwen3vl-2b
```

--------------------------------

### Deploy Worker to Fly.io

Source: https://docs.openadapt.ai/packages/openadapt-wright

Instructions for deploying the worker application to Fly.io, including details on its automatic scaling and shutdown behavior.

```bash
# Deploy worker
cd apps/worker
fly deploy

# The worker automatically:
# - Starts on HTTP request (Fly.io auto-start)
# - Polls Supabase for queued jobs
# - Shuts down after 5 minutes idle (scale-to-zero)
# - Re-queues jobs on SIGTERM (graceful shutdown)

```

--------------------------------

### End-to-End Evaluation Pipeline

Source: https://docs.openadapt.ai/packages/openadapt-evals

Automate the entire evaluation flow, including demo generation, VM management, and evaluation. Supports various configurations like cloud provider, specific tasks, and deterministic desktop settings.

```bash
python scripts/run_eval_pipeline.py
```

```bash
python scripts/run_eval_pipeline.py --tasks 04d9aeaf
```

```bash
python scripts/run_eval_pipeline.py --tasks 04d9aeaf --dry-run
```

```bash
python scripts/run_eval_pipeline.py --cloud aws --vm-name waa-pool-00
```

```bash
python scripts/run_eval_pipeline.py \
  --tasks 04d9aeaf \
  --clean-desktop \
  --force-tray-icons \
  --waa-image-version win11-24h2-2026-03-04
```

--------------------------------

### View recordings

Source: https://docs.openadapt.ai/packages/openadapt

Open the viewer for a specific capture session.

```bash
openadapt capture view my-task
```

--------------------------------

### CLI Command Reference

Source: https://docs.openadapt.ai/packages/openadapt-herald

Summary of available CLI commands for managing the Herald pipeline.

```bash
herald collect   # Gather and display artifacts
herald compose   # Generate content (no posting)
herald preview   # Alias for compose
herald publish   # Full pipeline: collect → compose → post
```

--------------------------------

### Run Demo-Conditioned Evaluation

Source: https://docs.openadapt.ai/packages/openadapt-evals

Execute evaluations based on recorded and annotated demos. Specify demo directories and tasks.

```bash
python scripts/record_waa_demos.py eval \
  --demo_dir annotated_demos/ \
  --tasks 04d9aeaf,0a0faba3
```

--------------------------------

### GRPO Training with TRL

Source: https://docs.openadapt.ai/packages/openadapt-evals

Recommended RL training using TRL's GRPOTrainer. Supports various configurations including Unsloth, constrained decoding, and Weave tracing. Mock mode is available for pipeline validation without hardware.

```python
python scripts/train_trl_grpo.py \
    --task-dir ./example_tasks \
    --server-url http://localhost:5001 \
    --model Qwen/Qwen2.5-VL-7B-Instruct \
    --output ./grpo_output
```

```python
python scripts/train_trl_grpo.py \
    --task-dir ./example_tasks \
    --server-url http://localhost:5001 \
    --model Qwen/Qwen2.5-VL-7B-Instruct \
    --use-unsloth \
    --constrained-decoding \
    --output ./grpo_output
```

```python
python scripts/train_trl_grpo.py \
    --task-dir ./example_tasks \
    --mock \
    --output ./grpo_output_mock
```

```python
python scripts/train_trl_grpo.py \
    --task-dir ./example_tasks \
    --server-url http://localhost:5001 \
    --model Qwen/Qwen2.5-VL-7B-Instruct \
    --weave-project openadapt-grpo \
    --output ./grpo_output
```

--------------------------------

### Council Ask Method with Options

Source: https://docs.openadapt.ai/packages/openadapt-consilium

The ask method accepts optional parameters for images, budget, system prompts, skipping review, and requesting JSON output.

```python
result = council.ask(
    "Your question here",
    images=[open("screenshot.png", "rb").read()],  # optional
    budget=0.50,        # max USD spend
    system="Be concise",  # system prompt for all models
    skip_review=False,  # skip Stages 2-3
    json_schema={...},  # request JSON output
)
```

--------------------------------

### OpenAdapt AI Project Architecture Overview

Source: https://docs.openadapt.ai/packages/openadapt-evals

This tree displays the directory structure of the OpenAdapt AI project, detailing the organization of agents, adapters, infrastructure management, evaluation utilities, benchmarks, and deployment tools.

```tree
openadapt_evals/
├── agents/               # Agent implementations
│   ├── base.py           #   BenchmarkAgent ABC
│   ├── api_agent.py      #   ApiAgent (Claude, GPT)
│   ├── claude_computer_use_agent.py  # ClaudeComputerUseAgent (coord clamping, fail-safe)
│   ├── retrieval_agent.py#   RetrievalAugmentedAgent
│   └── policy_agent.py   #   PolicyAgent (trained models)
├── adapters/             # Benchmark adapters
│   ├── base.py           #   BenchmarkAdapter ABC + data classes
│   ├── rl_env.py         #   RLEnvironment (Gymnasium-style wrapper for GRPO/PPO)
│   └── waa/              #   WAA live, mock, and local adapters
├── infrastructure/       # Cloud VM and pool management
│   ├── azure_vm.py       #   AzureVMManager
│   ├── aws_vm.py         #   AWSVMManager
│   ├── vm_provider.py    #   VMProvider protocol (multi-cloud abstraction)
│   ├── pool.py           #   PoolManager
│   ├── probe.py          #   4-layer WAA probe (screenshot, a11y, action, score)
│   ├── ssh_tunnel.py     #   SSHTunnelManager
│   └── vm_monitor.py     #   VMMonitor dashboard
├── evaluation/           # Shared evaluation utilities
│   └── metrics.py        #   fuzzy_match and scoring functions
├── benchmarks/           # Evaluation runner, CLI, viewers
│   ├── runner.py         #   evaluate_agent_on_benchmark()
│   ├── cli.py            #   Benchmark CLI (run, mock, live, view, probe)
│   ├── vm_cli.py         #   VM/Pool CLI (oa-vm, 50+ commands)
│   ├── viewer.py         #   HTML results viewer
│   ├── pool_viewer.py    #   Pool results viewer
│   └── trace_export.py   #   Training data export
├── waa_deploy/           # WAA Docker image & task setup
│   ├── evaluate_server.py#   Flask server (port 5050): /setup, /evaluate, /task
│   ├── Dockerfile        #   QEMU + Windows 11 + pre-downloaded apps
│   └── tools_config.json #   App installer URLs and configs
├── annotation.py         # VLM-based demo annotation pipeline
├── vlm.py                # VLM provider abstraction (OpenAI, Anthropic)
├── server/               # WAA server extensions
├── config.py             # Settings (pydantic-settings, .env)
└── __init__.py
scripts/
├── run_eval_pipeline.py      # End-to-end eval: demo gen + VM + ZS/DC eval
├── record_waa_demos.py       # Record demos via VNC
├── generate_demo_review.py   # Markdown review artifacts with thumbnails
├── run_grpo_rollout.py       # Example: collect RL rollouts from WAA
├── refine_demo.py            # Two-pass LLM demo refinement
└── run_dc_eval.py            # Demo-conditioned evaluation

```

--------------------------------

### Create Interactive HTML Viewer

Source: https://docs.openadapt.ai/packages/openadapt-capture

Generates an interactive HTML viewer from a recorded capture file. Optionally include audio.

```python
from openadapt_capture import Capture, create_html

capture = Capture.load("./my_capture")
create_html(capture, output="viewer.html", include_audio=True)
```

--------------------------------

### Create Animated GIF Demo

Source: https://docs.openadapt.ai/packages/openadapt-capture

Generates an animated GIF demo from a recorded capture file. Specify output file, frames per second, and maximum duration.

```python
from openadapt_capture import Capture, create_demo

capture = Capture.load("./my_capture")
create_demo(capture, output="demo.gif", fps=10, max_duration=15)
```

--------------------------------

### Evaluate a model

Source: https://docs.openadapt.ai/packages/openadapt

Run a benchmark evaluation using a trained model checkpoint.

```bash
openadapt eval run --checkpoint training_output/model.pt --benchmark waa
```

--------------------------------

### Train Model Locally

Source: https://docs.openadapt.ai/packages/openadapt-ml

Train a model locally using CUDA or Apple Silicon.

```bash
uv run python -m openadapt_ml.cloud.local train \
  --capture ~/captures/my-workflow --open
```

--------------------------------

### CLI Reference

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Command-line interface for interacting with Consilium.

```APIDOC
## CLI Reference

### Description
Command-line interface for interacting with Consilium.

### Usage
`consilium "prompt" [OPTIONS]`

### Options
- **--models TEXT** - Comma-separated model IDs (default: `gpt-5.2,claude-sonnet-4-6,gemini-3.1-pro`).
- **--chairman TEXT** - Chairman model for synthesis (default: `claude-sonnet-4-6`).
- **--image PATH** - Image file to include (repeatable).
- **--budget FLOAT** - Max spend in USD.
- **--no-review** - Skip Stages 2-3 (faster, cheaper).
- **--system TEXT** - System prompt for all models.
- **--json** - Output raw JSON.

### Examples
```bash
# Full 3-stage pipeline
consilium "Compare Python and Rust for CLI tools"

# Fast mode (Stage 1 only)
consilium "Summarize this" --no-review

# With screenshot
consilium "What's on this screen?" --image screenshot.png

# Budget-limited
consilium "Write a haiku about AI" --budget 0.10

# JSON output for piping
consilium "List 3 colors" --json | jq '.final_answer'

# Custom models
consilium "Hello" --models gpt-5.2,gemini-3.1-pro --chairman gpt-5.2
```
```

--------------------------------

### Run and View Evaluations

Source: https://docs.openadapt.ai/packages/openadapt-evals

Execute an evaluation using a specified agent and task, then view the results of a specific run.

```bash
openadapt-evals run --agent api-claude --task notepad_1
```

```bash
openadapt-evals view --run-name live_eval
```

--------------------------------

### CLI Smoke Tests

Source: https://docs.openadapt.ai/packages/openadapt-desktop

Run basic command-line interface (CLI) smoke tests to verify the core functionality of the OpenAdapt engine.

```bash
uv run python -m engine list
uv run python -m engine storage
uv run python -m engine health
```

--------------------------------

### Local Machine to Cloud VM Interaction Diagram

Source: https://docs.openadapt.ai/packages/openadapt-evals

Illustrates the communication flow between a local machine running `oa-vm` and `openadapt-evals` CLI tools, and a cloud VM hosting Dockerized services like `evaluate_server` and WAA.

```diagram
LOCAL MACHINE                          CLOUD VM (Azure or AWS, Ubuntu)
┌─────────────────────┐                ┌──────────────────────────────┐
│  oa-vm CLI          │   SSH Tunnel   │  Docker                      │
│  (pool management)  │ ─────────────> │  ├─ evaluate_server (:5050)  │
│                     │  :5001 → :5000 │  │  └─ /setup, /evaluate     │
│  openadapt-evals    │  :5051 → :5050 │  ├─ Samba share (/tmp/smb/)  │
│  (benchmark runner) │  :8006 → :8006 │  └─ QEMU (Win 11)           │
│                     │                │     ├─ WAA Flask API (:5000) │
│                     │                │     ├─ \host.lan\Data\      │
│                     │                │     └─ Agent                 │
└─────────────────────┘                └──────────────────────────────┘

```

--------------------------------

### Set API Keys

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Export your API keys as environment variables before using the library.

```bash
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="AI..."
```

--------------------------------

### Capture GUI Interactions

Source: https://docs.openadapt.ai/packages/openadapt-capture

Use the Recorder context manager to capture mouse, keyboard, and screen events.

```python
from openadapt_capture import Recorder

# Record GUI interactions
with Recorder("./my_capture", task_description="Demo task") as recorder:
    # Captures mouse, keyboard, and screen until context exits
    input("Press Enter to stop recording...")
```

--------------------------------

### OpenAdapt Configuration: Environment Variables

Source: https://docs.openadapt.ai/packages/openadapt-evals

Configure API keys and cloud settings using environment variables or a .env file. Pydantic-settings loads these automatically.

```dotenv
# .env
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

# Azure (for --cloud azure VM management)
AZURE_SUBSCRIPTION_ID=...
AZURE_ML_RESOURCE_GROUP=...
AZURE_ML_WORKSPACE_NAME=...
```

--------------------------------

### Council.ask() Method

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Use the ask method to query the council with a prompt, optionally providing images, budget, system prompt, review skipping, or JSON schema.

```APIDOC
## Council.ask() Method

### Description
Queries the council with a given prompt. Supports optional parameters for images, budget, system prompt, skipping review stages, and requesting JSON output.

### Method
POST (conceptual, as it's a method call)

### Endpoint
N/A (Python method)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **prompt** (str) - Required - The question or prompt to send to the LLMs.
- **images** (list[bytes]) - Optional - A list of image data to include with the prompt.
- **budget** (float) - Optional - The maximum USD spend allowed for the query.
- **system** (str) - Optional - A system prompt to guide all models' responses.
- **skip_review** (bool) - Optional - If True, skips the review and synthesis stages (Stages 2 and 3).
- **json_schema** (dict) - Optional - A schema to request the output in JSON format.

### Request Example
```python
result = council.ask(
    "What are the key differences between REST and GraphQL?",
    images=[open("screenshot.png", "rb").read()],  # optional
    budget=0.50,        # max USD spend
    system="Be concise",  # system prompt for all models
    skip_review=False,  # skip Stages 2-3
    json_schema={...},  # request JSON output
)
```

### Response
#### Success Response (200)
Returns a `CouncilResult` object containing the final answer and other details.
- **final_answer** (str) - The synthesized best answer from the council.
- **individual_responses** (list) - A list of each model's Stage 1 response.
- **reviews** (list) - A list of each model's Stage 2 review.
- **total_cost** (float) - The total estimated cost in USD.
- **total_latency_seconds** (float) - The total wall-clock time in seconds.

#### Response Example
```json
{
  "final_answer": "The key differences between REST and GraphQL are...",
  "individual_responses": [...],
  "reviews": [...],
  "total_cost": 0.15,
  "total_latency_seconds": 15.5
}
```

### Error Handling
Specific error handling details are not provided in the source text.
```

--------------------------------

### Replay and Analyze Captures

Source: https://docs.openadapt.ai/packages/openadapt-capture

Load a capture directory and iterate through time-aligned events and screenshots.

```python
from openadapt_capture import Capture

# Load and iterate over time-aligned events
capture = Capture.load("./my_capture")

for action in capture.actions():
    # Each action has an associated screenshot
    print(f"{action.timestamp}: {action.type} at ({action.x}, {action.y})")
    screenshot = action.screenshot  # PIL Image at time of action
```

--------------------------------

### Configure Council with Specific Models

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Instantiate the `Council` class and specify the desired models using their provider/model format. This allows for fine-grained control over which AI models are used in the pipeline.

```python
council = Council(models=["openai/gpt-5.2", "anthropic/claude-sonnet-4-6"])

```

--------------------------------

### Use Schema for GUI Trajectories

Source: https://docs.openadapt.ai/packages/openadapt-ml

Define and construct GUI trajectories using Pydantic models for Episodes, Steps, Actions, and Observations. Supports various ActionTypes.

```python
from openadapt_ml.schema import Episode, Step, Action, Observation, ActionType

episode = Episode(
    episode_id="demo_001",
    instruction="Open Notepad and type Hello World",
    steps=[
        Step(
            step_index=0,
            observation=Observation(screenshot_path="step_0.png"),
            action=Action(type=ActionType.CLICK, coordinates={"x": 100, "y": 200}),
        ),
        Step(
            step_index=1,
            observation=Observation(screenshot_path="step_1.png"),
            action=Action(type=ActionType.TYPE, text="Hello World"),
        ),
    ],
    success=True,
)

```

--------------------------------

### Python API for Evaluations

Source: https://docs.openadapt.ai/packages/openadapt-evals

Use the OpenAdapt AI Python API to set up an adapter, define an agent, and run evaluations. Compute and print success rate metrics.

```python
from openadapt_evals import (
    ApiAgent,
    WAALiveAdapter,
    WAALiveConfig,
    evaluate_agent_on_benchmark,
    compute_metrics,
)

adapter = WAALiveAdapter(WAALiveConfig(server_url="http://localhost:5001"))
agent = ApiAgent(provider="anthropic")

results = evaluate_agent_on_benchmark(agent, adapter, task_ids=["notepad_1"])
metrics = compute_metrics(results)
print(f"Success rate: {metrics['success_rate']:.1%}")
```

--------------------------------

### Monorepo Project Structure

Source: https://docs.openadapt.ai/packages/openadapt-wright

Overview of the directory structure for the OpenAdapt AI project, including worker, bot, and shared packages.

```tree
wright/
  apps/
    worker/       # Fly.io: generalized dev loop (scale-to-zero)
      src/
        index.ts           # HTTP server (health, drain, cancel)
        queue-poller.ts    # Supabase job queue polling + claiming
        dev-loop.ts        # Ralph Loop orchestrator
        claude-session.ts  # Claude Agent SDK wrapper
        test-runner.ts     # Auto-detect + run test suites
        github-ops.ts      # Clone, branch, commit, push, PR
        __tests__/         # 53 tests across 6 test files
    bot/          # Fly.io: always-on Telegram bot (grammY)
  packages/
    shared/       # Shared types + constants
  supabase/
    migrations/   # Database schema (job_queue, job_events, test_results)

```

--------------------------------

### End-to-End Dev Loop Pipeline

Source: https://docs.openadapt.ai/packages/openadapt-wright

Outlines the sequential steps of the development loop, from repository cloning to pull request creation.

```text
1. cloneRepo()          → Creates a real git repo with package.json + tests
2. createFeatureBranch() → Creates wright/test-1234 branch
3. detectTestRunner()    → Detects 'jest' from package.json
4. detectPackageManager()→ Detects 'npm' from package.json
5. installDependencies() → Runs 'npm install'
6. runClaudeSession()    → Mocked: returns $0.05 cost, 3 turns
7. runTests()            → Executes real 'npx jest --forceExit'
8. commitAndPush()       → Mocked: returns commit SHA abc123def
9. createPullRequest()   → Mocked: returns PR URL
10. cleanup()            → Verifies workdir deleted after completion
```

--------------------------------

### Parallel VM Pool Creation and Task Distribution

Source: https://docs.openadapt.ai/packages/openadapt-evals

Create and manage VM pools for parallel task execution on Azure and AWS. Use `pool-create` to set up workers, `pool-wait` to ensure readiness, and `pool-run` to distribute tasks. Specify the cloud provider with `--cloud aws`.

```bash
# Create a pool of VMs and distribute tasks (Azure)
oa-vm pool-create --workers 5
oa-vm pool-wait
oa-vm pool-run --tasks 50
```

```bash
# Same workflow on AWS
oa-vm pool-create --cloud aws --workers 5
oa-vm pool-wait --cloud aws
oa-vm pool-run --cloud aws --tasks 50
```

```bash
# Or use Azure ML orchestration
openadapt-evals azure --workers 10 --waa-path /path/to/WindowsAgentArena
```

--------------------------------

### Council Class Initialization

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Initialize the Council class with optional parameters to customize the LLM models, chairman model, and maximum workers.

```APIDOC
## Council Class Initialization

### Description
Initialize the main orchestrator class `Council` with customizable parameters.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from consilium import Council

council = Council(
    models=["gpt-5.2", "claude-sonnet-4-6", "gemini-3.1-pro"],
    chairman="claude-sonnet-4-6",
    max_workers=8,
)
```

### Response
#### Success Response (200)
Initialization of the Council object.

#### Response Example
None (initialization does not return a value in this format)
```

--------------------------------

### Run Performance Test

Source: https://docs.openadapt.ai/packages/openadapt-capture

Executes a performance test with synthetic input for 10 seconds, recording metrics like time, memory, and event counts. Output includes a memory usage plot.

```bash
uv run python scripts/perf_test.py
```

--------------------------------

### Budget Control in Consilium SDK

Source: https://docs.openadapt.ai/packages/openadapt-consilium

Set a budget for AI queries using the `budget` parameter in the `council.ask` method. If Stage 1 costs exceed the budget, subsequent stages are automatically skipped, and the best Stage 1 response is returned.

```python
result = council.ask("Expensive question", budget=0.10)

# If Stage 1 costs > $0.10, Stages 2-3 are automatically skipped
# The best Stage 1 response is returned as the final answer

```

--------------------------------

### Run Integration Tests

Source: https://docs.openadapt.ai/packages/openadapt-capture

Executes integration tests, specifically targeting performance-related tests. Requires accessibility permissions.

```bash
uv run pytest tests/test_performance.py -v -m slow
```

--------------------------------

### Run Tests with Pytest

Source: https://docs.openadapt.ai/packages/openadapt-ml

Execute the project's tests using pytest.

```bash
uv run pytest
```

--------------------------------

### Clean Up VM Pool

Source: https://docs.openadapt.ai/packages/openadapt-evals

Stop billing by cleaning up the virtual machine pool. Use the `-y` flag to confirm without prompting.

```bash
oa-vm pool-cleanup -y
```

--------------------------------

### Share Recording (Send)

Source: https://docs.openadapt.ai/packages/openadapt-capture

Initiates sharing a recording from the sending machine using Magic Wormhole. A code will be displayed for the receiving machine.

```bash
# On the sending machine
capture share send ./my_capture
```

--------------------------------

### Implement BenchmarkAgent Interface

Source: https://docs.openadapt.ai/packages/openadapt-evals

Define a custom agent by inheriting from BenchmarkAgent and implementing the act and reset methods.

```python
from openadapt_evals import BenchmarkAgent, BenchmarkAction, BenchmarkObservation, BenchmarkTask

class MyAgent(BenchmarkAgent):
    def act(
        self,
        observation: BenchmarkObservation,
        task: BenchmarkTask,
        history: list[tuple[BenchmarkObservation, BenchmarkAction]] | None = None,
    ) -> BenchmarkAction:
        # Your agent logic here
        return BenchmarkAction(type="click", x=0.5, y=0.5)

    def reset(self) -> None:
        pass
```
Tokens:
Raw