### Install Podman on Windows

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Podman on Windows using winget. Requires initializing and starting the Podman machine.

```powershell
# Option B — Podman
winget install RedHat.Podman
podman machine init
podman machine start
```

--------------------------------

### Install Docker on Windows

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Docker Desktop on Windows using winget. Requires launching Docker Desktop after installation.

```powershell
# Option A — Docker Desktop (WSL2 backend)
winget install Docker.DockerDesktop
# then launch Docker Desktop from the Start menu and wait for it to be ready
```

--------------------------------

### Install ClawBench using pipx

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Alternative installation method for ClawBench using pipx.

```bash
pipx install clawbench-eval
```

--------------------------------

### Install ClawBench using pip

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Alternative installation method for ClawBench using pip.

```bash
python -m pip install clawbench-eval
```

--------------------------------

### Install ClawBench using uv

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs ClawBench from PyPI using the uv tool. This is the recommended method for normal use.

```bash
uv tool install clawbench-eval
```

--------------------------------

### Clone and Run ClawBench

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Clone the repository, navigate to the directory, and execute the run script to set up and start ClawBench. This is the primary method for initiating the benchmark.

```bash
git clone https://github.com/reacher-z/ClawBench.git && cd ClawBench && ./run.sh
```

--------------------------------

### Run Chrome Extension Setup Script

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/chrome-extension/README.md

Execute this script to load the Chrome extension for local development. Provide the target URL as an argument.

```bash
./setup.sh https://example.com
```

--------------------------------

### Setup CLAWBENCH Agent

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Follow these steps to set up the CLAWBENCH agent. Ensure you have the prerequisites and configure your models before launching.

```bash
# 1. PurelyMail credentials are provided in the committed .env

# 2. Configure at least one model
cp models/models.example.yaml models/models.yaml
# Edit models.yaml: add API key and base URL for your model(s)

# 3. Launch
./run.sh
```

--------------------------------

### Install Podman on Linux (Ubuntu/Debian)

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Podman on Ubuntu/Debian systems. This is the recommended rootless option.

```bash
# Option A — Podman (rootless by default, recommended)
sudo apt update && sudo apt install -y podman
```

--------------------------------

### Copy and Edit Model Configuration

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Command to copy the example model configuration file and open it for editing.

```bash
cp models/models.example.yaml models/models.yaml
$EDITOR models/models.yaml
```

--------------------------------

### Rescoring an Existing Trace Bundle

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/eval/scoring.md

This bash script demonstrates the steps to re-grade an existing trace bundle without re-running the agent. It includes package installation, trace bundle download, API key setup, and the rescoring command.

```bash
# 1. install the package
pip install clawbench-eval

# 2. download the trace bundle for the model you want to re-score
hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace \
  --include "*-claude-sonnet-4-6-*" \
  --local-dir ./v2-traces

# 3. set your judge model's API key in env
export DEEPSEEK_API_KEY=sk-...

# 4. (one-time) add the judge model to models.yaml — see docs/models.md

# 5. rescore
python scripts/clawbench_rescore.py \
  --judge-model deepseek-v4-pro \
  --only-batch ./v2-traces \
  --force        # re-judge existing judge.json files
```

--------------------------------

### Model Configuration Example

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Example of a model configuration entry in models.yaml, specifying API details and thinking level.

```yaml
qwen3.5-397b-a17b:
  api_key: "sk-or-v1-..."
  base_url: https://openrouter.ai/api/v1
  api_type: openai-completions
  thinking_level: medium
```

--------------------------------

### Example: OpenCLAW via OpenRouter/OpenAI-compatible API

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Demonstrates running OpenCLAW using an OpenRouter endpoint with a specific model and disabling thinking. This is an example of agent and model configuration.

```bash
-a openclaw -m openai/deepseek/deepseek-v4-flash --ak thinking=off
```

--------------------------------

### Install Docker on Linux (Ubuntu/Debian)

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Docker on Ubuntu/Debian systems. Requires adding the user to the docker group.

```bash
# Option B — Docker
sudo apt install -y docker.io
sudo usermod -aG docker $USER  # log out / back in so your shell picks up the group
```

--------------------------------

### Install Docker on macOS

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Docker Desktop on macOS using Homebrew. This is the easiest option and includes a GUI.

```bash
# Option A — Docker Desktop (easiest, includes GUI)
brew install --cask docker
open -a Docker                 # launch and wait for the whale icon to settle
```

--------------------------------

### Example: Hermes via OpenRouter

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Shows how to configure the 'hermes' agent to use a specific model ('deepseek/deepseek-v4-flash') through OpenRouter.

```bash
-a hermes -m deepseek/deepseek-v4-flash
```

--------------------------------

### Install and Run ClawBench TUI

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Install ClawBench via PyPI or run from a source checkout. The interactive TUI requires a terminal. For non-TTY environments, use `clawbench-run` or `clawbench-batch`.

```bash
clawbench         # PyPI install
uv run clawbench  # source checkout
```

--------------------------------

### Reproduce Scoring with ClawBench CLI

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/docs/scoring.md

This bash script demonstrates how to install the ClawBench evaluation package, download trace bundles, set API keys, and re-score existing trace data.

```bash
# 1. install the package
pip install clawbench-eval

# 2. download the trace bundle for the model you want to re-score
hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace \
  --include "*-claude-sonnet-4-6-*" \
  --local-dir ./v2-traces

# 3. set your judge model's API key in env
export DEEPSEEK_API_KEY=sk-...

# 4. (one-time) add the judge model to models.yaml — see docs/models.md

# 5. rescore
python scripts/clawbench_rescore.py \
  --judge-model deepseek-v4-pro \
  --only-batch ./v2-traces \
  --force        # re-judge existing judge.json files
```

--------------------------------

### Install ClawBench Evaluation Package

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Install the clawbench-eval package using pip. This is required to run evaluations on your own models.

```bash
pip install clawbench-eval
```

--------------------------------

### Run CLAWBENCH Agent Interactively

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Launch the interactive Text User Interface (TUI) for CLAWBENCH. This is the recommended way to start.

```bash
./run.sh
```

--------------------------------

### Run ClawBench with Harbor Framework

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Execute ClawBench tasks using the Harbor framework. This command starts the ClawBench browser runtime within a Harbor environment and connects the specified agent. Judge credentials are passed via `--ve` flags.

```bash
# Run with Harbor. Use "harbor run" directly if Harbor is already installed.
uvx --from harbor==0.15.0 harbor run \
  -p ./harbor-datasets/clawbench-v2 \
  -a "<agent>" \
  -m "<model>" \
  --env-file .env \
  --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \
  --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \
  --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \
  --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}"
```

--------------------------------

### Install Podman on macOS

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Installs Podman on macOS using Homebrew. This is a rootless, daemonless, CLI-only option. Requires a one-time VM initialization.

```bash
# Option B — Podman (rootless, no daemon, CLI only)
brew install podman
podman machine init            # one-time: downloads the Linux VM image
podman machine start           # must be running before any podman command
```

--------------------------------

### Run ClawBench Runtime Server Locally

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/runtime-server/README.md

This command starts the ClawBench runtime server locally for debugging purposes. It sets the data directory, display, and runs the FastAPI application using uvicorn.

```bash
cd src/clawbench/runtime/runtime-server
CLAWBENCH_DATA_DIR=./data DISPLAY=:99 uv run --frozen uvicorn server:app --host 0.0.0.0 --port 7878
```

--------------------------------

### Run Hermes with OpenRouter

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Configure and run Hermes using OpenRouter. Set your OPENROUTER_API_KEY before execution. This example uses the deepseek-v4-flash model.

```bash
# Hermes through OpenRouter.
export OPENROUTER_API_KEY="your-openrouter-key"

uvx --from harbor==0.15.0 harbor run \
  -p ./harbor-datasets/clawbench-v2 \
  -a hermes \
  -m deepseek/deepseek-v4-flash \
  --env-file .env \
  --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \
  --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \
  --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \
  --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" \
  --jobs-dir ./harbor-jobs/hermes-deepseek-flash
```

--------------------------------

### Example Task JSON Structure

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

This JSON defines a single task for ClawBench, including the agent's instruction, time limit, and evaluation schema. The url_pattern is a placeholder that will not match any URL, indicating that the evaluation relies on agentic judgment rather than URL matching.

```json
{
  "instruction": "On Uber Eats, order delivery: one Pad Thai, deliver to home address, note \"no peanuts\"",
  "time_limit": 30,
  "eval_schema": {
    "url_pattern": "__PLACEHOLDER_WILL_NOT_MATCH__",
    "method": "POST"
  }
}
```

--------------------------------

### Develop ClawBench from Source

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Clone the ClawBench repository and set up the development environment. This is recommended for contributors modifying the driver, test cases, or container build.

```bash
git clone https://github.com/reacher-z/ClawBench.git && cd ClawBench
cp models/models.example.yaml models/models.yaml   # edit: add your model API keys
# .env is already provided for PurelyMail; edit only for your own creds or HF upload
./run.sh                                           # interactive TUI
```

```bash
uv run clawbench-run \
  test-cases/v1/001-daily-life-food-uber-eats claude-sonnet-4-6   # single run
```

```bash
uv run clawbench-run \
  test-cases/v1/001-daily-life-food-uber-eats --human             # human mode
```

--------------------------------

### Running Agents and Evaluation Workflow

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

This diagram illustrates the two-step workflow: first running agents using the root uv package, then performing evaluation using the eval/ directory. It shows the input and output for each stage.

```text
 1. Run agents (root uv package)   2. Evaluate (eval/)
 ─────────────────────────         ────────────────────────────────
 ./run.sh / clawbench-batch ──►    Claude Code subagents compare
 produces test-output/             agent vs human trajectories
   with 5-layer recordings         under eval/agentic_eval.md rubric
```

--------------------------------

### Run Agent and Evaluate

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/docs/README.zh-CN.md

This diagram illustrates the two-step process: first running the agent (using the root uv package) and then performing the evaluation. The evaluation compares Claude Code sub-agent outputs against human references.

```text
 1. 运行智能体 (根 uv 包)            2. 测评 (eval/)            
 ────────────────────────           ────────────────────────────────            
 ./run.sh / clawbench-batch   ──►    Claude Code 子代理对比            
 生成 test-output/                  智能体 vs 人类轨迹            
   含五层录制数据                    按 eval/agentic_eval.md rubric 判定            
```

--------------------------------

### Run Interactive TUI

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Execute the interactive terminal UI for ClawBench. This command starts the main TUI interface.

```bash
./run.sh
```

```bash
uv run clawbench
```

--------------------------------

### Run Harbor Dataset with Clawbench

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Executes a Harbor dataset using the 'uvx' command. Replace '<agent>' and '<model>' with your specific agent and model identifiers. Ensure the .env file is configured.

```bash
uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a "<agent>" -m "<model>" --env-file .env
```

--------------------------------

### Download ClawBench V1 Traces

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Use this command to download the execution traces for ClawBench V1 model runs. This includes recordings, requests, actions, and agent messages.

```bash
hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace
```

--------------------------------

### Configure models using TUI

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Launches the ClawBench TUI to configure models. This command creates local templates under 'models/' if they don't exist.

```bash
clawbench
```

--------------------------------

### Download ClawBench V2 Traces

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Use this command to download the execution traces for ClawBench V2 model runs. This dataset is continuously updated with new model evaluations.

```bash
hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace
```

--------------------------------

### Download ClawBench Dataset

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Use this command to download the main ClawBench dataset, which includes task definitions and metadata for V1 and V2.

```bash
hf download --repo-type dataset NAIL-Group/ClawBench
```

--------------------------------

### Run CLAWBENCH Agent Single Test Case

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Execute a single test case for a specified model. Use the `--harness` flag to select the appropriate harness.

```bash
uv run clawbench-run test-cases/v1/<case-dir> <model-name> --harness openclaw
```

```bash
uv run clawbench-run test-cases/v1/<case-dir> <model-name> --harness claude-code
```

--------------------------------

### Adapt V2 Tasks for Harbor Framework

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Convert V2 test cases into a local Harbor dataset. This prepares the tasks for execution within the Harbor framework, which manages the browser runtime and agent connection.

```bash
# Convert all V2 tasks into Harbor-compatible task directories.
uv run clawbench-harbor-adapt \
  --output-dir ./harbor-datasets/clawbench-v2 \
  --overwrite

# Optional smoke dataset with one generated task.
uv run clawbench-harbor-adapt \
  --output-dir ./harbor-datasets/clawbench-v2-smoke \
  --limit 1 \
  --overwrite
```

--------------------------------

### Download ClawBench Data from Hugging Face

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Download the ClawBench dataset, including tasks, rubrics, and metadata, from Hugging Face using the `hf download` command.

```bash
hf download NAIL-Group/ClawBench
```

--------------------------------

### Run Specific Model Patterns and Case Globs

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

This command runs jobs for models matching 'qwen*' and 'claude*', and for test cases matching 'test-cases/v1/886-*' and 'test-cases/v1/872-*', with 2 concurrent jobs. Use this for targeted testing.

```bash
uv run clawbench-batch \
  --models "qwen*" "claude*" \
  --cases "test-cases/v1/886-*" "test-cases/v1/872-*" \
  --max-concurrent 2
```

--------------------------------

### Run CLAWBENCH Agent Batch Test Cases (V2)

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Execute batch runs for V2 test cases using a specified model suite. The `--all-cases` flag ensures all cases within the suite are processed.

```bash
uv run clawbench-batch \
  --models <model-name> --cases-suite v2 --all-cases --max-concurrent 3 --harness hermes
```

--------------------------------

### Testing V1 Task with Human Mode

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/CONTRIBUTING.md

Run this command to test a V1 task in human mode before submitting a PR. This verifies task completability.

```bash
uv run clawbench-run test-cases/v1/887-daily-life-food-grubhub --human
```

--------------------------------

### Run OpenClaw via OpenRouter/OpenAI-compatible API

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Execute a Harbor run using OpenClaw with an OpenAI-compatible API. Ensure the .env file is configured for authentication.

```bash
uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a openclaw -m openai/deepseek/deepseek-v4-flash --ak thinking=off --env-file .env
```

--------------------------------

### Run Single Test Case with Human Reference

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Execute a single test case and include a human reference for comparison. This helps in evaluating model performance against human behavior.

```bash
uv run clawbench-run test-cases/v1/886-entertainment-hobbies-experience-topgolf --human
```

--------------------------------

### Run OpenClaw with OpenRouter

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Configure and run OpenClaw using OpenRouter's OpenAI-compatible endpoint. Ensure OPENROUTER_API_KEY is set in your environment.

```bash
# OpenClaw through OpenRouter's OpenAI-compatible endpoint.
export OPENAI_BASE_URL="https://openrouter.ai/api/v1"
export OPENAI_API_KEY="$OPENROUTER_API_KEY"

uvx --from harbor==0.15.0 harbor run \
  -p ./harbor-datasets/clawbench-v2 \
  -a openclaw \
  -m openai/deepseek/deepseek-v4-flash \
  --ak thinking=off \
  --env-file .env \
  --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \
  --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \
  --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \
  --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" \
  --jobs-dir ./harbor-jobs/openclaw-deepseek-flash
```

--------------------------------

### Directory Naming Convention for V1 Tasks

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/CONTRIBUTING.md

Follow this naming convention when creating a new directory for a V1 test case.

```bash
test-cases/v1/<id>-<metaclass>-<class>-<platform>/
```

--------------------------------

### Preview Job Matrix Without Execution

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

This command simulates the execution by printing the job matrix for all models and all cases without actually running any jobs. It's useful for verifying configurations.

```bash
uv run clawbench-batch --all-models --all-cases --dry-run
```

--------------------------------

### Testing V2 Task with Human Mode

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/CONTRIBUTING.md

Run this command to test a V2 task in human mode before submitting a PR. This verifies task completability.

```bash
uv run clawbench-run test-cases/v2/v2-887-daily-life-food-grubhub --human
```

--------------------------------

### Directory Structure

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Overview of the Clawbench project's directory layout.

```text
src/
  clawbench/
    tui.py                       # Interactive menu for configuring and launching runs
    runner/
      run.py                     # Single test-case driver
      batch.py                   # Concurrent batch runner
      run_support/
        resume.py                # Runner-private resume PDF renderer
        resume_template.json     # Synthetic resume source template
    utils/
      paths.py                   # Shared PROJECT_ROOT / HARNESS_ROOT discovery
      hf_upload.py               # Optional HuggingFace upload helpers
  runtime-server/
    server.py                    # CDP instrumentation server for actions, screenshots, requests, recording
    pyproject.toml               # Container-only uv project
    uv.lock
    README.md
  chrome-extension/
    manifest.json
    stealth.js                   # Browser fingerprint hardening patches
    setup.sh                     # Local extension launch helper
    README.md
  harnesses/
    harnesses.yaml              # Central harness registry: images, scripts, usage emitters
    harness.schema.json         # JSON Schema for harnesses.yaml
    base/
      Dockerfile.base            # Shared Chromium, Xvfb, noVNC, server, extension image
      entrypoint.sh              # Shared container startup logic
    openclaw/
      usage-emitter.py           # Harness-local JSONL usage artifact emitter
    ...                          # other harnesses
```

--------------------------------

### Adapt Harbor V2 Dataset

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Convert V2 tasks into a Harbor dataset format. This command is used to prepare datasets for use with Harbor.

```bash
uv run clawbench-harbor-adapt --output-dir ./harbor-datasets/clawbench-v2 --overwrite
```

--------------------------------

### Directory Naming Convention for V2 Tasks

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/CONTRIBUTING.md

Follow this naming convention when creating a new directory for a V2 test case. Note the 'v2-' prefix.

```bash
test-cases/v2/v2-<id>-<metaclass>-<class>-<platform>/
```

--------------------------------

### Run CLAWBENCH Agent in Human Mode

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Launch CLAWBENCH in human mode for manual control via a noVNC browser interface. No harness is required for this mode.

```bash
uv run clawbench-run test-cases/v1/<case-dir> --human
```

--------------------------------

### Export Container Engine

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/docs/README.zh-CN.md

To use Podman instead of Docker, set the CONTAINER_ENGINE environment variable to 'podman'. The framework automatically detects the available engine.

```bash
export CONTAINER_ENGINE=podman
```

--------------------------------

### Action Logging Endpoint

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/runtime-server/README.md

A compatibility endpoint for logging browser DOM actions.

```APIDOC
## POST /api/action

### Description
This is a compatibility endpoint. CDP capture writes actions directly to storage.

### Method
POST

### Endpoint
/api/action

### Parameters
#### Request Body
- **action_data** (object) - Required - Data representing the browser DOM action.
```

--------------------------------

### Run Batch Clawbench for All V1 Tasks

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Executes all tasks within the 'test-cases/v1/' directory for a specified model ('claude-sonnet-4-6') with a concurrency limit of 3.

```bash
uv run clawbench-batch --models claude-sonnet-4-6 --all-cases --max-concurrent 3
```

--------------------------------

### Run Hermes via OpenRouter

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/src/README.md

Execute a Harbor run using Hermes with the OpenRouter service. This command requires an .env file for configuration.

```bash
uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a hermes -m deepseek/deepseek-v4-flash --env-file .env
```

--------------------------------

### Run CLAWBENCH Agent Batch Test Cases (V1)

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Perform batch runs for V1 test cases, creating a cross-product of specified models and all available cases. Control concurrency with `--max-concurrent`.

```bash
uv run clawbench-batch \
  --models <model-name> --all-cases --max-concurrent 3 --harness openclaw
```

--------------------------------

### ClawBench Project Structure

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/AGENTS.md

Overview of the ClawBench project directory layout, including entry points, source code, model configurations, test cases, and evaluation scripts.

```plaintext
ClawBench/
  run.sh                          # Entry point -- launches interactive TUI
  pyproject.toml                  # Root uv package metadata and CLI scripts
  .env                            # Shared PurelyMail credentials packaged with the CLI
  src/
    clawbench/
      tui.py                      # Interactive TUI (called by run.sh)
      runner/
        run.py                    # Single test-case runner
        batch.py                  # Batch runner (model x case cross-product)
        run_support/              # Runner-private helpers for single runs
      utils/
        hf_upload.py              # Optional HuggingFace upload helpers
      runtime/
        shared/
          alex_green_personal_info.json
        runtime-server/           # Container-only uv project for server deps
        chrome-extension/         # Recording extension
        harnesses/                # Dockerfiles + setup/run scripts
  models/
    models.yaml                   # Model API configs (gitignored -- copy from example)
    models.example.yaml           # Template with placeholder keys
    model.schema.json             # JSON schema for model entries
  test-cases/
    task.schema.json              # JSON schema for task.json
    v1/                           # V1: 153 task directories
    v2/                           # V2: 130 task directories
    v1-lite/                      # Lite: 20 V1 task directories with linked task files
  eval/
    README.md                     # Evaluation guide + Claude Code prompt template
    agentic_eval.md               # Evaluator rubric for judging agent success
```

--------------------------------

### Run Batch Clawbench for All V2 Tasks

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/README.md

Executes all tasks within the 'test-cases/v2/' directory for a specified model ('claude-sonnet-4-6') with a concurrency limit of 3.

```bash
uv run clawbench-batch --models claude-sonnet-4-6 --cases-suite v2 --all-cases --max-concurrent 3
```

--------------------------------

### Hugging Face Space README Front Matter

Source: https://github.com/tiger-ai-lab/clawbench/blob/main/docs/huggingface-space-metadata.md

This YAML block should be placed at the top of the Space README.md. Update placeholder values for title, emoji, color scheme, SDK, app file, pinning, license, thumbnail, models, datasets, tags, and last_refresh.

```yaml
---
title: ClawBench
emoji: 🐾
colorFrom: indigo
colorTo: yellow
sdk: gradio
sdk_version: 5.30.0
app_file: app.py
pinned: true
license: apache-2.0
thumbnail: assets/social-card.png
models:
  - TIGER-Lab/ClawBench
datasets:
  - NAIL-Group/ClawBench
  - TIGER-Lab/ClawBench
  - NAIL-Group/ClawBenchV1Trace
  - NAIL-Group/ClawBenchV2Trace
tags:
  - benchmark
  - browser-agents
  - computer-use
  - leaderboard
last_refresh: 2026-06-08
---
```