SkillsBench

https://skillsbench.ai/docs
Admin
SkillsBench is the first benchmark that tests whether agent skills can improve agent performance,...
Tokens:2,942
Snippets:46
Trust Score:7.5
Update:5 months ago
Show doc for...
Tokens:
Raw
### Clone Harbor and Install Dependencies

Source: https://skillsbench.ai/docs/getting-started

Clone the Harbor repository and install its development dependencies using uv.

```bash
# Clone Harbor and install
git clone https://github.com/laude-institute/harbor.git
cd harbor
uv sync --extra dev
```

--------------------------------

### Instruction Style Example

Source: https://skillsbench.ai/docs/contributing

Demonstrates the direct, terminal-bench style for instructions, including conversational tone, context, and sequential steps.

```text
Build a sales report from the spreadsheet data.

1. Load sales data from /app/data/sales.csv
2. Calculate total revenue by region
3. Generate /app/output/report.xlsx with summary sheet
4. Create /app/output/chart.png showing revenue breakdown
```

--------------------------------

### Run Full SkillsBench Benchmark

Source: https://skillsbench.ai/docs/getting-started

Execute the full SkillsBench benchmark. Use the 'oracle' agent for setup verification or specify your agent and model for evaluation.

```bash
# Run with oracle (reference solution) to verify setup
uv run harbor jobs start -d skillsbench

# Run with your agent
uv run harbor jobs start -d skillsbench -a <agent_name> -m "<model_name>"

# Example: Claude Code with Sonnet 4.5
uv run harbor jobs start -d skillsbench -a claude-code -m "anthropic/claude-sonnet-4-5"
```

--------------------------------

### Harbor Framework Commands

Source: https://skillsbench.ai/docs/contributing

Provides key commands for interacting with the Harbor framework, including running tasks with datasets and checking task formats.

```bash
harbor run --dataset <path> --agent <agent-name>    # run tasks
```

```bash
harbor tasks check                                  # validate task format
```

--------------------------------

### Run a Single Task in SkillsBench

Source: https://skillsbench.ai/docs/getting-started

Execute a single task within SkillsBench. Specify the task path for the oracle solution or for evaluation with your agent.

```bash
# Oracle (reference solution)
uv run harbor trials start -p datasets/skillsbench/<task_id>

# With your agent
uv run harbor trials start -p datasets/skillsbench/<task_id> -a <agent_name> -m "<model_name>"
```

--------------------------------

### Task Format Structure

Source: https://skillsbench.ai/docs/contributing

Defines the directory structure and required files for a SkillsBench task submission. Ensure all specified files are present and correctly formatted.

```text
task-name/
├── instruction.md          # REQUIRED - Task description
├── task.toml               # REQUIRED - Metadata, timeouts, required/distractor skills
├── environment/
│   ├── Dockerfile          # REQUIRED - Container with dependencies
│   └── skills/
│       └── skill-name/
│           ├── SKILL.md    # REQUIRED (per skill)
│           ├── scripts/    # OPTIONAL
│           ├── references/ # OPTIONAL
│           └── assets/     # OPTIONAL
├── solution/
│   └── solve.sh            # REQUIRED - Oracle solution (must pass 100%)
└── tests/
    ├── test.sh             # REQUIRED - Runs pytest
    └── test_outputs.py     # REQUIRED - Writes reward to /logs/verifier/reward.txt
```

--------------------------------

### Free up Docker space

Source: https://skillsbench.ai/docs/getting-started

Run 'docker system prune' to free up disk space used by Docker. This is recommended when Docker build failures occur due to insufficient space.

```bash
# Free up Docker space if builds fail
docker system prune
```

--------------------------------

### Export External API Keys

Source: https://skillsbench.ai/docs/getting-started

Export necessary API keys as environment variables to enable tasks that interact with external services.

```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
export GEMINI_API_KEY=...
export ELEVENLABS_API_KEY=...
export GH_AUTH_TOKEN=ghp_...
export HUGGINGFACE_API_TOKEN=hf_...
export MODAL_TOKEN_ID=ak-...
export MODAL_TOKEN_SECRET=as-...
```

--------------------------------

### Run Self-Contained Subset via CLI Flags

Source: https://skillsbench.ai/docs/getting-started

Alternatively, exclude specific tasks using CLI flags to run only the self-contained subset of SkillsBench.

```bash
uv run harbor jobs start -d skillsbench -a <agent_name> -m "<model_name>" \
  -x gh-repo-analytics \
  -x mhc-layer-impl \
  -x pedestrian-traffic-counting \
  -x pg-essay-to-audiobook \
  -x scheduling-email-assistant \
  -x speaker-diarization-subtitles \
  -x trend-anomaly-causal-inference \
  -x video-filler-word-remover \
  -x video-tutorial-indexer
```

--------------------------------

### Configure Self-Contained Subset via YAML

Source: https://skillsbench.ai/docs/getting-started

Use a YAML configuration file to exclude tasks requiring external API keys or having build issues, running only the self-contained tasks.

```yaml
jobs_dir: jobs
n_attempts: 1
timeout_multiplier: 3.0
orchestrator:
  type: local
  n_concurrent_trials: 4
  quiet: false
environment:
  type: docker
  force_build: true
  delete: true
agents:
  - name: oracle
    model_name: oracle
datasets:
  - path: datasets/skillsbench
    exclude_task_names:
      - gh-repo-analytics          # requires GH_AUTH_TOKEN
      - mhc-layer-impl             # requires MODAL_TOKEN_ID/SECRET
      - pedestrian-traffic-counting # requires OPENAI/GEMINI/ANTHROPIC API keys
      - pg-essay-to-audiobook       # requires OPENAI_API_KEY + ELEVENLABS_API_KEY
      - scheduling-email-assistant  # hardcoded volume mount + HUGGINGFACE_API_TOKEN
      - speaker-diarization-subtitles # Docker build OOM (Whisper large-v3)
      - trend-anomaly-causal-inference # requires ANTHROPIC + OPENAI API keys
      - video-filler-word-remover   # requires OPENAI_API_KEY
      - video-tutorial-indexer      # requires OPENAI_API_KEY
```

--------------------------------

### Reduce concurrent trials for API rate limiting

Source: https://skillsbench.ai/docs/getting-started

Adjust the 'n_concurrent_trials' setting under 'orchestrator' to reduce concurrency and avoid API rate limiting errors. This is useful for tasks calling external APIs.

```yaml
orchestrator:
  type: local
  n_concurrent_trials: 2  # reduce from 4 to avoid rate limits
```

--------------------------------

### Exclude tasks in job config YAML

Source: https://skillsbench.ai/docs/getting-started

Configure task exclusions directly within your job configuration YAML file using the 'exclude_task_names' list. This provides a declarative way to manage skipped tasks.

```yaml
datasets:
  - path: datasets/skillsbench
    exclude_task_names:
      - gh-repo-analytics
      - mhc-layer-impl
      - pedestrian-traffic-counting
      - pg-essay-to-audiobook
      - scheduling-email-assistant
      - speaker-diarization-subtitles
      - trend-anomaly-causal-inference
      - video-filler-word-remover
      - video-tutorial-indexer
```

--------------------------------

### Increase timeout multiplier

Source: https://skillsbench.ai/docs/getting-started

Set 'timeout_multiplier' to a higher value in your YAML config to increase agent and build timeouts. Use this for tasks with long builds or heavy computation.

```yaml
timeout_multiplier: 3.0  # multiplies both agent and build timeouts
```
Tokens:
Raw