### Clone Harbor and Install Dependencies Source: https://skillsbench.ai/docs/getting-started Clone the Harbor repository and install its development dependencies using uv. ```bash # Clone Harbor and install git clone https://github.com/laude-institute/harbor.git cd harbor uv sync --extra dev ``` -------------------------------- ### Instruction Style Example Source: https://skillsbench.ai/docs/contributing Demonstrates the direct, terminal-bench style for instructions, including conversational tone, context, and sequential steps. ```text Build a sales report from the spreadsheet data. 1. Load sales data from /app/data/sales.csv 2. Calculate total revenue by region 3. Generate /app/output/report.xlsx with summary sheet 4. Create /app/output/chart.png showing revenue breakdown ``` -------------------------------- ### Run Full SkillsBench Benchmark Source: https://skillsbench.ai/docs/getting-started Execute the full SkillsBench benchmark. Use the 'oracle' agent for setup verification or specify your agent and model for evaluation. ```bash # Run with oracle (reference solution) to verify setup uv run harbor jobs start -d skillsbench # Run with your agent uv run harbor jobs start -d skillsbench -a -m "" # Example: Claude Code with Sonnet 4.5 uv run harbor jobs start -d skillsbench -a claude-code -m "anthropic/claude-sonnet-4-5" ``` -------------------------------- ### Harbor Framework Commands Source: https://skillsbench.ai/docs/contributing Provides key commands for interacting with the Harbor framework, including running tasks with datasets and checking task formats. ```bash harbor run --dataset --agent # run tasks ``` ```bash harbor tasks check # validate task format ``` -------------------------------- ### Run a Single Task in SkillsBench Source: https://skillsbench.ai/docs/getting-started Execute a single task within SkillsBench. Specify the task path for the oracle solution or for evaluation with your agent. ```bash # Oracle (reference solution) uv run harbor trials start -p datasets/skillsbench/ # With your agent uv run harbor trials start -p datasets/skillsbench/ -a -m "" ``` -------------------------------- ### Task Format Structure Source: https://skillsbench.ai/docs/contributing Defines the directory structure and required files for a SkillsBench task submission. Ensure all specified files are present and correctly formatted. ```text task-name/ ├── instruction.md # REQUIRED - Task description ├── task.toml # REQUIRED - Metadata, timeouts, required/distractor skills ├── environment/ │ ├── Dockerfile # REQUIRED - Container with dependencies │ └── skills/ │ └── skill-name/ │ ├── SKILL.md # REQUIRED (per skill) │ ├── scripts/ # OPTIONAL │ ├── references/ # OPTIONAL │ └── assets/ # OPTIONAL ├── solution/ │ └── solve.sh # REQUIRED - Oracle solution (must pass 100%) └── tests/ ├── test.sh # REQUIRED - Runs pytest └── test_outputs.py # REQUIRED - Writes reward to /logs/verifier/reward.txt ``` -------------------------------- ### Free up Docker space Source: https://skillsbench.ai/docs/getting-started Run 'docker system prune' to free up disk space used by Docker. This is recommended when Docker build failures occur due to insufficient space. ```bash # Free up Docker space if builds fail docker system prune ``` -------------------------------- ### Export External API Keys Source: https://skillsbench.ai/docs/getting-started Export necessary API keys as environment variables to enable tasks that interact with external services. ```bash export OPENAI_API_KEY=sk-... export ANTHROPIC_API_KEY=sk-ant-... export GEMINI_API_KEY=... export ELEVENLABS_API_KEY=... export GH_AUTH_TOKEN=ghp_... export HUGGINGFACE_API_TOKEN=hf_... export MODAL_TOKEN_ID=ak-... export MODAL_TOKEN_SECRET=as-... ``` -------------------------------- ### Run Self-Contained Subset via CLI Flags Source: https://skillsbench.ai/docs/getting-started Alternatively, exclude specific tasks using CLI flags to run only the self-contained subset of SkillsBench. ```bash uv run harbor jobs start -d skillsbench -a -m "" \ -x gh-repo-analytics \ -x mhc-layer-impl \ -x pedestrian-traffic-counting \ -x pg-essay-to-audiobook \ -x scheduling-email-assistant \ -x speaker-diarization-subtitles \ -x trend-anomaly-causal-inference \ -x video-filler-word-remover \ -x video-tutorial-indexer ``` -------------------------------- ### Configure Self-Contained Subset via YAML Source: https://skillsbench.ai/docs/getting-started Use a YAML configuration file to exclude tasks requiring external API keys or having build issues, running only the self-contained tasks. ```yaml jobs_dir: jobs n_attempts: 1 timeout_multiplier: 3.0 orchestrator: type: local n_concurrent_trials: 4 quiet: false environment: type: docker force_build: true delete: true agents: - name: oracle model_name: oracle datasets: - path: datasets/skillsbench exclude_task_names: - gh-repo-analytics # requires GH_AUTH_TOKEN - mhc-layer-impl # requires MODAL_TOKEN_ID/SECRET - pedestrian-traffic-counting # requires OPENAI/GEMINI/ANTHROPIC API keys - pg-essay-to-audiobook # requires OPENAI_API_KEY + ELEVENLABS_API_KEY - scheduling-email-assistant # hardcoded volume mount + HUGGINGFACE_API_TOKEN - speaker-diarization-subtitles # Docker build OOM (Whisper large-v3) - trend-anomaly-causal-inference # requires ANTHROPIC + OPENAI API keys - video-filler-word-remover # requires OPENAI_API_KEY - video-tutorial-indexer # requires OPENAI_API_KEY ``` -------------------------------- ### Reduce concurrent trials for API rate limiting Source: https://skillsbench.ai/docs/getting-started Adjust the 'n_concurrent_trials' setting under 'orchestrator' to reduce concurrency and avoid API rate limiting errors. This is useful for tasks calling external APIs. ```yaml orchestrator: type: local n_concurrent_trials: 2 # reduce from 4 to avoid rate limits ``` -------------------------------- ### Exclude tasks in job config YAML Source: https://skillsbench.ai/docs/getting-started Configure task exclusions directly within your job configuration YAML file using the 'exclude_task_names' list. This provides a declarative way to manage skipped tasks. ```yaml datasets: - path: datasets/skillsbench exclude_task_names: - gh-repo-analytics - mhc-layer-impl - pedestrian-traffic-counting - pg-essay-to-audiobook - scheduling-email-assistant - speaker-diarization-subtitles - trend-anomaly-causal-inference - video-filler-word-remover - video-tutorial-indexer ``` -------------------------------- ### Increase timeout multiplier Source: https://skillsbench.ai/docs/getting-started Set 'timeout_multiplier' to a higher value in your YAML config to increase agent and build timeouts. Use this for tasks with long builds or heavy computation. ```yaml timeout_multiplier: 3.0 # multiplies both agent and build timeouts ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.