Harbor

https://www.harborframework.com/docs
Admin
Harbor is a framework for evaluating and optimizing agents and models in container environments,...
Tokens:57,835
Snippets:780
Trust Score:7.2
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install Harbor CLI

Source: https://www.harborframework.com/docs/getting-started

Standard installation command for the Harbor tool using uv.

```bash
uv tool install harbor
```

--------------------------------

### Install All Reward Kit Extras

Source: https://www.harborframework.com/docs/rewardkit/built-in-criteria

Command to install all optional dependencies for Reward Kit.

```bash
uv add harbor-rewardkit[all]
```

--------------------------------

### adapter_metadata.json Schema Example

Source: https://www.harborframework.com/docs/datasets/adapters

Example configuration for the adapter_metadata.json file, detailing benchmark splits and parity experiment metrics.

```json
[
  {
    "adapter_name": "my-benchmark",
    "adapter_builders": ["Jane Doe (jane@example.com)"],
    "original_benchmark": [
      {
        "split": "full",
        "size": 500,
        "harness": "agent",
        "supported_agents": ["codex@0.1.0"],
        "adaptable": true,
        "notes": "None"
      }
    ],
    "harbor_adapter": [
      {
        "split": "full",
        "adapted_benchmark_size": 500,
        "parity_benchmark_size": 100,
        "parity_sampling_rate": 0.2,
        "registry_benchmark_size": 500,
        "added_agents": ["None"],
        "parity_matching_agents": ["codex@0.1.0+gpt-5-2025-06-01"],
        "parity_unmatching_agents": ["None"],
        "parity_costs": "$150",
        "notes": "Parity on 100-task subset. Averaged over 3 trials."
      }
    ]
  }
]
```

--------------------------------

### Task configuration example

Source: https://www.harborframework.com/docs/tasks

A comprehensive example of a task.toml file defining metadata, verifier settings, agent requirements, and environment resources.

```toml
schema_version = "1.4"

[task]
name = "<org>/<name>"
version = "1.0.0"
description = "A short description of the task"
authors = [{ name = "Steve Jobs", email = "steve@apple.com" }]
keywords = ["trivial", "programming"]

[metadata]
difficulty_explanation = "Trivial task for demonstration"
category = "programming"

[verifier]
timeout_sec = 120.0
env = { API_KEY = "sk-test-123" }
user = "root"  # optional: run the verifier as this OS user

[agent]
timeout_sec = 120.0
user = "agent"  # optional: run the agent as this OS user

[solution]
env = { API_KEY = "sk-test-123" }

[environment]
network_mode = "no-network"  # baseline; defaults to "public" when omitted
build_timeout_sec = 600.0
docker_image = "some-org/some-name:some-tag"
os = "linux"  # or "windows" to target Windows containers
cpus = 1
memory_mb = 2048
storage_mb = 10240
gpus = 0
gpu_types = ["H100", "A100"]
env = { SOME_ENV_VAR = "${SOME_ENV_VAR}" } # harbor run requests approval from the user for these env vars

[environment.tpu]                   # optional; omit the table if you don't need TPUs
type = "v6e"                        # alias (v3, v4, v5e, v5p, v6e, v7, trillium, ironwood) or canonical GKE label
topology = "2x4"                    # required; per-pod chip count = product of dimensions (here, 8)
# A task allocates one TPU slice per pod; specify a single spec rather than a list.
# Currently only the GKE environment honors this field.

[[environment.mcp_servers]]
name = "mcp-server"
transport = "streamable-http"
url = "http://mcp-server:8000/mcp"

[environment.healthcheck]
command = "curl -f http://localhost:8080/health"
interval_sec = 5.0
timeout_sec = 30.0
retries = 3
```

--------------------------------

### Install Harbor Nightly Build

Source: https://www.harborframework.com/docs/getting-started

Install the latest development release from the main branch using the pre-release flag.

```bash
uv tool install --prerelease explicit 'harbor>=0.dev0'
```

--------------------------------

### Install Publish Skill

Source: https://www.harborframework.com/docs/datasets/publishing

Install the 'publish' skill for your coding agent to assist with publishing datasets.

```bash
npx skills add harbor-framework/harbor --skill publish
```

--------------------------------

### Install Reward Kit Package

Source: https://www.harborframework.com/docs/rewardkit

Install the standalone Reward Kit package or the version with all document processing extras.

```bash
uv tool install harbor-rewardkit
```

```bash
uv tool install harbor-rewardkit[all]
```

--------------------------------

### Install create-task skill

Source: https://www.harborframework.com/docs/tasks/multi-step

Use this command to add the create-task skill to your Harbor project for guided multi-step task creation.

```bash
npx skills add harbor-framework/harbor --skill create-task
```

--------------------------------

### Clone and branch the Harbor repository

Source: https://www.harborframework.com/docs/datasets/adapters-human

Initial setup steps to prepare a local environment for adapter development.

```bash
git clone https://github.com/{you}/harbor.git
cd harbor
git checkout -b {adapter-name}-adapter
```

--------------------------------

### Run Terminal-Bench Oracle Solutions

Source: https://www.harborframework.com/docs/tutorials/running-terminal-bench

Installs and runs the default oracle solutions for Terminal-Bench 2.0 using Harbor. Ensure Docker is installed and running.

```bash
harbor run -d terminal-bench/terminal-bench-2 -a oracle
```

--------------------------------

### Configure Datasets

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Example of defining a git repository dataset with a specified path and task limit.

```json
{
  "datasets": [
    {
      "repo": "my-org/my-tasks@main",
      "path": "tasks",
      "n_tasks": 5
    }
  ]
}
```

--------------------------------

### Quick Start: Run Dataset from Git Repository

Source: https://www.harborframework.com/docs/datasets/git-repos

This command clones a repository, finds the dataset manifest (registry.json), and resolves the specified dataset and model.

```bash
harbor run --repo org/repo-name -d my-dataset -a claude-code -m anthropic/claude-sonnet-4
```

--------------------------------

### Start Harbor Task Environment

Source: https://www.harborframework.com/docs/tasks/task-tutorial

Command to start a Harbor task environment interactively using Docker. This allows manual testing of commands within the container.

```bash
harbor task start-env -p ssh-key-pair -e docker -a -i

```

--------------------------------

### Windows Dockerfile Example

Source: https://www.harborframework.com/docs/tasks/windows-container-support

A minimal Dockerfile configuration using a Windows Server Core base image.

```dockerfile
FROM mcr.microsoft.com/windows/servercore:ltsc2022
WORKDIR C:\\app
```

--------------------------------

### Self-deleting setup script

Source: https://www.harborframework.com/docs/tasks/multi-step

A template for a workdir/setup.sh script that performs initialization and removes itself before the agent runs.

```bash
#!/usr/bin/env bash
set -euo pipefail

# ... your setup work ...

rm -- "$0"
```

--------------------------------

### Example comparison output

Source: https://www.harborframework.com/docs/rewardkit

The output format generated when comparing multiple verifier directories.

```text
Comparison:
------------------------------------------
reward          v1      v2    diff
------------------------------------------
correctness  0.7500  0.9000  -0.1500
structure    1.0000  1.0000  +0.0000
------------------------------------------
```

--------------------------------

### Publish with tags and visibility

Source: https://www.harborframework.com/docs/tasks/publishing

Example of publishing a task with a custom tag and public visibility.

```bash
harbor publish "<path/to/task>" -t benchmark-baseline --public
```

--------------------------------

### Install Image Processing Extras

Source: https://www.harborframework.com/docs/rewardkit/built-in-criteria

Command to install the necessary dependencies for image-related criteria.

```bash
uv add harbor-rewardkit[image]
```

--------------------------------

### Install Reward Kit Skill

Source: https://www.harborframework.com/docs/rewardkit

Add the Reward Kit skill to your coding agent using npx.

```bash
npx skills add harbor-framework/harbor --skill rewardkit
```

--------------------------------

### Job Secret Injection Example

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Demonstrates how job secrets are resolved when provided in the launch configuration.

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY"]
}
"job_secrets": {}
```

```json
{
  "name": "claude-code",
  "model_name": "anthropic/claude-opus-4-1",
  "secrets": ["ANTHROPIC_API_KEY"]
}
"job_secrets": {"ANTHROPIC_API_KEY": "sk-ant-job-secret"}
```

--------------------------------

### Configure Individual Tasks

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Example of a job configuration defining both a git-based task and a registry-based task.

```json
{
  "config": {
    "job_name": "individual-tasks",
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-5-mini",
        "secrets": ["OPENAI_API_KEY"]
      }
    ],
    "tasks": [
      {
        "git_url": "https://github.com/owner/private-tasks.git",
        "git_ref": "main",
        "path": "tasks/my-task"
      },
      {
        "name": "terminal-bench/torch-tensor-parallelism",
        "ref": "latest"
      }
    ]
  },
  "job_secrets": {
    "OPENAI_API_KEY": "sk-..."
  },
  "dry_run": false
}
```

--------------------------------

### Example reward.json output

Source: https://www.harborframework.com/docs/rewardkit

The resulting JSON file containing both per-dimension scores and aggregated keys.

```json
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6, "reward": 0.0, "soft_score": 0.78 }
```

--------------------------------

### Docker Compose Volume Mount Syntax

Source: https://www.harborframework.com/docs/tasks/windows-container-support

Examples of short-form and long-form volume mount syntax for Docker Compose.

```yaml
volumes:
  - C:/host/path:/container/path
```

```yaml
services:
  main:
    volumes:
      - type: bind
        source: C:/git/harbor/jobs/<job>/trials/<trial>/verifier
        target: C:/logs/verifier
```

--------------------------------

### Windows Test Script Implementation

Source: https://www.harborframework.com/docs/tasks/windows-container-support

A batch script example for writing a reward value to the required log location.

```batch
@echo off
echo 1 > "%LOGS_DIR%\verifier\reward.txt"
```

--------------------------------

### Job Configuration YAML

Source: https://www.harborframework.com/docs/hosted-harbor/cli

Example configuration file for defining agents, datasets, tasks, and job-specific secrets for a Harbor job.

```yaml
# config.yaml
job_name: opus-vs-codex-sweep
organization: my-org          # omit to use your personal org
credential_mode: gateway      # or "direct"

n_attempts: 3                 # 1-10
n_concurrent_trials: 32       # 1-1000

# Every agent runs against every task. Trials = n_attempts x tasks x agents.
agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-1
    secrets: [ANTHROPIC_API_KEY]

  - name: claude-code
    model_name: anthropic/claude-sonnet-4-5
    secrets: [ANTHROPIC_API_KEY]

  - name: codex
    model_name: openai/gpt-5
    secrets: [OPENAI_API_KEY, HF_TOKEN]
    n_concurrent: 8           # per-agent sub-limit under n_concurrent_trials
    env:
      RUST_LOG: info          # nonsensitive only - secrets go in `secrets`

  - name: oracle              # needs no inference credential
    secrets: []

datasets:
  - name: terminal-bench/terminal-bench-2-1
    ref: "6"                  # `ref` or `version`, never both
    n_tasks: 25
    exclude_task_names:
      - "flaky-*"

tasks:
  - name: harbor/hello-world
    ref: latest

# Supplied for this job only, read from your local environment at launch.
job_secrets:
  HF_TOKEN:
    from_env: HF_TOKEN

retry:
  max_retries: 2
  include_exceptions:
    - EnvironmentStartError
```

--------------------------------

### Launch Harbor Results Viewer

Source: https://www.harborframework.com/docs/run-jobs/run-evals

Start the local web-based viewer to analyze job results. Point it to your 'jobs' directory to access stored configurations, results, and trial data.

```bash
harbor view jobs
```

--------------------------------

### Turn 3 Conversation History Example with Context Summarization

Source: https://www.harborframework.com/docs/agents/terminus-2

Demonstrates the conversation history for Turn 3 after context summarization, including original task context and previous agent summaries.

```json
// Context summarization happens here

Turn 3:
  {"user": "You are an AI assistant tasked with solving command-line tasks...\n           Task Description: Create a file called hello.txt\n           Current terminal state: ..."}
  {"user": "You are picking up work from a previous AI agent on this task:\n           **Original Task:** Create a file called hello.txt\n           **Summary from Previous Agent:** ...\n           **Current Terminal Screen:** ...\n           Please begin by asking several questions..."}
  {"assistant": "1. What files have been created so far?\n2. ..."}
  {"user": "Here are the answers the other agent provided.\n\n[answers]\n\nContinue working on this task..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
```

--------------------------------

### Apply Enforcement Policies via CLI

Source: https://www.harborframework.com/docs/tasks/managing-resources

Use command-line flags to set CPU and memory enforcement policies when running a Harbor task. This example sets CPU to 'limit' and memory to 'guarantee'.

```bash
harbor run -p "<path/to/dataset>" -m "<model>" -a "<agent>" \
  -e docker --cpus limit --memory guarantee

```

--------------------------------

### Install Document Processing Extras

Source: https://www.harborframework.com/docs/rewardkit/built-in-criteria

Command to install the necessary dependencies for xlsx_cell_equals.

```bash
uv add   harbor-rewardkit[documents]
```

--------------------------------

### Example metric.py Script

Source: https://www.harborframework.com/docs/datasets/metrics

This Python script calculates the mean reward from a JSONL input file and writes the result to a JSON output file. It includes argument parsing for input and output paths.

```python
# /// script
# dependencies = []
# ///
# To add a dependency: uv add --script metric.py <dependency>

import argparse
import json
from pathlib import Path


def main(input_path: Path, output_path: Path):
    rewards = []

    for line in input_path.read_text().splitlines():
        reward = json.loads(line)
        if reward is None:
            rewards.append(0)
        elif len(reward) != 1:
            raise ValueError(
                f"Expected exactly one key in reward dictionary, got {len(reward)}"
            )
        else:
            rewards.extend(reward.values())

    mean = sum(rewards) / len(rewards) if rewards else 0

    output_path.write_text(json.dumps({"mean": mean}))


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "-i",
        "--input-path",
        type=Path,
        required=True,
        help="Path to a jsonl file containing rewards, one json object per line.",
    )
    parser.add_argument(
        "-o",
        "--output-path",
        type=Path,
        required=True,
        help="Path to a json file where the metric will be written as a json object.",
    )
    args = parser.parse_args()
    main(args.input_path, args.output_path)
```

--------------------------------

### Initialize Dataset Configuration

Source: https://www.harborframework.com/docs/datasets/adapters

Create the dataset.toml file within the dataset directory.

```bash
cd /path/to/harbor-datasets/datasets/<adapter-name>
harbor init  # select "dataset"; enter the dataset name as <org>/<dataset>
```

--------------------------------

### Initialize Adapter Repository

Source: https://www.harborframework.com/docs/datasets/adapters

Commands to clone the Harbor repository and create a new branch for adapter development.

```bash
git clone https://github.com/{your-github-username}/harbor.git
cd harbor
git checkout -b {your-adapter-name}-adapter
```

--------------------------------

### Manage Harbor datasets and adapters

Source: https://www.harborframework.com/docs/datasets/adapters-human

Commands for listing datasets and initializing new adapter projects.

```bash
# List available datasets
harbor dataset list

# Scaffold a new adapter interactively
harbor adapter init

# Or with arguments
harbor adapter init my-adapter --name "My Benchmark"
```

--------------------------------

### Execute the adapter CLI

Source: https://www.harborframework.com/docs/datasets/adapters-human

Command to run the adapter and generate task directories.

```bash
uv run python -m {adapter_name}.main --output-dir <path>
```

--------------------------------

### View Harbor Help

Source: https://www.harborframework.com/docs/getting-started

Display the list of all available CLI commands.

```bash
harbor --help
```

--------------------------------

### JSON Reward Output Example

Source: https://www.harborframework.com/docs/tutorials/llm-as-a-judge

Example of a JSON object written to `reward.json` containing a single score. Harbor supports both `reward.txt` and `reward.json`.

```json
{ "funny": 0.75 }
```

--------------------------------

### Initialize a new task with a metadata template

Source: https://www.harborframework.com/docs/tasks

Use this command to create a new task directory pre-populated with metadata and configuration defaults from a specified TOML file.

```bash
harbor tasks init "<task-name>" --metadata-template task-template.toml
```

--------------------------------

### GET /secrets

Source: https://www.harborframework.com/docs/hosted-harbor/secrets

Lists your secrets as metadata only.

```APIDOC
## GET /secrets

### Description
Lists secrets associated with the user or organization. Supports filtering by scope, job_id, and status.

### Method
GET

### Endpoint
/secrets

### Query Parameters
- **scope** (string) - Optional - Filter by scope.
- **job_id** (string) - Optional - Filter by job ID.
- **status** (string) - Optional - Filter by status (active, revoked, all). Defaults to active.

### Response
#### Success Response (200)
- **secrets** (array) - List of secret metadata objects.
```

--------------------------------

### GET /registry-credentials

Source: https://www.harborframework.com/docs/hosted-harbor/registry-credentials

Lists active registry credentials and returns their metadata.

```APIDOC
## GET /registry-credentials

### Description
Lists active credentials and returns metadata only. Use status parameters to filter results.

### Method
GET

### Endpoint
/registry-credentials

### Parameters
#### Query Parameters
- **status** (string) - Optional - Filter by status (e.g., 'active', 'revoked', 'all').
```

--------------------------------

### Define Job Secrets

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Example of the job_secrets map structure for environment variables.

```json
{ "OPENAI_API_KEY": "sk-..." }
```

--------------------------------

### GET /job-status

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Retrieves trial counts and status information for one or more jobs.

```APIDOC
## GET /job-status

### Description
Returns trial counts and status for one or more jobs. Supports both GET and POST methods.

### Method
GET / POST

### Endpoint
/job-status

### Parameters
#### Query Parameters
- **job_id** (string) - Optional - A single job UUID.
- **job_ids** (string) - Optional - A comma-separated list of job UUIDs.
- **force_combined** (boolean) - Optional - Set to true to get the combined overview shape for a single job.

### Request Body (POST)
- **job_ids** (array) - Required - List of job UUIDs.
- **force_combined** (boolean) - Optional - Whether to return the combined overview shape.

### Response
#### Success Response (200)
- **pending** (integer) - Count of pending trials.
- **running** (integer) - Count of running trials.
- **completed** (integer) - Count of completed trials.
- **failed** (integer) - Count of failed trials.
- **canceled** (integer) - Count of canceled trials.
- **total** (integer) - Total count of trials.
```

--------------------------------

### Initialize a new task

Source: https://www.harborframework.com/docs/tasks

Creates a new task directory structure using the specified organization and task name.

```bash
harbor init --task "<org>/<name>"
```

--------------------------------

### Initialize Agent configuration

Source: https://www.harborframework.com/docs/agents/trajectory-format

Configures agent metadata and class information.

```python
from harbor.models.trajectories import Agent

agent = Agent(
    name="openhands",
    version="0.9.0",
    model_name="gpt-4",
    extra={"agent_class": "CodeActAgent"}
)
```

--------------------------------

### Handle Credential Response

Source: https://www.harborframework.com/docs/hosted-harbor/registry-credentials

Example of a successful API response containing credential metadata.

```json
{
  "id": "00000000-0000-0000-0000-000000000003",
  "org_id": "00000000-0000-0000-0000-00000000000a",
  "provider": "gar",
  "registry_host": "us-east1-docker.pkg.dev",
  "display_name": "prod-puller",
  "fingerprint": "harbor-puller@my-project.iam.gserviceaccount.com",
  "status": "active",
  "created_at": "2026-08-13T14:30:00Z"
}
```

--------------------------------

### View Run Command Help

Source: https://www.harborframework.com/docs/getting-started

Display help information for the primary evaluation command.

```bash
harbor run --help
```

--------------------------------

### Initialize metric.py for a Dataset

Source: https://www.harborframework.com/docs/datasets/metrics

Use this command to create a `metric.py` file and automatically add it to your `dataset.toml` when initializing a dataset.

```bash
harbor init --dataset "<org>/<name>" --with-metric
```

--------------------------------

### Initialize a Dataset Manifest

Source: https://www.harborframework.com/docs/datasets/publishing

Initializes a new dataset manifest in the specified directory. Use the --with-metric flag to include a metric script.

```bash
harbor dataset init "<org>/<dataset>" \
  --description "A short description" \
  --author "Your Name <your@email.com>"
```

```bash
harbor dataset init "<org>/<dataset>" --with-metric
```

--------------------------------

### Initialize Dataset Configuration

Source: https://www.harborframework.com/docs/datasets/adapters-human

Commands to initialize the dataset configuration file within the target directory.

```bash
cd harbor-datasets/datasets/<adapter-name>
harbor init
# Select "dataset" when prompted, then enter the dataset name as <org>/<dataset>.
```

--------------------------------

### Run a Hello World simulated user trial

Source: https://www.harborframework.com/docs/run-jobs/simulated-user

Executes a basic trial using claude-code for both the target and user agents with the ACP bridge.

```bash
harbor run -t hello-world/hello-world \
  --agent claude-code --model anthropic/claude-sonnet-5 \
  --user-agent claude-code --user-model anthropic/claude-sonnet-5 \
  --bridge acp
```

--------------------------------

### Task Dockerfile Template

Source: https://www.harborframework.com/docs/datasets/adapters

A base Dockerfile structure for setting up the agent's workspace and installing dependencies.

```dockerfile
FROM python:3.13-slim
WORKDIR /workspace

RUN apt-get update && apt-get install -y \
    git \
    && rm -rf /var/lib/apt/lists/*

# Install benchmark-specific dependencies
# RUN pip install --no-cache-dir <packages>

# Copy task-specific files
# COPY . /workspace/
```

--------------------------------

### Implement an installed agent

Source: https://www.harborframework.com/docs/agents

Implement the BaseInstalledAgent interface to execute agents directly within the container environment.

```python
from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template
from harbor.environments.base import BaseEnvironment
from harbor.models.agent.context import AgentContext

class MyInstalledAgent(BaseInstalledAgent):
    async def install(self, environment: BaseEnvironment) -> None:
        """
        Install the agent in the environment. Use exec_as_root for system
        packages and exec_as_agent for user-level installs.
        """
        await self.exec_as_root(environment, command="apt-get update && apt-get install -y curl")
        await self.exec_as_agent(environment, command="pip install my-agent")

    @with_prompt_template
    async def run(
        self, instruction: str, environment: BaseEnvironment, context: AgentContext
    ) -> None:
        """
        Run the agent in the environment. The @with_prompt_template decorator
        automatically applies prompt template rendering to the instruction.
        Use exec_as_agent to execute commands as the configured agent user.
        """
        await self.exec_as_agent(
            environment,
            command=f"my-agent run {shlex.quote(instruction)}",
        )

    def populate_context_post_run(self, context: AgentContext) -> None:
        """
        Populate the context with the results of the agent execution.
        Called after run() completes. Typically involves parsing trajectory files.
        """
        pass
```

--------------------------------

### Configure local ACP agent

Source: https://www.harborframework.com/docs/agents/acp

Define an ACP agent that is already installed within the task environment.

```yaml
agents:
  - name: acp
    kwargs:
      registry_entry:
        id: local-agent
        name: Local Agent
        version: local
        description: ACP agent provided by the task image
        distribution:
          local:
            cmd: node
            args: [/opt/local-agent/server.js]
```

--------------------------------

### Registry Configuration Conversion

Source: https://www.harborframework.com/docs/datasets/adapters

Shows the transition from a JSON-based registry file to the Harbor initialization workflow.

```json
{
    "name": "my-adapter",
    "version": "head",
    "description": "...",
    "github_url": "https://github.com/laude-institute/terminal-bench-datasets.git",
    "dataset_path": "datasets/my-adapter",
    "task_id_subset": null
}
```

```bash
harbor init  # select "dataset", creates dataset.toml
# Edit dataset.toml with descriptions, authors, credits
# Then submit to Harbor team for publishing
```

--------------------------------

### Initialize Observation model

Source: https://www.harborframework.com/docs/agents/trajectory-format

Captures environment feedback resulting from tool calls.

```python
from harbor.models.trajectories import Observation, ObservationResult

observation = Observation(
    results=[
        ObservationResult(
            source_call_id="call_price_1",
            content="GOOGL is currently trading at $185.35"
        )
    ]
)
```

--------------------------------

### View multi-reward JSON output

Source: https://www.harborframework.com/docs/rewardkit

Example of the JSON structure produced when multiple reward dimensions are evaluated.

```json
{ "correctness": 0.75, "structure": 1.0, "quality": 0.6 }
```

--------------------------------

### Define Scoring Aggregation

Source: https://www.harborframework.com/docs/rewardkit/judge-criteria

Example of configuring the scoring aggregation method and threshold in the judge TOML file.

```toml
[scoring]
aggregation = "all-pass"  # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7           # only used with "threshold" aggregation
```

--------------------------------

### Initialize Step model

Source: https://www.harborframework.com/docs/agents/trajectory-format

Represents an interaction step, supporting both user messages and agent tool calls with associated metrics.

```python
from harbor.models.trajectories import Step

# User step
user_step = Step(
    step_id=1,
    timestamp="2025-01-15T10:30:00Z",
    source="user",
    message="Create a file called hello.txt with 'Hello, world!' as the content."
)

# Agent step with tool calls
agent_step = Step(
    step_id=2,
    timestamp="2025-01-15T10:30:02Z",
    source="agent",
    model_name="claude-3-5-sonnet-20241022",
    message="I'll create the file for you.",
    reasoning_content="The user wants a simple text file. I'll use the file_write tool.",
    tool_calls=[
        ToolCall(
            tool_call_id="call_1",
            function_name="file_write",
            arguments={"path": "hello.txt", "content": "Hello, world!"}
        )
    ],
    observation=Observation(
        results=[
            ObservationResult(
                source_call_id="call_1",
                content="File created successfully"
            )
        ]
    ),
    metrics=Metrics(
        prompt_tokens=520,
        completion_tokens=80,
        cached_tokens=200,
        cost_usd=0.00045
    )
)
```

--------------------------------

### Regrade Delta Summary Output

Source: https://www.harborframework.com/docs/run-jobs/regrade

Example of the output summary printed by Harbor after completing a job regrade.

```text
Regrade delta over 1 trial(s): 1 changed (0 up, 1 down), mean reward 1.000 → 0.500
```

--------------------------------

### Check Job Status

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Retrieve trial counts for jobs using a GET request with authorization headers.

```bash
curl -sS "$BASE/job-status?job_id=<uuid>" \
  -H "Authorization: Bearer $KEY"
```

--------------------------------

### Run RewardKit with configuration flags

Source: https://www.harborframework.com/docs/rewardkit

Execute tests with custom workspace, output path, and concurrency limits.

```bash
rewardkit <tests_dirs...> \
    --workspace /app \
    --output /logs/verifier/reward.json \
    --max-concurrent-programmatic 8 \
    --max-concurrent-llm 8 \
    --max-concurrent-agent 2
```

--------------------------------

### Execute Harbor experiments

Source: https://www.harborframework.com/docs/datasets/adapters

Command-line interface usage for running experiments using the specified configuration file.

```bash
# Oracle verification (default)
harbor run -c adapters/<adapter-name>/run_<adapter-name>.yaml

# Switch agent by uncommenting the desired agent block
```

--------------------------------

### Job Submission Request Body

Source: https://www.harborframework.com/docs/hosted-harbor/submitting-jobs

Example JSON payload for launching a job with a registry dataset and agent configuration.

```json
{
  "config": {
    "job_name": "I-love-harbor",
    "agents": [
      {
        "name": "terminus-2",
        "model_name": "openai/gpt-5.6-luna",
        "secrets": ["OPENAI_API_KEY"]
      }
    ],
    "datasets": [
      {
        "name": "harbor/hello-world",
        "ref": "latest",
        "n_tasks": 5
      }
    ],
    "n_attempts": 1,
    "n_concurrent_trials": 20
  },
  "job_secrets": {
    "OPENAI_API_KEY": "sk-..."
  },
  "dry_run": false
}
```

--------------------------------

### Run Dataset from Registry

Source: https://www.harborframework.com/docs/datasets/adapters-human

Command to execute a published dataset directly from the registry.

```bash
harbor run -d <organization-name>/<adapter-name>
```

--------------------------------

### Define run_{adapter-name}.yaml configuration

Source: https://www.harborframework.com/docs/datasets/adapters

Configuration file defining datasets, agents, environment settings, and orchestrator parameters for experiments.

```yaml
datasets:
  - path: datasets/<adapter-name>

# Default: oracle agent for verification
agents:
  - name: oracle

# Uncomment to run with other agents:
# agents:
#   - name: codex
#     model_name: openai/gpt-5-mini
#
# agents:
#   - name: claude-code
#     model_name: claude-sonnet-4-5-20250929

environment:
  type: docker
  delete: true

orchestrator:
  type: local
  n_concurrent_trials: 4
```

--------------------------------

### Configure Individual Mode Criteria

Source: https://www.harborframework.com/docs/rewardkit/judge-criteria

Example of setting the judge mode to individual and defining specific files for each criterion.

```toml
[judge]
judge = "anthropic/claude-sonnet-5"
mode = "individual"

[[criterion]]
description = "Is the analysis correct?"
files = ["/app/analysis.pdf"]

[[criterion]]
description = "Is the spreadsheet well-structured?"
files = ["/app/data.xlsx"]
```

--------------------------------

### Docker Volume Specification Error

Source: https://www.harborframework.com/docs/tasks/windows-container-support

Example of an invalid volume specification error caused by colon-based path parsing on Windows.

```text
invalid volume specification: 'C:/git/harbor/jobs/.../verifier:/logs/verifier:rw'
```

--------------------------------

### Initialize Metrics model

Source: https://www.harborframework.com/docs/agents/trajectory-format

Tracks LLM operational metrics including token usage and costs.

```python
from harbor.models.trajectories import Metrics

metrics = Metrics(
    prompt_tokens=520,
    completion_tokens=80,
    cached_tokens=200,
    cost_usd=0.00045,
    logprobs=[-0.1, -0.05, -0.02],  # Optional
    completion_token_ids=[1722, 310, 5533]  # Optional
)
```

--------------------------------

### Verifier Test Script

Source: https://www.harborframework.com/docs/datasets/adapters

A shell script example for running tests and writing a numeric reward to the required log location.

```bash
#!/bin/bash
pytest /tests/test_*.py
if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

--------------------------------

### POST /job-submit

Source: https://www.harborframework.com/docs/hosted-harbor/api

Launch a remote rollout and check its status.

```APIDOC
## POST /job-submit

### Description
Launch a remote rollout and check its status.

### Method
POST

### Endpoint
https://ofhuhcpkvzjlejydnvyd.supabase.co/functions/v1/job-submit
```

--------------------------------

### Example metric.py Output Format

Source: https://www.harborframework.com/docs/datasets/metrics

The `metric.py` script should output a JSON object. Multiple metric values can be included in the output.

```json
{"mean": 0.81, "pass_rate": 0.5}
```

--------------------------------

### Run SWE-Bench Verified Dataset

Source: https://www.harborframework.com/docs/run-jobs/run-evals

Execute the 'swe-bench/swe-bench-verified' dataset. Similar to other benchmarks, you need to specify the model and agent for the run.

```bash
harbor run -d swe-bench/swe-bench-verified -m "<model>" -a "<agent>"
```

--------------------------------

### Run Parity Experiments with Adapters

Source: https://www.harborframework.com/docs/datasets/adapters

Commands to execute parity experiments using either a representative subset or the full dataset.

```bash
uv run python -m <adapter_name>.main --split parity --output-dir /path/to/output  # parity subset
uv run python -m <adapter_name>.main --output-dir /path/to/output                 # full dataset
```

--------------------------------

### Dockerfile for Task Environment

Source: https://www.harborframework.com/docs/tasks/task-tutorial

This Dockerfile sets up an Ubuntu 24.04 environment and installs the openssh-client, which is necessary for SSH operations within the task.

```docker
FROM ubuntu:24.04

# Create working directory
WORKDIR /app

# Install openssh-client for the task
RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/lists/*

```

--------------------------------

### Execute tests and generate reward file

Source: https://www.harborframework.com/docs/tasks

Example of a test script that runs pytest and writes the result to the required reward file location.

```bash
#!/bin/bash

uvx pytest /tests/test.py

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi
```

--------------------------------

### JSON Reward Output with Multiple Scores

Source: https://www.harborframework.com/docs/tutorials/llm-as-a-judge

Example of a JSON object containing multiple named metrics, such as creativity, humor, and grammar scores.

```json
{ "creativity": 0.9, "humor": 0.7, "grammar": 1.0 }
```

--------------------------------

### MCP Server Client Instruction

Source: https://www.harborframework.com/docs/tutorials/mcp-server-task

Provides step-by-step instructions for a client to connect to an MCP server, call the 'get_secret' tool, and write the returned secret to a file.

```markdown
There is an MCP server running at `http://mcp-server:8000/sse`
that exposes a tool called `get_secret`.

1. Connect to the MCP server
2. Call the `get_secret` tool
3. Write the returned secret value to `/app/secret.txt`
```

--------------------------------

### Turn 1 Conversation History Example

Source: https://www.harborframework.com/docs/agents/terminus-2

Illustrates the structure of the conversation history for Turn 1, including user prompts and assistant responses.

```json
Turn 1:
  {"user": "You are an AI assistant tasked with solving command-line tasks...\n           Task Description: Create a file called hello.txt\n           Current terminal state: ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
```

--------------------------------

### Run Parity Experiments

Source: https://www.harborframework.com/docs/datasets/adapters-human

Execute the Harbor parity experiment command using the specified adapter, agent, and model.

```bash
# Harbor side
harbor run -p datasets/<adapter-name> -a <agent> -m <model>
```

--------------------------------

### Adapter package structure

Source: https://www.harborframework.com/docs/datasets/adapters-human

The directory layout generated by the adapter initialization command.

```text
adapters/{adapter-name}/
├── .python-version              # (Optional)
├── pyproject.toml               # Python package config
├── README.md                    # Requirements checklist and final documentation
├── adapter_metadata.json        # Structured metadata about the adapter
├── parity_experiment.json       # Parity results (filled in later)
├── run_{adapter-name}.yaml      # Reference config to run the full adapted dataset
└── src/{adapter_name}/          # adapter-name with dashes replaced by underscores
    ├── __init__.py
    ├── adapter.py               # Core logic: parse benchmark data, generate task dirs
    ├── main.py                  # CLI entry point (--output-dir, --limit, --overwrite, --task-ids)
    └── task-template/           # Template files copied into each generated task
        ├── task.toml
        ├── instruction.md
        ├── environment/
        │   └── Dockerfile
        ├── solution/
        │   └── solve.sh
        └── tests/
            └── test.sh
```

--------------------------------

### Turn 2 Conversation History Example

Source: https://www.harborframework.com/docs/agents/terminus-2

Shows the conversation history for Turn 2, including user prompts, assistant responses, and new terminal output.

```json
Turn 2:
  {"user": "You are an AI assistant tasked with solving command-line tasks...\n           Task Description: Create a file called hello.txt\n           Current terminal state: ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
  {"user": "New Terminal Output:\nroot@container:/app# ..."}
  {"assistant": "{'analysis': '...', 'plan': '...', 'commands': [...]}"}
```

--------------------------------

### Initialize New Harbor Task

Source: https://www.harborframework.com/docs/tasks/task-tutorial

Create a new task directory with essential files for a Harbor task. This command initializes a task named 'ssh-key-pair'.

```bash
harbor task init ssh-key-pair
```

--------------------------------

### Configure simple artifact collection

Source: https://www.harborframework.com/docs/run-jobs/results-and-artifacts

List specific paths in job.yaml to mirror them under the trial's artifacts directory.

```yaml
artifacts:
  - /app/hello.txt
  - /workspace/output.csv
  - /data/results
```

--------------------------------

### Execute Python Judge Script with uv

Source: https://www.harborframework.com/docs/tutorials/llm-as-a-judge

A shell script to run the Python judge script using `uv run`. Ensure `uv` is installed and the Python script is accessible.

```bash
#!/bin/bash
uv run /tests/llm_judge.py
```
Tokens:
Raw