BenchLocal

https://github.com/stevibe/benchlocal
Admin
Local-first desktop app for running, comparing, and managing installable LLM Bench Packs against...
Tokens:34,639
Snippets:350
Trust Score:7.4
License:MIT
Update:5 months ago
Show doc for...
Tokens:
Raw
### Setup macOS Release Environment

Source: https://github.com/stevibe/benchlocal/blob/main/docs/macos-release.md

Initiates the interactive setup helper for the macOS release environment. This command guides through the necessary configurations for local signing and notarization.

```bash
npm run release:setup:mac
```

--------------------------------

### HTTP Workflow Example: Start Run

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to start a new run for a specific tab. Allows configuration of execution mode and runs per test. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/tabs/$TAB_ID/runs" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{"executionMode":"serial_by_model","runsPerTest":1}'
```

--------------------------------

### Install @benchlocal/sdk

Source: https://github.com/stevibe/benchlocal/blob/main/packages/benchlocal-sdk/README.md

Install the SDK using npm.

```bash
npm install @benchlocal/sdk
```

--------------------------------

### Get Installable Bench Packs Registry

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Fetches a list of Bench Packs available for installation from the registry. The response may be an empty array if the registry is empty or inaccessible.

```json
{
  "registry": []
}
```

--------------------------------

### HTTP Workflow Example: Fetch Benchpacks

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to fetch available benchpacks from the agent. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl "$BENCHLOCAL_AGENT_BASE_URL/v1/benchpacks" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN"
```

--------------------------------

### HTTP Workflow Example: Fetch Workspaces

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to fetch available workspaces from the agent. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl "$BENCHLOCAL_AGENT_BASE_URL/v1/workspaces" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN"
```

--------------------------------

### Provider Configuration Example

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Example of configuring an 'openrouter' provider, specifying its kind, name, base URL, and API key environment variable.

```toml
[providers.openrouter]
kind = "openrouter"
name = "OpenRouter"
enabled = true
base_url = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

```

--------------------------------

### Verifier Preferences Configuration

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Example of configuring verifier preferences within an installed Bench Pack, specifying the mode ('docker') and auto-start setting.

```toml
[benchpacks.structoutput-15.verifiers.verifier]
mode = "docker"
auto_start = true

```

--------------------------------

### MCP Client Configuration Example

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example JSON configuration for an MCP client, specifying the 'benchlocal' server details including type, URL, and authorization headers.

```json
{
  "mcpServers": {
    "benchlocal": {
      "type": "streamable-http",
      "url": "http://127.0.0.1:50060/mcp",
      "headers": {
        "Authorization": "Bearer <token>"
      }
    }
  }
}
```

--------------------------------

### GET /v1/benchpacks/registry

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Fetches a list of available Bench Packs from the registry that can be installed.

```APIDOC
## GET /v1/benchpacks/registry

### Description
Returns registry entries for installable Bench Packs.

### Method
GET

### Endpoint
/v1/benchpacks/registry

### Response
#### Success Response (200)
- **registry** (array) - An array of registry entries for Bench Packs.

### Response Example
```json
{
  "registry": []
}
```
```

--------------------------------

### Get Installed Bench Packs

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of installed Bench Packs and their associated scenario metadata. The response may be an empty array if no Bench Packs are installed.

```json
{
  "benchPacks": []
}
```

--------------------------------

### Install @benchlocal/core

Source: https://github.com/stevibe/benchlocal/blob/main/packages/benchlocal-core/README.md

Install the core package using npm. This package contains essential types and runtime contracts for the BenchLocal ecosystem.

```bash
npm install @benchlocal/core
```

--------------------------------

### Third-Party Bench Pack Install Configuration

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Configuration for a third-party Bench Pack installed from an archive URL, including its ID, enabled status, source, URL, and version.

```toml
[benchpacks.third-party-pack]
enabled = true
source = "archive"
url = "https://example.com/benchpack.tar.gz"
version = "1.0.0"

```

--------------------------------

### GET /v1/benchpacks

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of installed Bench Packs and their associated scenario metadata.

```APIDOC
## GET /v1/benchpacks

### Description
Returns installed Bench Packs and scenario metadata.

### Method
GET

### Endpoint
/v1/benchpacks

### Response
#### Success Response (200)
- **benchPacks** (array) - An array of installed Bench Pack objects.

### Response Example
```json
{
  "benchPacks": []
}
```
```

--------------------------------

### HTTP Workflow Example: Create Tab

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to create a new tab within a workspace. Requires workspace ID, benchpack ID, and a title. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/workspaces/$WORKSPACE_ID/tabs" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{"benchPackId":"toolcall-15","title":"ToolCall-15"}'
```

--------------------------------

### Agent Guide

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Provides agent-readable Markdown documentation for runtime bootstrapping.

```APIDOC
## GET /v1/agent-guide

### Description
Auth required. Returns agent-readable Markdown generated by the running app. This is intentionally shorter than this developer reference and is meant for runtime agent bootstrapping.

### Method
GET

### Endpoint
/v1/agent-guide
```

--------------------------------

### HTTP Workflow Example: Select Models

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to select models for a specific tab. Requires tab ID and a list of model IDs. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/tabs/$TAB_ID/select-models" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{"modelIds":["huggingface:Qwen/Qwen3.5-9B"]}'
```

--------------------------------

### Official Bench Pack Install Configuration

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Configuration for an officially installed Bench Pack, specifying its ID, enabled status, source ('registry'), and version.

```toml
[benchpacks.toolcall-15]
enabled = true
source = "registry"
version = "1.0.0"

```

--------------------------------

### Check macOS Release Environment

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Run this command to ensure the local macOS environment is properly configured for releasing. If setup is required, use the corresponding setup command.

```bash
npm run release:doctor:mac
```

```bash
npm run release:setup:mac
```

--------------------------------

### Start Tab Run

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Starts a new run for a tab. This is an asynchronous operation.

```APIDOC
## POST /v1/tabs/:tabId/runs

### Description
Starts a run for a tab. Run commands are asynchronous.

### Method
POST

### Endpoint
/v1/tabs/:tabId/runs

### Parameters
#### Path Parameters
- **tabId** (string) - Required - The ID of the tab to start a run for.

### Request Body
- **benchPackId** (string) - Optional - Defaults to the tab's selected Bench Pack.
- **modelIds** (string[]) - Optional - Defaults to the tab's selected models.
- **executionMode** (BenchLocalExecutionMode) - Optional - Defaults to the tab's execution mode.
- **runsPerTest** (number) - Optional - Defaults to the tab's runs-per-test.
- **generation** (GenerationRequest) - Optional - Defaults to the tab's sampling overrides.

### Response
#### Success Response (202)
- **accepted** (boolean) - Indicates if the run was accepted.
- **tabId** (string) - The ID of the tab.

The run will set `loadedRunId` on the tab after a summary is produced.
```

--------------------------------

### Benchlocal Polling for Events

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Tools that start benchmark work return immediately. Poll for results and updates using `benchlocal_get_recent_events`, the `benchlocal://state/events/recent` URI, or by making a `GET /v1/events` request.

```text
benchlocal_get_recent_events
benchlocal://state/events/recent
GET /v1/events
```

--------------------------------

### Get All Models

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of all configured models.

```APIDOC
## GET /v1/models

### Description
Returns configured models.

### Method
GET

### Endpoint
/v1/models

### Response
#### Success Response (200)
- **models** (array) - An array of model configurations.

### Response Example
```json
{
  "models": [
    {
      "id": "huggingface:Qwen/Qwen3.5-9B",
      "provider": "huggingface",
      "model": "Qwen/Qwen3.5-9B",
      "label": "Qwen3.5-9B",
      "group": "primary",
      "enabled": true
    }
  ]
}
```
```

--------------------------------

### Typical Bench Pack Usage with @benchlocal/sdk

Source: https://github.com/stevibe/benchlocal/blob/main/packages/benchlocal-sdk/README.md

Example of defining a Bench Pack using the SDK. This includes loading the manifest, defining scenarios, preparing the runtime context, and scoring model results. Ensure benchmark logic is kept in separate repo modules.

```typescript
import {
  createHostHelpers,
  defineBenchPack,
  loadBenchPackManifest,
  requireScoredResults
} from "@benchlocal/sdk";

const manifest = loadBenchPackManifest(__dirname);

export { manifest };

export default defineBenchPack({
  manifest,
  async listScenarios() {
    return [];
  },
  async prepare(context) {
    const helpers = createHostHelpers(context);

    return {
      async runScenario(input) {
        const provider = helpers.getRequiredProvider(input.model.provider, { enabledOnly: true });
        const inference = helpers.getInferenceEndpoint(input.model.id);

        return {
          scenarioId: input.scenario.id,
          status: "pass",
          summary: inference?.status === "running" ? inference.baseUrl : provider.baseUrl
        };
      },
      async dispose() {}
    };
  },
  scoreModelResults(results) {
    const scored = requireScoredResults(results);

    return {
      totalScore: scored.reduce((sum, result) => sum + result.score, 0),
      categories: []
    };
  }
});
```

--------------------------------

### Start Tab Run

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Starts a new run for a tab. This is an asynchronous operation. Defaults for `benchPackId`, `modelIds`, `executionMode`, `runsPerTest`, and `generation` are inherited from the tab's settings if not provided.

```json
{
  "accepted": true,
  "tabId": "tab-..."
}
```

--------------------------------

### Model Configuration Example

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Defines a specific model, linking it to a provider and assigning an ID, label, and group. The 'id' must remain stable.

```toml
[[models]]
id = "openrouter:openai/gpt-4.1"
provider = "openrouter"
model = "openai/gpt-4.1"
label = "GPT-4.1 via OpenRouter"
group = "primary"
enabled = true

```

--------------------------------

### Bench Pack Manifest Example (JSON)

Source: https://github.com/stevibe/benchlocal/blob/main/BENCH_PROTOCOL_V1.md

This JSON structure defines the metadata for a Bench Pack, including its ID, version, entry point, and capabilities. Ensure 'id', 'name', 'version', and 'entry' fields are correctly set.

```json
{
  "schemaVersion": 1,
  "protocolVersion": 1,
  "id": "bugfind-15",
  "name": "BugFind-15",
  "author": "stevibe",
  "version": "1.0.0",
  "description": "Execution-backed benchmark for bug finding and bug fixing.",
  "entry": "./dist/benchlocal/index.js",
  "requirements": {
    "benchlocal": {
      "minVersion": "0.2.0"
    },
    "hostFeatures": ["inferenceEndpoints", "dockerInferenceEndpoints"]
  },
  "samplingDefaults": {
    "temperature": 0
  },
  "capabilities": {
    "tools": false,
    "multiTurn": false,
    "streamingProgress": true,
    "verification": true
  },
  "verifiers": [
    {
      "id": "verifier",
      "transport": "http",
      "required": true,
      "defaultMode": "docker",
      "docker": {
        "buildContext": "./verification",
        "listenPort": 4010,
        "healthcheckPath": "/health"
      }
    }
  ]
}
```

--------------------------------

### Get Configured Models

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of all models that are currently configured.

```json
{
  "models": [
    {
      "id": "huggingface:Qwen/Qwen3.5-9B",
      "provider": "huggingface",
      "model": "Qwen/Qwen3.5-9B",
      "label": "Qwen3.5-9B",
      "group": "primary",
      "enabled": true
    }
  ]
}
```

--------------------------------

### Benchlocal/index.ts Adapter Layer Example

Source: https://github.com/stevibe/benchlocal/blob/main/BENCH_PACK_AUTHORING.md

Implement the BenchLocal adapter layer to load manifest, list scenarios, and bridge host context. This file should remain thin.

```typescript
import {
  createHostHelpers,
  defineBenchPack,
  loadBenchPackManifest,
  requireScoredResults,
  type ScenarioRunInput,
  type ScenarioResult
} from "@benchlocal/sdk";

import { SCENARIOS, getScenarioCards, scoreModelResults } from "../lib/benchmark";
import { runScenarioForModel } from "../lib/orchestrator";

const manifest = loadBenchPackManifest(__dirname);

export { manifest };

export default defineBenchPack({
  manifest,

  async listScenarios() {
    return SCENARIOS.map((scenario) => ({
      id: scenario.id,
      title: scenario.title,
      category: scenario.category,
      detailCards: getScenarioCards(scenario)
    }));
  },

  async prepare(context) {
    const helpers = createHostHelpers(context);

    return {
      async runScenario(input: ScenarioRunInput): Promise<ScenarioResult> {
        return runScenarioForModel(input, helpers);
      },
      async dispose() {}
    };
  },

  scoreModelResults(results) {
    return scoreModelResults(requireScoredResults(results));
  }
});
```

--------------------------------

### Registering a Tool with MCP

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example of how to register a tool with the MCP. This allows the agent to expose functionality through the MCP interface.

```APIDOC
## Register Tool

### Description
Registers a tool with the MCP, making it available for use by models.

### Method
`server.registerTool(name, definition, handler)`

### Parameters
- **name** (string) - The unique identifier for the tool.
- **definition** (object) - An object containing the tool's metadata:
  - **title** (string) - A human-readable title for the tool.
  - **description** (string) - A detailed explanation of what the tool does.
  - **inputSchema** (object) - A Zod schema defining the expected input for the tool.
  - **annotations** (object) - Additional metadata for the tool (e.g., `readOnlyHint`, `openWorldHint`).
- **handler** (function) - An asynchronous function that executes the tool's logic. It receives the parsed input and should return a tool result.

### Example
```typescript
server.registerTool(
  "benchlocal_example_action",
  {
    title: "Example Action",
    description: "Do the same operation exposed by the UI and HTTP API.",
    inputSchema: {
      id: z.string()
    },
    annotations: { readOnlyHint: false, openWorldHint: false }
  },
  async ({ id }) => jsonToolResult(await controller.example(id))
);
```
```

--------------------------------

### Benchlocal Start Run Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Initiate a benchmark run using `benchlocal_start_run`. Requires `tabId` and can optionally include `benchPackId`, `modelIds`, `executionMode`, `runsPerTest`, and `generation` settings. Returns `{ accepted: true, tabId }`.

```json
{ "tabId", "benchPackId"?, "modelIds"?, "executionMode"?, "runsPerTest"?, "generation"? }
```

--------------------------------

### HTTP Workflow Example: Refresh Model Availability

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Example using curl to refresh the availability of models for a specific tab. Requires tab ID. Ensure BENCHLOCAL_AGENT_BASE_URL and BENCHLOCAL_AGENT_TOKEN are set.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/tabs/$TAB_ID/models/availability/refresh" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{}'
```

--------------------------------

### Benchlocal Get Run Summary Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Get a summary of a specific completed benchmark run using `benchlocal_get_run_summary`. Requires `benchPackId` and `runId`. Returns the saved run summary.

```json
{ "benchPackId", "runId" }
```

--------------------------------

### List Models with Authorization

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

List available models by making a GET request to the /v1/models endpoint, including the necessary Authorization header.

```bash
curl "$BENCHLOCAL_AGENT_BASE_URL/v1/models" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN"

```

--------------------------------

### Top-Level Configuration Schema

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Defines the basic schema version and storage directory paths for BenchLocal. Fresh configurations start blank.

```toml
schema_version = 1
default_benchpack = ""
run_storage_dir = "~/.benchlocal/runs"
benchpack_storage_dir = "~/.benchlocal/benchpacks"
log_storage_dir = "~/.benchlocal/logs"
cache_dir = "~/.benchlocal/cache"

[registry]
official_url = "https://raw.githubusercontent.com/stevibe/benchlocal-registry/main/registry.json"

[ui]
theme = "system"

[agent]
enabled = false
access = "localhost"

```

--------------------------------

### Get Specific Model

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves details for a single, specified configured model.

```json
{
  "model": {
    "id": "huggingface:Qwen/Qwen3.5-9B",
    "provider": "huggingface",
    "model": "Qwen/Qwen3.5-9B",
    "label": "Qwen3.5-9B",
    "group": "primary",
    "enabled": true
  }
}
```

--------------------------------

### GET /v1/workspaces

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Fetches the complete state of all workspaces managed by the BenchLocal agent.

```APIDOC
## GET /v1/workspaces

### Description
Returns the full workspace state.

### Method
GET

### Endpoint
/v1/workspaces

### Response
#### Success Response (200)
- **path** (string) - The file path to the workspace state.
- **created** (boolean) - Indicates if the workspace state was newly created.
- **state** (object) - The workspace state object.

### Response Example
```json
{
  "path": "/Users/me/.benchlocal/state.json",
  "created": false,
  "state": {
    "schema_version": 1,
    "activeWorkspaceId": "workspace-main",
    "workspaceOrder": ["workspace-main"],
    "workspaces": {},
    "tabs": {}
  }
}
```
```

--------------------------------

### Run Management Tools

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Tools for starting, resuming, retrying, and stopping benchmark runs, as well as retrieving run history and summaries.

```APIDOC
## `benchlocal_start_run`

### Description
Starts a new benchmark run.

### Method
POST

### Endpoint
`/benchlocal/runs/start`

### Parameters
#### Request Body
- **tabId** (string) - Required - The ID of the tab to run.
- **benchPackId** (string) - Optional - The ID of the Bench Pack to use.
- **modelIds** (array) - Optional - An array of model IDs to include in the run.
- **executionMode** (string) - Optional - The execution mode for the run.
- **runsPerTest** (number) - Optional - The number of runs per test.
- **generation** (object) - Optional - Generation parameters.
```

```APIDOC
## `benchlocal_resume_run`

### Description
Resumes a previously stopped or failed benchmark run.

### Method
POST

### Endpoint
`/benchlocal/runs/{runId}/resume`

### Parameters
#### Path Parameters
- **runId** (string) - Required - The ID of the run to resume.
#### Request Body
- **tabId** (string) - Required - The ID of the tab associated with the run.
- **executionMode** (string) - Optional - The execution mode for the run.
- **runsPerTest** (number) - Optional - The number of runs per test.
- **generation** (object) - Optional - Generation parameters.
```

```APIDOC
## `benchlocal_retry_scenario`

### Description
Retries a specific scenario within a run for a given model.

### Method
POST

### Endpoint
`/benchlocal/runs/{runId}/scenarios/{scenarioId}/retry`

### Parameters
#### Path Parameters
- **runId** (string) - Required - The ID of the run.
- **scenarioId** (string) - Required - The ID of the scenario to retry.
- **modelId** (string) - Required - The ID of the model to use for retrying.
#### Request Body
- **tabId** (string) - Required - The ID of the tab associated with the run.
- **runsPerTest** (number) - Optional - The number of runs per test.
- **generation** (object) - Optional - Generation parameters.
```

```APIDOC
## `benchlocal_retry_provider_errors`

### Description
Retries all scenarios in a run that failed due to provider errors.

### Method
POST

### Endpoint
`/benchlocal/runs/{runId}/retry-provider-errors`

### Parameters
#### Path Parameters
- **runId** (string) - Required - The ID of the run.
#### Request Body
- **tabId** (string) - Required - The ID of the tab associated with the run.
- **runsPerTest** (number) - Optional - The number of runs per test.
- **generation** (object) - Optional - Generation parameters.
```

```APIDOC
## `benchlocal_retry_failed_results`

### Description
Retries all scenarios in a run that have failed results.

### Method
POST

### Endpoint
`/benchlocal/runs/{runId}/retry-failed-results`

### Parameters
#### Path Parameters
- **runId** (string) - Required - The ID of the run.
#### Request Body
- **tabId** (string) - Required - The ID of the tab associated with the run.
- **runsPerTest** (number) - Optional - The number of runs per test.
- **generation** (object) - Optional - Generation parameters.
```

```APIDOC
## `benchlocal_stop_run`

### Description
Stops a currently running benchmark.

### Method
POST

### Endpoint
`/benchlocal/runs/stop`

### Parameters
#### Request Body
- **tabId** (string) - Required - The ID of the tab associated with the run to stop.
```

```APIDOC
## `benchlocal_list_run_history`

### Description
Retrieves the run history for a specific Bench Pack.

### Method
GET

### Endpoint
`/benchlocal/benchpacks/{benchPackId}/history`

### Parameters
#### Path Parameters
- **benchPackId** (string) - Required - The ID of the Bench Pack.
```

```APIDOC
## `benchlocal_get_run_summary`

### Description
Retrieves a saved summary for a specific run of a Bench Pack.

### Method
GET

### Endpoint
`/benchlocal/benchpacks/{benchPackId}/runs/{runId}/summary`

### Parameters
#### Path Parameters
- **benchPackId** (string) - Required - The ID of the Bench Pack.
- **runId** (string) - Required - The ID of the run.
```

--------------------------------

### Get Saved Run Summary

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a summary of a specific saved run.

```json
{
  "run": {}
}
```

--------------------------------

### Launch App with Local Update URL

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Launch the installed BenchLocal application with the BENCHLOCAL_UPDATE_URL environment variable set to your local HTTP feed. This is intended for packaged app testing; dev mode disables self-update.

```bash
BENCHLOCAL_UPDATE_URL=http://127.0.0.1:9000/ /Applications/BenchLocal.app/Contents/MacOS/BenchLocal
```

--------------------------------

### GET /v1/config

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves the current BenchLocal agent configuration, with sensitive information like API keys redacted.

```APIDOC
## GET /v1/config

### Description
Returns redacted BenchLocal config.

### Method
GET

### Endpoint
/v1/config

### Response
#### Success Response (200)
- **config** (object) - The agent configuration object.

### Response Example
```json
{
  "config": {
    "schema_version": 1,
    "ui": { "theme": "system" },
    "agent": { "enabled": true, "access": "localhost", "port": 50060 },
    "providers": {
      "huggingface": {
        "kind": "huggingface",
        "name": "Hugging Face",
        "enabled": true,
        "base_url": "https://router.huggingface.co/v1",
        "api_key_env": "HF_TOKEN",
        "has_api_key": false,
        "has_api_key_env": true
      }
    },
    "models": []
  }
}
```
```

--------------------------------

### Get Commit Log Since Previous Tag

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Retrieve a one-line summary of commits from the previous tag up to the current HEAD. This is useful for generating release notes.

```bash
git log --oneline <previous-tag>..HEAD
```

--------------------------------

### Build All Release Artifacts

Source: https://github.com/stevibe/benchlocal/blob/main/docs/windows-release.md

Execute this command from the repository root to build all release artifacts, including Windows.

```bash
npm run release:all
```

--------------------------------

### Build Windows Release Artifacts

Source: https://github.com/stevibe/benchlocal/blob/main/docs/windows-release.md

Run this command from the BenchLocal directory to build the Windows release artifacts.

```bash
cd BenchLocal
npm run build:win
```

--------------------------------

### Build Linux Release Artifacts

Source: https://github.com/stevibe/benchlocal/blob/main/docs/linux-release.md

Run this command from the repository root to build the Linux release artifacts.

```bash
cd BenchLocal
npm run build:linux
```

--------------------------------

### Get Verifier Status

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves the current runtime status of all verifiers.

```APIDOC
## GET /v1/verifiers

### Description
Returns verifier runtime status.

### Method
GET

### Endpoint
/v1/verifiers

### Response
#### Success Response (200)
- **verifiers** (array) - An array of verifier statuses.

### Response Example
```json
{
  "verifiers": []
}
```
```

--------------------------------

### Get Specific Model

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a single configured model by its ID.

```APIDOC
## GET /v1/models/:modelId

### Description
Returns one configured model.

### Method
GET

### Endpoint
/v1/models/:modelId

### Parameters
#### Path Parameters
- **modelId** (string) - Required - The ID of the model to retrieve.

### Response
#### Success Response (200)
- **model** (object) - The configuration of the model.

### Response Example
```json
{
  "model": {
    "id": "huggingface:Qwen/Qwen3.5-9B",
    "provider": "huggingface",
    "model": "Qwen/Qwen3.5-9B",
    "label": "Qwen3.5-9B",
    "group": "primary",
    "enabled": true
  }
}
```
```

--------------------------------

### MCP Initialize

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Initializes the MCP connection with the agent, setting up communication parameters.

```APIDOC
## MCP Initialize

### Description
Initializes the Model Communication Protocol (MCP) connection with the BenchLocal agent. This is the first step in establishing communication.

### Method
POST

### Endpoint
`/mcp`

### Headers
- **Authorization**: Bearer $BENCHLOCAL_AGENT_TOKEN
- **content-type**: application/json
- **accept**: application/json, text/event-stream

### Request Body
- **jsonrpc** (string) - Required - Must be "2.0".
- **id** (number) - Required - A unique identifier for the request.
- **method** (string) - Required - Must be "initialize".
- **params** (object) - Required - Parameters for the initialize method:
  - **protocolVersion** (string) - Required - The version of the MCP protocol.
  - **capabilities** (object) - Optional - Client capabilities.
  - **clientInfo** (object) - Optional - Information about the client:
    - **name** (string) - Required - The name of the client.
    - **version** (string) - Required - The version of the client.

### Request Example
```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/mcp" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -H "accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2025-03-26",
      "capabilities": {},
      "clientInfo": {
        "name": "curl-smoke",
        "version": "0.0.0"
      }
    }
  }'
```
```

--------------------------------

### Create Provider

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Creates a new provider. The `id` is optional and will be generated if not provided. `api_key` and `api_key_env` are mutually exclusive.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/providers" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "id": "huggingface",
    "kind": "huggingface",
    "name": "Hugging Face",
    "enabled": true,
    "base_url": "https://router.huggingface.co/v1",
    "api_key_env": "HF_TOKEN"
  }'
```

--------------------------------

### Get All Providers

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of all configured providers, with their secrets redacted.

```APIDOC
## GET /v1/providers

### Description
Returns configured providers with secrets redacted.

### Method
GET

### Endpoint
/v1/providers

### Response
#### Success Response (200)
- **providers** (object) - An object containing provider configurations.

### Response Example
```json
{
  "providers": {
    "huggingface": {
      "kind": "huggingface",
      "name": "Hugging Face",
      "enabled": true,
      "base_url": "https://router.huggingface.co/v1",
      "api_key_env": "HF_TOKEN",
      "has_api_key": false,
      "has_api_key_env": true
    }
  }
}
```
```

--------------------------------

### Get Verifier Status

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves the current runtime status of all verifiers.

```json
{
  "verifiers": []
}
```

--------------------------------

### Benchlocal Create Provider Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Create a new provider using `benchlocal_create_provider`. Requires `kind`, `base_url`, and optionally `id`, `name`, `enabled`, `api_key`, or `api_key_env`. The `kind` must be one of the specified types.

```json
{ "id"?, "kind", "name"?, "enabled"?, "base_url", "api_key"?, "api_key_env"? }
```

--------------------------------

### Build Production App

Source: https://github.com/stevibe/benchlocal/blob/main/docs/macos-release.md

Compiles and packages the production app, generating DMG and ZIP artifacts. Use this for creating distributable releases.

```bash
npm run pack
```

--------------------------------

### Local Development Packaging

Source: https://github.com/stevibe/benchlocal/blob/main/BENCHLOCAL_REGISTRY_V1.md

Command to package a Bench Pack directly from a local disk during development, useful for testing registry integration.

```bash
cd /path/to/StructOutput-15
npm run build:benchlocal
```

--------------------------------

### Push Commit and Create Tag

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Push the release preparation commit to the main branch and create a Git tag for the release. The tag format must be `v<version>` to match the `electron-updater` configuration.

```bash
git push origin main
git tag v<version>
git push origin v<version>
```

--------------------------------

### Get Bench Pack History

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves the run history for a specific Bench Pack.

```APIDOC
## GET /v1/benchpacks/:benchPackId/history

### Description
Returns run history entries for a Bench Pack.

### Method
GET

### Endpoint
/v1/benchpacks/:benchPackId/history

### Parameters
#### Path Parameters
- **benchPackId** (string) - Required - The ID of the Bench Pack.

### Response
#### Success Response (200)
- **history** (array) - An array of run history entries.

### Response Example
```json
{
  "history": []
}
```
```

--------------------------------

### Get Active Benchmark Runs

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of currently active benchmark runs.

```APIDOC
## GET /v1/runs/active

### Description
Returns active benchmark runs.

### Method
GET

### Endpoint
/v1/runs/active

### Response
#### Success Response (200)
- **activeRuns** (array) - An array of active benchmark runs.

### Response Example
```json
{
  "activeRuns": []
}
```
```

--------------------------------

### Build Unpacked App Bundle

Source: https://github.com/stevibe/benchlocal/blob/main/docs/macos-release.md

Compiles the application and produces an unpacked local .app bundle. This is useful for local development and testing.

```bash
npm run build:dir
```

--------------------------------

### Get Specific Provider

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a single configured provider by its ID, with secrets redacted.

```APIDOC
## GET /v1/providers/:providerId

### Description
Returns one redacted provider.

### Method
GET

### Endpoint
/v1/providers/:providerId

### Parameters
#### Path Parameters
- **providerId** (string) - Required - The ID of the provider to retrieve.

### Response
#### Success Response (200)
- **providerId** (string) - The ID of the provider.
- **provider** (object) - The configuration of the provider.

### Response Example
```json
{
  "providerId": "huggingface",
  "provider": {
    "kind": "huggingface",
    "name": "Hugging Face",
    "enabled": true,
    "base_url": "https://router.huggingface.co/v1",
    "api_key_env": "HF_TOKEN",
    "has_api_key": false,
    "has_api_key_env": true
  }
}
```
```

--------------------------------

### Get Active Benchmark Runs

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of currently active benchmark runs.

```json
{
  "activeRuns": []
}
```

--------------------------------

### `benchlocal-run-benchpack` Command

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

A command-line interface for running a Bench Pack with specified models and workspace.

```APIDOC
## `benchlocal-run-benchpack`

### Description
Executes a Bench Pack, allowing selection of models and workspace.
This command is part of the preferred workflow for teaching an MCP-capable agent.

### Arguments
#### Request Body (assumed structure for CLI arguments)
- **benchPackId** (string) - Required - The ID of the Bench Pack to run.
- **modelIds** (string) - Required - A comma-separated string of model IDs to use.
- **workspaceId** (string) - Optional - The ID of the workspace to use.
```

--------------------------------

### Get Configured Providers

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a list of all configured providers, with their sensitive information redacted.

```json
{
  "providers": {
    "huggingface": {
      "kind": "huggingface",
      "name": "Hugging Face",
      "enabled": true,
      "base_url": "https://router.huggingface.co/v1",
      "api_key_env": "HF_TOKEN",
      "has_api_key": false,
      "has_api_key_env": true
    }
  }
}
```

--------------------------------

### Create Model

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Creates a new model. The `id` is optional and will be generated if not provided. Ensure the `provider` field matches an existing provider.

```bash
curl -X POST "$BENCHLOCAL_AGENT_BASE_URL/v1/models" \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "id": "huggingface:Qwen/Qwen3.5-9B",
    "provider": "huggingface",
    "model": "Qwen/Qwen3.5-9B",
    "label": "Qwen3.5-9B",
    "group": "primary",
    "enabled": true
  }'
```

--------------------------------

### Get Specific Provider

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves details for a single, specified provider, with sensitive information redacted.

```json
{
  "providerId": "huggingface",
  "provider": {
    "kind": "huggingface",
    "name": "Hugging Face",
    "enabled": true,
    "base_url": "https://router.huggingface.co/v1",
    "api_key_env": "HF_TOKEN",
    "has_api_key": false,
    "has_api_key_env": true
  }
}
```

--------------------------------

### Get Bench Pack Run History

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves the run history entries for a specific Bench Pack.

```json
{
  "history": []
}
```

--------------------------------

### Benchlocal Create Model Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Create a new model configuration using `benchlocal_create_model`. Requires `provider` and `model` identifiers, and optionally `id`, `label`, `group`, and `enabled` status.

```json
{ "id"?, "provider", "model", "label"?, "group"?, "enabled"? }
```

--------------------------------

### Get Specific Run Summary

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieves a summary for a specific run within a Bench Pack's history.

```APIDOC
## GET /v1/benchpacks/:benchPackId/history/:runId

### Description
Returns a saved run summary.

### Method
GET

### Endpoint
/v1/benchpacks/:benchPackId/history/:runId

### Parameters
#### Path Parameters
- **benchPackId** (string) - Required - The ID of the Bench Pack.
- **runId** (string) - Required - The ID of the run.

### Response
#### Success Response (200)
- **run** (object) - An object containing the run summary.

### Response Example
```json
{
  "run": {}
}
```
```

--------------------------------

### Commit Release Preparation

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

After bumping the version, review the changes and commit them. This includes adding the updated package files and creating a commit message indicating the release version.

```bash
git status --short
git add package.json app/package.json package-lock.json
git commit -m "Release BenchLocal v<version>"
```

--------------------------------

### UI Theme Configuration

Source: https://github.com/stevibe/benchlocal/blob/main/CONFIG_SCHEMA_V1.md

Sets the UI theme for BenchLocal. Supported values include 'system', 'light', 'dark', and 'night'.

```toml
[ui]
theme = "system"

```

--------------------------------

### Build Release Artifacts

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Build the release artifacts for different platforms. Use `release:mac` for macOS and `build:win`/`build:linux` for Windows and Linux respectively. `npm run release:all` executes all three builds.

```bash
npm run release:mac
```

```bash
npm run build:win
```

```bash
npm run build:linux
```

--------------------------------

### Initialize BenchLocal Agent Access

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Sets environment variables for accessing the BenchLocal agent. Ensure the token is correctly retrieved from agent settings.

```bash
export BENCHLOCAL_AGENT_BASE_URL="http://127.0.0.1:50060"
export BENCHLOCAL_AGENT_TOKEN="<token from Settings > Agent Access>"
```

--------------------------------

### Create Model

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Creates a new model configuration.

```APIDOC
## POST /v1/models

### Description
Creates a model.

### Method
POST

### Endpoint
/v1/models

### Parameters
#### Request Body
- **id** (string) - Optional - The unique identifier for the model.
- **provider** (string) - Required - The ID of the provider this model belongs to.
- **model** (string) - Required - The model identifier within the provider.
- **label** (string) - Optional - A display label for the model.
- **group** (string) - Optional - A group identifier for the model.
- **enabled** (boolean) - Optional - Whether the model is enabled.

### Request Example
```json
{
  "id": "huggingface:Qwen/Qwen3.5-9B",
  "provider": "huggingface",
  "model": "Qwen/Qwen3.5-9B",
  "label": "Qwen3.5-9B",
  "group": "primary",
  "enabled": true
}
```

Returns `201`.
```

--------------------------------

### Benchlocal Get Model Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Fetch details for a specific model by its ID using `benchlocal_get_model`. Requires a JSON object with `modelId`.

```json
{ "modelId": string }
```

--------------------------------

### Benchlocal Get Provider Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Retrieve a single redacted provider by its ID using `benchlocal_get_provider`. Requires a JSON object with `providerId`.

```json
{ "providerId": string }
```

--------------------------------

### Connect to SSE Event Stream

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Opens a Server-Sent Events stream to receive real-time updates from the BenchLocal agent. Requires agent token authentication.

```bash
curl -N \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  "$BENCHLOCAL_AGENT_BASE_URL/v1/events"
```

--------------------------------

### GET /v1/events

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Opens a Server-Sent Events (SSE) stream for real-time agent events. Includes initial metadata and subsequent event data.

```APIDOC
## GET /v1/events

### Description
Opens a Server-Sent Events stream. The initial response includes a comment, and each subsequent event is formatted with an ID, event type, and data payload.

### Method
GET

### Endpoint
/v1/events

### Request Example
```bash
curl -N \
  -H "Authorization: Bearer $BENCHLOCAL_AGENT_TOKEN" \
  "$BENCHLOCAL_AGENT_BASE_URL/v1/events"
```

### Response
Initial response includes a comment:
```text
: BenchLocal agent event stream
```

Each event is sent as:
```text
id: evt-...
event: benchpack.run.event
data: {"eventId":"evt-வுகளை","createdAt":"...","type":"benchpack.run.event","payload":{}}
```

Event envelope:
```ts
type BenchLocalAgentEvent<TPayload = unknown> = {
  eventId: string;
  createdAt: string;
  type: BenchLocalAgentEventType;
  payload: TPayload;
};
```

Current event types:
```text
agent.state.updated
config.updated
workspace.updated
models.availability.updated
benchpack.run.started
benchpack.run.event
benchpack.run.finished
benchpack.run.error
verifier.event
```
```

--------------------------------

### Bump Desktop Client Version

Source: https://github.com/stevibe/benchlocal/blob/main/RELEASING.md

Use this command to increment the desktop client version without creating a Git tag. Ensure you replace `<version>` with the desired version number.

```bash
npm version <version> --workspace app --include-workspace-root --no-git-tag-version
```

```bash
npm version 0.2.2 --workspace app --include-workspace-root --no-git-tag-version
```

--------------------------------

### GET /v1/health Endpoint Response

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

The health endpoint provides runtime status, BenchLocal version, and agent configuration details. No authentication is required.

```json
{
  "ok": true,
  "benchLocalVersion": "0.2.6",
  "agent": {
    "enabled": true,
    "running": true,
    "access": "localhost",
    "host": "127.0.0.1",
    "port": 50060,
    "baseUrl": "http://127.0.0.1:50060",
    "connectedClients": 0,
    "message": "Agent API is listening on http://127.0.0.1:50060.",
    "startedAt": "2026-05-18T00:00:00.000Z"
  },
  "docs": {
    "agentGuide": "/v1/agent-guide",
    "openapi": "/v1/openapi.json",
    "mcp": "/mcp"
  }
}
```

--------------------------------

### Create Provider

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Creates a new provider configuration.

```APIDOC
## POST /v1/providers

### Description
Creates a provider.

### Method
POST

### Endpoint
/v1/providers

### Parameters
#### Request Body
- **id** (string) - Optional - The unique identifier for the provider.
- **kind** (BenchLocalProviderKind) - Required - The type or kind of the provider.
- **name** (string) - Optional - A display name for the provider.
- **enabled** (boolean) - Optional - Whether the provider is enabled.
- **base_url** (string) - Required - The base URL for the provider's API.
- **api_key** (string) - Optional - The API key for authentication.
- **api_key_env** (string) - Optional - The environment variable name for the API key.

### Request Example
```json
{
  "id": "huggingface",
  "kind": "huggingface",
  "name": "Hugging Face",
  "enabled": true,
  "base_url": "https://router.huggingface.co/v1",
  "api_key_env": "HF_TOKEN"
}
```

Returns `201`.
```

--------------------------------

### Benchlocal Get Health Tool

Source: https://github.com/stevibe/benchlocal/blob/main/docs/agent-control-api.md

Use `benchlocal_get_health` to check the runtime compatibility and health status of the Benchlocal system. This tool is read-only and requires no input.

```json
{}
```

--------------------------------

### BenchLocal Project Structure

Source: https://github.com/stevibe/benchlocal/blob/main/ARCHITECTURE.md

Illustrates the directory structure of the BenchLocal project, including the Electron main process, preload scripts, renderer, and shared packages.

```text
app/
  src/main/      Electron main process
  src/preload/   secure renderer bridge
  src/renderer/  desktop UI
packages/
  benchlocal-core/
    shared protocol, config, workspaces, themes
  benchlocal-sdk/
    authoring helpers for Bench Packs
  benchpack-host/
    install, inspection, verifier, and run orchestration
themes/
  built-in desktop themes
scripts/
  local macOS release helpers
```
Tokens:
Raw