### Run Example PDF and MCP Suites

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Execute example suites for PDF processing and MCP (Model-Centric Programming) demos. These commands are useful for quickly testing the workbench's capabilities with pre-configured examples.

```bash
npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
```

```bash
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1
```

--------------------------------

### Suite YAML Configuration

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Example of a suite YAML file defining evaluation parameters, models, environment variables, setup commands, and cases.

```yaml
# examples/workbench/pdf/suite.yml
name: pdf-workbench-example
references: ./references           # copied into /work for every case
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY             # forwarded unchanged into all containers
timeoutSeconds: 600
setup:
  - mkdir -p input
  - node $CASE/checks/_pdf.mjs write-inputs input
  - cp input/statement.pdf statement.pdf

cases:
  # Inline case — inherits suite defaults
  - name: extract-pdf-facts
    task: |
      Extract the key facts from statement.pdf and write answer.json with this exact schema:
      { "account": string, "quarter": string, "totalRevenue": number,
        "riskFlags": string[], "approvalCode": string }
      Use numeric dollars without commas or a currency symbol for totalRevenue.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

  # External case file (path relative to suite.yml)
  - ./cases/advanced-extraction/case.yml
```

--------------------------------

### Install Gemini CLI extension

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Install the Gemini extension from GitHub. This command fetches and installs the skill-optimizer as a Gemini CLI extension.

```bash
gemini extensions install https://github.com/fastxyz/skill-optimizer
```

--------------------------------

### Install and build skill-optimizer locally

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Install project dependencies and build the skill-optimizer project locally. This is a prerequisite for running the local CLI.

```bash
npm install
npm run build
```

--------------------------------

### CLI Command Examples

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Demonstrates how to run evaluation suites using the skill-optimizer CLI with TypeScript execution.

```bash
npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1

```

--------------------------------

### MCP Command Examples

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Demonstrates how to interact with MCP services using the 'mcp' command-line tool for listing and calling services.

```bash
mcp list calculator
mcp call calculator.add a=17 b=25
```

--------------------------------

### Install Skill Optimizer via Gemini CLI

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Commands to install and update the skill-optimizer extension using the Gemini CLI.

```bash
gemini extensions install https://github.com/fastxyz/skill-optimizer
gemini extensions update skill-optimizer   # to update
```

--------------------------------

### Install Skill Optimizer into Multiple Agents

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Use this command to install the skill-optimizer into multiple supported agent environments simultaneously.

```bash
npx skills add fastxyz/skill-optimizer \
  --skill skill-optimizer \
  -a claude-code -a opencode -a codex -a cursor -y
```

--------------------------------

### Directory Layout Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Illustrates the standard directory structure for an evaluation project, including references, checks, binaries, and workspace.

```text
my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
    my-skill/references/api.md
  checks/
    create-inputs.mjs
    grade-output.mjs
  bin/
    fake-product-cli
  workspace/
    starter-repo/
```

--------------------------------

### Case Schema Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

This YAML defines a test case, including its name, references, task, graders, setup, cleanup, environment variables, MCP servers/services, default model, and timeout. Relative paths are resolved from the case file directory.

```yaml
name: extract-pdf-facts
references: ./references
task: |
  Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
graders:
  - name: answer-json
    command: node $CASE/checks/extract-pdf-facts.mjs
setup:
  - node $CASE/checks/create-inputs.mjs
cleanup: []
env:
  - OPENROUTER_API_KEY
mcpServers:
  calculator:
    baseUrl: http://calculator:3000/mcp
mcpServices:
  calculator:
    command: node
    args:
      - calculator-server.mjs
model: openrouter/google/gemini-2.5-flash
timeoutSeconds: 600

```

--------------------------------

### Install skill-optimizer CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Installs the skill-optimizer skill for common agents. Use this command to add the skill to your agent's capabilities.

```bash
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor
```

--------------------------------

### Skill Installation Command

Source: https://github.com/fastxyz/skill-optimizer/blob/development/AGENTS.md

Command to add the skill-optimizer skill using npm.

```bash
npx skills add fastxyz/skill-optimizer --skill skill-optimizer ...
```

--------------------------------

### Skill-Only Install using open skills CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Install only the skill files without plugin metadata using the open skills CLI. This command is useful if you only need the core skill functionality.

```bash
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor -y
```

--------------------------------

### Add skill-optimizer as a Skill-Only Install

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/README.codex.md

Install only the skill files for skill-optimizer using the open skills CLI. This method is suitable when you do not need to manage it as a full plugin.

```bash
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a codex -y
```

--------------------------------

### Install Skill Optimizer via Claude Code Marketplace

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Instructions for adding the skill-optimizer plugin through the Claude Code marketplace.

```bash
/plugin marketplace add fastxyz/skill-optimizer
/plugin install skill-optimizer@skill-optimizer
```

--------------------------------

### Run Workbench Suite SDK Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Demonstrates how to use the `runWorkbenchSuite` SDK function to execute a suite of test cases with specified trials and concurrency.

```typescript
import { runWorkbenchSuite } from 'skill-optimizer';

await runWorkbenchSuite({
  suitePath: 'examples/workbench/pdf/suite.yml',
  trials: 3,
  concurrency: 2,
});
```

--------------------------------

### Install skill-optimizer Plugin in OpenCode

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/README.opencode.md

Add this JSON configuration to your opencode.json file to install the skill-optimizer plugin. Restart OpenCode after adding the plugin.

```json
{
  "plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"]
}
```

--------------------------------

### Install skill-optimizer plugin for Claude Code

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Install the skill-optimizer plugin after registering it as a marketplace. This makes the skill available for use within your agent.

```text
/plugin install skill-optimizer@skill-optimizer
```

--------------------------------

### Local Development Workflow

Source: https://github.com/fastxyz/skill-optimizer/blob/development/CONTRIBUTING.md

Commands to clone the repository, install dependencies, and run essential build and test scripts. Ensure all these commands pass before submitting code changes.

```bash
git clone https://github.com/fastxyz/skill-optimizer
cd skill-optimizer
npm install
npm run typecheck
npm test
npm run build
```

--------------------------------

### Trace JSONL Entry Examples

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Illustrates common entry shapes found in trace.jsonl files, used for debugging model failures and verifying tool usage.

```json
{ "type": "trace_start", "caseName": "extract-pdf-facts", "model": "openrouter/google/gemini-2.5-flash" }
```

```json
{ "type": "message", "role": "assistant", "text": "..." }
```

```json
{ "type": "tool_call", "name": "bash", "arguments": { "command": "node script.mjs" } }
```

```json
{ "type": "tool_call", "name": "read", "arguments": { "path": "/work/pdf-skill/SKILL.md" } }
```

```json
{ "type": "tool_result", "name": "bash", "text": "...", "isError": false }
```

--------------------------------

### MCP Server and Service Configuration

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Example suite configuration for MCP servers and services, including base URLs, headers, and local service commands. Environment variables are used for authentication.

```yaml
mcpServers:
  calculator:
    baseUrl: http://calculator:3000/mcp
  context7:
    baseUrl: https://mcp.context7.com/mcp
    headers:
      Authorization: "Bearer ${CONTEXT7_API_KEY}"
env:
  - OPENROUTER_API_KEY
  - CONTEXT7_API_KEY
mcpServices:
  calculator:
    command: node
    args:
      - calculator-server.mjs
```

--------------------------------

### Minimal Suite Configuration

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

A basic YAML configuration for a skill evaluation suite, defining the name, references, models, environment variables, timeout, setup commands, and cases.

```yaml
name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs
```

--------------------------------

### Add skill-optimizer as a Cursor skill

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Install the skill-optimizer skill using the open skills CLI for Cursor. This command adds the skill to your Cursor environment.

```bash
npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a cursor -y
```

--------------------------------

### Add skill-optimizer Plugin Marketplace with Git Ref

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/README.codex.md

This command allows pinning a specific Git reference (e.g., a branch or tag) when adding the skill-optimizer marketplace. This is useful for ensuring reproducible installations.

```bash
codex plugin marketplace add fastxyz/skill-optimizer --ref main
```

--------------------------------

### Minimal Suite Configuration (YAML)

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Defines a basic evaluation suite for PDF fact extraction, specifying models, environment variables, setup scripts, and case tasks with grading commands.

```yaml
name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

```

--------------------------------

### Open plugin search interface in OpenAI Codex

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Access the plugin search interface in Codex to find and install available plugins. After registering, you can search for 'skill-optimizer'.

```text
/plugins
```

--------------------------------

### Suite Schema Definition

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Defines a workbench suite with inline cases, model references, environment variables, and setup commands. External case files can also be referenced.

```yaml
name: pdf-workbench-example
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/_pdf.mjs write-inputs input
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs
  - cases/external-case/case.yml
```

--------------------------------

### Run MCP Calculator Demo Suite

Source: https://github.com/fastxyz/skill-optimizer/blob/development/examples/workbench/README.md

Executes the MCP calculator demonstration suite. This command initiates a local MCP calculator server as a hidden Docker service and exposes it via the workbench 'mcp' command.

```bash
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1
```

--------------------------------

### Run PDF Skill Demo Suite

Source: https://github.com/fastxyz/skill-optimizer/blob/development/examples/workbench/README.md

Executes the PDF skill demonstration suite using the workbench CLI. This command runs the configured model matrix and outputs trace and result files.

```bash
npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
```

--------------------------------

### Update Gemini CLI extension

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Update the skill-optimizer extension for the Gemini CLI. Use this command to ensure you have the latest version installed.

```bash
gemini extensions update skill-optimizer
```

--------------------------------

### Workbench Suite Running

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Runs a suite matrix.

```APIDOC
## runWorkbenchSuite(params)

### Description
Runs a suite matrix.

### Parameters
#### Request Body
- **params** (object) - Required - Parameters for running the suite.
  - **suitePath** (string) - Required - Path to the suite file.
  - **trials** (number) - Optional - Number of trials to run.
  - **concurrency** (number) - Optional - Number of concurrent trials.
```

--------------------------------

### Grader JSON Output Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Preferred grader output is a JSON object on stdout. If no JSON is printed, the exit code determines pass/fail.

```json
{ "pass": false, "score": 0, "evidence": ["answer.json missing approvalCode"] }
```

--------------------------------

### Run Suite with Custom Image and Output

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Configure a custom Docker image for the evaluation environment and specify a custom output directory for results.

```bash
npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml \
  --image my-registry/skill-optimizer-workbench:v2 \
  --out ./ci-results
```

--------------------------------

### CLI Help Commands

Source: https://github.com/fastxyz/skill-optimizer/blob/development/CLAUDE.md

Commands to display help information for the skill-optimizer CLI and its subcommands. Useful for understanding available options and usage.

```bash
npx tsx src/cli.ts --help
```

```bash
npx tsx src/cli.ts run-case --help
```

```bash
npx tsx src/cli.ts run-suite --help
```

--------------------------------

### Build Docker Image for Skill Optimizer Workbench

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Build the Docker image required for the skill-optimizer workbench runner.

```bash
npm install && npm run build
docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .
```

--------------------------------

### Filter PDF Pages by Content

Source: https://github.com/fastxyz/skill-optimizer/blob/development/examples/workbench/pdf/references/pdf-skill/SKILL.md

This example filters PDF pages based on their extracted text content. It includes pages that contain 'CUSTOMER COPY' but not 'INTERNAL NOTES'.

```python
from pypdf import PdfReader, PdfWriter

reader = PdfReader("customer-packet.pdf")
writer = PdfWriter()

for page in reader.pages:
    text = page.extract_text() or ""
    if "CUSTOMER COPY" in text and "INTERNAL NOTES" not in text:
        writer.add_page(page)

with open("customer-copy.pdf", "wb") as output:
    writer.write(output)
```

--------------------------------

### Build, Typecheck, and Test Project

Source: https://github.com/fastxyz/skill-optimizer/blob/development/CLAUDE.md

Standard npm commands for building the project, checking TypeScript types, and running tests. Ensure these are executed before committing changes.

```bash
npm run build
```

```bash
npm run typecheck
```

```bash
npm test
```

--------------------------------

### Run Suite with Trials and Concurrency

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Execute all cases defined in a suite YAML file, with a specified number of trials per case/model and concurrent containers.

```bash
npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml \
  --trials 3 \
  --concurrency 4
```

--------------------------------

### Run Suite with Trials and Concurrency CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Execute a test suite, running multiple trials per case/model with a specified concurrency level. This is ideal for large-scale evaluation of multiple cases.

```bash
npx tsx src/cli.ts run-suite <suite.yml> --trials 3 --concurrency 2
```

--------------------------------

### Run Case with Multiple Models and Trials CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Run a single case against multiple specified OpenRouter models, with a set number of independent trials per model and a defined level of concurrency. This allows for comprehensive testing and comparison.

```bash
npx tsx src/cli.ts run-case <case.yml> --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-5.4 --trials 3 --concurrency 2
```

--------------------------------

### Structured Extraction from PDF

Source: https://github.com/fastxyz/skill-optimizer/blob/development/examples/workbench/pdf/references/pdf-skill/SKILL.md

This example demonstrates parsing extracted PDF text to find specific fields like account, quarter, and revenue. Inspect the text first to determine field formats and labels.

```python
from pypdf import PdfReader
import json

reader = PdfReader("statement.pdf")
text = "\n".join(page.extract_text() or "" for page in reader.pages)
lines = [line.strip() for line in text.splitlines() if line.strip()]

answer = {"riskFlags": []}
for line in lines:
    if line.startswith("Account:"):
        answer["account"] = line.split(":", 1)[1].strip()
    elif line.startswith("Quarter:"):
        answer["quarter"] = line.split(":", 1)[1].strip()
    elif line.startswith("Total Revenue:"):
        raw = line.split(":", 1)[1].strip().replace("$", "").replace(",", "")
        answer["totalRevenue"] = float(raw)
    elif line.startswith("Risk Flag:"):
        answer["riskFlags"].append(line.split(":", 1)[1].strip())
    elif line.startswith("Approval Code:"):
        answer["approvalCode"] = line.split(":", 1)[1].strip()

with open("answer.json", "w") as output:
    json.dump(answer, output, indent=2)
```

--------------------------------

### Configure Skill Optimizer for OpenCode

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Add the skill-optimizer to your `opencode.json` configuration file.

```json
{ "plugin": ["skill-optimizer@git+https://github.com/fastxyz/skill-optimizer.git"] }
```

--------------------------------

### Deterministic Grader Script (JavaScript)

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

An example deterministic grader script written in Node.js. It checks for the existence and validity of 'answer.json' and verifies specific fields. Ensure necessary helper functions like 'printResult' are available.

```javascript
// checks/extract-pdf-facts.mjs — example deterministic grader
import { existsSync } from 'node:fs';
import { join } from 'node:path';
import { printResult, readJson, requireEnv } from './_pdf.mjs';

const answerPath = join(requireEnv('WORK'), 'answer.json');

if (!existsSync(answerPath)) {
  printResult(false, 'answer.json was not created');
}

let answer;
try {
  answer = readJson(answerPath);
} catch (error) {
  printResult(false, `answer.json is not valid JSON: ${error.message}`);
}

const failures = [];
if (answer.account !== 'Delta Orchard Cooperative') failures.push('account mismatch');
if (answer.quarter !== 'Q4 2025')                    failures.push('quarter mismatch');
if (answer.totalRevenue !== 128430)                  failures.push('totalRevenue must be numeric 128430');
if (!Array.isArray(answer.riskFlags))                failures.push('riskFlags must be an array');
if (answer.approvalCode !== 'PDF-7429')              failures.push('approvalCode mismatch');

// Grader stdout must be a JSON object; exit code is secondary
printResult(failures.length === 0, failures.length > 0 ? failures : 'answer.json matched');
// stdout → { "pass": true, "score": 1, "evidence": ["answer.json matched"] }
// stdout → { "pass": false, "score": 0, "evidence": ["account mismatch", "totalRevenue must be numeric 128430"] }

```

--------------------------------

### JSON Output Grader Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

This JavaScript grader checks for the existence and content of 'answer.json'. It parses the file, validates the 'approvalCode', and outputs a JSON object indicating pass/fail status, score, and evidence. Use this when grading structured JSON outputs.

```javascript
import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const path = join(process.env.WORK, 'answer.json');
const failures = [];

if (!existsSync(path)) {
  failures.push('answer.json was not created');
} else {
  const answer = JSON.parse(readFileSync(path, 'utf-8'));
  if (answer.approvalCode !== 'PDF-7429') failures.push('approvalCode mismatch');
}

console.log(JSON.stringify({
  pass: failures.length === 0,
  score: failures.length === 0 ? 1 : 0,
  evidence: failures.length === 0 ? ['answer.json matched'] : failures,
}));
process.exit(failures.length === 0 ? 0 : 1);
```

--------------------------------

### Run Single Case with CLI

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Execute a single case YAML file against a specified OpenRouter model. This is the default behavior.

```bash
npx tsx src/cli.ts run-case ./my-eval/case.yml
```

--------------------------------

### Trace Guard Grader Example

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

This JavaScript grader inspects the 'trace.jsonl' file to ensure a specific skill was not read by the agent. It parses each line of the trace file, checks for 'tool_call' entries with the 'read' name and a forbidden path, and reports the findings. Use this to enforce constraints on agent tool usage.

```javascript
import { existsSync, readFileSync } from 'node:fs';
import { join } from 'node:path';

const tracePath = join(process.env.RESULTS, 'trace.jsonl');
const lines = existsSync(tracePath) ? readFileSync(tracePath, 'utf-8').trim().split(/\r?\n/) : [];
const readForbiddenSkill = lines.some((line) => {
  try {
    const entry = JSON.parse(line);
    const path = entry?.arguments?.path ?? entry?.arguments?.filePath;
    return entry.type === 'tool_call' && entry.name === 'read' && /\/pdf-skill\/SKILL\.md$/.test(path);
  } catch {
    return false;
  }
});

console.log(JSON.stringify({
  pass: !readForbiddenSkill,
  score: readForbiddenSkill ? 0 : 1,
  evidence: readForbiddenSkill ? ['agent read the PDF skill'] : ['no forbidden skill read'],
}));
process.exit(readForbiddenSkill ? 1 : 0);
```

--------------------------------

### Run a Suite of Cases

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Execute a collection of evaluation cases defined in a suite YAML file. Configure the number of trials and concurrency for parallel execution. Ideal for comprehensive testing of multiple skills or scenarios.

```bash
npx tsx src/cli.ts run-suite ./suite.yml --trials 3 --concurrency 2
```

--------------------------------

### Single-Trial Run Case Output Structure

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Illustrates the directory structure and key files generated for a single-trial `run-case` execution. `trace.jsonl` is the primary source for debugging.

```text
case/.results/<run-id>/
  trace.jsonl
  result.json
  summary.json
  workspace/        # on failure or --keep-workspace
```

--------------------------------

### Workbench Suite Loading

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Parses and validates a suite file.

```APIDOC
## loadWorkbenchSuite(path)

### Description
Parses and validates a suite file.

### Parameters
#### Path Parameters
- **path** (string) - Required - The path to the suite file.
```

--------------------------------

### Standalone Case YAML Configuration

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Defines a single user task with optional setup/cleanup commands and grader commands. Ensure 'references' points to a valid directory.

```yaml
# Standalone case with all supported fields
name: split-customer-packet
references: ./references           # required; must be a directory
model: openrouter/google/gemini-2.5-flash
timeoutSeconds: 300
env:
  - OPENROUTER_API_KEY
  - STRIPE_TEST_KEY
setup:
  - node $CASE/checks/_pdf.mjs write-inputs $WORK
cleanup:
  - rm -f $WORK/tmp-*
task: |
  Create customer-copy.pdf from customer-packet.pdf.
  Include only the pages marked CUSTOMER COPY, in their original order.
  Exclude the page marked INTERNAL NOTES. The output must be a PDF.
graders:
  - name: customer-copy-pdf
    command: node $CASE/checks/split-customer-packet.mjs
  - name: no-internal-notes
    command: node $CASE/checks/no-internal-notes.mjs

```

--------------------------------

### Docker Workbench Case Running

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Lower-level Docker case runner.

```APIDOC
## runDockerWorkbenchCase(params)

### Description
Lower-level Docker case runner.

### Parameters
#### Request Body
- **params** (object) - Required - Parameters for the Docker runner.
```

--------------------------------

### Run Case Matrix with Multiple Models and Trials

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Execute a case against multiple models, each with a specified number of trials. Concurrency can be controlled to manage resource usage.

```bash
npx tsx src/cli.ts run-case ./my-eval/case.yml \
  --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-4o \
  --trials 3 \
  --concurrency 2
```

--------------------------------

### Run a suite with skill-optimizer CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Executes an entire evaluation suite using the skill-optimizer CLI. The suite definition in the YAML file determines the cases and models to run.

```bash
npx tsx src/cli.ts run-suite <suite.yml>
```

--------------------------------

### Run a single case directly with a specific model

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Execute a single test case directly using the CLI. This command specifies the case file and the OpenRouter model to use.

```bash
npx tsx src/cli.ts run-case ./case.yml --model openrouter/google/gemini-2.5-flash
```

--------------------------------

### Add skill-optimizer Plugin Marketplace

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/README.codex.md

Use this command to register the skill-optimizer repository as a plugin marketplace in Codex. This allows for managing the skill as a plugin.

```bash
codex plugin marketplace add fastxyz/skill-optimizer
```

--------------------------------

### Run one case with skill-optimizer CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Executes a single evaluation case using the skill-optimizer CLI. Specify the case YAML file to run.

```bash
npx tsx src/cli.ts run-case <case.yml>
```

--------------------------------

### Single Trial Output Layout

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Illustrates the file structure generated for a single trial evaluation, including trace, result, summary, and workspace.

```text
# Output layout (single trial):
# my-eval/.results/<run-id>/
#   trace.jsonl      ← agent messages, tool calls, tool results
#   result.json      ← pass, score, evidence, graders, metrics, cost
#   summary.json     ← final assistant text, failed graders, bash commands
#   workspace/       ← preserved on failure or --keep-workspace
```

--------------------------------

### CLI Help Verification

Source: https://github.com/fastxyz/skill-optimizer/blob/development/AGENTS.md

Command to verify CLI help output, useful after changes to CLI behavior or documentation.

```bash
npx tsx src/cli.ts --help
```

--------------------------------

### Workbench Case Loading

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Parses and validates a case file.

```APIDOC
## loadWorkbenchCase(path)

### Description
Parses and validates a case file.

### Parameters
#### Path Parameters
- **path** (string) - Required - The path to the case file.
```

--------------------------------

### Workbench Case Running

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Runs a single case or a model/trial matrix.

```APIDOC
## runWorkbenchCase(params)

### Description
Runs one case or a model/trial matrix.

### Parameters
#### Request Body
- **params** (object) - Required - Parameters for running the case.
  - **casePath** (string) - Required - Path to the case file.
  - **trials** (number) - Optional - Number of trials to run.
  - **concurrency** (number) - Optional - Number of concurrent trials.
```

--------------------------------

### Run development tasks

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Execute common development tasks such as type checking, running tests, and building the project. These commands are essential for development workflows.

```bash
npm run typecheck
npm test
npm run build
```

--------------------------------

### Run Development Checks

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Execute these commands after code or documentation changes that affect behavior. Ensure all checks pass before committing.

```bash
npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help
```

--------------------------------

### Directory Layout for Evaluation

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Illustrates the standard directory structure for an evaluation project, including suite configuration, references, grading scripts, binaries, and workspace.

```text
my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
  checks/
    create-inputs.mjs
    extract-pdf-facts.mjs
  bin/
    fake-cli
  workspace/
    starter-app/

```

--------------------------------

### Run a Workbench Suite Programmatically

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Use `runWorkbenchSuite` as a programmatic alternative to the `run-suite` CLI. It iterates through all case, model, and trial combinations with configurable concurrency and output options.

```typescript
import { runWorkbenchSuite } from 'skill-optimizer';

await runWorkbenchSuite({
  suitePath: './examples/workbench/pdf/suite.yml',
  trials: 3,
  concurrency: 4,
  outDir: './ci-results',
  image: 'skill-optimizer-workbench:local',
  keepWorkspace: true,
});
// Writes ./ci-results/<run-id>/suite-result.json:
// {
//   name: 'pdf-workbench-example',
//   models: ['openrouter/google/gemini-2.5-flash'],
//   cases: ['extract-pdf-facts', 'split-customer-packet', ...],
//   summary: { total: 4, passed: 3, failed: 1, passRate: 0.75, ... },
//   results: [{ caseName, model, trialPassRate, passAtK, passHatK, trials }]
// }
```

--------------------------------

### Output Directory Structure

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Details the directory structure for run results, including aggregate summaries, trial-specific data, traces, and preserved workspaces.

```text
.results/<run-id>/
  suite-result.json                  # run-suite aggregate
  run-result.json                    # run-case matrix aggregate
  trials/<case>--<model>--001/
    trace.jsonl                      # agent messages and tool calls
    result.json                      # pass, score, evidence, graders, metrics
    summary.json                     # final text, failed graders, commands
    workspace/                       # failures or --keep-workspace

```

--------------------------------

### runWorkbenchCase

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Serves as the programmatic equivalent of the `run-case` CLI command. It executes a single case through Docker, with options for running across a model matrix and multiple trials.

```APIDOC
## SDK: `runWorkbenchCase`

Programmatic equivalent of the `run-case` CLI command. Runs one case through Docker, optionally across a model matrix with multiple trials.

```typescript
import { runWorkbenchCase } from 'skill-optimizer';

// Single model, single trial
await runWorkbenchCase({
  casePath: './my-eval/case.yml',
  model: 'openrouter/google/gemini-2.5-flash',
});
// stdout: "Results: ./my-eval/.results/<run-id>"
// stdout: "Grade: PASS"
// process.exitCode set to 1 on failure

// Multi-model matrix
await runWorkbenchCase({
  casePath: './my-eval/case.yml',
  models: [
    'openrouter/google/gemini-2.5-flash',
    'openrouter/openai/gpt-4o',
    'openrouter/anthropic/claude-sonnet-4-5',
  ],
  trials: 5,
  concurrency: 3,
  outDir: './ci-results',
  keepWorkspace: false,
});
// Writes ./ci-results/<run-id>/run-result.json with:
// { summary: { passRate, passedTrials, failedTrials, meanScore },
//   results: [{ model, trialPassRate, passAtK, passHatK, trials: [...] }]
// }
```
```

--------------------------------

### Run Suite Output Structure

Source: https://github.com/fastxyz/skill-optimizer/blob/development/docs/workbench.md

Details the output directory layout for a `run-suite` command. This structure is used for organizing results from multiple cases and models within a suite.

```text
suite/.results/<run-id>/
  suite-result.json
  trials/<case-slug>--<model-slug>--001/trace.jsonl
  trials/<case-slug>--<model-slug>--001/result.json
```

--------------------------------

### Custom Output Directory

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Specify a custom directory for storing evaluation results, including traces and summaries.

```bash
npx tsx src/cli.ts run-case ./my-eval/case.yml --out ./ci-results
```

--------------------------------

### Model List Parsing

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/references/workbench.md

Parses comma-separated OpenRouter references.

```APIDOC
## parseModelList(raw)

### Description
Parse comma-separated OpenRouter refs.

### Parameters
#### Request Body
- **raw** (string) - Required - Comma-separated string of model references.
```

--------------------------------

### Run one case across multiple models with skill-optimizer CLI

Source: https://github.com/fastxyz/skill-optimizer/blob/development/skills/skill-optimizer/SKILL.md

Runs a single evaluation case against a specified list of models using the skill-optimizer CLI. This is useful for comparing model performance on the same task.

```bash
npx tsx src/cli.ts run-case <case.yml> --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-5.4
```

--------------------------------

### Add skill-optimizer as a Claude Code plugin marketplace

Source: https://github.com/fastxyz/skill-optimizer/blob/development/README.md

Register the repository as a Claude Code plugin marketplace. This command adds the skill-optimizer to your agent's available plugins.

```text
/plugin marketplace add fastxyz/skill-optimizer
```

--------------------------------

### Run a Workbench Case Programmatically

Source: https://context7.com/fastxyz/skill-optimizer/llms.txt

Use `runWorkbenchCase` as a programmatic alternative to the `run-case` CLI. It runs a single case through Docker, with options for multi-model matrices and multiple trials.

```typescript
import { runWorkbenchCase } from 'skill-optimizer';

// Single model, single trial
await runWorkbenchCase({
  casePath: './my-eval/case.yml',
  model: 'openrouter/google/gemini-2.5-flash',
});
// stdout: "Results: ./my-eval/.results/<run-id>"
// stdout: "Grade: PASS"
// process.exitCode set to 1 on failure

// Multi-model matrix
await runWorkbenchCase({
  casePath: './my-eval/case.yml',
  models: [
    'openrouter/google/gemini-2.5-flash',
    'openrouter/openai/gpt-4o',
    'openrouter/anthropic/claude-sonnet-4-5',
  ],
  trials: 5,
  concurrency: 3,
  outDir: './ci-results',
  keepWorkspace: false,
});
// Writes ./ci-results/<run-id>/run-result.json with:
// { summary: { passRate, passedTrials, failedTrials, meanScore },
//   results: [{ model, trialPassRate, passAtK, passHatK, trials: [...] }] }
```