### Experiment Configuration with MCP Server Setup

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Configures an agent experiment that includes a setup step to install and configure a custom MCP server. This is useful for testing agents with specific tool integrations.

```typescript
// experiments/with-mcp.ts
import type { ExperimentConfig } from '@vercel/agent-eval';

const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  model: 'opus',
  runs: 10,
  earlyExit: false,
  setup: async (sandbox) => {
    await sandbox.runCommand('npm', ['install', '-g', '@myframework/mcp-server']);
    await sandbox.writeFiles({
      '.claude/settings.json': JSON.stringify({
        mcpServers: { myframework: { command: 'myframework-mcp' } }
      })
    });
  },
};

export default config;
```

--------------------------------

### Copy .env.example to .env

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Copy the example environment file to .env to start configuring your API keys and tokens. The framework uses dotenv to load environment variables from .env.local and .env.

```bash
cp .env.example .env
```

--------------------------------

### Configure Playground UI Options

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Example of how to specify custom directories and ports when launching the playground UI. This allows for more flexible result browsing.

```bash
npx @vercel/agent-eval playground --results-dir ./results --evals-dir ./evals --port 3001
```

--------------------------------

### Full Experiment Configuration Example

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Defines a comprehensive experiment configuration, including agent, model, runs, scripts, validation, setup, prompt editing, completion hooks, brands, and sandbox settings.

```typescript
// experiments/my-experiment.ts
import type { ExperimentConfig } from '@vercel/agent-eval';

const config: ExperimentConfig = {
  // Required: which agent to use
  agent: 'vercel-ai-gateway/claude-code',

  // Model to use. Omit this to use the underlying agent CLI's native default.
  // Provide an array to run the same experiment across multiple models.
  model: 'opus',

  // How many times to run each eval (default: 1)
  runs: 10,

  // Stop after first success? (default: true)
  earlyExit: false,

  // npm scripts that must pass after agent finishes (default: [])
  scripts: ['build', 'lint'],

  // Validation mode after the agent finishes (default: 'vitest')
  // 'vitest' - run EVAL.ts/EVAL.tsx plus configured scripts
  // 'none' - response-only mode; skip EVAL.ts/EVAL.tsx, run scripts if provided
  validation: 'vitest',

  // Timeout per run in seconds (default: 600)
  timeout: 600,

  // Filter which evals to run (default: '*' for all)
  evals: '*',
  // evals: ['specific-eval'],
  // evals: (name) => name.startsWith('api-'),

  // Setup function for sandbox pre-configuration
  setup: async (sandbox) => {
    await sandbox.writeFiles({ '.env': 'API_KEY=test' });
    await sandbox.runCommand('npm', ['run', 'setup']);
  },

  // Rewrite the prompt before running
  editPrompt: (prompt) => `Use the skill.\n\n${prompt}`,

  // Custom post-run analysis hook. Can attach analysis/metadata to result.json.
  onRunComplete: async ({ runData }) => ({
    ...runData,
    result: {
      ...runData.result,
      analysis: { mentionedBrands: ['Vercel'] },
    },
  }),

  // Optional brands to compare in downstream analysis.
  brands: [
    {
      id: 'vercel',
      name: 'Vercel',
      domain: 'vercel.com',
      aliases: ['Vercel Platform'],
      isYourBrand: true,
    },
  ],

  // Sandbox backend (default: 'auto' -- Vercel if token present, else Docker)
  sandbox: 'auto',

  // Copy project files to results directory (default: 'none')
  // 'none' - don't copy files
  // 'changed' - copy only files modified by the agent
  // 'all' - copy the entire project including original fixture files
  copyFiles: 'changed',
};

export default config;

```

--------------------------------

### Minimal Setup: Claude Code with Docker Sandbox

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Example TypeScript configuration for running Claude Code via its direct API using the Docker sandbox. This setup requires only the Anthropic API key and does not need a Vercel token.

```typescript
// experiments/my-eval.ts
import type { ExperimentConfig } from '@vercel/agent-eval';

const config: ExperimentConfig = {
  agent: 'claude-code',  // Direct API (not vercel-ai-gateway/...)
  model: 'opus',
  runs: 1,
  sandbox: 'docker',     // No VERCEL_TOKEN needed
};

export default config;
```

--------------------------------

### Launch Results Viewer

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Start the web-based viewer for results. This command provides a visual interface to analyze experiment outcomes.

```bash
npx @vercel/agent-eval playground
```

--------------------------------

### Initialize a New Agent Eval Project

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Use this command to scaffold a new agent evaluation project. Follow this with dependency installation and API key configuration.

```bash
npx @vercel/agent-eval init my-agent-evals
cd my-agent-evals

npm install

cp .env.example .env
# Edit .env with your AI_GATEWAY_API_KEY and VERCEL_TOKEN
```

--------------------------------

### Agent Selection Examples

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Illustrates how to select agents, distinguishing between Vercel AI Gateway and direct API integrations. Ensure provider API keys are set for direct API usage.

```typescript
// Vercel AI Gateway (recommended -- unified billing and observability)
agent: 'vercel-ai-gateway/claude-code'  // Claude Code via AI Gateway
agent: 'vercel-ai-gateway/codex'        // OpenAI Codex via AI Gateway
agent: 'vercel-ai-gateway/opencode'     // OpenCode via AI Gateway

// Direct API (uses provider keys directly)
agent: 'claude-code'  // requires ANTHROPIC_API_KEY
agent: 'codex'        // requires OPENAI_API_KEY
agent: 'gemini'       // requires GEMINI_API_KEY
agent: 'cursor'       // requires CURSOR_API_KEY

```

--------------------------------

### Perform a Smoke Test

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Use the --smoke flag for a quick setup verification. It runs the first experiment alphabetically once per model.

```bash
npx @vercel/agent-eval --smoke
```

--------------------------------

### Environment Variable for Direct API Key

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Set the Anthropic API key as an environment variable when using the minimal setup example for Claude Code with a Docker sandbox.

```bash
ANTHROPIC_API_KEY=sk-ant-...
```

--------------------------------

### Smoke Test a Single Experiment

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Apply the --smoke flag to a specific experiment to quickly verify its setup and basic functionality.

```bash
npx @vercel/agent-eval cc --smoke
```

--------------------------------

### Preview Experiments Without Running

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Use the --dry flag to see which experiments would run and what actions would be taken, without incurring any API costs or making actual calls.

```bash
npx @vercel/agent-eval --dry
```

--------------------------------

### Sandbox Options for Direct API Usage

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Choose one of these sandbox options when using direct API keys. Docker is free and requires no account, while Vercel requires a free account and a VERCEL_TOKEN.

```bash
# Option 1: Use Docker (free, no account needed)
# Just set sandbox: 'docker' in your experiment config, that's it!

# Option 2: Use Vercel (requires free account)
VERCEL_TOKEN=your-vercel-token
```

--------------------------------

### Launch Results Viewer in Watch Mode

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Enable live mode for the results viewer. This option automatically watches for and displays new results as they become available.

```bash
npx @vercel/agent-eval playground --watch
```

--------------------------------

### Run All Experiments

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Execute all discovered experiments in parallel. Results with matching fingerprints are automatically reused to save time and cost.

```bash
npx @vercel/agent-eval
```

--------------------------------

### Preview a Single Experiment

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Combine the experiment filename with the --dry flag to preview a specific experiment without execution.

```bash
npx @vercel/agent-eval cc --dry
```

--------------------------------

### Direct API Keys for AI Models

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Use these environment variables for direct API access to AI models if you do not have a Vercel account or prefer not to use the Vercel AI Gateway. You will need the specific API key for the model provider.

```bash
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-proj-...
```

--------------------------------

### Run a Single Experiment

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Execute a specific experiment by providing its filename (without the .ts extension) as an argument. This command targets a single test case.

```bash
npx @vercel/agent-eval cc
```

--------------------------------

### Vercel AI Gateway Environment Variables

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Set these environment variables to use the Vercel AI Gateway for all models. This is the recommended approach and requires an AI Gateway API key and a Vercel token.

```bash
AI_GATEWAY_API_KEY=your-ai-gateway-api-key
VERCEL_TOKEN=your-vercel-token
```

--------------------------------

### Generate a Changeset

Source: https://github.com/vercel-labs/agent-eval/blob/main/CONTRIBUTING.md

Use this command to create a new changeset file when your changes affect users. This is required for features, fixes, and breaking changes.

```bash
npx changeset
```

--------------------------------

### Run Agent Eval Playground

Source: https://github.com/vercel-labs/agent-eval/blob/main/packages/playground/README.md

Execute the agent-eval playground from your project's root directory. You can also specify custom directories for results and evals, and a port for the development server.

```bash
npx @vercel/agent-eval-playground
```

```bash
npx @vercel/agent-eval-playground --results-dir ./results --evals-dir ./evals --port 3001
```

--------------------------------

### OpenCode Model Format

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Specifies the required `vercel/{provider}/{model}` format for OpenCode models when using Vercel AI Gateway. Omitting the `vercel/` prefix will result in an error.

```typescript
model: 'vercel/anthropic/claude-sonnet-4'
model: 'vercel/openai/gpt-4o'
model: 'vercel/moonshotai/kimi-k2'
model: 'vercel/minimax/minimax-m2.1'

```

--------------------------------

### Summary JSON Structure

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Illustrates the structure of the summary.json file found in each evaluation directory. This file contains key metrics and failure classifications for an experiment run.

```json
{
  "totalRuns": 2,
  "passedRuns": 0,
  "passRate": "0%",
  "meanDuration": 45.2,
  "fingerprint": "a1b2c3...",
  "classification": {
    "failureType": "infra",
    "failureReason": "Rate limited (HTTP 429) — model never ran"
  },
  "valid": false
}
```

--------------------------------

### File Copying Configuration

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Demonstrates the `copyFiles` option within an ExperimentConfig. This setting controls which project files generated or modified by the agent are saved with the evaluation results.

```typescript
const config: ExperimentConfig = {
  copyFiles: 'changed',  // or 'all' or 'none' (default)
};
```

--------------------------------

### Response-Only Evals Configuration

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Sets up an experiment for response-only evaluation using `validation: 'none'`, suitable for tasks where only the agent's answer is critical. Requires `PROMPT.md` and `package.json`.

```typescript
const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  model: 'sonnet',
  validation: 'none',
  runs: 10,
  earlyExit: false,
  brands: [
    { id: 'vercel', name: 'Vercel', aliases: ['Vercel Platform'], isYourBrand: true },
    { id: 'netlify', name: 'Netlify' },
    { id: 'railway', name: 'Railway' },
  ],
  onRunComplete: async ({ runData }) => {
    // Add custom brand/recommendation analysis here.
    return runData;
  },
};

```

--------------------------------

### Multi-Model Experiment Configuration

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Configures an experiment to run across multiple models by providing an array to the `model` property. Results are segregated by model.

```typescript
const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  model: ['opus', 'sonnet'],
  runs: 10,
};

```

--------------------------------

### Force Re-run All Experiments

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Use the --force flag to ignore cached fingerprints and re-run all experiments, regardless of previous results. This option is only applicable when running all experiments.

```bash
npx @vercel/agent-eval --force
```

--------------------------------

### Native Agent Default Model Configuration

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Demonstrates omitting the `model` property to utilize the agent CLI's native default model. The `observedModel` will reflect the runtime model.

```typescript
const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  runs: 10,
};

```

--------------------------------

### Control Experiment Configuration

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

Defines a baseline experiment configuration for agent evaluation. This is used as a control in A/B testing.

```typescript
// experiments/control.ts
import type { ExperimentConfig } from '@vercel/agent-eval';

const config: ExperimentConfig = {
  agent: 'vercel-ai-gateway/claude-code',
  model: 'opus',
  runs: 10,
  earlyExit: false,
};

export default config;
```

--------------------------------

### EVAL.ts for Button Component Test

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

This TypeScript file contains tests to verify the creation of a Button component. It checks for the existence of the component file, ensures it has the required props, and confirms the project builds successfully.

```typescript
import { test, expect } from 'vitest';
import { readFileSync, existsSync } from 'fs';
import { execSync } from 'child_process';

test('Button component exists', () => {
  expect(existsSync('src/components/Button.tsx')).toBe(true);
});

test('has required props', () => {
  const content = readFileSync('src/components/Button.tsx', 'utf-8');
  expect(content).toContain('label');
  expect(content).toContain('onClick');
});

test('project builds', () => {
  execSync('npm run build', { stdio: 'pipe' });
});
```

--------------------------------

### Asserting Agent Shell Command Usage

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

This TypeScript test verifies that the agent used a specific scaffolding command. It reads the agent's execution results and checks if the expected command is present in the list of shell commands executed.

```typescript
import { test, expect } from 'vitest';
import { readFileSync } from 'fs';

test('agent used the correct scaffolding command', () => {
  const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));
  const commands = results.o11y.shellCommands.map((c: { command: string }) => c.command);
  expect(commands).toContain('npx create-next-app project');
});
```

--------------------------------

### Asserting Agent Tool Call Limits

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

This TypeScript test ensures the agent does not make an excessive number of tool calls. It parses the agent's results and asserts that the total number of tool calls is below a specified threshold.

```typescript
import { test, expect } from 'vitest';
import { readFileSync } from 'fs';

test('agent did not make excessive tool calls', () => {
  const results = JSON.parse(readFileSync('__agent_eval__/results.json', 'utf-8'));
  expect(results.o11y.totalToolCalls).toBeLessThan(50);
});
```

--------------------------------

### Acknowledge Non-Model Failures

Source: https://github.com/vercel-labs/agent-eval/blob/main/README.md

The --ack-failures flag allows non-model-related errors to be kept as final results instead of being discarded.

```bash
npx @vercel/agent-eval --ack-failures
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.