Agentest

https://r-prem.github.io/agentest/
Admin
Agentest provides scenario-based testing for AI agents with simulated users, tool-call mocks, and...
Tokens:58,687
Snippets:702
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Setup and Run CrewAI Agent Demo (HTTP Endpoint)

Source: https://r-prem.github.io/agentest/examples/demos.html

For the CrewAI demo, first start the Python server in one terminal, then in another, install Node.js dependencies, set up environment variables, and run the simulation.

```bash
# Terminal 1: start the Python server
cd demos/crewai-agent && pip install -r requirements.txt
npm run serve

# Terminal 2: run scenarios
cd demos/crewai-agent && npm install && cp .env.example .env
npm run sim
```

--------------------------------

### Install and Run LangChain Tool Agent Demo

Source: https://r-prem.github.io/agentest/examples/demos.html

Navigate to the demo directory, install dependencies, set up environment variables, and run the simulation for the LangChain Tool Agent.

```bash
cd demos/langchain-tool-agent && npm install && cp .env.example .env
npm run sim
```

--------------------------------

### GitHub Actions Workflow Example

Source: https://r-prem.github.io/agentest/guide/reporters.html

A sample GitHub Actions workflow to run Agentest. It includes setup for Node.js and execution of the agentest run command.

```yaml
name: Agent Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm install

      - name: Run agent tests
        run: npx agentest run
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          AGENT_API_KEY: ${{ secrets.AGENT_API_KEY }}
```

--------------------------------

### Install and Run Vercel AI SDK Agent Demo

Source: https://r-prem.github.io/agentest/examples/demos.html

Navigate to the demo directory, install dependencies, set up environment variables, and run the simulation for the Vercel AI SDK Agent.

```bash
cd demos/vercel-ai-agent && npm install && cp .env.example .env
npm run sim
```

--------------------------------

### Hybrid Setup - Two Pass Execution

Source: https://r-prem.github.io/agentest/guide/local-llms.html

This bash command demonstrates a workaround for a hybrid setup where a local model is used for the simulated user and a cloud model for evaluation. It involves running two separate passes.

```bash
# Pass 1: Run simulation with local LLM
agents run --config-file agent/config.local.yaml --output-dir results/local-sim

# Pass 2: Run evaluation with cloud LLM
agents run --config-file agent/config.cloud-eval.yaml --output-dir results/cloud-eval --resume results/local-sim
```

--------------------------------

### Console Reporter Live Progress Example

Source: https://r-prem.github.io/agentest/guide/reporters.html

Demonstrates the live progress spinner shown by the console reporter during simulation and evaluation.

```text
⠹ user books a morning slot → simulating conv-1-a3f8b2c1 turn 2/8 — calling agent
⠸ user books a morning slot → evaluating conv-1-a3f8b2c1 turn 2 — running metrics
```

--------------------------------

### Examples of User Profiles

Source: https://r-prem.github.io/agentest/guide/scenarios.html

Be specific with profiles to generate diverse conversations. Examples include technical users, beginners, busy executives, and edge case testers.

```typescript
// Technical user
profile: 'Senior developer who knows React and TypeScript. Prefers concise technical answers.'
```

```typescript
// Non-technical user
profile: 'First-time user unfamiliar with coding. Needs step-by-step guidance.'
```

```typescript
// Impatient user
profile: 'Busy executive. Types short messages. Expects quick, direct answers.'
```

```typescript
// Edge case tester
profile: 'QA engineer testing edge cases. Will try unusual inputs and corner cases.'
```

--------------------------------

### Example Tool Call Mock

Source: https://r-prem.github.io/agentest/guide/what-is-agentest.html

Demonstrates how to intercept and resolve tool calls within Agentest. This example shows a mock for a `check_availability` tool returning available slots and a `create_booking` tool confirming a booking.

```javascript
check_availability → { slots: ['09:00', '10:30'] }
create_booking → { success: true, bookingId: 'BK-001' }
```

--------------------------------

### Configure Tool Mocks

Source: https://r-prem.github.io/agentest/reference/scenario-api.html

Set up mocks for tools used in the scenario. This example shows a mock for 'my_tool' that returns a static result.

```typescript
mocks: {
  tools: {
    my_tool: (args, ctx) => ({ result: 'value' }),
  },
}
```

--------------------------------

### Install Agentest CLI

Source: https://r-prem.github.io/agentest/guide/getting-started.html

Install the Agentest CLI as a development dependency using npm.

```bash
npm install @agentesting/agentest --save-dev
```

--------------------------------

### Console Reporter Verbose Output Example

Source: https://r-prem.github.io/agentest/guide/reporters.html

An example of the detailed output when running Agentest in verbose mode, including full conversation turns, tool calls, and metrics.

```text
✓ user books a morning slot
  ✓ conv-1-a3f8b2c1

    Turn 1:
      User: I'd like to book a haircut for next Tuesday morning.
      Agent: I can help you with that! Let me check our availability for Tuesday morning.
      Tool calls:
        - check_availability({ date: '2026-03-25' })
          → { available: true, slots: ['09:00', '09:45', '10:30'] }
      Agent: We have several slots available on Tuesday morning: 9:00 AM, 9:45 AM, and 10:30 AM. Which time works best for you?

    Turn 2:
      User: 9:00 AM would be perfect.
      Agent: Great! I'll book you for 9:00 AM on Tuesday, March 25th.
      Tool calls:
        - create_booking({ date: '2026-03-25', time: '09:00', service: 'haircut' })
          → { success: true, bookingId: 'BK-001', confirmationSent: true }
      Agent: All set! Your appointment is confirmed for Tuesday, March 25th at 9:00 AM. Confirmation email sent.

    Metrics:
      helpfulness: 4.5 — Directly addressed user's goal with specific, actionable steps
      coherence: 5.0 — Perfectly consistent throughout the conversation
      relevance: 5.0 — Every response was on-topic and goal-oriented
      goal_completion: 1 — Goal fully achieved
```

--------------------------------

### Comparison Mode Output Example

Source: https://r-prem.github.io/agentest/guide/comparison-mode.html

Agentest displays a detailed per-metric comparison between agents after each scenario, followed by a summary of scenario pass rates.

```text
user books a morning slot
  ✓ gpt-4o
    ✓ conv-1-a3f8b2c1 (2 turns, goal met, helpfulness: 4.5)
    ✓ conv-2-d9e1f4a7 (3 turns, goal met, helpfulness: 4.2)
  ✓ gpt-4o-mini
    ✓ conv-1-b7d2e4f9 (3 turns, goal met, helpfulness: 3.8)
    ✓ conv-2-c1a4b7e3 (4 turns, goal met, helpfulness: 3.5)

  ── comparison ──
  helpfulness: gpt-4o: 4.35 | gpt-4o-mini: 3.65
  coherence:   gpt-4o: 5.0  | gpt-4o-mini: 4.2
  goal_completion: gpt-4o: 1.0 | gpt-4o-mini: 1.0
  ── end comparison ──

Comparison Summary
  gpt-4o: 5/5 scenarios passed
  gpt-4o-mini: 4/5 scenarios passed
```

--------------------------------

### Example Trajectory Assertion

Source: https://r-prem.github.io/agentest/guide/what-is-agentest.html

Illustrates how to assert the order and presence of tool calls made by the agent. This example uses the 'contains' match mode to verify that `check_availability` and `create_booking` were called.

```javascript
matchMode: 'contains', expected: [check_availability, create_booking]
```

--------------------------------

### Basic Setup with YAML

Source: https://r-prem.github.io/agentest/guide/comparison-mode.html

Configure comparison mode using YAML syntax. This achieves the same result as the TypeScript configuration, allowing for parallel testing of different agents.

```yaml
agent:
  name: gpt-4o
  endpoint: http://localhost:3000/api/chat
  body:
    model: gpt-4o
    temperature: 0.7

compare:
  - name: gpt-4o-mini
    body:
      model: gpt-4o-mini
  - name: claude-sonnet
    body:
      model: claude-sonnet-4-20250514
```

--------------------------------

### vLLM Provider Configuration

Source: https://r-prem.github.io/agentest/guide/local-llms.html

Example configuration for vLLM using the 'openai-compatible' provider. Specify the model name and the baseURL for the local server.

```typescript
provider: 'openai-compatible',
model: 'meta-llama/Llama-3.1-8B-Instruct',
providerOptions: { baseURL: 'http://localhost:8000/v1' },
```

--------------------------------

### Console Reporter Output Example

Source: https://r-prem.github.io/agentest/guide/reporters.html

Illustrates the detailed, human-readable output of the console reporter, including scenario status, scores, and error summaries.

```text
Agentest running 2 scenario(s)

  ⠹ user books a morning slot → simulating conv-1-a3f8b2c1 turn 2/8 — calling agent

✓ user books a morning slot
  ✓ conv-1-a3f8b2c1 (2 turns, goal met, trajectory matched)
    helpfulness: 4.5, coherence: 5.0, relevance: 5.0
  ✓ conv-2-d9e1f4a7 (3 turns, goal met, trajectory matched)
    helpfulness: 4.2, coherence: 4.8, relevance: 5.0
  ✓ conv-3-e8f2g5h9 (2 turns, goal met, trajectory matched)
    helpfulness: 4.8, coherence: 5.0, relevance: 5.0

  Average scores:
    helpfulness: 4.5 (threshold: 3.5 ✓)
    coherence: 4.9
    relevance: 5.0

✗ user cancels booking
  ✗ conv-1-b4c5d6e7 (3 turns, goal NOT met, trajectory matched)
    helpfulness: 3.1, coherence: 4.0, relevance: 4.5
  ✓ conv-2-f7g8h9i0 (2 turns, goal met, trajectory matched)
    helpfulness: 4.0, coherence: 4.5, relevance: 5.0

  Average scores:
    helpfulness: 3.55 (threshold: 3.5 ✓)
    goal_completion: 0.5 (threshold: 0.8 ✗)

  Unique errors:
    [high] Agent Failed to Complete Cancellation (1 occurrence)
      Agent provided cancellation information but didn't actually cancel the booking.

FAILED
1/2 scenarios passed
```

--------------------------------

### Registering Custom Metrics

Source: https://r-prem.github.io/agentest/reference/metrics-api.html

Example of how to register custom metrics using `defineConfig`. Custom metrics are provided in the `customMetrics` array, and thresholds can be set.

```APIDOC
## Registration 

```typescript
import { MyMetric } from './metrics/my-metric'

export default defineConfig({
  customMetrics: [new MyMetric()],
  thresholds: {
    my_metric: 4.0,
  },
})
```
```

--------------------------------

### Basic Setup with TypeScript

Source: https://r-prem.github.io/agentest/guide/comparison-mode.html

Configure comparison mode by adding a `compare` array to your Agentest config. Each entry overrides specific fields from the primary `agent`.

```typescript
export default defineConfig({
  agent: {
    name: 'gpt-4o',
    endpoint: 'http://localhost:3000/api/chat',
    body: { model: 'gpt-4o', temperature: 0.7 },
  },

  compare: [
    { name: 'gpt-4o-mini', body: { model: 'gpt-4o-mini' } },
    { name: 'claude-sonnet', body: { model: 'claude-sonnet-4-20250514' } },
  ],
})
```

--------------------------------

### Overall Run Failure Example

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Illustrates an overall Agentest run failing because one of the scenarios ('user cancels booking') failed.

```text
✓ user books a morning slot (3/3 conversations passed)
✗ user cancels booking (1/3 conversations errored)
✓ user reschedules appointment (3/3 conversations passed)

2/3 scenarios passed
FAIL: 1 scenario failed
```

--------------------------------

### Example Threshold Calculation

Source: https://r-prem.github.io/agentest/guide/evaluation-metrics.html

Demonstrates how average scores are calculated and compared against a threshold to determine scenario pass or fail status.

```yaml
thresholds: {
  helpfulness: 3.5,
}
```

--------------------------------

### LM Studio Provider Configuration

Source: https://r-prem.github.io/agentest/guide/local-llms.html

Example configuration for LM Studio using the 'openai-compatible' provider. Specify the model name and the baseURL for the local server.

```typescript
provider: 'openai-compatible',
model: 'lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF',
providerOptions: { baseURL: 'http://localhost:1234/v1' },
```

--------------------------------

### JSON Reporter Output Example

Source: https://r-prem.github.io/agentest/reference/cli.html

An example of the JSON output format generated by the JSON reporter. This structure contains detailed results for scenarios, including conversations and summaries, suitable for programmatic analysis.

```json
{
  "scenarios": [
    {
      "scenarioId": "...",
      "scenarioName": "user books a morning slot",
      "conversations": [...],
      "summary": {...}
    }
  ]
}
```

--------------------------------

### llama.cpp Server Provider Configuration

Source: https://r-prem.github.io/agentest/guide/local-llms.html

Example configuration for llama.cpp server using the 'openai-compatible' provider. Use 'default' for the model name and specify the baseURL.

```typescript
provider: 'openai-compatible',
model: 'default',  // llama.cpp uses a single model
providerOptions: { baseURL: 'http://localhost:8080/v1' },
```

--------------------------------

### Console Reporter Non-TTY Mode Example

Source: https://r-prem.github.io/agentest/guide/reporters.html

Shows how the console reporter adapts to non-TTY environments by disabling the spinner and displaying progress line-by-line.

```text
Running scenario: user books a morning slot
  Simulating conv-1-a3f8b2c1 turn 1/8
  Simulating conv-1-a3f8b2c1 turn 2/8
  Evaluating conv-1-a3f8b2c1
  ...
```

--------------------------------

### Full Agent Configuration Example

Source: https://r-prem.github.io/agentest/guide/yaml-config.html

This snippet shows a comprehensive YAML configuration for an agent, including its name, type, endpoint, headers, and body for chat completions. It also defines LLM provider settings, simulation parameters, evaluation metrics, file inclusion/exclusion patterns, and reporter configurations.

```yaml
agent:
  name: booking-agent
  type: chat_completions  # default
  endpoint: http://localhost:3000/api/chat
  streaming: false
  headers:
    Authorization: Bearer ${AGENT_API_KEY}
    Content-Type: application/json
  body:
    model: gpt-4o
    temperature: 0.7
    messages:
      - role: system
        content: You are a helpful booking assistant.

# LLM provider for simulated user and evaluation
provider: anthropic
model: claude-sonnet-4-20250514

# Simulation settings
conversationsPerScenario: 3
maxTurns: 8
concurrency: 20

# Evaluation
metrics:
  - helpfulness
  - coherence
  - relevance
  - faithfulness
  - goal_completion
  - agent_behavior_failure

thresholds:
  helpfulness: 3.5
  coherence: 4.0
  goal_completion: 0.8

failOnErrorSeverity: critical

# File discovery
include:
  - tests/**/*.sim.ts
  - scenarios/**/*.sim.ts
exclude:
  - node_modules/**
  - dist/**

# Reporters
reporters:
  - console
  - json
  - github-actions

# Mock behavior
unmockedTools: error
```

--------------------------------

### Full Agentest Configuration Example

Source: https://r-prem.github.io/agentest/guide/configuration.html

This snippet shows a comprehensive configuration for Agentest, including agent details, LLM provider, simulation settings, evaluation metrics, custom metrics, file discovery rules, output reporters, and mock behavior.

```typescript
import { defineConfig } from '@agentesting/agentest'
import { ToneMetric } from './metrics/tone.js'

export default defineConfig({
  // Agent configuration
  agent: {
    name: 'support-bot',
    endpoint: 'http://localhost:3000/api/chat',
    streaming: false,
    headers: {
      Authorization: 'Bearer ${AGENT_API_KEY}',
      'X-Org-ID': '${ORG_ID}',
    },
    body: {
      model: 'gpt-4o',
      temperature: 0.7,
      messages: [
        { role: 'system', content: 'You are a customer support agent for Acme Inc.' },
      ],
    },
  },

  // LLM provider for simulation and evaluation
  provider: 'anthropic',
  model: 'claude-sonnet-4-20250514',

  // Simulation settings
  conversationsPerScenario: 5,
  maxTurns: 10,
  concurrency: 10,

  // Evaluation
  metrics: [
    'helpfulness',
    'coherence',
    'relevance',
    'faithfulness',
    'goal_completion',
    'agent_behavior_failure',
    'tool_call_behavior_failure',
  ],

  thresholds: {
    helpfulness: 3.5,
    coherence: 4.0,
    goal_completion: 0.8,
  },

  failOnErrorSeverity: 'high',

  customMetrics: [new ToneMetric()],

  // File discovery
  include: ['tests/**/*.sim.ts'],
  exclude: ['node_modules/**', 'dist/**'],

  // Output
  reporters: ['console', 'json', 'github-actions'],

  // Mock behavior
  unmockedTools: 'error',
})
```

--------------------------------

### Examples of Concrete Goals

Source: https://r-prem.github.io/agentest/guide/scenarios.html

Define concrete and measurable goals for the simulated user. Vague goals make it harder for the LLM to judge completion.

```typescript
// Good goals (concrete and measurable)
goal: 'Book a haircut for next Tuesday morning.'
goal: 'Cancel order #12345 and get a refund confirmation.'
goal: 'Find restaurants near me that are open now.'
```

```typescript
// Vague goals (harder for LLM to judge completion)
goal: 'Use the booking system.'  // Too vague
goal: 'Ask about features.'      // No clear end state
```

--------------------------------

### Example Scenario Failure Due to Metric Threshold Violations

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Shows a scenario failing because 'helpfulness' and 'goal_completion' averages fell below their configured thresholds.

```text
Average scores:
  helpfulness: 3.2 (threshold: 3.5 ✗)
  goal_completion: 0.67 (threshold: 0.8 ✗)
  coherence: 4.5 (threshold: 4.0 ✓)
```

--------------------------------

### Configure Cloud LLM Providers

Source: https://r-prem.github.io/agentest/guide/configuration.html

Examples for configuring Agentest to use cloud-based LLM providers like Anthropic, OpenAI, and Google.

```typescript
// Anthropic (default)
provider: 'anthropic',
model: 'claude-sonnet-4-20250514',
```

```typescript
// OpenAI
provider: 'openai',
model: 'gpt-4o',
```

```typescript
// Google
provider: 'google',
model: 'gemini-2.0-flash',
```

--------------------------------

### Set Environment Variables and Run Agent Tests

Source: https://r-prem.github.io/agentest/reference/cli.html

Sets necessary environment variables for LLM provider API keys and agent authentication, then runs agent tests. This is a common setup for CI/CD pipelines or local development.

```bash
export ANTHROPIC_API_KEY=your-key-here
export AGENT_API_KEY=your-agent-key
npx agentest run
```

--------------------------------

### YAML Configuration Validation Error Example

Source: https://r-prem.github.io/agentest/guide/yaml-config.html

Illustrates a typical error message received when a YAML configuration fails validation against the Zod schema.

```text
Error: Invalid configuration
  - agent.endpoint must be a valid URL
  - thresholds.helpfulness must be between 0 and 5
  - conversationsPerScenario must be a positive integer
```

--------------------------------

### Simulation Parameters

Source: https://r-prem.github.io/agentest/guide/configuration.html

Configure the number of conversations, maximum turns, and concurrency for simulations. Use debounceMs to control the delay between scenario starts.

```yaml
conversationsPerScenario: 3,  // run 3 independent conversations per scenario
maxTurns: 8,                  // max user↔agent exchanges per conversation
concurrency: 20,              // max parallel LLM calls across all scenarios
debounceMs: 0                // optional delay (ms) between scenario starts
```

--------------------------------

### Implement a Quantitative Metric for Tone

Source: https://r-prem.github.io/agentest/guide/custom-metrics.html

Extend `QuantitativeMetric` and implement the `score()` method to evaluate the tone of an assistant's response. This example uses an LLM to rate the tone on a 1-5 scale.

```typescript
import { QuantitativeMetric, type ScoreInput, type QuantResult } from '@agentesting/agentest'
import { z } from 'zod'

export class ToneMetric extends QuantitativeMetric {
  readonly name = 'tone'

  async score(input: ScoreInput): Promise<QuantResult> {
    const result = await this.llm.generateObject({
      schema: z.object({
        score: z.number().min(1).max(5),
        reason: z.string(),
      }),
      messages: [
        {
          role: 'system',
          content: `Rate the tone of the assistant's response on a scale of 1-5.
1 = Hostile or dismissive
2 = Cold or robotic
3 = Neutral
4 = Friendly and professional
5 = Warm, empathetic, and engaging`,
        },
        {
          role: 'user',
          content: `User said: ${input.userMessage}\n\nAssistant replied: ${input.agentMessage}`,
        },
      ],
    })

    return { value: result.object.score, reason: result.object.reason }
  }
}
```

--------------------------------

### Example JSON Test Results

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Illustrates the structure of the JSON output from agentest, including scenario details, conversation data, and evaluation metrics.

```json
{
  "scenarios": [
    {
      "scenarioName": "user books a morning slot",
      "conversations": [
        {
          "conversationId": "conv-1-a3f8b2c1",
          "turns": [...],
          "error": null,
          "trajectoryResult": {
            "matched": false,
            "missingCalls": ["create_booking"],
            "extraCalls": [],
            "orderingIssues": []
          },
          "evaluation": {
            "helpfulness": { "value": 3.2, "reason": "..." },
            ...
          }
        }
      ],
      "summary": {
        "passed": false,
        "failureReasons": ["Trajectory assertion failed in 1 conversation"]
      }
    }
  ]
}
```

--------------------------------

### Custom Template: Multi-language Testing

Source: https://r-prem.github.io/agentest/guide/scenarios.html

Adapt the user prompt for multi-language testing by specifying the desired language for communication, such as German in this example.

```typescript
userPromptTemplate: `Du bist: {{profile}}

Dein Ziel: {{goal}}

Bekannte Fakten:
{{knowledge}}

Kommuniziere ausschließlich auf Deutsch.
Setze shouldStop:true wenn das Ziel erreicht wurde.`
```

--------------------------------

### Example Scenario Failure Due to Error Severity

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Demonstrates a scenario failing because detected errors ('critical', 'high') meet or exceed the configured 'high' failOnErrorSeverity.

```text
Unique errors:
  [critical] Security Violation: Leaked API Key (1 occurrence)
  [high] False Information About Pricing (2 occurrences)
  [medium] Unnecessary Clarification Requests (3 occurrences)
  [low] Slightly Verbose Responses (5 occurrences)
```

--------------------------------

### Example Merged Request Body

Source: https://r-prem.github.io/agentest/guide/configuration.html

Illustrates how the agent's default `body` configuration is merged with user messages for a complete request sent to the chat completions endpoint.

```json
{
  "model": "gpt-4o",
  "temperature": 0.7,
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "User's first message" },
    ...
  ]
}
```

--------------------------------

### Define Scenario with Knowledge Base

Source: https://r-prem.github.io/agentest/guide/scenarios.html

Provide `knowledge` items to give the simulated user realistic context and ground truth for metrics like `faithfulness`. This example includes tool mocks.

```typescript
scenario('user asks about weather', {
  profile: 'Casual user checking the weather.',
  goal: 'Get today's weather forecast for Seattle.',

  knowledge: [
    { content: 'Today is March 24, 2026.' },
    { content: 'The user is located in Seattle, WA.' },
  ],

  mocks: {
    tools: {
      get_weather: (args) => ({
        location: args.location,
        temperature: 58,
        condition: 'cloudy',
        forecast: 'Rain expected this afternoon',
      }),
    },
  },
})
```

--------------------------------

### Happy Path Scenario Example

Source: https://r-prem.github.io/agentest/guide/writing-effective-scenarios.html

Test the expected successful flow of a user completing a checkout process. Includes mocks for tool calls and assertions for expected tool usage.

```typescript
scenario('user completes checkout', {
  profile: 'Straightforward shopper. Provides payment info when asked.',
  goal: 'Purchase the item in cart and receive order confirmation.',
  mocks: {
    tools: {
      get_cart: () => ({ items: [{ name: 'Widget', price: 29.99 }] }),
      process_payment: () => ({ success: true, orderId: 'ORD-001' }),
    },
  },
  assertions: {
    toolCalls: {
      matchMode: 'contains',
      expected: [
        { name: 'get_cart', argMatchMode: 'ignore' },
        { name: 'process_payment', argMatchMode: 'ignore' },
      ],
    },
  },
})
```

--------------------------------

### Example Scenario Failure Due to Trajectory Assertion Mismatch

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Shows a scenario failing because one conversation missed a required tool call ('create_booking') according to trajectory assertions.

```text
✓ user books a morning slot
  ✓ conv-1-a3f8b2c1 (2 turns, goal met, trajectory matched)
  ✗ conv-2-d9e1f4a7 (2 turns, goal met, trajectory FAILED)
    Missing tool calls: create_booking
  ✓ conv-3-e8f2g5h9 (2 turns, goal met, trajectory matched)

2/3 conversations passed trajectory assertions
FAIL: Trajectory assertion failed in 1 conversation
```

--------------------------------

### Custom Agent Handler with TypeScript

Source: https://r-prem.github.io/agentest/guide/yaml-config.html

Example of a custom agent configuration using TypeScript, which is required for custom handler functions. This cannot be achieved with YAML configuration.

```typescript
// agentest.config.ts
import { defineConfig } from '@agentesting/agentest'
import { myAgent } from './src/agent.js'

export default defineConfig({
  agent: {
    type: 'custom',
    name: 'my-agent',
    handler: async (messages) => {
      const result = await myAgent.chat(messages)
      return { role: 'assistant', content: result.text }
    },
  },
})
```

--------------------------------

### Scenario Organization: Multiple Scenarios Per File

Source: https://r-prem.github.io/agentest/guide/writing-effective-scenarios.html

Group related scenarios within a single file to maintain organization. This example shows booking-related scenarios.

```typescript
// tests/booking.sim.ts
scenario('happy path — morning slot', { ... })
scenario('happy path — afternoon slot', { ... })
scenario('slot unavailable after check', { ... })
scenario('agent handles booking failure', { ... })
```

--------------------------------

### Agentest Scenario Execution Output

Source: https://r-prem.github.io/agentest/examples/basic-scenario.html

Example output showing a successful test run. It details the progress of scenarios, individual conversation analysis, and overall score thresholds.

```text
Agentest running 1 scenario(s)

  ⠹ user books a morning slot → simulating conv-1-a3f8b2c1 turn 2/8 — calling agent

✓ user books a morning slot
  ✓ conv-1-a3f8b2c1 (2 turns, goal met, trajectory matched)
    helpfulness: 4.5, coherence: 5.0, relevance: 5.0, goal_completion: 1
  ✓ conv-2-d9e1f4a7 (3 turns, goal met, trajectory matched)
    helpfulness: 4.2, coherence: 4.8, relevance: 5.0, goal_completion: 1
  ✓ conv-3-e8f2g5h9 (2 turns, goal met, trajectory matched)
    helpfulness: 4.8, coherence: 5.0, relevance: 5.0, goal_completion: 1

  Average scores:
    helpfulness: 4.5 (threshold: 3.5 ✓)
    coherence: 4.9
    relevance: 5.0
    goal_completion: 1.0 (threshold: 0.8 ✓)

1/1 scenarios passed
```

--------------------------------

### Run Agent Tests with Custom Configuration

Source: https://r-prem.github.io/agentest/reference/cli.html

Overrides the default configuration discovery by specifying a custom configuration file. This is useful for managing different testing environments or setups.

```bash
npx agentest run --config custom-config.ts
```

--------------------------------

### LLM Access in Metrics

Source: https://r-prem.github.io/agentest/reference/metrics-api.html

Custom metrics have access to `this.llm` for generating responses using an LLM. This example shows how to use `generateObject` to get a structured response.

```APIDOC
## LLM Access 

Custom metrics have access to `this.llm`:

```typescript
async score(input: ScoreInput): Promise<QuantResult> {
  const result = await this.llm.generateObject({
    schema: z.object({ score: z.number(), reason: z.string() }),
    messages: [{ role: 'user', content: 'prompt' }],
  })
  return { value: result.object.score, reason: result.object.reason }
}
```
```

--------------------------------

### Run Basic Agentest Scenario

Source: https://r-prem.github.io/agentest/examples/basic-scenario.html

Execute the default test suite for the Agentest project. This command initiates the testing process for all defined scenarios.

```bash
npx agentest run
```

--------------------------------

### Configure Local LLM Providers

Source: https://r-prem.github.io/agentest/guide/configuration.html

Examples for configuring Agentest to use local LLM providers such as Ollama or OpenAI-compatible endpoints.

```typescript
// Ollama (defaults to http://localhost:11434/v1)
provider: 'ollama',
model: 'llama3.2',
```

```typescript
// OpenAI-compatible (LM Studio, vLLM, llama.cpp)
provider: 'openai-compatible',
model: 'my-model',
providerOptions: {
  baseURL: 'http://localhost:1234/v1',  // required
  apiKey: 'optional-key',                // optional, defaults to 'not-needed'
},
```

--------------------------------

### LangChain Agent Integration

Source: https://r-prem.github.io/agentest/guide/framework-integration.html

Integrate a LangChain ChatOpenAI model as a custom Agentest agent. Ensure you have the necessary LangChain dependencies installed.

```typescript
import { ChatOpenAI } from '@langchain/openai'
import { defineConfig, type ChatMessage } from '@agentesting/agentest'

const model = new ChatOpenAI({ model: 'gpt-4o' })

export default defineConfig({
  agent: {
    type: 'custom',
    name: 'langchain-agent',
    handler: async (messages: ChatMessage[]) => {
      const response = await model.invoke(
        messages.map((m) => ({ role: m.role, content: m.content })),
      )
      return { role: 'assistant' as const, content: response.content as string }
    },
  },
})
```

--------------------------------

### Show Default Prompts

Source: https://r-prem.github.io/agentest/guide/scenarios.html

Use this command to view the default system prompts generated by Agentest when `userPromptTemplate` is not provided.

```bash
npx agentest show-prompts
```

--------------------------------

### Running Agentest with a Custom Config Path

Source: https://r-prem.github.io/agentest/guide/yaml-config.html

Demonstrates how to specify a custom configuration file path using the --config flag with the Agentest CLI.

```bash
npx agentest run --config custom-config.yaml
```

```bash
npx agentest run --config configs/ci.yaml
```

```bash
npx agentest run --config agentest.prod.yml
```

--------------------------------

### Testing Tool Call Order with Trajectory Assertions

Source: https://r-prem.github.io/agentest/guide/mocks.html

Example of using trajectory assertions to verify the exact order in which tools are called during a scenario.

```typescript
scenario('user completes checkout', {
  profile: 'Customer ready to purchase.',
  goal: 'Complete the checkout process.',

  mocks: {
    tools: {
      validate_cart: (args) => ({ valid: true, total: 49.99 }),
      process_payment: (args) => ({ success: true, transactionId: 'TXN-001' }),
      send_confirmation: (args) => ({ sent: true, emailId: 'EMAIL-001' }),
    },
  },

  assertions: {
    toolCalls: {
      matchMode: 'strict',  // exact order required
      expected: [
        { name: 'validate_cart', argMatchMode: 'ignore' },
        { name: 'process_payment', argMatchMode: 'ignore' },
        { name: 'send_confirmation', argMatchMode: 'ignore' },
      ],
    },
  },
})
```

--------------------------------

### CLI Reference

Source: https://r-prem.github.io/agentest/reference/api.html

Documentation for the Agentest command-line interface.

```APIDOC
## CLI Reference

### Description
Provides documentation for the Agentest command-line interface commands and usage.
```

--------------------------------

### File Discovery Configuration

Source: https://r-prem.github.io/agentest/reference/config-api.html

Configure glob patterns for discovering scenario files.

```APIDOC
## File Discovery

### `include`

**Type:** `string[]`
**Default:** `['**/*.sim.ts']`

Glob patterns for scenario files.

### `exclude`

**Type:** `string[]`
**Default:** `['node_modules/**']`

Glob patterns to exclude.
```

--------------------------------

### Simulation Configuration

Source: https://r-prem.github.io/agentest/reference/config-api.html

Configure parameters for running simulations.

```APIDOC
## Simulation

### `conversationsPerScenario`

**Type:** `number`
**Default:** `3`

Number of independent conversations per scenario.

### `maxTurns`

**Type:** `number`
**Default:** `8`

Maximum turns per conversation.

### `concurrency`

**Type:** `number`
**Default:** `20`

Max parallel LLM calls.

### `debounceMs`

**Type:** `number`
**Default:** `0`

Minimum delay in milliseconds between scenario starts. Set this to avoid rate limits when running many scenarios (e.g., `debounceMs: 2000` for a 2-second gap).
```

--------------------------------

### Define Unique Error Structure

Source: https://r-prem.github.io/agentest/guide/how-it-works.html

Defines the structure for a unique error, including its label, description, severity, number of occurrences, and associated examples.

```typescript
interface UniqueError {
  label: string          // "Providing weather info without real-time access"
  description: string    // Detailed explanation
  severity: 'low' | 'medium' | 'high' | 'critical'
  occurrences: number
  examples: TurnResult[]
}
```

--------------------------------

### Configure OpenAI-Compatible Provider

Source: https://r-prem.github.io/agentest/guide/local-llms.html

Use this configuration for any server exposing an OpenAI-compatible API, such as LM Studio, vLLM, or llama.cpp. Specify the baseURL and optionally an apiKey.

```typescript
export default defineConfig({
  provider: 'openai-compatible',
  model: 'my-local-model',
  providerOptions: {
    baseURL: 'http://localhost:1234/v1',  // required
    apiKey: 'not-needed',                 // optional, defaults to placeholder
  },
})
```

--------------------------------

### Development vs. CI Configuration

Source: https://r-prem.github.io/agentest/guide/writing-effective-scenarios.html

Configure scenario settings for faster development cycles or thorough CI runs. Uses environment variables to switch between modes.

```typescript
// Development — fast and cheap
export default defineConfig({
  conversationsPerScenario: 1,
  maxTurns: 4,
  metrics: ['helpfulness', 'goal_completion'],
  concurrency: 5,
})

// CI — thorough
export default defineConfig({
  conversationsPerScenario: 5,
  maxTurns: 8,
  // all metrics (default)
  concurrency: 20,
})
```

```typescript
const isDev = process.env.CI !== 'true'

export default defineConfig({
  conversationsPerScenario: isDev ? 1 : 5,
  maxTurns: isDev ? 4 : 8,
  metrics: isDev ? ['helpfulness', 'goal_completion'] : undefined,  // undefined = all
})
```

--------------------------------

### Passthrough Mode with Scenario Mocking

Source: https://r-prem.github.io/agentest/guide/mocks.html

Example of setting 'passthrough' mode and mocking only critical tools within a scenario, letting others return undefined.

```typescript
export default defineConfig({
  unmockedTools: 'passthrough',
  // ...
})

scenario('test booking flow', {
  mocks: {
    tools: {
      // Only mock the critical tools
      check_availability: (args) => ({ available: true }),
      create_booking: (args) => ({ success: true }),
      // send_email, log_analytics, etc. will return undefined
    },
  },
})
```

--------------------------------

### Sequence Mock Behavior Example

Source: https://r-prem.github.io/agentest/guide/mocks.html

Illustrates how sequence() returns values from the provided array for successive calls, repeating the last value for subsequent calls.

```typescript
create_order: sequence([
  { success: true, orderId: 'ORD-001' },
  { success: true, orderId: 'ORD-002' },
])

// Calls:
// 1st call → { success: true, orderId: 'ORD-001' }
// 2nd call → { success: true, orderId: 'ORD-002' }
// 3rd call → { success: true, orderId: 'ORD-002' }  ← repeats last
// 4th call → { success: true, orderId: 'ORD-002' }  ← repeats last
```

--------------------------------

### Set LLM Provider API Keys

Source: https://r-prem.github.io/agentest/guide/getting-started.html

Configure API keys for different LLM providers (Anthropic, OpenAI, Google) using environment variables.

```bash
# For Anthropic (default provider)
export ANTHROPIC_API_KEY=your-key-here

# For OpenAI
export OPENAI_API_KEY=your-key-here

# For Google
export GOOGLE_GENERATIVE_AI_API_KEY=your-key-here
```

--------------------------------

### Example Agentest Error Output

Source: https://r-prem.github.io/agentest/guide/evaluation-metrics.html

Illustrates how Agentest groups and reports unique errors with severity and occurrence counts, instead of listing repeated failures.

```text
Unique errors:
  [high] Providing Weather Info Without Real-Time Access (3 occurrence(s))
    Agent gave specific forecasts without live data, leading to false information.
    Examples: turn 2, turn 5, turn 7

  [medium] Unnecessary Clarification When Context Is Available (2 occurrence(s))
    Agent asked for details already present in the conversation.
    Examples: turn 3, turn 6
```

--------------------------------

### Example Scenario Failure Due to Conversation Error

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Illustrates a scenario failing because one conversation encountered an error, even though others succeeded.

```text
✗ user books a morning slot
  ✗ conv-1-a3f8b2c1 ERROR: Agent endpoint returned 500
  ✓ conv-2-d9e1f4a7 (2 turns, goal met)
  ✓ conv-3-e8f2g5h9 (3 turns, goal met)

1/3 conversations succeeded
FAIL: 1 conversation errored
```

--------------------------------

### YAML Configuration with Environment Variable Interpolation

Source: https://r-prem.github.io/agentest/guide/yaml-config.html

Demonstrates how to use environment variables within YAML configuration for sensitive information like API keys and custom headers. Ensure referenced variables are set to avoid loading failures.

```yaml
agent:
  endpoint: https://api.example.com/chat
  headers:
    Authorization: Bearer ${AGENT_API_KEY}
    X-Custom-Header: ${CUSTOM_VALUE}

```

--------------------------------

### Mocking Discount Logic with Knowledge Items

Source: https://r-prem.github.io/agentest/guide/mocks.html

Combine knowledge items with mocks to define and test complex business logic, such as discount application based on customer type or codes.

```typescript
scenario('test discount application', {
  knowledge: [
    { content: 'VIP customers get 20% discount on all orders.' },
    { content: 'Discount code SAVE10 gives 10% off.' },
  ],

  mocks: {
    tools: {
      apply_discount: (args) => {
        if (args.code === 'SAVE10') {
          return { discountPercent: 10, finalPrice: 44.99 }
        }
        if (args.customerType === 'VIP') {
          return { discountPercent: 20, finalPrice: 39.99 }
        }
        return { discountPercent: 0, finalPrice: 49.99 }
      },
    },
  },
})
```

--------------------------------

### Configure Reporters in Agentest

Source: https://r-prem.github.io/agentest/reference/cli.html

Defines the reporters to be used for test results in the Agentest configuration file. This example includes console, JSON, and GitHub Actions reporters.

```typescript
export default defineConfig({
  reporters: ['console', 'json', 'github-actions'],
})
```

--------------------------------

### JSON Reporter Output Structure

Source: https://r-prem.github.io/agentest/guide/reporters.html

Example structure of the JSON output generated by the JSON reporter. This file contains detailed test results for programmatic analysis.

```json
{
  "runId": "run-2026-03-29T10:30:00Z",
  "startedAt": "2026-03-29T10:30:00.000Z",
  "completedAt": "2026-03-29T10:32:15.432Z",
  "durationMs": 135432,
  "config": {
    "conversationsPerScenario": 3,
    "maxTurns": 8,
    "provider": "anthropic",
    "model": "claude-sonnet-4-20250514"
  },
  "scenarios": [
    {
      "scenarioId": "scenario-1",
      "scenarioName": "user books a morning slot",
      "passed": true,
      "conversations": [
        {
          "conversationId": "conv-1-a3f8b2c1",
          "passed": true,
          "turns": [
            {
              "turnIndex": 0,
              "userMessage": "I'd like to book a haircut for next Tuesday morning.",
              "agentMessage": "I can help you with that!",
              "toolCalls": [
                {
                  "name": "check_availability",
                  "arguments": { "date": "2026-03-25" },
                  "result": { "available": true, "slots": ["09:00", "09:45", "10:30"] }
                }
              ],
              "evaluation": {
                "helpfulness": { "value": 4.5, "reason": "..." },
                "coherence": { "value": 5.0, "reason": "..." },
                "relevance": { "value": 5.0, "reason": "..." }
              }
            }
          ],
          "trajectoryResult": {
            "matched": true,
            "missingCalls": [],
            "extraCalls": [],
            "orderingIssues": []
          },
          "error": null
        }
      ],
      "summary": {
        "conversationCount": 3,
        "passedCount": 3,
        "errorCount": 0,
        "averageScores": {
          "helpfulness": 4.5,
          "coherence": 4.9,
          "relevance": 5.0,
          "goal_completion": 1.0
        },
        "thresholdViolations": [],
        "uniqueErrors": []
      }
    }
  ],
  "summary": {
    "totalScenarios": 2,
    "passedScenarios": 1,
    "failedScenarios": 1,
    "totalConversations": 6,
    "passedConversations": 5,
    "errorConversations": 0
  }
}
```

--------------------------------

### Run Agentest with Local Config

Source: https://r-prem.github.io/agentest/guide/local-llms.html

Execute Agentest with a local configuration, typically not requiring an API key.

```bash
ANTHROPIC_API_KEY=not-needed npx agentest run  # with local config
```

--------------------------------

### Define All Scenarios with defineSimSuite

Source: https://r-prem.github.io/agentest/guide/vitest-integration.html

Use `defineSimSuite` to run all Agentest scenarios as a single Vitest test. This is the simplest way to integrate.

```typescript
import { defineSimSuite } from '@agentesting/agentest/vitest'

defineSimSuite({
  agent: { name: 'my-agent', endpoint: 'http://localhost:3000/api/chat' },
})
```

--------------------------------

### Run Agentest Scenarios

Source: https://r-prem.github.io/agentest/reference/cli.html

Use this command to execute all defined scenarios or to filter them by name or file path. You can also specify a custom configuration file or working directory.

```bash
npx agentest run [options]
```

```bash
# Run all scenarios
npx agentest run
```

```bash
# Custom config file (TypeScript or YAML)
npx agentest run --config path/to/config.ts
npx agentest run --config agentest.config.yaml
```

```bash
# Different working directory
npx agentest run --cwd ./packages/my-agent
```

```bash
# Filter scenarios by name
npx agentest run --scenario "booking"
npx agentest run --scenario "cancel"
```

```bash
# Filter by file path
npx agentest run --file "auth.sim.ts"
npx agentest run --file "booking"
```

```bash
# Print full conversation transcripts
npx agentest run --verbose
```

```bash
# Watch mode
npx agentest run --watch
npx agentest run -w
```

```bash
# Combine flags
npx agentest run --watch --verbose --scenario "booking"
```

```bash
# Combine file and scenario filters
npx agentest run --file "auth" --scenario "login"
```

--------------------------------

### Define Critical Path Scenarios

Source: https://r-prem.github.io/agentest/guide/writing-effective-scenarios.html

Start by defining the 3-5 most critical user journeys for your agent. Each scenario should focus on a single, specific behavior or edge case.

```typescript
scenario('happy path — user books successfully', { ... })
scenario('slot becomes unavailable mid-conversation', { ... })
scenario('user provides ambiguous information', { ... })
scenario('agent handles tool failure gracefully', { ... })
```

--------------------------------

### Run a Specific Scenario

Source: https://r-prem.github.io/agentest/guide/pass-fail-logic.html

Execute only the 'booking' scenario to speed up iteration when focusing on specific failing tests.

```bash
npx agentest run --scenario "booking"
```

--------------------------------

### GitHub Actions Reporter Annotations

Source: https://r-prem.github.io/agentest/guide/reporters.html

Example of GitHub Actions annotations emitted by the reporter for errors, warnings, and notices. These appear inline in the 'Files changed' tab of PRs.

```bash
::error file=tests/booking.sim.ts,line=42::Scenario 'user cancels booking' failed: Threshold violation: goal_completion (0.5 < 0.8)
::warning::Agent behavior failure detected: repetition (3 occurrences)
::notice::Scenario 'user books a morning slot' passed with average helpfulness: 4.5
```

--------------------------------

### Define Trajectory Assertions

Source: https://r-prem.github.io/agentest/reference/scenario-api.html

Specify assertions for tool calls within a scenario's trajectory. This example uses 'contains' match mode to check for a specific tool call.

```typescript
assertions: {
  toolCalls: {
    matchMode: 'contains',
    expected: [
      { name: 'tool1', argMatchMode: 'ignore' },
    ],
  },
}
```

--------------------------------

### Set Environment Variables for API Keys

Source: https://r-prem.github.io/agentest/guide/configuration.html

How to set environment variables for cloud providers to automatically provide API keys to Agentest.

```bash
export ANTHROPIC_API_KEY=your-key-here
export OPENAI_API_KEY=your-key-here
export GOOGLE_GENERATIVE_AI_API_KEY=your-key-here
```
Tokens:
Raw