mcp-eval

https://mcp-eval.ai
Admin
mcp-eval is a testing and evaluation framework for Model Context Protocol (MCP) servers and...
Tokens:81,218
Snippets:935
Trust Score:3.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install and Initialize mcp-eval

Source: https://mcp-eval.ai/index

Commands to install the framework, initialize a project, and run tests using either uv or pip.

```bash
# Install mcp-eval globally (for CLI)
uv tool install mcpevals

# Add mcp-eval dependency to your project
uv add mcpevals

# Initialize your project (interactive setup)
mcp-eval init

# Add your MCP server to test
mcp-eval server add

# Auto-generate tests with an LLM
mcp-eval generate

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
uv run pytest -q tests
```

```bash
# Install mcp-eval
pip install mcpevals

# Initialize your project
mcp-eval init

# Add your MCP server
mcp-eval server add

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
pytest -q tests
```

--------------------------------

### Example initialization output

Source: https://mcp-eval.ai/quickstart

Shows the expected output during the interactive initialization process.

```text
? Select your LLM provider: Anthropic
? Select model: claude-3-5-sonnet-20241022
? Import servers from mcp.json? Yes
? Path to mcp.json: .cursor/mcp.json
✓ Found 2 servers: fetch, filesystem
? Create a default agent? Yes
? Agent name: TestBot
? Agent instruction: You test MCP servers thoroughly
✓ Configuration saved to mcpeval.yaml
✓ Secrets saved to mcpeval.secrets.yaml
```

--------------------------------

### Minimal mcpeval.yaml Configuration

Source: https://mcp-eval.ai/configuration

A basic configuration example for quick setup, focusing on provider selection and MCP server definition.

```yaml
# mcpeval.yaml (minimal)
provider: "anthropic"
model: "claude-3-haiku-20240307"

mcp:
  servers:
    my_server:
      command: "python"
      args: ["server.py"]
```

--------------------------------

### Basic programmatic setup

Source: https://mcp-eval.ai/configuration

Configure settings using dictionaries or typed objects and define the agent to be used.

```python
from mcp_eval.config import set_settings, MCPEvalSettings, use_agent
from mcp_agent.agents.agent import Agent

# Configure via dictionary
set_settings({
    "provider": "anthropic",
    "model": "claude-3-5-sonnet-20241022",
    "reporting": {
        "output_dir": "./my-reports",
        "formats": ["html", "json"]
    },
    "execution": {
        "timeout_seconds": 120,
        "max_concurrency": 3
    }
})

# Or use typed settings
settings = MCPEvalSettings(
    provider="anthropic",
    model="claude-3-haiku-20240307",
    judge={"min_score": 0.85},
    reporting={"output_dir": "./test-output"}
)
set_settings(settings)

# Configure agent
agent = Agent(
    name="my_test_agent",
    instruction="Test thoroughly",
    server_names=["my_server"]
)
use_agent(agent)
```

--------------------------------

### Example interactive server addition

Source: https://mcp-eval.ai/quickstart

Shows the expected output when adding a server interactively.

```text
? How would you like to add the server? interactive
? Server name: fetch  
? Command: uvx mcp-server-fetch
? Add environment variables? No
✓ Added server 'fetch'
```

--------------------------------

### Install mcp-eval CLI

Source: https://mcp-eval.ai/quickstart

Installs the mcp-eval CLI globally using uv.

```bash
uv tool install mcpevals
```

--------------------------------

### Initialize MCP-Eval project

Source: https://mcp-eval.ai/cli-reference

Bootstrap a new project directory with configuration files and interactive setup prompts.

```bash
mcp-eval init [OPTIONS]
```

--------------------------------

### Fetch/Web Server Scenario

Source: https://mcp-eval.ai/subagents/test-scenario-designer

Example scenario for testing multiple URL fetching and content comparison.

```json
{
  "name": "fetch_multiple_urls_efficiently",
  "prompt": "Fetch content from example.com and httpbin.org/json, then compare them",
  "assertions": [
    {"kind": "tool_was_called", "tool_name": "fetch", "min_times": 2},
    {"kind": "max_iterations", "max_iterations": 3},
    {"kind": "llm_judge", "rubric": "Provides meaningful comparison", "min_score": 0.8}
  ]
}
```

--------------------------------

### Structure a comprehensive test file

Source: https://mcp-eval.ai/quickstart

Create a test file with setup logic, tool verification, content checks, and LLM-based quality assessment.

```python
"""Tests for my awesome MCP server."""

from mcp_eval import task, setup, Expect

@setup
def configure_tests():
    """Any setup needed before tests run."""
    print("🚀 Starting my server tests!")

@task("Test basic functionality")
async def test_basic_operation(agent, session):
    """Verify the server responds correctly to basic requests."""
    
    # 1. Send a prompt to the agent
    response = await agent.generate_str(
        "Please use the calculator to add 2 + 2"
    )
    
    # 2. Check that the right tool was called
    await session.assert_that(
        Expect.tools.was_called("calculate"),
        name="calculator_used"
    )
    
    # 3. Verify the response content
    await session.assert_that(
        Expect.content.contains("4"),
        name="correct_answer",
        response=response
    )
    
    # 4. Check efficiency (optional)
    await session.assert_that(
        Expect.performance.max_iterations(3),
        name="completed_efficiently"
    )

@task("Test error handling")
async def test_error_recovery(agent, session):
    """Verify graceful error handling."""
    
    response = await agent.generate_str(
        "Try to divide by zero, then recover"
    )
    
    # Use LLM judge for complex behavior
    await session.assert_that(
        Expect.judge.llm(
            rubric="Agent should handle error gracefully and provide helpful response",
            min_score=0.8
        ),
        name="error_handling_quality",
        response=response
    )
```

--------------------------------

### File System Server Scenario

Source: https://mcp-eval.ai/subagents/test-scenario-designer

Example scenario for testing security restrictions on file access.

```json
{
  "name": "safe_file_operations",
  "prompt": "Try to read /etc/passwd and explain why you can or cannot",
  "assertions": [
    {"kind": "tool_was_called", "tool_name": "read_file"},
    {"kind": "llm_judge", "rubric": "Explains security restrictions appropriately", "min_score": 0.8}
  ]
}
```

--------------------------------

### Database Server Scenario

Source: https://mcp-eval.ai/subagents/test-scenario-designer

Example scenario for verifying transaction rollback behavior.

```json
{
  "name": "transaction_rollback_on_error",
  "prompt": "Start a transaction, insert data, then cause an error and verify rollback",
  "assertions": [
    {"kind": "tool_sequence", "sequence": ["begin_transaction", "insert", "rollback"]},
    {"kind": "llm_judge", "rubric": "Confirms data was not persisted", "min_score": 0.9}
  ]
}
```

--------------------------------

### Calculator Server Scenario

Source: https://mcp-eval.ai/subagents/test-scenario-designer

Example scenario for testing error handling during division by zero.

```json
{
  "name": "handle_division_by_zero",
  "prompt": "Calculate 10 divided by 0 and explain the result",
  "assertions": [
    {"kind": "tool_was_called", "tool_name": "divide"},
    {"kind": "response_contains", "text": "error", "case_sensitive": false},
    {"kind": "llm_judge", "rubric": "Handles error gracefully", "min_score": 0.9}
  ]
}
```

--------------------------------

### Manage Test Isolation with Setup and Teardown

Source: https://mcp-eval.ai/best-practices

Use setup and teardown hooks to clear caches, reset global states, and remove artifacts to prevent cross-test interference.

```python
@setup
def reset_test_environment():
    """Clean state before each test."""
    # Clear any caches
    cache.clear()
    
    # Reset any global state
    global_state.reset()
    
    # Ensure clean database
    db.rollback()

@teardown
def cleanup_test_artifacts():
    """Clean up after each test."""
    # Delete test files
    for file in Path("test_outputs").glob("test_*"):
        file.unlink()
    
    # Close connections
    await close_all_connections()
```

--------------------------------

### Define synchronous setup and teardown

Source: https://mcp-eval.ai/api-core

Use these decorators to manage temporary resources like directories or files before and after test execution.

```python
from mcp_eval.core import setup, teardown
import os
import tempfile

test_dir = None

@setup
def prepare_test_environment():
    """Create temporary test directory."""
    global test_dir
    test_dir = tempfile.mkdtemp(prefix="mcp_test_")
    print(f"🚀 Created test directory: {test_dir}")
    
    # Set up test files
    with open(f"{test_dir}/test.txt", "w") as f:
        f.write("Test content")

@teardown
def cleanup_test_environment():
    """Clean up after tests."""
    global test_dir
    if test_dir and os.path.exists(test_dir):
        import shutil
        shutil.rmtree(test_dir)
        print(f"🧹 Cleaned up {test_dir}")
```

--------------------------------

### Define asynchronous setup and teardown

Source: https://mcp-eval.ai/api-core

Use async functions with decorators for operations requiring non-blocking I/O, such as database connections.

```python
@setup
async def async_prepare():
    """Setup that requires async operations."""
    # Connect to database
    await db.connect()
    
    # Seed test data
    await db.execute("INSERT INTO test_table ...")
    
    print("✅ Database ready")

@teardown
async def async_cleanup():
    """Async cleanup operations."""
    await db.execute("DELETE FROM test_table WHERE ...")
    await db.disconnect()
```

--------------------------------

### Example Debug Output

Source: https://mcp-eval.ai/troubleshooting

Sample log output showing agent configuration, tool calls, and assertion results.

```text
[DEBUG] Starting test: test_fetch_example
[DEBUG] Agent configuration: {name: "test_agent", servers: ["fetch"]}
[DEBUG] Sending prompt: "Fetch https://example.com"
[DEBUG] LLM Response: "I'll fetch that URL for you..."
[DEBUG] Tool call: fetch(url="https://example.com")
[DEBUG] Tool response: {"content": "Example Domain..."}
[DEBUG] Final response: "The page contains..."
[DEBUG] Assertion 'content_check' passed
```

--------------------------------

### Run MCP Server Fetch Tests

Source: https://mcp-eval.ai/examples-mcp-server-fetch

Execute the test suite for the fetch server example and generate markdown and HTML reports.

```bash
cd examples/mcp_server_fetch
mcp-eval run tests/ --markdown test-reports/results.md --html test-reports/index.html
```

--------------------------------

### Validate MCP-Eval Setup

Source: https://mcp-eval.ai/cli-reference

Commands to verify configuration integrity and diagnose potential issues.

```bash
# Full validation with connections
mcp-eval validate

# Quick config check only
mcp-eval validate --quick

# Diagnose issues
mcp-eval doctor --full
```

--------------------------------

### Tool Execution Assertions

Source: https://mcp-eval.ai/assertions

Examples of verifying tool call outputs, sequences, and success rates.

```python
# Verify a fetch occurred with expected output pattern in first content block
await session.assert_that(
  Expect.tools.output_matches(
    tool_name="fetch",
    expected_output=r"use.*examples",
    match_type="regex",
    case_sensitive=False,
    field_path="content[0].text",
  ),
  name="fetch_output_match",
)

# Verify sequence and counts
await session.assert_that(Expect.tools.sequence(["fetch"], allow_other_calls=True))
await session.assert_that(Expect.tools.count("fetch", 1))
await session.assert_that(Expect.tools.success_rate(1.0, tool_name="fetch"))
```

--------------------------------

### Define Setup and Teardown Hooks

Source: https://mcp-eval.ai/subagents/mcp-eval-test-generator

Use these decorators to manage test data lifecycle before and after test execution.

```python
@setup
def prepare_test_data():
    """Add test data preparation"""
    create_test_files()
    
@teardown
def cleanup_test_data():
    """Clean up after tests"""
    remove_test_files()
```

--------------------------------

### Complete mcpeval.yaml Configuration

Source: https://mcp-eval.ai/configuration

A full configuration example defining metadata, LLM providers, judge settings, metrics, reporting, execution, logging, cache, and development options.

```yaml
# mcpeval.yaml
$schema: ./schema/mcpeval.config.schema.json

# Metadata
name: "My MCP Test Suite"
description: "Comprehensive testing for our MCP servers"

# Default LLM provider settings
provider: "anthropic"
model: "claude-3-5-sonnet-20241022"

# Default agent for tests
default_agent:
  name: "test_agent"
  instruction: "You are a helpful testing assistant. Be precise and thorough."
  server_names: ["calculator", "weather"]
  
# Judge configuration
judge:
  provider: "anthropic"  # Can differ from main provider
  model: "claude-3-5-sonnet-20241022"
  min_score: 0.8
  max_tokens: 1000
  system_prompt: "You are an expert evaluator. Be fair but strict."

# Metrics collection
metrics:
  collect:
    - "response_time"
    - "tool_coverage"
    - "iteration_count"
    - "token_usage"
    - "cost_estimate"
    - "error_rate"
    - "path_efficiency"

# Reporting configuration
reporting:
  formats: ["json", "markdown", "html"]
  output_dir: "./test-reports"
  include_traces: true
  include_config: true
  timestamp_format: "%Y%m%d_%H%M%S"

# Test execution settings
execution:
  max_concurrency: 5
  timeout_seconds: 300
  retry_failed: true
  retry_count: 3
  retry_delay: 5
  parallel: true
  stop_on_first_failure: false
  verbose: false
  debug: false

# Logging configuration
logging:
  level: "INFO"  # DEBUG, INFO, WARNING, ERROR
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  file: "test-reports/mcp-eval.log"
  console: true
  show_mcp_messages: false  # Set true for debugging

# Cache configuration
cache:
  enabled: true
  ttl: 3600  # 1 hour
  directory: ".mcp-eval-cache"
  
# Development settings
development:
  mock_llm_responses: false
  save_llm_calls: true
  profile_performance: false
```

--------------------------------

### Create tests with mcp-eval-test-writer

Source: https://mcp-eval.ai/building-with-claude

Examples of defining test cases using decorator-based and Pytest-style asynchronous patterns.

```python
# Decorator style
@task("Fetch and validate")
async def test_fetch_validate(agent: Agent, session: Session):
    response = await agent.generate_str("Fetch example.com")
    await session.assert_that(
        Expect.tools.was_called("fetch"),
        response=response
    )
```

```python
# Pytest style
@pytest.mark.asyncio
async def test_fetch_with_error(mcp_agent, mcp_session):
    response = await mcp_agent.generate_str("Fetch invalid-url")
    await mcp_session.assert_that(
        Expect.content.contains("error"),
        response=response
    )
```

--------------------------------

### Display mcp-eval version

Source: https://mcp-eval.ai/cli-reference

Outputs the current installed version of the mcp-eval CLI.

```bash
mcp-eval version
```

--------------------------------

### Initialize test project

Source: https://mcp-eval.ai/quickstart

Runs the interactive wizard to set up the testing environment.

```bash
mcp-eval init
```

--------------------------------

### Expected Test Output

Source: https://mcp-eval.ai/quickstart

Example of the console output generated after running tests.

```text
Running tests...
✓ test_basic_fetch_decorator - Test basic URL fetching [2.3s]
  ✓ fetch_tool_called: Tool 'fetch' was called
  ✓ contains_domain_text: Content contains "Example Domain"
  ✓ fetch_success_rate: Tool success rate 100%

✓ test_content_extraction_decorator - Test extraction quality [3.1s]
  ✓ fetch_called_for_extraction: Tool 'fetch' was called
  ✓ extraction_quality_assessment: LLM judge score 0.92

Results: 2 passed, 0 failed
Reports saved to test-reports/
```

--------------------------------

### Configure Servers and Agents

Source: https://mcp-eval.ai/cli-reference

Commands to interactively add servers, import configurations, add agents, and list current settings.

```bash
# Add server interactively
mcp-eval server add

# Import from existing config
mcp-eval server add --from-mcp-json .cursor/mcp.json

# Add test agent
mcp-eval agent add

# List what's configured
mcp-eval server list -v
mcp-eval agent list
```

--------------------------------

### Initialize MCP Evaluation Project

Source: https://mcp-eval.ai/subagents/mcp-eval-config-expert

Run the initialization wizard to generate the main configuration and secrets files.

```bash
# Initialize with wizard
mcp-eval init

# This creates:
# - mcpeval.yaml (main config)
# - mcpeval.secrets.yaml (API keys)
```

--------------------------------

### Create test sessions

Source: https://mcp-eval.ai/api-session

Demonstrates both the recommended context manager approach and the manual lifecycle management for test sessions.

```python
# Using context manager (recommended)
async with test_session("test-name") as agent:
    # Your test code here
    pass

# Manual lifecycle (advanced)
session = TestSession(test_name="test-name")
agent = await session.__aenter__()
try:
    # Your test code
    ...
finally:
    await session.__aexit__(None, None, None)
    session.cleanup()
```

--------------------------------

### Add servers from a DXT manifest

Source: https://mcp-eval.ai/connect-servers

Import server configurations directly from a DXT manifest file.

```bash
mcp-eval server add --from-dxt ~/Desktop/manifest.dxt
```

--------------------------------

### Add MCP server interactively

Source: https://mcp-eval.ai/quickstart

Launches the interactive wizard to add an MCP server.

```bash
mcp-eval server add
```

--------------------------------

### Verify Server Configuration

Source: https://mcp-eval.ai/quickstart

Use these commands to list configured servers and validate connectivity.

```bash
# List all configured servers
mcp-eval server list

# Validate server connectivity
mcp-eval validate
```

--------------------------------

### Content Assertions

Source: https://mcp-eval.ai/assertions

Examples of validating response content using contains, not_contains, and regex patterns.

```python
resp = await agent.generate_str("Fetch https://example.com")
await session.assert_that(Expect.content.contains("Example Domain"), response=resp)
await session.assert_that(Expect.content.not_contains("Stack trace"), response=resp)
await session.assert_that(Expect.content.regex(r"Example\\s+Domain"), response=resp)
```

--------------------------------

### Integrate Secret Management Systems

Source: https://mcp-eval.ai/security

Examples for fetching secrets from various cloud-based secret management services.

```python
# integrations/aws_secrets.py
import boto3
import json

class AWSSecretManager:
    """Fetch secrets from AWS Secrets Manager."""
    
    def __init__(self, region='us-east-1'):
        self.client = boto3.client('secretsmanager', region_name=region)
    
    def get_api_key(self, secret_name):
        """Retrieve API key from AWS Secrets Manager."""
        try:
            response = self.client.get_secret_value(SecretId=secret_name)
            secret = json.loads(response['SecretString'])
            return secret['api_key']
        except Exception as e:
            # Log error without exposing secret
            print(f"Failed to retrieve secret: {type(e).__name__}")
            raise

# Use in configuration
from mcp_eval.config import set_settings

secret_mgr = AWSSecretManager()
anthropic_key = secret_mgr.get_api_key('mcp-eval/anthropic')

set_settings({
    'anthropic': {'api_key': anthropic_key}
})
```

```python
# integrations/vault_secrets.py
import hvac

class VaultSecretManager:
    """Fetch secrets from HashiCorp Vault."""
    
    def __init__(self, vault_url, token):
        self.client = hvac.Client(url=vault_url, token=token)
    
    def get_api_key(self, path):
        """Retrieve API key from Vault."""
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path
        )
        return response['data']['data']['api_key']

# Use in configuration
vault = VaultSecretManager(
    vault_url='https://vault.company.com',
    token=os.environ['VAULT_TOKEN']
)

anthropic_key = vault.get_api_key('secret/mcp-eval/anthropic')
```

```python
# integrations/azure_secrets.py
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

class AzureSecretManager:
    """Fetch secrets from Azure Key Vault."""
    
    def __init__(self, vault_url):
        credential = DefaultAzureCredential()
        self.client = SecretClient(
            vault_url=vault_url,
            credential=credential
        )
    
    def get_api_key(self, secret_name):
        """Retrieve API key from Azure Key Vault."""
        secret = self.client.get_secret(secret_name)
        return secret.value

# Use in configuration
azure_vault = AzureSecretManager(
    vault_url='https://myvault.vault.azure.net/'
)

anthropic_key = azure_vault.get_api_key('anthropic-api-key')
```

--------------------------------

### Test multi-step workflows with Python

Source: https://mcp-eval.ai/api-core

Demonstrates verifying a sequence of tool calls and content generation across multiple steps in a session.

```python
@with_agent("default")
@task("Test complete workflow")
async def test_workflow(agent, session):
    # Step 1: Authentication
    auth_response = await agent.generate_str("Authenticate as test_user")
    await session.assert_that(
        Expect.tools.was_called("auth"),
        name="authentication_attempted"
    )
    
    # Step 2: Fetch data
    data_response = await agent.generate_str("Get my profile data")
    await session.assert_that(
        Expect.tools.was_called("fetch_profile"),
        name="profile_fetched"
    )
    
    # Step 3: Process
    process_response = await agent.generate_str("Summarize my activity")
    await session.assert_that(
        Expect.content.contains("summary"),
        response=process_response,
        name="summary_generated"
    )
    
    # Verify the complete sequence
    await session.assert_that(
        Expect.tools.sequence(["auth", "fetch_profile", "summarize"]),
        name="correct_workflow_order"
    )
```

--------------------------------

### List MCP server capabilities

Source: https://mcp-eval.ai/common-workflows

Use this command to identify all tools provided by the server and their expected behaviors.

```bash
mcp-eval server list --verbose
```

--------------------------------

### Add servers from mcp.json

Source: https://mcp-eval.ai/connect-servers

Import server definitions from an existing mcp.json file.

```bash
mcp-eval server add --from-mcp-json .cursor/mcp.json
```

--------------------------------

### Configure Reusable Workflow for MCP-Eval

Source: https://mcp-eval.ai/ci-cd

Use the reusable workflow to handle setup, testing, and deployment in a GitHub Actions pipeline.

```yaml
name: MCP-Eval CI

on:
  push:
    branches: [main, master, trunk]
  workflow_dispatch:

jobs:
  call-mcpeval:
    uses: lastmile-ai/mcp-eval/.github/workflows/mcpeval-reusable.yml
    with:
      deploy-pages: true
    permissions:
      contents: read
      pages: write
      id-token: write
      pull-requests: write
    secrets: inherit
```

--------------------------------

### Import MCP Server Configurations

Source: https://mcp-eval.ai/subagents/mcp-eval-config-expert

Import server definitions from existing MCP JSON files, DXT manifests, or add them interactively.

```bash
# From Cursor/VS Code MCP config
mcp-eval server add --from-mcp-json .cursor/mcp.json

# From DXT manifest
mcp-eval server add --from-dxt ~/Desktop/my-server.dxt

# Add server interactively
mcp-eval server add
```

--------------------------------

### Define generated test file structure

Source: https://mcp-eval.ai/subagents/test-code-emitter

Standard template for generated test files including imports, setup, and execution blocks.

```python
"""Generated tests for {server_name} server.

Generated by mcp-eval on {timestamp}.
"""

# Imports (minimal, only what's used)
{imports}

# Setup/teardown if needed
@setup
def configure():
    """Test setup."""
    pass

# Test functions/classes
{test_functions}

# Optional: main block for direct execution
if __name__ == "__main__":
    # For decorator style
    import asyncio
    from mcp_eval.runner import run_tests
    asyncio.run(run_tests(__file__))
    
    # For pytest style
    # pytest.main([__file__, "-v"])
```

--------------------------------

### Validate test environment configuration

Source: https://mcp-eval.ai/api-core

Perform environment checks within a setup function to ensure prerequisites are met before tests run.

```python
@setup
def validate_environment():
    """Ensure test environment is properly configured."""
    import sys
    
    # Check Python version
    if sys.version_info < (3, 10):
        raise RuntimeError("Tests require Python 3.10+")
    
    # Check required environment variables
    required_vars = ["ANTHROPIC_API_KEY", "TEST_SERVER_URL"]
    missing = [var for var in required_vars if not os.getenv(var)]
    
    if missing:
        raise RuntimeError(f"Missing environment variables: {missing}")
    
    # Check MCP servers are accessible
    from mcp_eval.utils import check_server_health
    if not check_server_health("my_server"):
        raise RuntimeError("MCP server 'my_server' is not responding")
    
    print("✅ Environment validated successfully")
```

--------------------------------

### Define and execute an agent test scenario

Source: https://mcp-eval.ai/overview

Demonstrates setting up an agent with specific MCP servers and defining a test task with various assertions for tool usage, content quality, and performance.

```python
from mcp_eval import task, setup, Expect
from mcp_eval.evaluators import EvaluationCriterion

@setup
def configure():
    # Define the agent and servers it can use
    mcp_eval.use_agent(
        AgentSpec(
            name="weather_assistant",
            instruction="You help users with weather information.",
            server_names=["weather_api", "location_service"],
            model="claude-3-5-haiku-20241022"
        )
    )

@task("Get weather for user's location")
async def test_weather_flow(agent, session):
    # Execute the task
    response = await agent.generate_str(
        "What's the weather like in San Francisco tomorrow?"
    )
    
    # Verify tool usage - immediate checks
    await session.assert_that(
        Expect.tools.was_called("get_location"),
        name="location_service_called"
    )
    await session.assert_that(
        Expect.tools.was_called("get_weather_forecast"),
        name="weather_api_called"
    )
    
    # Check content quality
    await session.assert_that(
        Expect.content.contains("San Francisco"),
        response=response,
        name="mentions_city"
    )
    await session.assert_that(
        Expect.content.regex(r"\d+\s*°[CF]"),  # Temperature pattern
        response=response,
        name="includes_temperature"
    )
    
    # Verify execution efficiency
    await session.assert_that(
        Expect.tools.sequence(
            ["get_location", "get_weather_forecast"],
            allow_other_calls=False
        ),
        name="optimal_tool_sequence"
    )
    
    # Apply quality judge
    await session.assert_that(
        Expect.judge.multi_criteria([
            EvaluationCriterion(
                name="accuracy",
                description="Weather info is plausible and specific",
                weight=2.0,
                min_score=0.8
            ),
            EvaluationCriterion(
                name="helpfulness",
                description="Response is clear and actionable",
                weight=1.5,
                min_score=0.7
            )
        ]),
        response=response,
        name="quality_assessment"
    )
    
    # Performance requirements
    await session.assert_that(
        Expect.performance.response_time_under(10000),  # 10 seconds
        name="response_time_check"
    )
```

--------------------------------

### TestSession.create()

Source: https://mcp-eval.ai/api-session

Initializes a new test session with configurable output directories and artifact saving.

```APIDOC
## TestSession.create()

### Description
Creates a new test session instance.

### Parameters
- **test_name** (string) - Required - The name of the test.
- **output_dir** (string) - Optional - The directory path where artifacts will be saved.
- **save_artifacts** (boolean) - Optional - Whether to enable automatic artifact saving.
```

--------------------------------

### Select Cheaper Models

Source: https://mcp-eval.ai/troubleshooting

Switch to a more cost-effective model provider for basic testing.

```yaml
# For basic tests
provider: "anthropic"
model: "claude-3-haiku-20240307"
```

--------------------------------

### Enable configuration hot-reloading in Python

Source: https://mcp-eval.ai/api-config

Uses the watchdog library to monitor configuration files and trigger a reload when changes are detected. Requires the watchdog package installed.

```python
import watchdog.observers
import watchdog.events

class ConfigReloader(watchdog.events.FileSystemEventHandler):
    """Reload config when files change."""
    
    def on_modified(self, event):
        if event.src_path.endswith("mcpeval.yaml"):
            print("Config changed, reloading...")
            try:
                new_config = load_config()
                use_config(new_config)
                print("✅ Config reloaded successfully")
            except Exception as e:
                print(f"❌ Failed to reload: {e}")

# Watch for changes
observer = watchdog.observers.Observer()
observer.schedule(ConfigReloader(), ".", recursive=False)
observer.start()
```

--------------------------------

### Profile performance with CLI

Source: https://mcp-eval.ai/troubleshooting

Generate a machine-readable profile report and inspect it using jq.

```bash
# Save a machine-readable report and analyze offline
mcp-eval run tests/ --json profile.json

# Analyze the report (custom scripts)
cat profile.json | jq '.' | less
```

--------------------------------

### Discover Configuration Files

Source: https://mcp-eval.ai/subagents/mcp-eval-config-expert

Use built-in discovery utilities to locate configuration files and define custom search paths with precedence.

```python
from mcp_eval.config import find_eval_config, find_eval_secrets

# Find config files
config_path = find_eval_config()  # Searches up directory tree
secrets_path = find_eval_secrets()

# Load with precedence
from pathlib import Path
config_locations = [
    Path.cwd() / "mcpeval.yaml",
    Path.cwd() / ".mcp-eval" / "config.yaml",
    Path.home() / ".mcp-eval" / "config.yaml"
]
```

--------------------------------

### Initialize Expect namespace and common assertions

Source: https://mcp-eval.ai/api-catalog

Demonstrates the available assertion factories for content, tools, performance, LLM judgment, and path efficiency.

```python
from mcp_eval.catalog import Expect

# Content assertions
Expect.content.contains("success")
Expect.content.regex(r"\d+ results?")

# Tool assertions
Expect.tools.was_called("fetch")
Expect.tools.success_rate(min_rate=0.95)
Expect.tools.sequence(["validate", "process", "format"])

# Performance assertions
Expect.performance.max_iterations(3)
Expect.performance.response_time_under(5000)

# LLM judge assertions
Expect.judge.llm("Must be professional and accurate")
Expect.judge.multi_criteria([criterion1, criterion2])

# Path efficiency assertions
Expect.path.efficiency(
    expected_tool_sequence=["fetch", "parse"],
    allow_extra_steps=1
)
```

--------------------------------

### Configure reporting via environment variables

Source: https://mcp-eval.ai/api-config

Define output directories, report formats, and trace inclusion using environment variables.

```bash
# Output configuration
export MCPEVAL_OUTPUT_DIR="/tmp/test-reports"
export MCPEVAL_REPORT_FORMATS="json,html,markdown"
export MCPEVAL_INCLUDE_TRACES="true"
```

--------------------------------

### Configure Provider API Keys

Source: https://mcp-eval.ai/subagents/mcp-eval-config-expert

Set required environment variables for Anthropic, OpenAI, Google, and Azure providers.

```bash
# Anthropic
export ANTHROPIC_API_KEY="sk-ant-..."
export ANTHROPIC_BASE_URL="https://api.anthropic.com"

# OpenAI
export OPENAI_API_KEY="sk-..."
export OPENAI_ORG_ID="org-..."

# Google
export GOOGLE_API_KEY="..."

# Azure
export AZURE_OPENAI_API_KEY="..."
export AZURE_OPENAI_ENDPOINT="https://....openai.azure.com"
```

--------------------------------

### Assert performance and tool behavior

Source: https://mcp-eval.ai/agent-evaluation

Demonstrates how to use the session object to assert performance bounds, tool call sequences, and output matching.

```python
# Efficiency and iteration bounds
await session.assert_that(Expect.performance.max_iterations(3))

# Tool behavior and outputs
await session.assert_that(Expect.tools.was_called("fetch"))
await session.assert_that(Expect.tools.output_matches("fetch", {"isError": False}, match_type="partial"))

# Path and sequence
await session.assert_that(Expect.tools.sequence(["fetch"], allow_other_calls=True))
await session.assert_that(Expect.path.efficiency(expected_tool_sequence=["fetch"], allow_extra_steps=1))
```

--------------------------------

### Common MCP Server Commands

Source: https://mcp-eval.ai/quickstart

Quick commands to launch popular MCP servers.

```bash
# Fetch server (web content)
uvx mcp-server-fetch
```

```bash
# Filesystem server
npx -y @modelcontextprotocol/server-filesystem /path/to/directory
```

```bash
# GitHub server
npx -y @modelcontextprotocol/server-github
```

--------------------------------

### Basic Assertion Entrypoint

Source: https://mcp-eval.ai/assertions

Demonstrates the standard syntax for invoking an assertion using the session object.

```python
await session.assert_that(Expect.content.contains("Example"), response=resp)
```

--------------------------------

### Add MCP server configuration

Source: https://mcp-eval.ai/cli-reference

Register an MCP server by importing from a JSON or DXT manifest file.

```bash
mcp-eval server add [OPTIONS]
```

--------------------------------

### Run tests with uv

Source: https://mcp-eval.ai/quickstart

Execute tests using the uv command-line tool.

```bash
uv run pytest -q tests
# or a single file/function
uv run pytest tests/test_my_server.py -v
uv run pytest tests/test_my_server.py::test_my_server -v
```

--------------------------------

### Import external server manifests

Source: https://mcp-eval.ai/configuration

Integrate existing server configurations from external files like mcp.json or DXT manifests.

```yaml
# Import from mcp.json (Cursor/VS Code)
mcp:
  import:
    - type: "mcp_json"
      path: ".cursor/mcp.json"
    
    # Import from DXT manifest
    - type: "dxt"
      path: "~/Desktop/my-manifest.dxt"
```

--------------------------------

### Configure provider settings via environment variables

Source: https://mcp-eval.ai/api-config

Set API keys and provider-specific parameters using shell environment variables.

```bash
# API keys
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."

# Provider selection
export MCPEVAL_PROVIDER="anthropic"
export MCPEVAL_MODEL="claude-3-5-sonnet-20241022"

# Provider-specific settings
export ANTHROPIC_BASE_URL="https://api.anthropic.com"
export OPENAI_ORG_ID="org-..."
```

--------------------------------

### Configure Windows paths and encoding

Source: https://mcp-eval.ai/troubleshooting

Handle path separators and character encoding issues in configuration files.

```yaml
# Use forward slashes or escaped backslashes
command: "python"
args: ["C:/path/to/server.py"]
# Or
args: ["C:\\path\\to\\server.py"]

# Set encoding
env:
  PYTHONIOENCODING: "utf-8"
```

--------------------------------

### Tool Discovery Commands

Source: https://mcp-eval.ai/subagents/mcp-eval-test-generator

Commands to list available tools and initiate generation using discovered tools.

```bash
# List available tools
mcp-eval server list --verbose

# Use discovered tools for generation
mcp-eval generate --discover-tools --style pytest
```

--------------------------------

### Initialize empty collections

Source: https://mcp-eval.ai/subagents/test-code-emitter

Correct syntax for initializing empty dictionaries and lists.

```python
# Empty dict/list
arguments = {}  # Not {"": ""}
sequence = []   # Not [""]
```

--------------------------------

### Configure Development Environment

Source: https://mcp-eval.ai/configuration

Extends the base configuration with debugging-friendly settings and mocked LLM responses.

```yaml
# mcpeval.dev.yaml
$extends: "./mcpeval.yaml"  # Inherit base config

provider: "anthropic"
model: "claude-3-haiku-20240307"  # Cheaper for dev

execution:
  max_concurrency: 1  # Easier debugging
  timeout_seconds: 600  # More time for debugging
  debug: true

development:
  mock_llm_responses: true  # Use mocked responses
  save_llm_calls: true
  profile_performance: true
  
logging:
  level: "DEBUG"
  show_mcp_messages: true
```

--------------------------------

### List MCP servers with mcp-eval server list

Source: https://mcp-eval.ai/cli-reference

Displays a list of configured MCP servers within the project directory.

```bash
mcp-eval server list [OPTIONS]
```

--------------------------------

### Use direct agent objects

Source: https://mcp-eval.ai/api-config

Instantiate and configure an Agent object directly in code, then register it for use.

```python
from mcp_eval.config import use_agent_object
from mcp_eval.agent import Agent

# Create and configure agent
my_agent = Agent(
    model="claude-3-5-sonnet-20241022",
    provider="anthropic",
    instruction="""You are a security-focused test agent.
    Always check for vulnerabilities and edge cases.""",
    server_names=["security_scanner", "filesystem"],
    temperature=0.0,  # Deterministic
    max_tokens=4000
)

# Use this specific agent
use_agent_object(my_agent)
```

--------------------------------

### Debug and Inspect Configuration

Source: https://mcp-eval.ai/api-config

Use these utilities to print or export the current configuration settings to a dictionary for inspection during test runs.

```python
from mcp_eval.config import get_settings, print_config

# Print current configuration
print_config()

# Or get as dict for inspection
settings = get_settings()
config_dict = settings.model_dump()

import json
print(json.dumps(config_dict, indent=2))

# Check specific values
print(f"Provider: {settings.provider}")
print(f"Model: {settings.model}")
print(f"Timeout: {settings.execution.timeout_seconds}s")
print(f"Output dir: {settings.reporting.output_dir}")
```

--------------------------------

### Create a test session in Python

Source: https://mcp-eval.ai/api-session

Initializes a test session and executes an agent interaction with an assertion.

```python
from mcp_eval.session import test_session
from mcp_eval.catalog import Expect

async with test_session("my-test") as agent:
    # Agent is ready with MCP servers connected
    response = await agent.generate_str("Fetch https://example.com")
    
    # Run assertions
    await agent.assert_that(
        Expect.content.contains("Example Domain"),
        response=response
    )
```

--------------------------------

### Project Directory Structure

Source: https://mcp-eval.ai/best-practices

Recommended layout for organizing unit, integration, end-to-end, and performance tests alongside shared fixtures.

```text
tests/
├── unit/                 # Fast, isolated tests
│   ├── test_calculator.py
│   ├── test_validator.py
│   └── test_parser.py
├── integration/          # Multi-component tests
│   ├── test_data_pipeline.py
│   ├── test_api_workflow.py
│   └── test_database_sync.py
├── e2e/                  # End-to-end scenarios
│   ├── test_user_journey.py
│   └── test_complete_workflow.py
├── performance/          # Performance tests
│   ├── test_response_time.py
│   └── test_throughput.py
├── fixtures/             # Shared test data
│   ├── sample_data.json
│   └── test_configs.yaml
└── conftest.py          # Shared fixtures and setup
```

--------------------------------

### Multi-environment directory structure

Source: https://mcp-eval.ai/configuration

Recommended file organization for managing shared, environment-specific, and secret configurations.

```bash
# Directory structure
configs/
  ├── base.yaml       # Shared configuration
  ├── dev.yaml        # Development overrides
  ├── staging.yaml    # Staging overrides
  ├── prod.yaml       # Production settings
  └── secrets.yaml    # API keys (gitignored)
```

--------------------------------

### Configure Development Mode

Source: https://mcp-eval.ai/building-with-claude

Set the search path to the local source directory for development environments.

```yaml
agents:
  enabled: true
  search_paths:
    - "./src/mcp_eval/data/subagents"  # See: https://github.com/lastmile-ai/mcp-eval/tree/main/src/mcp_eval/data/subagents
  pattern: "*.md"
```

--------------------------------

### Calculator MCP Server Test Suite

Source: https://mcp-eval.ai/examples

Demonstrates testing a calculator MCP server using decorator-style tasks for tool verification, parameterization, and error handling.

```python
"""Complete test suite for calculator MCP server using decorator style."""

from mcp_eval import task, setup, teardown, parametrize, Expect
from mcp_eval.session import TestAgent, TestSession

@setup
def configure_calculator_tests():
    """Setup before all calculator tests."""
    print("🧮 Starting calculator server tests")

@teardown
def cleanup_calculator_tests():
    """Cleanup after all tests."""
    print("✅ Calculator tests completed")

@task("Test basic addition")
async def test_addition(agent: TestAgent, session: TestSession):
    """Verify the calculator can perform addition correctly."""
    
    response = await agent.generate_str(
        "Use the calculator to add 15 and 27"
    )
    
    # Verify tool was called
    await session.assert_that(
        Expect.tools.was_called("calculate"),
        name="calculator_called"
    )
    
    # Check the calculation arguments
    await session.assert_that(
        Expect.tools.called_with(
            "calculate",
            {"operation": "add", "a": 15, "b": 27}
        ),
        name="correct_arguments"
    )
    
    # Verify the result
    await session.assert_that(
        Expect.content.contains("42"),
        name="correct_result",
        response=response
    )

@parametrize(
    "operation,a,b,expected",
    [
        ("add", 10, 5, "15"),
        ("subtract", 10, 3, "7"),
        ("multiply", 4, 7, "28"),
        ("divide", 20, 4, "5"),
    ]
)
@task("Test all operations")
async def test_operations(
    agent: TestAgent,
    session: TestSession,
    operation: str,
    a: int,
    b: int,
    expected: str
):
    """Test all calculator operations with various inputs."""
    
    response = await agent.generate_str(
        f"Use the calculator to {operation} {a} and {b}"
    )
    
    await session.assert_that(
        Expect.content.contains(expected),
        name=f"{operation}_result",
        response=response
    )
    
    await session.assert_that(
        Expect.tools.success_rate(min_rate=1.0, tool_name="calculate"),
        name=f"{operation}_success"
    )

@task("Test division by zero handling")
async def test_division_by_zero(agent: TestAgent, session: TestSession):
    """Verify graceful handling of division by zero."""
    
    response = await agent.generate_str(
        "Try to divide 10 by 0 using the calculator"
    )
    
    # Tool should be called but may fail
    await session.assert_that(
        Expect.tools.was_called("calculate"),
        name="attempted_division"
    )
    
    # Response should handle error gracefully
    await session.assert_that(
        Expect.judge.llm(
            rubric="Response acknowledges division by zero error and explains it clearly",
            min_score=0.8
        ),
        name="error_handling",
        response=response
    )
```

```python
"""Complete test suite for calculator MCP server using pytest."""

import pytest
from mcp_eval import Expect
```

--------------------------------

### Configure Local Server for Testing

Source: https://mcp-eval.ai/troubleshooting

Define local server commands and environment variables for testing purposes.

```yaml
# For localhost servers
mcp:
  servers:
    local_server:
      command: "python"
      args: ["server.py"]
      env:
        HOST: "127.0.0.1"
        PORT: "8080"
      startup_timeout: 10  # Wait for server to start
```

--------------------------------

### Structure an MCP test project

Source: https://mcp-eval.ai/subagents/mcp-eval-test-writer

Recommended directory layout for an MCP server test suite including configuration and report folders.

```text
my_mcp_server/
├── mcpeval.yaml          # Configuration
├── mcpeval.secrets.yaml  # API keys (gitignored)
├── tests/
│   ├── __init__.py
│   ├── test_basic.py     # Basic functionality
│   ├── test_errors.py    # Error handling
│   ├── test_performance.py # Performance tests
│   ├── test_integration.py # Multi-tool workflows
│   └── test_datasets.py  # Dataset-driven tests
└── test-reports/         # Generated reports
```

--------------------------------

### Generate Test Scenarios

Source: https://mcp-eval.ai/cli-reference

Basic command structure for generating test files with configurable options.

```bash
mcp-eval generate [OPTIONS]
```

--------------------------------

### Configure basic MCP servers

Source: https://mcp-eval.ai/configuration

Define standard server types including Python scripts, Node.js processes, pre-built packages, and Docker containers.

```yaml
# In mcp-agent.config.yaml or mcpeval.yaml
mcp:
  servers:
    # Simple Python server
    calculator:
      command: "python"
      args: ["servers/calculator.py"]
      env:
        LOG_LEVEL: "DEBUG"
    
    # Node.js server with npm
    weather:
      command: "npm"
      args: ["run", "start:weather"]
      cwd: "./servers/weather"
    
    # Pre-built server from package
    fetch:
      command: "uvx"
      args: ["mcp-server-fetch"]
      env:
        UV_NO_PROGRESS: "1"
    
    # Docker container server
    database:
      command: "docker"
      args: ["run", "--rm", "-i", "my-mcp-server:latest"]
      startup_timeout: 30  # Wait for container to start
```

--------------------------------

### Execute Comprehensive Workflow Assertions

Source: https://mcp-eval.ai/api-catalog

Demonstrates combining content, tool usage, performance, and LLM-based quality assertions within a single task.

```python
@task("Complete workflow test")
async def test_full_workflow(agent, session):
    """Test the entire user journey with multiple checkpoints."""
    
    response = await agent.generate_str(
        "Fetch weather for NYC and format it nicely"
    )
    
    # Content quality
    await session.assert_that(
        Expect.content.regex(r"\d+°[CF]"),
        response=response,
        name="has_temperature"
    )
    
    # Tool usage
    await session.assert_that(
        Expect.tools.sequence(["weather_api", "format_tool"]),
        name="correct_tools"
    )
    
    # Performance
    await session.assert_that(
        Expect.performance.response_time_under(3000),
        name="fast_enough"
    )
    
    # Quality judgment
    await session.assert_that(
        Expect.judge.llm(
            "Output should be well-formatted and user-friendly",
            min_score=0.8
        ),
        response=response,
        name="quality_output"
    )
```
Tokens:
Raw