### Quick Start Example: Running a Simple Test

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/README.md

A step-by-step guide to running a basic test from scratch. This includes cloning the example, setting the API key, and running a test using 'uv run', which automatically handles dependencies.

```bash
# 1. Clone and navigate to this example
cd examples/mcp_server_fetch

# 2. Set your API key
export ANTHROPIC_API_KEY="your-key-here"

# 3. Run a simple test (uv will handle dependencies automatically)
uv run mcp-eval run tests/test_decorator_style.py

# Expected output:
# ✓ test_basic_fetch_decorator [4.123s]
#   └─ Test basic URL fetching with decorator style
#      ├─ fetch_tool_called: ✓ Tool 'fetch' was called at least 1 time(s)
#      ├─ contains_domain_text: ✓ Response contains 'Example Domain'
#      └─ fetch_success_rate: ✓ Tool success rate is at least 100.0%
```

--------------------------------

### Minimal mcpeval.yaml Configuration Example

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

A minimal `mcpeval.yaml` configuration for quick setup. It specifies the LLM provider, model, and basic MCP server details. This is useful for getting started with mcp-eval without extensive configuration.

```yaml
# mcpeval.yaml (minimal)
provider: "anthropic"
model: "claude-3-haiku-20240307"

mcp:
  servers:
    my_server:
      command: "python"
      args: ["server.py"]

```

--------------------------------

### MCP Server Examples

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/quickstart.mdx

Examples of common MCP servers that can be tested, including fetch, filesystem, and GitHub servers. These commands are used to start different types of MCP servers for evaluation.

```bash
# Fetch server (web content)
uvx mcp-server-fetch
```

```bash
# Filesystem server
npx -y @modelcontextprotocol/server-filesystem /path/to/directory
```

```bash
# GitHub server
npx -y @modelcontextprotocol/server-github
```

--------------------------------

### Simple Synchronous Setup and Teardown (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/api-core.mdx

Implement simple setup and teardown logic using the `@setup` and `@teardown` decorators. These functions run once before and after all tests, respectively. The example demonstrates creating and cleaning up a temporary directory and its contents.

```python
from mcp_eval.core import setup, teardown
import os
import tempfile

test_dir = None

@setup
def prepare_test_environment():
    """Create temporary test directory."""
    global test_dir
    test_dir = tempfile.mkdtemp(prefix="mcp_test_")
    print(f"🚀 Created test directory: {test_dir}")
    
    # Set up test files
    with open(f"{test_dir}/test.txt", "w") as f:
        f.write("Test content")

@teardown
def cleanup_test_environment():
    """Clean up after tests."""
    global test_dir
    if test_dir and os.path.exists(test_dir):
        import shutil
        shutil.rmtree(test_dir)
        print(f"🧹 Cleaned up {test_dir}")
```

--------------------------------

### Run MCP-Eval Example Tests

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/data/sample/README.md

Shows the command to execute the example test file using the MCP-Eval framework after dependencies have been installed.

```bash
mcp_eval run usage_example.py
```

--------------------------------

### Complete mcpeval.yaml Configuration Example

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

A comprehensive example of the `mcpeval.yaml` file, demonstrating settings for metadata, LLM providers, agents, judges, metrics, reporting, execution, logging, and caching. This file provides a full template for setting up the mcp-eval testing environment.

```yaml
# mcpeval.yaml
$schema: ./schema/mcpeval.config.schema.json

# Metadata
name: "My MCP Test Suite"
description: "Comprehensive testing for our MCP servers"

# Default LLM provider settings
provider: "anthropic"
model: "claude-3-5-sonnet-20241022"

# Default agent for tests
default_agent:
  name: "test_agent"
  instruction: "You are a helpful testing assistant. Be precise and thorough."
  server_names: ["calculator", "weather"]
  
# Judge configuration
judge:
  provider: "anthropic"  # Can differ from main provider
  model: "claude-3-5-sonnet-20241022"
  min_score: 0.8
  max_tokens: 1000
  system_prompt: "You are an expert evaluator. Be fair but strict."

# Metrics collection
metrics:
  collect:
    - "response_time"
    - "tool_coverage"
    - "iteration_count"
    - "token_usage"
    - "cost_estimate"
    - "error_rate"
    - "path_efficiency"

# Reporting configuration
reporting:
  formats: ["json", "markdown", "html"]
  output_dir: "./test-reports"
  include_traces: true
  include_config: true
  timestamp_format: "%Y%m%d_%H%M%S"

# Test execution settings
execution:
  max_concurrency: 5
  timeout_seconds: 300
  retry_failed: true
  retry_count: 3
  retry_delay: 5
  parallel: true
  stop_on_first_failure: false
  verbose: false
  debug: false

# Logging configuration
logging:
  level: "INFO"  # DEBUG, INFO, WARNING, ERROR
  format: "% (asctime)s - %(name)s - %(levelname)s - %(message)s"
  file: "test-reports/mcp-eval.log"
  console: true
  show_mcp_messages: false  # Set true for debugging

# Cache configuration
cache:
  enabled: true
  ttl: 3600  # 1 hour
  directory: ".mcp-eval-cache"
  
# Development settings
development:
  mock_llm_responses: false
  save_llm_calls: true
  profile_performance: false

```

--------------------------------

### MCP-Agent Configuration Example (YAML)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

Provides an example configuration file (mcp-agent.config.yaml) for MCP-Eval, detailing server setups, OpenTelemetry (otel) export settings, and provider configurations for LLMs.

```yaml
# mcp-agent.config.yaml
mcp:
  servers:
    fetch:
      command: "uvx"
      args: ["mcp-server-fetch"]
      env: { UV_NO_PROGRESS: "1" }
    my_server:
      command: "python"
      args: ["path/to/my_server.py"]

otel:
  enabled: true
  exporters: ["file"]

# Provider sections (e.g., Anthropic/OpenAI) are read from env/this file

```

--------------------------------

### MCP Eval Test File Structure (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/quickstart.mdx

Illustrates the structure of a typical test file for MCP Eval, including setup functions and multiple task definitions. This example shows how to define setup logic using `@setup` and create tests for basic operations and error handling using `@task`.

```python
"""Tests for my awesome MCP server."""

from mcp_eval import task, setup, Expect

@setup
def configure_tests():
    """Any setup needed before tests run."""
    print("🚀 Starting my server tests!")

@task("Test basic functionality")
async def test_basic_operation(agent, session):
    """Verify the server responds correctly to basic requests."""
    
    # 1. Send a prompt to the agent
    response = await agent.generate_str(
        "Please use the calculator to add 2 + 2"
    )
    
    # 2. Check that the right tool was called
    await session.assert_that(
        Expect.tools.was_called("calculate"),
        name="calculator_used"
    )
    
    # 3. Verify the response content
    await session.assert_that(
        Expect.content.contains("4"),
        name="correct_answer",
        response=response
    )
    
    # 4. Check efficiency (optional)
    await session.assert_that(
        Expect.performance.max_iterations(3),
        name="completed_efficiently"
    )

@task("Test error handling")
async def test_error_recovery(agent, session):
    """Verify graceful error handling."""
    
    response = await agent.generate_str(
        "Try to divide by zero, then recover"
    )
    
    # Use LLM judge for complex behavior
    await session.assert_that(
        Expect.judge.llm(
            rubric="Agent should handle error gracefully and provide helpful response",
            min_score=0.8
        ),
        name="error_handling_quality",
        response=response
    )
```

--------------------------------

### Asynchronous Setup and Teardown (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/api-core.mdx

Implement asynchronous setup and teardown logic using the `@setup` and `@teardown` decorators with `async def` functions. This is suitable for operations that involve I/O, such as database connections or asynchronous API calls. The example shows connecting to a database, seeding data, and cleaning up.

```python
from mcp_eval.core import setup, teardown
# Assuming 'db' is an async database client object

@setup
async def async_prepare():
    """Setup that requires async operations."""
    # Connect to database
    await db.connect()
    
    # Seed test data
    await db.execute("INSERT INTO test_table ...")
    
    print("✅ Database ready")

@teardown
async def async_cleanup():
    """Async cleanup operations."""
    await db.execute("DELETE FROM test_table WHERE ...")
    await db.disconnect()
```

--------------------------------

### MCP-Eval Quick Start: Fetch Example Test

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

A decorator-style test using MCP-Eval's unified assertion API to test an agent's ability to fetch a URL and summarize its content. It includes setup for the agent and assertions for content, tool usage, and success rate.

```python
# tests/test_fetch.py
import mcp_eval
from mcp_eval import task, setup
from mcp_eval import Expect
from mcp_agent.agents.agent import Agent

@setup
def configure():
    # Define servers on the Agent itself (preferred)
    from mcp_agent.agents.agent_spec import AgentSpec
    mcp_eval.use_agent(
        AgentSpec(
            name="fetch_tester",
            instruction="You can fetch URLs and summarize them.",
            server_names=["fetch"],
            # Optionally set per-agent LLM
            # provider="anthropic",
            # model="claude-3-5-haiku-20241022",
        )
    )

@task("Fetch example.com and verify")
async def test_fetch_example(agent, session):
    response = await agent.generate_str("Fetch https://example.com and summarize in one sentence")

    # Immediate content assertion (providing response) + deferred tool usage checks
    await session.assert_that(Expect.content.contains("Example Domain"), name="contains_example_domain", response=response)
    await session.assert_that(Expect.tools.was_called("fetch"), name="fetch_called")
    await session.assert_that(Expect.tools.success_rate(min_rate=1.0, tool_name="fetch"), name="fetch_success_rate")

```

--------------------------------

### Install and Initialize mcp-eval using uv

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/index.mdx

Installs the mcp-eval CLI tool globally using uv, adds it as a project dependency, and initializes a new mcp-eval project. This is the recommended setup for managing dependencies and CLI tools.

```bash
# Install mcp-eval globally (for CLI)
uv tool install mcpevals

# Add mcp-eval dependency to your project
uv add mcpevals

# Initialize your project (interactive setup)
mcp-eval init

# Add your MCP server to test
mcp-eval server add

# Auto-generate tests with an LLM
mcp-eval generate

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
uv run pytest -q tests
```

--------------------------------

### MCP Eval Server Configuration (YAML)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/quickstart.mdx

Shows an example of how to configure MCP servers in the `mcpeval.yaml` file. This configuration specifies the command and arguments needed to start the server, ensuring it's correctly recognized by MCP Eval.

```yaml
mcp:
  servers:
    my_server:
      command: "python"
      args: ["path/to/server.py"]
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/data/sample/README.md

Demonstrates how to install project dependencies using the 'uv' package manager. It assumes you are in the sample directory and have a requirements.txt file.

```bash
cd sample/
uv pip install -r requirements.txt
```

--------------------------------

### Run MCP Eval Examples from Command Line

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/examples.mdx

Provides command-line instructions for running MCP Eval examples. It covers running all examples in a directory, running pytest-style tests, and running specific Python test files.

```bash
# Run all examples (decorator/dataset)
mcp-eval run examples/

# Run pytest examples
uv run pytest -q examples/mcp_server_fetch/tests/test_pytest_style.py

# Run specific test file (decorators)
mcp-eval run examples/test_calculator.py
```

--------------------------------

### Basic Programmatic Setup for MCP-Eval (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

Demonstrates how to configure MCP-Eval using Python code. It shows two methods: setting global settings via a dictionary or `MCPEvalSettings` object, and configuring a specific agent using `use_agent`.

```python
from mcp_eval.config import set_settings, MCPEvalSettings, use_agent
from mcp_agent.agents.agent import Agent

# Configure via dictionary
set_settings({
    "provider": "anthropic",
    "model": "claude-3-5-sonnet-20241022",
    "reporting": {
        "output_dir": "./my-reports",
        "formats": ["html", "json"]
    },
    "execution": {
        "timeout_seconds": 120,
        "max_concurrency": 3
    }
})

# Or use typed settings
settings = MCPEvalSettings(
    provider="anthropic",
    model="claude-3-haiku-20240307",
    judge={"min_score": 0.85},
    reporting={"output_dir": "./test-output"}
)
set_settings(settings)

# Configure agent
agent = Agent(
    name="my_test_agent",
    instruction="Test thoroughly",
    server_names=["my_server"]
)
use_agent(agent)
```

--------------------------------

### File Discovery Structure Example

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

Illustrates the directory structure mcp-eval searches for configuration files. It includes the current directory, parent directories, and the user's home directory, showing the hierarchy for `mcpeval.yaml`, `mcpeval.secrets.yaml`, and related `mcp-agent` files.

```text
Current directory:
  ├── mcpeval.yaml
  ├── mcpeval.secrets.yaml
  ├── mcp-agent.config.yaml
  ├── mcp-agent.secrets.yaml
  └── .mcp-eval/
      ├── config.yaml
      └── secrets.yaml

Parent directories (recursive):
  └── (same structure)

Home directory:
  └── ~/.mcp-eval/
      ├── config.yaml
      └── secrets.yaml

```

--------------------------------

### Initialize MCP-Eval Configuration

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-config-expert.mdx

Start the MCP-Eval setup process using the `init` command. This command typically creates essential configuration files like `mcpeval.yaml` and `mcpeval.secrets.yaml`.

```bash
# Initialize with wizard
mcp-eval init

# This creates:
# - mcpeval.yaml (main config)
# - mcpeval.secrets.yaml (API keys)
```

--------------------------------

### Run Usage Example with mcp-eval

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/sample_server/README.md

Executes a usage example that employs dataset-style helpers and assertions with mcp-eval. This command runs the tests defined in `usage_example.py`.

```bash
uv run mcp-eval run usage_example.py
```

--------------------------------

### mcp-eval Multi-Environment Directory Structure (Bash)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

This bash example illustrates a common directory structure for managing mcp-eval configurations across multiple environments. It separates base settings from environment-specific overrides and sensitive secrets.

```bash
# Directory structure
configs/
  ├── base.yaml       # Shared configuration
  ├── dev.yaml        # Development overrides
  ├── staging.yaml    # Staging overrides
  ├── prod.yaml       # Production settings
  └── secrets.yaml    # API keys (gitignored)
```

--------------------------------

### Python: Test lifecycle setup and teardown

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/README.md

Shows how to define setup and teardown functions for tests using the `@setup` and `@teardown` decorators in Python. These functions are executed before and after all tests, respectively, and are useful for initializing and cleaning up test resources.

```python
from mcp_eval import setup, teardown, task

@setup
def prepare():
    """Run before all tests"""
    initialize_test_data()

@teardown  
def cleanup():
    """Run after all tests"""
    cleanup_test_data()

@task("Test with lifecycle")
async def test_something(agent, session):
    # Test implementation
    pass
```

--------------------------------

### Setup and Teardown: Test Lifecycle Management

Source: https://context7.com/lastmile-ai/mcp-eval/llms.txt

Defines setup and teardown functions that execute before and after all tests, respectively. These are useful for initializing and cleaning up test environments, such as creating temporary directories and files.

```python
from mcp_eval import setup, teardown, task
import tempfile
import shutil
from pathlib import Path

test_dir = None

@setup
def prepare_test_environment():
    """Create temporary test directory before tests run."""
    global test_dir
    test_dir = tempfile.mkdtemp(prefix="mcp_test_")
    print(f"Created test directory: {test_dir}")

    # Create test files
    Path(f"{test_dir}/test.txt").write_text("Test content")

@teardown
def cleanup_test_environment():
    """Clean up after all tests complete."""
    global test_dir
    if test_dir:
        shutil.rmtree(test_dir, ignore_errors=True)
        print(f"Cleaned up {test_dir}")

@task("Test with lifecycle")
async def test_with_lifecycle(agent, session):
    response = await agent.generate_str(f"Read file from {test_dir}")
    await session.assert_that(
        Expect.content.contains("content"),
        response=response
    )
```

--------------------------------

### Configure and Test Agents in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-debugger.mdx

Demonstrates how to configure and test different agent setups using the `mcp_eval.config.use_agent` function. It shows examples of using a minimal agent configuration and a more verbose agent designed for debugging.

```python
# Test different agent configs
from mcp_eval.config import use_agent
from mcp_agent.agents.agent import Agent

# Try minimal agent
minimal_agent = Agent(
    name="debug_agent",
    instruction="Simple test agent",
    server_names=["fetch"]
)
use_agent(minimal_agent)

# Test with verbose agent
verbose_agent = Agent(
    name="verbose_agent",
    instruction="Debug agent. Print all tool calls and responses.",
    server_names=["fetch"]
)
use_agent(verbose_agent)
```

--------------------------------

### Install Project Dependencies with pip

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/README.md

Installs the project dependencies in editable mode using pip. This is an alternative to using uv for dependency management.

```bash
pip install -e .
```

--------------------------------

### Install uv Package Manager

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/README.md

Installs the uv package manager using a curl script. uv is recommended for managing dependencies automatically during test execution.

```bash
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Configure and Debug mcp-eval Server Connections

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/troubleshooting.mdx

Troubleshoot 'MCP server not found' or 'failed to start' errors by verifying server configurations, testing server commands manually, and enabling verbose output for debugging.

```yaml
# mcpeval.yaml or mcp-agent.config.yaml
mcp:
  servers:
    my_server:
      command: "python"  # Ensure command exists
      args: ["path/to/server.py"]  # Check path is correct
      env:
        PYTHONPATH: "."  # Add if needed
```

```bash
# Run the server command directly
python path/to/server.py

# Check for errors or missing dependencies
```

```bash
mcp-eval run tests/ -vv
```

```bash
pip install -r requirements.txt
```

```bash
chmod +x server.py
```

--------------------------------

### Install uv Package Manager

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/data/sample/README.md

Instructions for installing the 'uv' package manager, a tool for dependency isolation. It provides commands for macOS/Linux using curl and a pip-based alternative.

```bash
# On macOS and Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or using pip
pip install uv
```

--------------------------------

### Install MCP-Eval CLI

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/quickstart.mdx

Installs the mcp-eval command-line interface globally using uv. This makes the mcp-eval CLI available system-wide for managing MCP server tests.

```bash
uv tool install mcpevals
```

--------------------------------

### Python: Add Setup Function for Tests

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-generator.mdx

Defines a setup function using the '@setup' decorator in Python to prepare test data before test execution. This is part of customizing generated tests.

```python
@setup
def prepare_test_data():
    """Add test data preparation"""
    create_test_files()
```

--------------------------------

### Install and Initialize mcp-eval using pip

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/index.mdx

Installs the mcp-eval package using pip and initializes a new mcp-eval project. This method is suitable for projects already using pip for dependency management.

```bash
# Install mcp-eval
pip install mcpevals

# Initialize your project
mcp-eval init

# Add your MCP server
mcp-eval server add

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
pytest -q tests
```

--------------------------------

### Calculator Server Scenario Example (JSON)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/test-scenario-designer.mdx

An example JSON test scenario for a Calculator Server. It focuses on handling division by zero, verifying that an error is detected and gracefully explained by the LLM.

```json
{
  "name": "handle_division_by_zero",
  "prompt": "Calculate 10 divided by 0 and explain the result",
  "assertions": [
    {"kind": "tool_was_called", "tool_name": "divide"},
    {"kind": "response_contains", "text": "error", "case_sensitive": false},
    {"kind": "llm_judge", "rubric": "Handles error gracefully", "min_score": 0.9}
  ]
}
```

--------------------------------

### Validate MCP-Eval Setup

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/cli-reference.mdx

Commands to validate the current MCP-Eval project setup. `validate` performs a full validation including connections, while `validate --quick` checks only the configuration. `doctor --full` can diagnose deeper issues.

```bash
# Full validation with connections
mcp-eval validate

# Quick config check only
mcp-eval validate --quick

# Diagnose issues
mcp-eval doctor --full
```

--------------------------------

### Install and Use mcp-eval CLI

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/README.md

Installs mcp-eval globally using uv or pip, initializes a project, adds an MCP server for testing, generates tests, and runs them. Requires Python 3.10+.

```bash
# Install mcp-eval globally (for CLI)
uv tool install mcpevals

# Add mcp-eval dependency to your project
uv add mcpevals

# Initialize your project (interactive setup)
mcp-eval init

# Add your MCP server to test
mcp-eval server add

# Auto-generate tests with an LLM
mcp-eval generate

# Run tests
mcp-eval run tests/

# Alternatively with pip:
# Install mcp-eval
pip install mcpevals

# Initialize your project
mcp-eval init

# Add your MCP server
mcp-eval server add

# Run tests
mcp-eval run tests/
```

--------------------------------

### Python Decorators for Test Setup and Teardown

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/data/subagents/mcp-eval-test-generator.md

Python code demonstrating the use of custom decorators `@setup` and `@teardown` for preparing test data before tests run and cleaning up after they complete. This enhances test isolation and reproducibility.

```python
@setup
def prepare_test_data():
    """Add test data preparation"""
    create_test_files()
    
@teardown
def cleanup_test_data():
    """Clean up after tests"""
    remove_test_files()
```

--------------------------------

### Validate MCP-Eval Configuration and Setup

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/troubleshooting.mdx

Run the `mcp-eval validate` command to check the overall configuration and setup of MCP-Eval. This command verifies that all components are correctly installed and configured, ensuring the environment is ready for testing.

```bash
# Validate everything
mcp-eval validate
```

--------------------------------

### Quick Start Evaluation

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/sample_server/README.md

A concise sequence of commands to quickly set up the environment, configure the API key, and run a basic evaluation using mcp-eval.

```bash
# 1) Navigate to this example
cd examples/sample_server

# 2) Set your API key
export ANTHROPIC_API_KEY="your-key-here"

# 3) Run a simple evaluation
uv run mcp-eval run sample_server.eval.py
```

--------------------------------

### Install MCP-Eval using pip or uv

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

Installs the MCP-Eval package in development mode. Requires Python 3.10+. Use pip or uv for installation.

```bash
pip install -e .
# or using uv
uv pip install -e .
```

--------------------------------

### Run Sample MCP Server

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/sample_server/README.md

Starts the toy MCP server using the uv run command. This server provides two tools: `get_current_time` and `summarize_text`.

```bash
uv run python sample_server.py
```

--------------------------------

### Example MCP-Eval Test Suite Structure

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-writer.mdx

This example shows a typical directory structure for an MCP-Eval test suite. It includes configuration files, test directories for different test types, and a directory for generated reports.

```directory
my_mcp_server/
├── mcpeval.yaml          # Configuration
├── mcpeval.secrets.yaml  # API keys (gitignored)
├── tests/
│   ├── __init__.py
│   ├── test_basic.py     # Basic functionality
│   ├── test_errors.py    # Error handling
│   ├── test_performance.py # Performance tests
│   ├── test_integration.py # Multi-tool workflows
│   └── test_datasets.py  # Dataset-driven tests
└── test-reports/         # Generated reports
```

--------------------------------

### Decorator Style Test in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-writer.mdx

Demonstrates the simplest test style using decorators for setup and tasks. It requires importing necessary components from 'mcp_eval' and defines setup and task functions with assertions for tool usage and content.

```python
from mcp_eval import task, setup, teardown, Expect
from mcp_eval.session import TestAgent, TestSession

@setup
def configure():
    """Setup runs before tests"""
    print("Starting tests")

@task("Test basic functionality")
async def test_basic(agent: TestAgent, session: TestSession):
    response = await agent.generate_str("Your prompt here")
    
    await session.assert_that(
        Expect.tools.was_called("tool_name"),
        name="tool_was_used"
    )
    
    await session.assert_that(
        Expect.content.contains("expected text"),
        response=response,
        name="has_expected_content"
    )
```

--------------------------------

### MCP-Eval: Generate Initial Tests

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-generator.mdx

Generates an initial set of tests with a specified number of examples and outputs them to a Python file. This is the starting point for creating a test suite.

```bash
mcp-eval generate --n-examples 10 --output tests/generated.py
```

--------------------------------

### Initialize mcp-eval Project with Sample Template

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/sample_server/README.md

Initializes a new mcp-eval project using the 'sample' template. This command generates configuration files like `mcpeval.yaml` and `mcpeval.secrets.yaml.example`.

```bash
# From your own project directory
mcp-eval init --template sample
```

--------------------------------

### Check MCP Server Fetch Availability

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/examples/sample_server/README.md

Verifies if the MCP Fetch server is installed and accessible by checking its help command. This is a prerequisite for examples that utilize the fetch functionality.

```bash
uvx mcp-server-fetch --help
```

--------------------------------

### Initialize MCP-Eval Project

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/cli-reference.mdx

Commands to initialize a new MCP-Eval project. The `init` command creates a new project structure, and the `--template sample` flag initializes it with sample files for quicker setup.

```bash
# Create new project
mcp-eval init

# Or with sample files
mcp-eval init --template sample
```

--------------------------------

### Content Assertion Patterns in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-writer.mdx

Provides examples of various content assertion methods available in mcp-eval, including checking for containment, exact matches, and regular expression patterns.

```python
# Expect.content.contains("text", case_sensitive=False)
# Expect.content.equals("exact match")
# Expect.content.regex(r"pattern")
```

--------------------------------

### React Get Root Component (JavaScript)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/report_generation/report_template.html

Retrieves the root component of a React tree starting from a given fiber node. It traverses up the tree to find the component with tag 3 (HostRoot).

```javascript
function f(e) {
  if (e.tag === 3) return e.stateNode.current === e ? e : null;
  for (var t = e.return; t !== null; ) {
    if (t.tag === 3) return t.stateNode.current === t ? e : null;
    e = t;
    t = t.return;
  }
  throw Error(s(188));
}
```

--------------------------------

### YAML Configuration for Secrets

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

Provides an example of a YAML file for storing secrets, such as API keys for cloud providers like Anthropic and OpenAI. This file is merged with the main configuration by MCP-Eval.

```yaml
# mcp-agent.secrets.yaml
anthropic:
  api_key: "sk-ant-..."
openai:
  api_key: "sk-openai-..."
```

--------------------------------

### Initialize MCP-Eval Project

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/quickstart.mdx

Initializes a new mcp-eval testing environment using an interactive wizard. This process configures LLM settings, API secrets, defines test agents, and imports existing MCP servers.

```bash
mcp-eval init
```

--------------------------------

### Advanced Programmatic Control for MCP-Eval (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/configuration.mdx

Illustrates advanced programmatic configuration in Python, including loading external config files, modifying settings at runtime, setting programmatic defaults for agents and servers, and using a context manager for temporary configuration overrides.

```python
from mcp_eval.config import (
    load_config,
    get_settings,
    use_config,
    ProgrammaticDefaults
)

# Load specific config file
config = load_config("configs/staging.yaml")
use_config(config)

# Modify settings at runtime
current = get_settings()
current.execution.timeout_seconds = 600
current.reporting.formats.append("csv")

# Set programmatic defaults
defaults = ProgrammaticDefaults()
defaults.set_agent_factory(lambda: create_custom_agent())
defaults.set_default_servers(["server1", "server2"])

# Context manager for temporary config
from mcp_eval.config import config_context

with config_context({"provider": "openai", "model": "gpt-4"}):
    # Tests here use OpenAI
    run_tests()
# Back to original config
```

--------------------------------

### Error Handling Test Pattern in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-writer.mdx

Provides an example of an error handling test case using the decorator style. This pattern focuses on verifying the agent's ability to gracefully handle invalid inputs and explain issues.

```python
@task("Test error recovery")
async def test_error_handling(agent, session):
    response = await agent.generate_str(
        "Try to use a tool with invalid input"
    )
    
    await session.assert_that(
        Expect.judge.llm(
            "Agent should handle error gracefully and explain the issue",
            min_score=0.8
        ),
        response=response,
        name="graceful_error_handling"
    )
```

--------------------------------

### Setup with Environment Validation (Python)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/api-core.mdx

Use the `@setup` decorator to perform environment validation before tests run. This function can check prerequisites like Python version, required environment variables, and the health of external services. If validation fails, it raises an error, preventing test execution.

```python
from mcp_eval.core import setup
import sys
import os
# Assuming 'check_server_health' is available from mcp_eval.utils

@setup
def validate_environment():
    """Ensure test environment is properly configured."""
    
    # Check Python version
    if sys.version_info < (3, 10):
        raise RuntimeError("Tests require Python 3.10+")
    
    # Check required environment variables
    required_vars = ["ANTHROPIC_API_KEY", "TEST_SERVER_URL"]
    missing = [var for var in required_vars if not os.getenv(var)]
    
    if missing:
        raise RuntimeError(f"Missing environment variables: {missing}")
    
    # Check MCP servers are accessible
    from mcp_eval.utils import check_server_health
    if not check_server_health("my_server"):
        raise RuntimeError("MCP server 'my_server' is not responding")
    
    print("✅ Environment validated successfully")
```

--------------------------------

### Multi-Criteria Judging with Advanced Options in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

This example illustrates how to perform multi-criteria judging for LLM responses, allowing for weighted criteria, minimum scores, and chain-of-thought reasoning. It's essential for comprehensive evaluation of LLM outputs against complex rubrics.

```python
from mcp_eval.evaluators import EvaluationCriterion

criteria = [
    EvaluationCriterion(
        name="accuracy",
        description="Factual correctness",
        weight=2.0,
        min_score=0.8
    ),
    EvaluationCriterion(
        name="completeness",
        description="Covers key points",
        weight=1.5,
        min_score=0.7
    ),
]
await session.assert_that(
    Expect.judge.multi_criteria(
        criteria=criteria,
        aggregate_method="weighted",
        require_all_pass=False,
        use_cot=True,
    ),
    response=response,
    inputs=original_input
)
```

--------------------------------

### Best Practice: Tool Discovery (Bash)

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/subagents/mcp-eval-test-generator.mdx

Demonstrates the best practice of discovering available tools before generating tests. This involves listing server tools verbosely and then using the discovered tools to guide test generation.

```bash
# List available tools
mcp-eval server list --verbose

# Use discovered tools for generation
mcp-eval generate --discover-tools --style pytest
```

--------------------------------

### Decorator Style Test Authoring

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

Implements tests using a decorator-based style, leveraging `@setup`, `@parametrize`, and `@task` decorators from `mcp_eval`. This style provides a unified API for defining test configurations, parameters, and asynchronous tasks, with the agent and session automatically injected.

```python
from mcp_eval import task, setup, parametrize, Expect

@setup
def essential_config():
    pass  # optional

@parametrize("url,expect", [
    ("https://example.com", "Example Domain"),
    ("https://httpbin.org/html", "Herman Melville"),
])
@task("Parametrized fetch scenarios")
async def test_fetch(agent, session, url, expect):
    response = await agent.generate_str(f"Fetch {url}")
    await session.assert_that(Expect.tools.was_called("fetch"), name="fetch_called")
    await session.assert_that(Expect.content.contains(expect), name="contains_expected", response=response)
```

--------------------------------

### Asserting Tool Output Matches with Regex in Python

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/GUIDE.md

This example demonstrates how to assert that a tool's output matches a specific regular expression, with options for case sensitivity and targeting specific fields within the output. It's useful for validating structured data returned by tools.

```python
await session.assert_that(
    Expect.tools.output_matches(
        tool_name="fetch",
        expected_output=r"use.*examples",
        match_type="regex",
        case_sensitive=False,
        field_path="content.0.text",
    ),
    name="fetch_output_match",
)
```

--------------------------------

### Python Example: Full MCP-Eval Test Case

Source: https://github.com/lastmile-ai/mcp-eval/blob/main/docs/overview.mdx

Demonstrates a complete test scenario using mcp-eval, including agent configuration, task execution, tool call verification, content assertion, sequence validation, quality assessment, and performance checks.

```python
from mcp_eval import task, setup, Expect
from mcp_eval.evaluators import EvaluationCriterion

@setup
def configure():
    # Define the agent and servers it can use
    mcp_eval.use_agent(
        AgentSpec(
            name="weather_assistant",
            instruction="You help users with weather information.",
            server_names=["weather_api", "location_service"],
            model="claude-3-5-haiku-20241022"
        )
    )

@task("Get weather for user's location")
async def test_weather_flow(agent, session):
    # Execute the task
    response = await agent.generate_str(
        "What's the weather like in San Francisco tomorrow?"
    )
    
    # Verify tool usage - immediate checks
    await session.assert_that(
        Expect.tools.was_called("get_location"),
        name="location_service_called"
    )
    await session.assert_that(
        Expect.tools.was_called("get_weather_forecast"),
        name="weather_api_called"
    )
    
    # Check content quality
    await session.assert_that(
        Expect.content.contains("San Francisco"),
        response=response,
        name="mentions_city"
    )
    await session.assert_that(
        Expect.content.regex(r"\d+\s*°[CF]"),  # Temperature pattern
        response=response,
        name="includes_temperature"
    )
    
    # Verify execution efficiency
    await session.assert_that(
        Expect.tools.sequence(
            ["get_location", "get_weather_forecast"],
            allow_other_calls=False
        ),
        name="optimal_tool_sequence"
    )
    
    # Apply quality judge
    await session.assert_that(
        Expect.judge.multi_criteria([
            EvaluationCriterion(
                name="accuracy",
                description="Weather info is plausible and specific",
                weight=2.0,
                min_score=0.8
            ),
            EvaluationCriterion(
                name="helpfulness",
                description="Response is clear and actionable",
                weight=1.5,
                min_score=0.7
            )
        ]),
        response=response,
        name="quality_assessment"
    )
    
    # Performance requirements
    await session.assert_that(
        Expect.performance.response_time_under(10000),  # 10 seconds
        name="response_time_check"
    )
```