### Install and Run MCP Evals CLI

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Installs the mcp-evals package and demonstrates how to run evaluations using either TypeScript or YAML configuration files via the CLI.

```bash
# Install the package
npm install mcp-evals

# Run evaluations with TypeScript config
npx mcp-eval path/to/evals.ts path/to/server.ts

# Run evaluations with YAML config
npx mcp-eval path/to/evals.yaml path/to/server.ts
```

--------------------------------

### YAML Evaluation Configuration Example

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Demonstrates how to define MCP evaluations declaratively using the YAML format. This approach simplifies configuration and supports both OpenAI and Anthropic models, with optional API key specification.

```yaml
# YAML configuration example (content not provided in the input text, only the header)
```

--------------------------------

### Start and Access Monitoring Stack Services

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Commands to start the docker-compose services in detached mode and access the web UIs for Prometheus, Grafana, and Jaeger. Default credentials for Grafana are provided.

```bash
# Start the monitoring stack
docker-compose up -d

# Access dashboards:
# - Prometheus: http://localhost:9090
# - Grafana: http://localhost:3000 (admin/admin)
# - Jaeger UI: http://localhost:16686
```

--------------------------------

### Start Monitoring Stack (Bash)

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Starts the Docker Compose-based monitoring stack for MCP Evals. This command assumes a docker-compose.yml file is present in the current directory.

```bash
docker-compose up -d
```

--------------------------------

### Install mcp-evals Node.js Package

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Installs the mcp-evals package using npm. This is the first step to using mcp-evals in your Node.js project.

```bash
npm install mcp-evals
```

--------------------------------

### Run MCP Evaluations using Node.js CLI

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Demonstrates how to execute MCP evaluations from the command line using the 'npx mcp-eval' command. It shows examples for both TypeScript and YAML configuration files, specifying the paths to the evaluation and server files.

```bash
# Using TypeScript configuration
npx mcp-eval path/to/your/evals.ts path/to/your/server.ts

# Using YAML configuration
npx mcp-eval path/to/your/evals.yaml path/to/your/server.ts
```

--------------------------------

### Initialize Metrics for MCP Server

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Initializes the metrics system for collecting Prometheus metrics and OpenTelemetry traces for MCP tool calls. This setup must be done before creating the MCP server instance. It configures tracing, an OpenTelemetry endpoint, and a service name.

```typescript
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { metrics } from 'mcp-evals';
import { z } from "zod";

// Initialize metrics BEFORE creating the MCP server
metrics.initialize(9090, {
  enableTracing: true,
  otelEndpoint: 'http://localhost:4318/v1/traces',
  serviceName: 'my-mcp-server'
});

// Create MCP server - tools are automatically instrumented
const server = new McpServer({
  name: "Demo",
  version: "1.0.0"
});

server.tool("add",
  { a: z.number(), b: z.number() },
  async ({ a, b }) => ({
    content: [{ type: "text", text: String(a + b) }]
  })
);

const transport = new StdioServerTransport();
await server.connect(transport);

// Metrics available at http://localhost:9090/metrics
// - mcp_tool_calls_total: Counter of tool calls by name
// - mcp_tool_errors_total: Counter of errors by tool name
// - mcp_tool_latency_seconds: Histogram of latency by tool name
```

--------------------------------

### TypeScript Evaluation Configuration Example

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Defines MCP evaluations programmatically using TypeScript and the EvalConfig interface. Each evaluation includes a name, description, and an asynchronous 'run' function that returns scored results.

```typescript
// evals.ts
import { EvalConfig, EvalFunction } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade } from "mcp-evals";

const weatherEval: EvalFunction = {
  name: 'Weather Tool Evaluation',
  description: 'Evaluates the accuracy and completeness of weather information retrieval',
  run: async (model) => {
    const result = await grade(model, "What is the weather in New York?");
    return JSON.parse(result);
  }
};

const calculatorEval: EvalFunction = {
  name: 'Calculator Evaluation',
  description: 'Tests basic arithmetic operations',
  run: async (model) => {
    const result = await grade(model, "Calculate 42 divided by 7");
    return JSON.parse(result);
  }
};

const config: EvalConfig = {
  model: openai("gpt-4"),
  evals: [weatherEval, calculatorEval]
};

export default config;
export const evals = [weatherEval, calculatorEval];
```

--------------------------------

### GitHub Action for MCP Evaluations

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Integrates MCP evaluations into a GitHub Actions CI/CD pipeline. This workflow automatically checks out code, sets up Node.js, installs dependencies, and runs MCP evaluations, posting results as comments on pull requests. It requires the path to evaluation and server files, and optionally an OpenAI API key, model, and timeout.

```yaml
# .github/workflows/mcp-evals.yml
name: Run MCP Evaluations
on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Run MCP Evaluations
        uses: mclenhard/mcp-evals@v1.0.9
        with:
          evals_path: 'src/evals/evals.ts'  # Or evals.yaml
          server_path: 'src/index.ts'
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}
          model: 'gpt-4'                    # Optional, defaults to gpt-4
          timeout: '5000'                   # Optional, timeout in ms
```

--------------------------------

### Configure GitHub Action for MCP Evals

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Sets up a GitHub Actions workflow to automatically run MCP evaluations on pull requests. It checks out code, sets up Node.js, installs dependencies, and then runs the mcp-evals action with specified evaluation and server paths.

```yaml
name: Run MCP Evaluations
on:
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          
      - name: Install dependencies
        run: npm install
        
      - name: Run MCP Evaluations
        uses: mclenhard/mcp-evals@v1.0.9
        with:
          evals_path: 'src/evals/evals.ts'    # Can also use .yaml files
          server_path: 'src/index.ts'
          openai_api_key: ${{ secrets.OPENAI_API_KEY }}
          model: 'gpt-4'  # Optional, defaults to gpt-4
```

--------------------------------

### YAML Evaluation Configuration for mcp-evals

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Provides a sample YAML configuration for mcp-evals. It specifies the LLM provider and model, and lists multiple evaluations, each with a name, description, prompt, and expected result. This format is simpler for basic evaluations.

```yaml
# Model configuration
model:
  provider: openai     # 'openai' or 'anthropic'
  name: gpt-4o        # Model name
  # api_key: sk-...   # Optional, uses OPENAI_API_KEY env var by default

# List of evaluations to run
evals:
  - name: weather_query_basic
    description: Test basic weather information retrieval
    prompt: "What is the current weather in San Francisco?"
    expected_result: "Should return current weather data for San Francisco including temperature, conditions, etc."

  - name: weather_forecast
    description: Test weather forecast functionality
    prompt: "Can you give me the 3-day weather forecast for Seattle?"
    expected_result: "Should return a multi-day forecast for Seattle"

  - name: invalid_location
    description: Test handling of invalid location requests
    prompt: "What's the weather in Atlantis?"
    expected_result: "Should handle invalid location gracefully with appropriate error message"
```

--------------------------------

### Monitoring Stack with Docker Compose

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Provides a Docker Compose configuration to deploy a comprehensive observability stack. This stack includes Prometheus for metrics collection, Grafana for visualization, and Jaeger for distributed tracing, enabling detailed monitoring of MCP server performance.

```yaml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.48.0
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

  grafana:
    image: grafana/grafana:10.4.1
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana-storage:/var/lib/grafana

  jaeger:
    image: jaegertracing/all-in-one:1.55.0
    container_name: jaeger
    ports:
      - "16686:16686"
      - "14268:14268"
      - "14250:14250"

volumes:
  grafana-storage:
```

--------------------------------

### Run Prompt and Stream Response with runEvals Function

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Executes a prompt against an MCP server using the 'runEvals' function and streams the response. This function establishes a connection via stdio transport, discovers tools, and uses an LLM to generate the answer.

```typescript
import { runEvals } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";

const model = openai("gpt-4o");
const prompt = "Add 5 and 3 together";
const serverPath = "./example-server/index.ts";

const response = await runEvals(model, prompt, serverPath);
console.log(response);
// Output: "The result of adding 5 and 3 is 8."
```

--------------------------------

### Load YAML Evaluation Configuration

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Loads a YAML configuration file for evaluations and converts it into an EvalConfig object. This object is then used to run evaluations. It requires the path to the YAML file and optionally the server file path.

```typescript
import { loadYamlEvalConfig } from 'mcp-evals/yaml-loader';

const config = loadYamlEvalConfig('./evals.yaml', './server.ts');

// config is now an EvalConfig with:
// - model: LanguageModel instance based on YAML config
// - evals: Array of EvalFunction objects created from YAML evals

console.log(`Loaded ${config.evals.length} evaluations`);
```

--------------------------------

### Docker Compose Configuration for Monitoring Stack

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Defines the services for Prometheus, OpenTelemetry Collector, Grafana, and Jaeger using docker-compose. Each service specifies its Docker image, exposed ports, and volume mounts for configuration and data persistence.

```yaml
version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    ports:
      - "4318:4318"
    volumes:
      - ./otel/otel-collector-config.yaml:/etc/otel/config.yaml

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "9411:9411"
```

--------------------------------

### Evaluate Prompt with MCP Server using grade Function

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Uses the 'grade' function to evaluate a given prompt against an MCP server. It connects to the server, executes the prompt, and returns LLM-scored results across five quality dimensions: accuracy, completeness, relevance, clarity, and reasoning.

```typescript
import { grade } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";

const model = openai("gpt-4");
const prompt = "What is the weather in New York?";
const serverPath = "./path/to/mcp-server.ts";

const result = await grade(model, prompt, serverPath);
const evalResult = JSON.parse(result);

console.log(evalResult);
// Output:
// {
//   "accuracy": 4,
//   "completeness": 5,
//   "relevance": 5,
//   "clarity": 4,
//   "reasoning": 4,
//   "overall_comments": "The answer provides accurate weather information..."
// }
```

--------------------------------

### Initialize MCP Evals Metrics (TypeScript)

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Initializes the metrics collection for MCP Evals. This function should be called early in your application's lifecycle before the MCP server is initialized. It requires a port number and optional OpenTelemetry configuration.

```typescript
import { metrics } from 'mcp-evals';
metrics.initialize(9090, { enableTracing: true, otelEndpoint: 'http://localhost:4318/v1/traces' });
```

--------------------------------

### TypeScript Evaluation Configuration for mcp-evals

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Defines an evaluation configuration using TypeScript. It imports necessary types and functions from 'mcp-evals', defines an evaluation function (e.g., 'weatherEval') that uses an LLM to grade a prompt, and exports a configuration object and an array of evaluations.

```typescript
import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction} from "mcp-evals";

const weatherEval: EvalFunction = {
    name: 'Weather Tool Evaluation',
    description: 'Evaluates the accuracy and completeness of weather information retrieval',
    run: async () => {
      const result = await grade(openai("gpt-4"), "What is the weather in New York?");
      return JSON.parse(result);
    }
};
const config: EvalConfig = {
    model: openai("gpt-4"),
    evals: [weatherEval]
  };
  
  export default config;
  
  export const evals = [
    weatherEval,
    // add other evals here
]; 
```

--------------------------------

### Execute All Evaluations with runAllEvals Function

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Runs all evaluations defined within an EvalConfig using the 'runAllEvals' function. It processes each evaluation sequentially, handles errors, and returns a Map containing the results for each evaluation.

```typescript
import { runAllEvals, EvalConfig, EvalFunction } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade } from 'mcp-evals';

const weatherEval: EvalFunction = {
  name: 'Weather Tool Evaluation',
  description: 'Evaluates weather information retrieval',
  run: async (model) => {
    const result = await grade(model, "What is the weather in New York?");
    return JSON.parse(result);
  }
};

const mathEval: EvalFunction = {
  name: 'Math Tool Evaluation',
  description: 'Evaluates basic arithmetic operations',
  run: async (model) => {
    const result = await grade(model, "What is 15 plus 27?");
    return JSON.parse(result);
  }
};

const config: EvalConfig = {
  model: openai("gpt-4"),
  evals: [weatherEval, mathEval]
};

const results = await runAllEvals(config, "./path/to/server.ts");

for (const [name, result] of results.entries()) {
  console.log(`${name}:`, JSON.stringify(result, null, 2));
}
```

--------------------------------

### TypeScript Interface for Evaluation Results

Source: https://github.com/mclenhard/mcp-evals/blob/main/README.md

Defines the TypeScript interface 'EvalResult' which outlines the structure of the results returned by each evaluation. It includes scores for accuracy, completeness, relevance, clarity, reasoning, and overall comments.

```typescript
interface EvalResult {
  accuracy: number;        // Score from 1-5
  completeness: number;    // Score from 1-5
  relevance: number;       // Score from 1-5
  clarity: number;         // Score from 1-5
  reasoning: number;       // Score from 1-5
  overall_comments: string; // Summary of strengths and weaknesses
}
```

--------------------------------

### EvalResult Interface Definition

Source: https://context7.com/mclenhard/mcp-evals/llms.txt

Defines the structure of an evaluation result object returned by the grade function and evaluation runs. It includes scores for accuracy, completeness, relevance, clarity, and reasoning, along with overall comments.

```typescript
interface EvalResult {
  accuracy: number;        // Score from 1-5: factual correctness
  completeness: number;    // Score from 1-5: full question coverage
  relevance: number;       // Score from 1-5: information relevance
  clarity: number;         // Score from 1-5: explanation quality
  reasoning: number;       // Score from 1-5: logical thinking
  overall_comments: string; // Summary of strengths and weaknesses
}

// Example result:
const result: EvalResult = {
  accuracy: 5,
  completeness: 4,
  relevance: 5,
  clarity: 4,
  reasoning: 4,
  overall_comments: "The response accurately retrieves and presents weather data. Minor improvement possible in formatting the humidity percentage."
};
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.