### Quick Start: Run Basic Filesystem Domain

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Run a quick start example using the FilesystemDomain with an OpenAI agent. Requires uv sync --extra domain-filesystem.

```python
from mcp_evals import DomainRunner, PlainGrouper
from mcp_evals.contrib.filesystem import FilesystemDomain
from pydantic_ai import Agent

async def main():
    agent = Agent("openai:gpt-4o")
    runner = DomainRunner(agent=agent, grouper=PlainGrouper())
    report = await runner.run(FilesystemDomain(), experiment_name="quickstart")
    report.print()
```

--------------------------------

### Setup Domain with Lifecycle Management

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Implement the setup method to initialize resources and register cleanup callbacks with AsyncExitStack. The stack handles cleanup automatically when the domain context exits.

```python
from contextlib import AsyncExitStack
from typing import Any

import shutil
import tempfile

from mcp_evals import Domain
from pydantic_ai.mcp import MCPServerStdio

class DatabaseDomain(Domain):
    name = "database"

    async def setup(self, stack: AsyncExitStack[Any]) -> None:
        """Called before MCP servers are started."""
        self._temp_dir = tempfile.mkdtemp(prefix="mcp_evals_")
        self._db_path = f"{self._temp_dir}/test.db"
        stack.callback(lambda: shutil.rmtree(self._temp_dir, ignore_errors=True))

        await self._seed_database()

    def mcp_servers(self):
        return [MCPServerStdio("uvx", "mcp-server-sqlite", self._db_path)]

    def tasks(self):
        return [CreateUsersTableTask(), InsertUserTask()]

    async def _seed_database(self) -> None:
        ... 
```

--------------------------------

### Install MCP Evals

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Install the MCP Evals library using pip for a basic setup.

```bash
pip install pydantic-ai-mcp-evals
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/deeppavlov/mcp-evals/blob/main/AGENTS.md

Use `uv sync` to install project dependencies. Add `--no-dev` to omit the development group. Use `--all-groups --all-extras` for full parity with CI.

```bash
uv sync
```

```bash
uv sync --no-dev
```

```bash
uv sync --all-groups --all-extras
```

--------------------------------

### Domain Setup and Toolset Availability

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Illustrates the context management for a domain, where `domain.setup(stack)` is called, MCP servers connect, and `domain.toolset` becomes available.

```python
async with domain:
    # Domain setup runs here
    # MCP servers connect
    # domain.toolset is ready
```

--------------------------------

### Bash Script for Installing Dependencies

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Installs project dependencies using uv sync. Ensure uv is installed and configured for your environment.

```bash
uv sync
```

--------------------------------

### Domain Runner Initialization and Execution

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Shows how to initialize `DomainRunner` with an agent and grouper, and then execute a domain run. The `async with domain` block handles setup and teardown.

```python
runner = DomainRunner(agent=agent, grouper=grouper)
await runner.run(domain, experiment_name="my-experiment")
```

--------------------------------

### Implement Task Lifecycle with AsyncExitStack

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Tasks can implement a `setup` method to perform initialization tasks, such as creating temporary directories or downloading fixtures. Cleanup is managed automatically by `AsyncExitStack`, which registers callbacks for resource deallocation when the task context exits. Environment variables can also be managed and restored.

```python
from contextlib import AsyncExitStack
from pathlib import Path
from typing import Any

import aiofiles.tempfile
import os

from mcp_evals import Task
from mcp_evals.contrib.filesystem.common_evaluators import ContentMatches, FileExists

class MusicReportTask(Task):
    name = "music_report"
    goal = "Analyze the music files and create music_analysis_report.txt"
    evaluators = [
        FileExists("music/music_analysis_report.txt"),
        ContentMatches("music/music_analysis_report.txt", pattern=r"晴天.*2\.576"),
    ]

    async def setup(self, stack: AsyncExitStack[Any]) -> None:
        # Create temp directory — stack.enter_async_context ensures cleanup
        temp_dir_ctx = aiofiles.tempfile.TemporaryDirectory()
        self.test_dir = Path(await stack.enter_async_context(temp_dir_ctx))

        # Download and extract test fixtures
        archive_path = await download_fixtures("music_collection.tar.gz")
        stack.callback(lambda: archive_path.unlink(missing_ok=True))

        await extract_archive(archive_path, self.test_dir)

        # Set env var for MCP server; restore on exit
        old_value = os.environ.get("FILESYSTEM_ROOT")
        os.environ["FILESYSTEM_ROOT"] = str(self.test_dir)
        stack.callback(lambda: _restore_env("FILESYSTEM_ROOT", old_value))

def _restore_env(key: str, old_value: str | None) -> None:
    if old_value is None:
        os.environ.pop(key, None)
    else:
        os.environ[key] = old_value
```

--------------------------------

### Run Custom Experiment with FilesystemDomain

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Run a benchmark experiment using DomainRunner with a specified agent and domain. Includes example of printing report and checking case results.

```python
from mcp_evals import DomainRunner, PlainGrouper
from mcp_evals.contrib.filesystem import FilesystemDomain
from pydantic_ai import Agent

async def main():
    agent = Agent("openai:gpt-4o", system_prompt="You are a helpful assistant.")
    runner = DomainRunner(agent=agent, grouper=PlainGrouper())
    report = await runner.run(FilesystemDomain(), experiment_name="my-experiment")
    report.print()
    for case in report.cases:
        passed = all(s.value == 1.0 for s in case.scores.values())
        print(f"{case.name}: {'✓' if passed else '✗'}")
```

--------------------------------

### Domain and Task Class Diagram

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Visual representation of the Domain and Task abstractions and their relationships, including example task implementations.

```mermaid
classDiagram
    direction TB

    class Domain {
        <<abstract>>
        Async context manager
        optional setup with stack
        abstract mcp_servers
    }

    class FilesystemDomain {
        tasks own MCP servers
    }

    class PostgresDomain {
        secrets_type PgConfig
        starts shared Postgres
    }

    Domain <|-- FilesystemDomain
    Domain <|-- PostgresDomain

    Domain "1" *-- "*" Task : tasks

    class Task {
        <<abstract>>
        name goal evaluators
        optional setup with stack
        optional per-task mcp_servers
    }

    class FilesystemTask {
        work_dir fixture
    }

    class PostgresTask {
        pg_config category_id
    }

    FilesystemDomain *-- FilesystemTask : tasks
    Task <|-- FilesystemTask
    Task <|-- PostgresTask

    class UppercaseTask {
        <<example>>
    }
    class AuthorFoldersTask {
        <<example>>
    }

    class DbaVectorAnalysisTask {
        <<example>>
    }
    class BaseballPlayerAnalysisTask {
        <<example>>
    }

    FilesystemTask <|-- UppercaseTask
    FilesystemTask <|-- AuthorFoldersTask
    PostgresTask <|-- DbaVectorAnalysisTask
    PostgresTask <|-- BaseballPlayerAnalysisTask

    PostgresDomain *-- PostgresTask : tasks
```

--------------------------------

### Example Usage of transfer_parts Function

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/transactional_inventory_transfer/description.md

Demonstrates various scenarios for calling the `transfer_parts` function, including a basic transfer, transferring to a new inventory, and expected failure cases due to insufficient quantity or self-transfer.

```sql
-- Basic transfer with reason
SELECT transfer_parts(14469, 14686, '3024', 15, 100, 'inventory_adjustment');

-- Transfer to new inventory (should create new record)
SELECT transfer_parts(11124, 14686, '3001', 4, 50, 'part_redistribution');

-- This should fail due to insufficient quantity
SELECT transfer_parts(14469, 14686, '3024', 15, 2000, 'large_transfer');

-- This should fail due to self-transfer
SELECT transfer_parts(14469, 14469, '3024', 15, 10, 'self_transfer');
```

--------------------------------

### Bash Script for Running Tests

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Executes the project's test suite using pytest. This command should be run after installing dependencies.

```bash
uv run pytest
```

--------------------------------

### Mermaid Sequence Diagram for Evaluation Pipeline

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Visualizes the interaction flow between User Code, DomainRunner, Domain, Agent, MCP Servers, Evaluator, and Logfire during an evaluation run. It details the setup, task execution, agent interaction, tool usage, evaluation, and cleanup phases.

```mermaid
sequenceDiagram
    participant U as User Code
    participant R as DomainRunner
    participant D as Domain
    participant A as Agent
    participant MCP as MCP Servers
    participant E as Evaluator
    participant LF as Logfire

    U->>R: runner.run(domain, experiment_name=...)
    
    R->>D: async with domain
    D->>D: domain.setup()
    D->>MCP: Connect to MCP servers (CombinedToolset)
    MCP-->>D: Combined toolset ready
    R->>D: domain.tasks()
    
    loop For each task
        Note over R: CaseLifecycle.setup enters task context
        R->>R: task.setup()
        R->>A: agent.run(task.goal, toolsets=[domain.toolset])
        A->>LF: Log agent span
        A->>MCP: Use tools
        MCP-->>A: Tool results
        A-->>R: Agent output
        
        R->>E: evaluator.evaluate(context)
        Note over E: Task context still active
        E->>LF: Log evaluator span
        E->>MCP: Check environment state
        E-->>R: EvaluatorOutput
        Note over R: CaseLifecycle.teardown exits task context
        R->>R: task stack cleanup
    end

    D->>MCP: Disconnect servers (CombinedToolset cleanup)
    D->>D: domain stack cleanup
    
    R-->>U: EvaluationReport
```

--------------------------------

### Using `GoalFromDescriptionMixin`

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Demonstrates how to use `GoalFromDescriptionMixin` to load the task's goal from a `description.md` file, avoiding duplication of the prompt.

```python
class MyTask(GoalFromDescriptionMixin, DomainBaseTask):
    name = "my-task"
    # Goal is loaded from description.md

    def __init__(self, ..., tool_retries=...):
        super().__init__(..., tool_retries=tool_retries)
        self.evaluators = (
            MyEvaluator1(),
            MyEvaluator2(),
        )

```

--------------------------------

### Typical Directory Structure for a New Domain

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Illustrates the standard file and directory layout for adding a new domain and its tasks within the `src/mcp_evals/contrib/` directory. Optional files are indicated.

```text
src/mcp_evals/contrib/<domain>/
├── __init__.py                 # export Domain, public helpers
├── domain.py
├── task.py                     # optional: domain base Task subclass
├── utils.py                    # optional: fixtures, config types, downloads
├── common_evaluators/
│   ├── __init__.py
│   └── ...                     # reusable Evaluator implementations
├── scenario_evaluators/        # optional (e.g. postgres): cross-task checks
│   ├── __init__.py
│   └── ...
└── tasks/
    ├── __init__.py             # import task classes; export __all__
    └── <task_slug>/
        ├── __init__.py
        ├── task.py             # concrete Task subclass
        ├── description.md      # common if using GoalFromDescriptionMixin
        ├── constants.py        # optional
        ├── utils.py            # optional
        ├── _setup.py           # optional (e.g. complex DB seed helpers)
        ├── custom_evaluator.py # optional: one module (postgres-style)
        └── custom_evaluators/  # optional: package (filesystem-style)
            ├── __init__.py
            └── ...
```

--------------------------------

### Run Development Commands with uv

Source: https://github.com/deeppavlov/mcp-evals/blob/main/AGENTS.md

Execute development tasks such as running tests, type checking, and code formatting using `uv run`. Ensure you are in the repository root.

```bash
uv run pytest
```

```bash
uv run mypy .
```

```bash
uv run ruff check --fix
```

```bash
uv run ruff format
```

--------------------------------

### File Structure for New Domain and Tasks

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Follow this file structure when adding a new domain and its associated tasks to the mcp-evals project. Key files include domain definition, task definitions, and runner configuration.

```text
contrib/<domain>/domain.py
contrib/<domain>/task.py
contrib/<domain>/utils.py
contrib/<domain>/common_evaluators/
contrib/<domain>/tasks/<task_slug>/task.py
contrib/<domain>/tasks/<task_slug>/
contrib/<domain>/tasks/__init__.py
contrib/<domain>/__init__.py
```

--------------------------------

### Define and Use Training/Testing Callbacks with DomainRunner

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Define asynchronous functions to execute before training or testing phases. These callbacks accept a run context and can be configured to rerun on resume if they are idempotent. Use `skip_training_tasks=True` to run the callback without executing training tasks.

```python
from loguru import logger

from mcp_evals import DomainRunner, HoldOutGrouper, PlainGrouper

async def before_training(run_ctx) -> None:
    logger.info("Starting training phase {}...", run_ctx.phase_name)
    # e.g. reset model, clear caches; phase_name helps with idempotent bookkeeping
    train_tasks = run_ctx.get_phase_tasks()
    logger.info("Train tasks in this phase: {}", len(train_tasks))

async def before_testing(run_ctx) -> None:
    logger.info("Starting testing phase {}...", run_ctx.phase_name)
    # e.g. load trained weights, persist model; can also access previous train phase tasks:
    train_tasks = run_ctx.get_training_tasks()
    logger.info("Training tasks for this split: {}", len(train_tasks))

runner = DomainRunner(
    agent=agent,
    grouper=HoldOutGrouper(test_ratio=0.2),
    start_training=before_training,
    start_testing=before_testing,
    # Optional: skip train task execution (callback still runs)
    # skip_training_tasks=True,
    # Optional: re-run callbacks when resuming from checkpoint (for idempotent callbacks)
    # rerun_start_training_on_resume=True,
    # rerun_start_testing_on_resume=True,
)
report = await runner.run(MyDomain(), experiment_name="exp")
```

--------------------------------

### CREATE Employee Performance Table

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/employee_hierarchy_management/description.md

Define a new table to store employee performance metrics. This includes setting up primary and foreign keys and appropriate data types.

```sql
CREATE TABLE employee_performance (
    employee_id INTEGER PRIMARY KEY REFERENCES Employee(EmployeeId),
    customers_assigned INTEGER,
    performance_score DECIMAL
);
```

--------------------------------

### Define Custom Domain with MCP Servers and Tasks

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Define a custom domain by inheriting from the Domain class. Specify MCP servers and tasks.

```python
from mcp_evals import Domain
from pydantic_ai.mcp import MCPServerStdio

class CustomDomain(Domain):
    name = "custom"
    
    def mcp_servers(self):
        return [MCPServerStdio("uvx", "mcp-server-filesystem", "/workspace")]
    
    def tasks(self):
        return [CreateConfigTask(), CreateReadmeTask()]
```

--------------------------------

### Build File System Docker Image

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Build the custom Docker image required for file system tasks in MCP Evals. This command downloads the necessary files from the specified GitHub repository and tag.

```bash
docker build -t mcp-filesystem-server:pydantic-ai-mcp-evals https://github.com/voorhs/mcp-filesystem-server.git#551c35a6661aec56f9610ca6c4eef7cb9a2b3eb0
```

--------------------------------

### Task Context Management during Agent and Evaluator Runs

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Explains how `CaseLifecycle` maintains the task context for both agent and evaluator runs, ensuring fixtures and MCP state persist.

```python
# Inside CaseLifecycle:
# Task context is open for agent run
# Task context is open for evaluator runs
# Exits task context (stack closes)
```

--------------------------------

### Create Vector Storage Consumption Table

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/dba_vector_analysis/description.md

Defines the schema for a table to track storage consumption by vector data compared to regular data within tables.

```sql
CREATE TABLE vector_analysis_storage_consumption (
    schema VARCHAR(50),
    table_name VARCHAR(100),
    total_size_bytes BIGINT,
    vector_data_bytes BIGINT,
    regular_data_bytes BIGINT,
    vector_storage_pct NUMERIC(5,2),
    row_count BIGINT
);
```

--------------------------------

### Running Validation Checks

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Execute these commands using `uv run` to perform static analysis and code formatting checks on your contributions. This ensures code quality and consistency.

```bash
uv run mypy .

```

```bash
uv run ruff format

```

```bash
uv run ruff check --fix

```

--------------------------------

### Run Contrib Tasks Script

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Execute the run_domain_tasks.py script to run contrib tasks. Use --domain pg for Postgres tasks.

```bash
uv run python scripts/run_domain_tasks.py
```

```bash
uv run python scripts/run_domain_tasks.py --domain pg
```

--------------------------------

### Define Custom Task for File Creation

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Define custom tasks by inheriting from the Task class. Specify the name, goal, and evaluators.

```python
from mcp_evals import Task
from mcp_evals.contrib.filesystem.common_evaluators import FileExists, ContentMatches

class CreateConfigTask(Task):
    name = "create_config"
    goal = "Create a config.json file with default settings"
    evaluators = [FileExists("config.json")]

class CreateReadmeTask(Task):
    name = "create_readme"
    goal = "Create a README.md with project description"
    evaluators = [
        FileExists("README.md"),
        ContentMatches("README.md", pattern=r"project description"),
    ]
```

--------------------------------

### UPDATE Employee Promotion and Salary

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/employee_hierarchy_management/description.md

Promote an employee by updating their title and add a new 'salary' column to the Employee table. Then, set specific salaries for promoted employees and a default for others.

```sql
ALTER TABLE "Employee" ADD COLUMN "salary" DECIMAL;
UPDATE "Employee" SET "salary" = 75000.00 WHERE "EmployeeId" = 8;
UPDATE "Employee" SET "salary" = 50000.00 WHERE "EmployeeId" != 8;
UPDATE "Employee" SET "Title" = 'Senior IT Specialist' WHERE "EmployeeId" = 8;
```

--------------------------------

### Provide Per-Task Dependencies with Deps Maker

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Implement deps_maker to provide task-specific dependencies like database connections or request-scoped state. The deps_maker returns an async context manager that yields dependencies, which are then passed to agent.run().

```python
from contextlib import asynccontextmanager
from typing import Any

from mcp_evals import DomainRunner, PlainGrouper
from mcp_evals.task import Task
from mcp_evals.types import DepsMaker

def db_deps_maker(task: Task[Any, Any]) -> Any:  # returns AbstractAsyncContextManager
    @asynccontextmanager
    async def _cm():
        conn = await get_db_connection()
        try:
            yield conn
        finally:
            await conn.close()

    return _cm()

runner = DomainRunner(
    agent=agent,
    grouper=PlainGrouper(),
    deps_maker=db_deps_maker,
)
report = await runner.run(MyDomain(), experiment_name="exp")
```

--------------------------------

### Process Agent Run Results with run_result_processor

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Implement a processor function to handle the result of each agent task execution. This function receives the task, the agent's run result, and dependencies, allowing for actions like logging or persisting outputs. Configure this processor when initializing `DomainRunner`.

```python
from typing import Any

from mcp_evals import DomainRunner, PlainGrouper
from pydantic_ai.run import AgentRunResult

async def log_and_persist(
    task: Any,
    result: AgentRunResult,
    deps: object,
) -> None:
    # Log, persist, or sync to your system
    print(f"Task {task.name}: {result.output}")
    await save_result_to_db(task.name, result, deps)

runner = DomainRunner(
    agent=agent,
    grouper=PlainGrouper(),
    run_result_processor=log_and_persist,
)
report = await runner.run(MyDomain(), experiment_name="exp")
```

--------------------------------

### Mermaid Flowchart for pydantic_evals Integration

Source: https://github.com/deeppavlov/mcp-evals/blob/main/README.md

Illustrates the internal conversion process from mcp_evals' Domain and Task instances to pydantic_evals' Dataset, Cases, and Evaluated Function. It highlights how each Case maps inputs and evaluators.

```mermaid
flowchart LR
    subgraph "mcp_evals (User API)"
        Domain --> Tasks[Task instances]
    end
    
    subgraph "pydantic_evals (Internal)"
        Dataset --> Cases[Case instances]
        Dataset --> EvalFn[Evaluated Function]
    end
    
    Domain -->|"converted to"| Dataset
    Tasks -->|"converted to"| Cases
    
    subgraph "Each Case"
        CaseInputs["inputs: Task (the instance itself)"]
        CaseEvals[evaluators: task.evaluators]
    end
```

--------------------------------

### Create Vector Indices Analysis Table

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/dba_vector_analysis/description.md

Defines the schema for a table to document all vector indexes, their types, and sizes.

```sql
CREATE TABLE vector_analysis_indices (
    schema VARCHAR(50),
    table_name VARCHAR(100),
    column_name VARCHAR(100),
    index_name VARCHAR(100),
    index_type VARCHAR(50), -- 'hnsw', 'ivfflat', etc.
    index_size_bytes BIGINT
);
```

--------------------------------

### INSERT Employee Performance Records

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/employee_hierarchy_management/description.md

Populate the newly created employee_performance table with data for new employees. This involves calculating assigned customers and assigning a performance score.

```sql
INSERT INTO employee_performance (employee_id, customers_assigned, performance_score)
SELECT
    e.EmployeeId,
    (SELECT COUNT(*) FROM "Customer" WHERE "SupportRepId" = e.EmployeeId) AS customers_assigned,
    CASE WHEN e.EmployeeId = 9 THEN 4.5 ELSE 4.2 END AS performance_score
FROM "Employee" e
WHERE e.EmployeeId IN (9, 10);
```

--------------------------------

### SQL Query for Participant Performance Report

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/participant_report_optimization/description.md

This query retrieves participant event and stat counts, along with the last event date. Analyze its execution plan to identify performance bottlenecks.

```sql
SELECT 
    pe.participant_id,
    COUNT(pe.event_id) as event_count,
    (SELECT COUNT(*) FROM stats s WHERE s.stat_holder_id = pe.participant_id AND s.stat_holder_type = 'persons') as stat_count,
    (SELECT COUNT(DISTINCT s.stat_repository_type) FROM stats s WHERE s.stat_holder_id = pe.participant_id AND s.stat_holder_type = 'persons') as stat_type_count,
    (SELECT MAX(e.start_date_time) FROM events e JOIN participants_events pe2 ON e.id = pe2.event_id WHERE pe2.participant_id = pe.participant_id) as last_event_date
FROM participants_events pe 
WHERE pe.participant_id <= 50
GROUP BY pe.participant_id
ORDER BY pe.participant_id;
```

--------------------------------

### Concrete Task Implementation in `task.py`

Source: https://github.com/deeppavlov/mcp-evals/blob/main/docs/adding_new_tasks.md

Defines a concrete task class that subclasses a domain-specific base task. It includes setting the task name, goal, and initializing evaluators.

```python
class MyTask(DomainBaseTask):
    name = "my-task"
    goal = "My task goal description."

    def __init__(self, ..., tool_retries=...):
        super().__init__(..., tool_retries=tool_retries)
        self.evaluators = (
            MyEvaluator1(),
            MyEvaluator2(),
        )

```

--------------------------------

### Create Vector Analysis Columns Table

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/dba_vector_analysis/description.md

Defines the schema for a table to store catalog information about vector columns found in the database.

```sql
CREATE TABLE vector_analysis_columns (
    schema VARCHAR(50),
    table_name VARCHAR(100),
    column_name VARCHAR(100),
    dimensions INTEGER,
    data_type VARCHAR(50),
    has_constraints BOOLEAN,
    rows BIGINT
);
```

--------------------------------

### INSERT New Employees into Employee Table

Source: https://github.com/deeppavlov/mcp-evals/blob/main/src/mcp_evals/contrib/postgres/tasks/employee_hierarchy_management/description.md

Use this SQL statement to add new employees with specified details. Ensure all fields, including foreign key references and contact information, are correctly formatted.

```sql
INSERT INTO "Employee" ("EmployeeId", "FirstName", "LastName", "Title", "ReportsTo", "BirthDate", "HireDate", "Address", "City", "State", "Country", "PostalCode", "Phone", "Fax", "Email") VALUES (9, 'Sarah', 'Johnson', 'Sales Support Agent', 2, '1985-03-15', '2009-01-10', '123 Oak Street', 'Calgary', 'AB', 'Canada', 'T2P 5G3', '+1 (403) 555-0123', '+1 (403) 555-0124', 'sarah.johnson@chinookcorp.com');
INSERT INTO "Employee" ("EmployeeId", "FirstName", "LastName", "Title", "ReportsTo", "BirthDate", "HireDate", "Address", "City", "State", "Country", "PostalCode", "Phone", "Fax", "Email") VALUES (10, 'Mike', 'Chen', 'Sales Support Agent', 2, '1982-08-22', '2009-01-10', '456 Pine Ave', 'Calgary', 'AB', 'Canada', 'T2P 5G4', '+1 (403) 555-0125', '+1 (403) 555-0126', 'mike.chen@chinookcorp.com');
```