### Comprehensive Multi-Tool Simulation Setup

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_simulators.md

This example demonstrates the complete setup for multi-tool simulation, including initializing the `ToolSimulator`, defining Pydantic models for responses, registering tools with shared state, and setting up an experiment.

```python
from typing import Any
from enum import Enum
from pydantic import BaseModel, Field
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.simulation.tool_simulator import ToolSimulator

# Initialize simulator
tool_simulator = ToolSimulator()
```

--------------------------------

### Example Environment State Object

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/types.md

An example of creating an EnvironmentState object to capture database state.

```python
state = EnvironmentState(
    name="database",
    state={"users": [{"id": 1, "name": "Alice"}]}
)
```

--------------------------------

### Install Pre-commit Hook

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Install the main pre-commit hook for automated quality checks.

```bash
pre-commit install
```

--------------------------------

### Install Strands Evals SDK

Source: https://github.com/strands-agents/evals/blob/main/README.md

Install the Strands Evals SDK using pip. This command installs the necessary package for using the evaluation framework.

```bash
# Install Strands Evals SDK
pip install strands-agents-evals
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Install pre-commit hooks for automated quality checks before each commit. This includes pre-commit and commit-msg hooks.

```bash
pre-commit install -t pre-commit -t commit-msg
```

--------------------------------

### Tool Simulation Setup

Source: https://github.com/strands-agents/evals/blob/main/README.md

Sets up ToolSimulator, defines an output schema for a tool, registers the tool, and runs an experiment to evaluate an agent using the simulated tool.

```python
from typing import Any
from enum import Enum
from pydantic import BaseModel, Field
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.simulation.tool_simulator import ToolSimulator

tool_simulator = ToolSimulator()

# Define output schema
class HVACMode(str, Enum):
    HEAT = "heat"
    COOL = "cool"
    AUTO = "auto"
    OFF = "off"

class HVACResponse(BaseModel):
    temperature: float = Field(..., description="Target temperature in Fahrenheit")
    mode: HVACMode = Field(..., description="HVAC mode")
    status: str = Field(default="success", description="Operation status")

# Register tool — the function body is never called; the LLM generates responses
@tool_simulator.tool(
    share_state_id="room_environment",
    initial_state_description="Room: 68°F, humidity 45%, HVAC off",
    output_schema=HVACResponse,
)
def hvac_controller(temperature: float, mode: str) -> dict[str, Any]:
    """Control heating/cooling system that affects room temperature and humidity."""
    pass

def task_function(case: Case) -> dict:
    hvac_tool = tool_simulator.get_tool("hvac_controller")
    agent = Agent(tools=[hvac_tool], callback_handler=None)
    response = agent(case.input)
    return {"output": str(response)}

cases = [Case(name="heat_control", input="Turn on the heat to 72 degrees")]
experiment = Experiment(cases=cases, evaluators=[GoalSuccessRateEvaluator()])
report = experiment.run_evaluations(task_function)
```

--------------------------------

### Pre-commit Hooks

Source: https://github.com/strands-agents/evals/blob/main/AGENTS.md

Install and run pre-commit hooks to ensure code quality and consistency before committing. This includes installing hooks and running all checks on all files.

```bash
pre-commit install -t pre-commit -t commit-msg
```

```bash
pre-commit run --all-files
```

--------------------------------

### Basic Case with Output Expectation

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance with a name, input, and expected output.

```APIDOC
## Usage Examples

### Basic Case with Output Expectation

```python
from strands_evals import Case

case = Case[str, str](
    name="Simple Math",
    input="What is 2x2?",
    expected_output="2x2 is 4.",
    metadata={"category": "math"}
)
```
```

--------------------------------

### Case with Environment State Expectations

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance that includes expected environment states.

```APIDOC
### Case with Environment State Expectations

```python
from strands_evals import Case
from strands_evals.types.evaluation import EnvironmentState

case = Case[str, dict](
    name="Database Update",
    input="Update user profile with name='Alice'",
    expected_environment_state=[
        EnvironmentState(
            name="database_state",
            state={"users": [{"id": 1, "name": "Alice"}]}
        )
    ]
)
```
```

--------------------------------

### Example Interaction Object

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/types.md

An example of how to structure an interaction object for a multi-agent system.

```python
interaction = {
    "node_name": "router",
    "dependencies": ["input_parser"],
    "messages": ["Request received: Process payment"]
}
```

--------------------------------

### Basic Case and Experiment Setup

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Defines a Case with input, expected output, and metadata, then creates an Experiment with this case and runs evaluations using a task function. The report can then be displayed.

```python
from strands_evals import Case, Experiment

case = Case[str, str](
    name="capital-france",
    input="What is the capital of France?",
    expected_output="The capital of France is Paris.",
    metadata={"category": "knowledge"},
)
experiment = Experiment[str, str](cases=[case], evaluators=[...])
report = experiment.run_evaluations(task_function)
report.run_display()
```

--------------------------------

### Set Up Development Environment

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Instructions for setting up the development environment using hatchling and hatch. This includes creating and activating a virtual environment and installing the package with various dependency sets.

```bash
# Create and activate a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install the package in development mode with dependencies
pip install -e .

# Install with test dependencies
pip install -e ".[test]"

# Install with both test and dev dependencies
pip install -e ".[test,dev]"
```

--------------------------------

### Save and Load Experiments

Source: https://github.com/strands-agents/evals/blob/main/README.md

Demonstrates how to serialize an experiment configuration to a file for later use and how to load it back. This ensures reproducibility of evaluation setups.

```python
# Save experiment with metadata
experiment.to_file("customer_service_eval", "json")

# Load experiment from file
loaded_experiment = Experiment.from_file("./experiment_files/customer_service_eval.json", "json")

# Experiment files include:
# - Test cases with metadata
# - Evaluator configuration
# - Expected outputs and trajectories
# - Versioning information
```

--------------------------------

### Multi-turn Conversation Simulation Setup

Source: https://github.com/strands-agents/evals/blob/main/README.md

Sets up telemetry, defines a task function to simulate multi-turn conversations using ActorSimulator, and runs an experiment with evaluators.

```python
from strands import Agent
from strands_evals import Case, Experiment, ActorSimulator
from strands_evals.evaluators import HelpfulnessEvaluator, GoalSuccessRateEvaluator
from strands_evals.mappers import StrandsInMemorySessionMapper
from strands_evals.telemetry import StrandsEvalsTelemetry

# Setup telemetry
telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()
memory_exporter = telemetry.in_memory_exporter

def task_function(case: Case) -> dict:
    # Create simulator to drive conversation
    simulator = ActorSimulator.from_case_for_user_simulator(
        case=case,
        max_turns=10
    )

    # Create agent to evaluate
    agent = Agent(
        trace_attributes={
            "gen_ai.conversation.id": case.session_id,
            "session.id": case.session_id
        },
        callback_handler=None
    )

    # Run multi-turn conversation
    all_spans = []
    user_message = case.input

    while simulator.has_next():
        memory_exporter.clear()
        agent_response = agent(user_message)
        turn_spans = list(memory_exporter.get_finished_spans())
        all_spans.extend(turn_spans)

        user_result = simulator.act(str(agent_response))
        user_message = str(user_result.structured_output.message)

    # Map to session for evaluation
    mapper = StrandsInMemorySessionMapper()
    session = mapper.map_to_session(all_spans, session_id=case.session_id)

    return {"output": str(agent_response), "trajectory": session}

# Use evaluators to assess simulated conversations
evaluators = [
    HelpfulnessEvaluator(),
    GoalSuccessRateEvaluator()
]

experiment = Experiment(cases=test_cases, evaluators=evaluators)
report = experiment.run_evaluations(task_function)
```

--------------------------------

### Install Strands Agents Evals in Development Mode

Source: https://github.com/strands-agents/evals/blob/main/README.md

Install the library in editable mode for development. Includes options to install with test or both test and development dependencies.

```bash
# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows use: .venv\Scripts\activate

# Install in development mode
pip install -e .

# Install with test dependencies
pip install -e ".[test]"

# Install with both test and dev dependencies
pip install -e ".[test,dev]"
```

--------------------------------

### Trace-Based Evaluator Setup

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

Sets up telemetry and runs an experiment using a task function that captures agent interactions as spans. The spans are then mapped to a session for analysis.

```python
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.telemetry import StrandsEvalsTelemetry
from strands_evals.mappers import StrandsInMemorySessionMapper

# Setup telemetry
telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()

def task_function(case: Case) -> dict:
    # Clear spans for this case
    telemetry.clear()
    
    # Create and run agent
    agent = Agent(
        trace_attributes={"session.id": case.session_id},
        callback_handler=None
    )
    response = agent(case.input)
    
    # Get spans and map to Session
    spans = telemetry.get_spans()
    mapper = StrandsInMemorySessionMapper()
    session = mapper.map_to_session(spans, session_id=case.session_id)
    
    return {"output": str(response), "trajectory": session}

# Evaluate with trace-based evaluator
experiment = Experiment(
    cases=test_cases,
    evaluators=[HelpfulnessEvaluator()]
)

report = experiment.run_evaluations(task_function)
```

--------------------------------

### OutputEvaluator Example

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluators.md

Example of creating an `OutputEvaluator` instance. This demonstrates how to specify the evaluation rubric, whether to include inputs, and the model to be used for assessment.

```python
from strands_evals.evaluators import OutputEvaluator

evaluator = OutputEvaluator(
    rubric="Score 1.0 if the response is accurate and complete. Score 0.5 if partially correct. Score 0.0 if incorrect.",
    include_inputs=True,
    model="global.anthropic.claude-sonnet-4-6"
)
```

--------------------------------

### Prepare Multimodal Input Data

Source: https://github.com/strands-agents/evals/blob/main/README.md

Create image data objects for multimodal evaluations. This example shows how to load an image from a file path.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import (
    MultimodalCorrectnessEvaluator,
    MultimodalFaithfulnessEvaluator,
    MultimodalInstructionFollowingEvaluator,
    MultimodalOverallQualityEvaluator,
)
from strands_evals.types import ImageData, MultimodalInput

# Create image data from various sources
image = ImageData(source="path/to/image.png")
```

--------------------------------

### Handle ImportError: OpenTelemetry not installed

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/errors.md

Demonstrates how to check for and install OpenTelemetry packages required for telemetry. An ImportError occurs if the 'otel' extra is not installed.

```bash
# ❌ Error when using telemetry without otel extra
python -c "from strands_evals.telemetry import StrandsEvalsTelemetry"

# ✅ Install with otel extra
pip install "strands-agents-evals[otel]"
```

--------------------------------

### Case with Custom Metadata

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance with detailed custom metadata.

```APIDOC
### Case with Custom Metadata

```python
from strands_evals import Case

case = Case[str, str](
    name="Knowledge Question",
    input="What is the capital of France?",
    expected_output="The capital of France is Paris.",
    metadata={
        "category": "knowledge",
        "difficulty": "easy",
        "source": "geography_kb",
        "tags": ["europe", "capitals"]
    }
)
```
```

--------------------------------

### Minimal Case Without Expectations

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a minimal Case instance with only the required input, relying on auto-generated session ID.

```APIDOC
### Minimal Case Without Expectations

```python
from strands_evals import Case

case = Case[str, str](
    input="What is the capital of France?"
)
# Uses auto-generated session_id and no name
```
```

--------------------------------

### Example Usage of Multimodal Evaluators

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluators.md

Demonstrates how to instantiate and use multimodal evaluators like MultimodalCorrectnessEvaluator and MultimodalFaithfulnessEvaluator with image data and a case.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import MultimodalCorrectnessEvaluator, MultimodalFaithfulnessEvaluator
from strands_evals.types import ImageData, MultimodalInput

image = ImageData(source="path/to/image.png")
case = Case[MultimodalInput, str](
    name="image-description",
    input=MultimodalInput(
        media=image,
        instruction="Describe this image in detail."
    )
)

evaluators = [
    MultimodalCorrectnessEvaluator(),
    MultimodalFaithfulnessEvaluator()
]
```

--------------------------------

### Case with Multi-Agent Interactions

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance for multi-agent systems, specifying expected interactions.

```APIDOC
### Case with Multi-Agent Interactions

```python
from strands_evals import Case

case = Case(
    name="Multi-Agent Coordination",
    input="Process this request across teams",
    expected_interactions=[
        {"node_name": "router", "messages": ["Received request"]},
        {"node_name": "processor", "messages": ["Processing..."]},
        {"node_name": "validator", "messages": ["Validating result"]}
    ],
    metadata={"agents": 3}
)
```
```

--------------------------------

### Setup HelpfulnessEvaluator with OpenTelemetry Traces

Source: https://github.com/strands-agents/evals/blob/main/README.md

Configure HelpfulnessEvaluator to assess agent helpfulness using OpenTelemetry traces. This involves setting up telemetry for trace capture and mapping spans to sessions for evaluation.

```python
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.telemetry import StrandsEvalsTelemetry
from strands_evals.mappers import StrandsInMemorySessionMapper

# Setup telemetry for trace capture
telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()

def user_task_function(case: Case) -> dict:
    telemetry.memory_exporter.clear()
    
    agent = Agent(
        trace_attributes={"session.id": case.session_id},
        callback_handler=None
    )
    response = agent(case.input)
    
    # Map spans to session for evaluation
    spans = telemetry.memory_exporter.get_finished_spans()
    mapper = StrandsInMemorySessionMapper()
    session = mapper.map_to_session(spans, session_id=case.session_id)
    
    return {"output": str(response), "trajectory": session}

# Seven-level scoring: Not helpful (0.0) to Above and beyond (1.0)
evaluators = [HelpfulnessEvaluator()]
experiment = Experiment[str, str](cases=test_cases, evaluators=evaluators)
```

--------------------------------

### Basic Output Evaluation

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_experiment.md

Set up and run a basic experiment to evaluate agent output accuracy. This requires defining test cases, an OutputEvaluator, and a function to get the agent's response.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator
from strands import Agent

test_cases = [
    Case[str, str](
        name="Simple Knowledge",
        input="What is the capital of France?",
        expected_output="The capital of France is Paris.",
        metadata={"category": "knowledge"}
    )
]

evaluators = [
    OutputEvaluator(
        rubric="Score 1.0 if accurate and complete. Score 0.5 if partially correct. Score 0.0 if incorrect."
    )
]

experiment = Experiment[str, str](cases=test_cases, evaluators=evaluators)

def get_response(case: Case) -> str:
    agent = Agent(callback_handler=None)
    return str(agent(case.input))

report = experiment.run_evaluations(get_response)
report.run_display()
```

--------------------------------

### Initialize In-Memory Telemetry Exporter

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

Configures OpenTelemetry tracing for evaluation with in-memory span export. Use this to start capturing traces during evaluation.

```python
from strands_evals.telemetry import StrandsEvalsTelemetry

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()
```

--------------------------------

### ActorSimulator: Simulate Actor with Predefined Profile

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_simulators.md

Initializes an ActorSimulator with a specific actor profile, initial query, and optional configurations for system prompt, tools, model, max turns, and structured output model. Use when you have a predefined actor persona and starting message.

```python
simulator = ActorSimulator(
    actor_profile=actor_profile, # Assume actor_profile is defined
    initial_query="Hello, I need assistance.",
    max_turns=7,
    structured_output_model=MyResponseModel # Optional Pydantic model
)
```

--------------------------------

### Configure Agent Trace Attributes

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

This example demonstrates how to set custom trace attributes when initializing an Agent. These attributes help in identifying and filtering traces, such as session ID, user ID, and request type.

```python
from strands_agents.agent import Agent

# Assuming case is defined elsewhere

agent = Agent(
    trace_attributes={
        "session.id": case.session_id,
        "gen_ai.user.id": "user-123",
        "request.type": "evaluation",
        "priority": "high"
    },
    callback_handler=None
)
```
```

--------------------------------

### Expand Existing Customer Service Experiment

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_generators.md

Loads an existing customer service experiment and expands it with additional test cases. This example demonstrates adding more data to a pre-defined evaluation suite.

```python
import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals import Experiment

async def expand_eval():
    # Load existing experiment
    existing = Experiment.from_file("./experiments/customer_service.json")
    
    generator = ExperimentGenerator[str, dict](
        input_type=str,
        output_type=dict,
        include_expected_output=True,
        include_metadata=True
    )
    
    # Add more test cases
    expanded = await generator.from_experiment_async(
        experiment=existing,
        num_new_cases=10,
        expand_coverage=True
    )
    
    expanded.to_file("customer_service_expanded", "json")
    return expanded

expanded = asyncio.run(expand_eval())
```

--------------------------------

### Map CloudWatch Spans to Session Format

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

This example demonstrates using `CloudWatchSessionMapper` to retrieve and convert spans from CloudWatch logs into the Session format. This requires `boto3` to be installed and configured.

```python
from strands_evals.mappers import CloudWatchSessionMapper

mapper = CloudWatchSessionMapper()
session = mapper.map_to_session(
    session_id="session-123",
    log_group="/aws/lambda/my-function"
)
```
```

--------------------------------

### Basic Report Display

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Demonstrates setting up an experiment with cases and evaluators, running evaluations, and displaying the basic report.

```python
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator
from strands import Agent

experiment = Experiment[
    cases=[
        Case(name="Test 1", input="What is 2+2?", expected_output="4"),
        Case(name="Test 2", input="What is 3+3?", expected_output="6")
    ],
    evaluators=[OutputEvaluator(rubric="Check if answer is correct")]
)

def task_function(case: Case) -> str:
    agent = Agent(callback_handler=None)
    return str(agent(case.input))

report = experiment.run_evaluations(task_function)
report.run_display()
```

--------------------------------

### Configure ActorSimulator

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/configuration.md

Set up an ActorSimulator for user simulation. A Case object is required, and optional parameters include system prompt templates, tools, the model to use, and the maximum number of conversation turns.

```python
from strands_evals import Case, ActorSimulator

case = Case(
    input="Initial user query",
    metadata={"task_description": "Task goal"}
)

simulator = ActorSimulator.from_case_for_user_simulator(
    case=case,
    system_prompt_template=None,           # Uses default if None
    tools=None,                            # Uses goal completion tool if None
    model="global.anthropic.claude-sonnet-4-6",
    max_turns=10                           # Maximum conversation turns
)
```

--------------------------------

### Initialize LangfuseProvider (Explicit Credentials)

Source: https://github.com/strands-agents/evals/blob/main/src/strands_evals/providers/README.md

Instantiate LangfuseProvider by explicitly passing public key, secret key, and host.

```python
from strands_evals.providers import LangfuseProvider

# Or pass credentials explicitly
provider = LangfuseProvider(
    public_key="pk-ாலத்தில்",
    secret_key="sk-ாலத்தில்",
    host="https://us.cloud.langfuse.com",
)
```

--------------------------------

### Enter Virtual Environment

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Use hatch to enter the recommended virtual environment for development.

```bash
hatch shell
```

--------------------------------

### Initialize OpenSearchProvider for Local Development

Source: https://github.com/strands-agents/evals/blob/main/src/strands_evals/providers/README.md

Instantiate OpenSearchProvider for local or development environments using basic authentication. Certificate verification is disabled.

```python
from strands_evals.providers import OpenSearchProvider

# Local / development (basic auth)
provider = OpenSearchProvider(
    host="https://localhost:9200",
    auth=("admin", "password"),
    verify_certs=False,
)
```

--------------------------------

### Initialize LangfuseProvider (Default Credentials)

Source: https://github.com/strands-agents/evals/blob/main/src/strands_evals/providers/README.md

Instantiate LangfuseProvider. It reads LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY from environment variables by default.

```python
from strands_evals.providers import LangfuseProvider

# Reads LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY from env by default
provider = LangfuseProvider()
```

--------------------------------

### Example: Generate Rubric Asynchronously

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_generators.md

Example of generating an evaluation rubric for agent response accuracy. The rubric defines scoring for accurate, partial, and incorrect responses.

```python
rubric = await generator.generate_rubric_async(
    "Evaluate how accurately the agent responds to user queries. Score 1.0 for accurate, complete responses. Score 0.5 for partial answers. Score 0.0 for incorrect or irrelevant responses."
)
```

--------------------------------

### TrajectoryEvaluator Example

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluators.md

Example of creating a `TrajectoryEvaluator` instance. This shows how to define the evaluation rubric and provide an initial description of available trajectory tools. It also demonstrates dynamic updates to these descriptions.

```python
from strands_evals.evaluators import TrajectoryEvaluator

evaluator = TrajectoryEvaluator(
    rubric="Score 1.0 if correct tools used in proper sequence. Use exact_match_scorer to verify trajectory matches expected sequence.",
    trajectory_description={"calculator": "Performs mathematical operations", "validator": "Validates results"}
)

# Update descriptions dynamically
evaluator.update_trajectory_description({
    "calculator": "Advanced calculator with history",
    "validator": "Strict validator"
})
```

--------------------------------

### Case with Trajectory Expectation

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance that includes an expected trajectory of actions.

```APIDOC
### Case with Trajectory Expectation

```python
from strands_evals import Case

case = Case[str, str](
    name="Calculator Task",
    input="What is 5 + 3 * 2?",
    expected_output="The answer is 11.",
    expected_trajectory=["calculator", "validator"],
    metadata={"category": "math", "requires_tools": True}
)
```
```

--------------------------------

### Report with Diagnosis and Recommendations

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Configures an experiment to trigger diagnosis on failure and displays the report with recommendations. It also shows how to access raw diagnosis details.

```python
from strands_evals import Experiment, DiagnosisConfig
from strands_evals.types.detector import DiagnosisTrigger, ConfidenceLevel

experiment = Experiment(
    cases=test_cases,
    evaluators=evaluators,
    diagnosis_config=DiagnosisConfig(
        trigger=DiagnosisTrigger.ON_FAILURE,
        confidence_threshold=ConfidenceLevel.MEDIUM
    )
)

report = experiment.run_evaluations(task_function)

# Display with recommendations
report.display(include_recommendations=True)

# Get recommendations
recommendations = report.get_recommendations()
print("\nFix Recommendations:")
for rec in recommendations:
    print(f"  - {rec}")

# Access diagnosis details
for case in report.cases:
    if case.get("diagnosis"):
        diagnosis = case["diagnosis"]
        print(f"\n{case['name']}:")
        print(f"  Failures: {len(diagnosis.failures)}")
        for rc in diagnosis.root_causes:
            print(f"    {rc.fix_type}: {rc.fix_recommendation}")
```

--------------------------------

### Run All Code Quality Checks

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Execute both formatting and linting checks using hatch.

```bash
hatch fmt --formatter
hatch fmt --linter
```

--------------------------------

### Case with Assertion for Goal-Based Evaluation

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_case.md

Example of creating a Case instance with an expected assertion for goal-based evaluation.

```APIDOC
### Case with Assertion for Goal-Based Evaluation

```python
from strands_evals import Case

case = Case[str, dict](
    name="User Registration",
    input="I'd like to create an account with email john@example.com",
    expected_assertion="create_user is called with {'email': 'john@example.com'}",
    metadata={"task": "registration"}
)
```
```

--------------------------------

### Register Sensor Tool

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_simulators.md

Register a tool simulator for getting temperature data. Specify a share_state_id and an output schema.

```python
@tool_simulator.tool(
    share_state_id="climate_control",
    initial_state_description="Room at 68°F, target 70°F, heat enabled",
    output_schema=Temperature
)
def get_temperature() -> dict[str, Any]:
    """Get current room temperature and target."""
    pass
```

--------------------------------

### Get Experiment Evaluators

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_experiment.md

Retrieves the list of evaluators used in the experiment. These are responsible for assessing the performance of test cases.

```python
@property
def evaluators(self) -> list[Evaluator[InputT, OutputT]]
```

--------------------------------

### Setup In-Memory Telemetry Exporter

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Captures spans using telemetry and maps them to a session. Clears the exporter before processing a case.

```python
from strands_evals.telemetry import StrandsEvalsTelemetry
from strands_evals.mappers import StrandsInMemorySessionMapper

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()

def task_function(case):
    telemetry.in_memory_exporter.clear()
    agent = Agent(
        trace_attributes={"session.id": case.session_id, "gen_ai.conversation.id": case.session_id},
        callback_handler=None,
    )
    response = agent(case.input)
    spans = telemetry.in_memory_exporter.get_finished_spans()
    session = StrandsInMemorySessionMapper().map_to_session(spans, session_id=case.session_id)
    return {"output": str(response), "trajectory": session}
```

--------------------------------

### Configure ToolSimulator for LLM-Powered Tools

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Sets up a ToolSimulator to generate LLM-powered, schema-validated responses for tools. Tools with shared state IDs maintain consistency.

```python
from pydantic import BaseModel, Field
from strands_evals.simulation.tool_simulator import ToolSimulator

tool_simulator = ToolSimulator()

class HVACResponse(BaseModel):
    temperature: float = Field(..., description="Target temp F")
    mode: str
    status: str = "success"

@tool_simulator.tool(
    share_state_id="room_environment",
    initial_state_description="Room: 68F, humidity 45%, HVAC off",
    output_schema=HVACResponse,
)
def hvac_controller(temperature: float, mode: str) -> dict: ...

agent = Agent(tools=[tool_simulator.get_tool("hvac_controller")])
```

--------------------------------

### Run Experiment and Display Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_simulators.md

Orchestrate the evaluation by creating an Experiment with cases and evaluators, running the evaluations, and displaying the report.

```python
experiment = Experiment(
    cases=cases,
    evaluators=[GoalSuccessRateEvaluator()]
)

report = experiment.run_evaluations(task_function)
report.run_display()
```

--------------------------------

### Run Evaluation with Output Evaluator

Source: https://github.com/strands-agents/evals/blob/main/README.md

Demonstrates how to set up and run an evaluation experiment using the OutputEvaluator. This includes creating test cases, defining a custom rubric for evaluation, and running the experiment.

```python
from strands import Agent
from strands_evals import Case, Experiment
from strands_evals.evaluators import OutputEvaluator

# Create test cases
test_cases = [
    Case[str, str](
        name="knowledge-1",
        input="What is the capital of France?",
        expected_output="The capital of France is Paris.",
        metadata={"category": "knowledge"}
    )
]

# Create evaluators with custom rubric
evaluators = [
    OutputEvaluator(
        rubric="""
        Evaluate based on:
        1. Accuracy - Is the information correct?
        2. Completeness - Does it fully answer the question?
        3. Clarity - Is it easy to understand?
        
        Score 1.0 if all criteria are met excellently.
        Score 0.5 if some criteria are partially met.
        Score 0.0 if the response is inadequate.
        """
    )
]

# Create experiment and run evaluation
experiment = Experiment[str, str](cases=test_cases, evaluators=evaluators)

def get_response(case: Case) -> str:
    agent = Agent(callback_handler=None)
    return str(agent(case.input))

# Run evaluations
report = experiment.run_evaluations(get_response)
report.run_display()
```

--------------------------------

### Initialize OpenSearchProvider for Amazon OpenSearch Service

Source: https://github.com/strands-agents/evals/blob/main/src/strands_evals/providers/README.md

Instantiate OpenSearchProvider for Amazon OpenSearch Service using SigV4 authentication. Requires boto3 and opensearch-py.

```python
from strands_evals.providers import OpenSearchProvider
from opensearchpy import RequestsAWSV4SignerAuth
import boto3

credentials = boto3.Session().get_credentials()
auth = RequestsAWSV4SignerAuth(credentials, "us-east-1", "es")
provider = OpenSearchProvider(
    host="https://my-domain.us-east-1.es.amazonaws.com",
    auth=auth,
)
```

--------------------------------

### Configuration Mechanisms

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/MANIFEST.txt

Details all configuration mechanisms for the Strands Evals SDK, including environment variables, constructor parameters, and setup for various components.

```APIDOC
## Configuration Mechanisms

### Description
Outlines all available methods for configuring the Strands Evals SDK.

### Methods
- **Environment Variables**: Configuration via environment variables.
- **Constructor Parameters**: Options passed during class instantiation for all classes.
- **Component-Specific Options**: Configuration for evaluators, simulators, `ExperimentGenerator`, and telemetry setup.
- **Model Selection**: Options for choosing underlying models.

### Best Practices
Includes recommendations for effective configuration and setup.
```

--------------------------------

### Run Unit Tests

Source: https://github.com/strands-agents/evals/blob/main/CONTRIBUTING.md

Execute unit tests using hatch. Use the -c flag to run tests with coverage.

```bash
hatch test
```

```bash
hatch test -c
```

--------------------------------

### Get Recommendations from Evaluation Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Extracts all unique fix recommendations generated from the diagnosis results. This provides a consolidated list of suggested improvements.

```python
recs = report.get_recommendations()
for rec in recs:
    print(f"  - {rec}")
```

--------------------------------

### Get Experiment Cases

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_experiment.md

Retrieves a deep copy of all test cases associated with the experiment. This prevents accidental modification of the original test cases.

```python
@property
def cases(self) -> list[Case[InputT, OutputT]]
```

--------------------------------

### Initialize HelpfulnessEvaluator

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluators.md

Instantiate the HelpfulnessEvaluator for assessing response helpfulness using OpenTelemetry traces. Requires setting up telemetry exporter.

```python
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.telemetry import StrandsEvalsTelemetry

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()

evaluator = HelpfulnessEvaluator()
```

--------------------------------

### from_context_async

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_generators.md

Generates a complete experiment, including test cases and rubrics, from a free-form context description. It allows specifying evaluators, a task description, and the number of topics for diversity.

```APIDOC
## Method: from_context_async

```python
async def from_context_async(
    self,
    context: str,
    num_cases: int = 5,
    evaluator: type[Evaluator] | list[type[Evaluator]] | None = None,
    task_description: str | None = None,
    num_topics: int | None = None
) -> Experiment[InputT, OutputT]
```

### Description
Generate a complete experiment from a free-form context description.

### Parameters
- `context` (required): Description of available tools, domain, or task context
- `num_cases`: Number of test cases to generate
- `evaluator`: Evaluator class(es) to include. Options:
  - Single class: `OutputEvaluator`
  - List of classes: `[OutputEvaluator, TrajectoryEvaluator]`
  - None: Auto-selects evaluators based on schema
- `task_description`: Optional description of the task being evaluated
- `num_topics`: Optional number of topics for diversity

### Returns
Experiment with generated cases and rubrics

### Example
```python
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import TrajectoryEvaluator, OutputEvaluator

context = """
Available tools:
- calculator(expression: str) -> float: Evaluates mathematical expressions
- web_search(query: str) -> str: Searches the web for information
- file_read(path: str) -> str: Reads file contents
"""

generator = ExperimentGenerator[str, str](str, str)

experiment = await generator.from_context_async(
    context=context,
    num_cases=10,
    evaluator=[OutputEvaluator, TrajectoryEvaluator],
    task_description="Math and research assistant with tool usage",
    num_topics=3
)

# Save the generated experiment
experiment.to_file("generated_experiment", "json")
```
```

--------------------------------

### Get Evaluator Results from Evaluation Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Filters and returns a list of case results for a specific evaluator. Useful for analyzing the performance of individual evaluators.

```python
output_eval_results = report.get_evaluator_results("OutputEvaluator")
print(f"OutputEvaluator results: {len(output_eval_results)} cases")
```

--------------------------------

### Create Custom Evaluator for Policy Compliance

Source: https://github.com/strands-agents/evals/blob/main/README.md

Define domain-specific evaluation logic by subclassing the base `Evaluator`. This example checks for policy violations in agent responses.

```python
from strands_evals.evaluators import Evaluator
from strands_evals.types import EvaluationData, EvaluationOutput

class PolicyComplianceEvaluator(Evaluator[str, str]):
    def evaluate(self, evaluation_case: EvaluationData[str, str]) -> EvaluationOutput:
        # Custom evaluation logic
        response = evaluation_case.actual_output
        
        # Check for policy violations
        violations = self._check_policy_violations(response)
        
        if not violations:
            return EvaluationOutput(
                score=1.0,
                test_pass=True,
                reason="Response complies with all policies",
                label="compliant"
            )
        else:
            return EvaluationOutput(
                score=0.0,
                test_pass=False,
                reason=f"Policy violations: {', '.join(violations)}",
                label="non_compliant"
            )
    
    def _check_policy_violations(self, response: str) -> list[str]:
        # Implementation details...
        return []
```

--------------------------------

### Running Full Readiness Check

Source: https://github.com/strands-agents/evals/blob/main/AGENTS.md

Perform a comprehensive readiness check that includes linting, formatting, and testing across all supported Python versions.

```bash
hatch run prepare                     # lint + format + test-lint + test --all
```

--------------------------------

### Custom Evaluator Implementation

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Implement a custom Evaluator by subclassing Evaluator and defining the evaluate method. This example shows a PolicyComplianceEvaluator that checks for violations in the actual output.

```python
from strands_evals.evaluators import Evaluator
from strands_evals.types import EvaluationData, EvaluationOutput

class PolicyComplianceEvaluator(Evaluator[str, str]):
    def evaluate(self, evaluation_case: EvaluationData[str, str]) -> list[EvaluationOutput]:
        violations = self._check(evaluation_case.actual_output)
        if not violations:
            return [EvaluationOutput(score=1.0, test_pass=True, reason="compliant", label="compliant")]
        return [EvaluationOutput(
            score=0.0,
            test_pass=False,
            reason=f"violations: {', '.join(violations)}",
            label="non_compliant",
        )]

    def _check(self, response: str) -> list[str]: ...
```

--------------------------------

### Create Math Evaluation Suite

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_generators.md

Generates a new experiment suite for a math solver agent using tool descriptions. It includes expected output and trajectory evaluation, and saves the experiment to a file.

```python
import asyncio
from strands_evals.generators import ExperimentGenerator
from strands_evals.evaluators import TrajectoryEvaluator

async def create_math_eval():
    context = """
    Available tools:
    - calculator(expression: str) -> float: Evaluate mathematical expressions
    - validate_answer(answer: float) -> bool: Validate if answer is correct
    
    The agent should use tools to solve math problems and validate answers.
    """
    
    generator = ExperimentGenerator[str, str](
        input_type=str,
        output_type=str,
        include_expected_output=True,
        include_expected_trajectory=True
    )
    
    experiment = await generator.from_context_async(
        context=context,
        num_cases=15,
        evaluator=TrajectoryEvaluator,
        task_description="Math problem solver",
        num_topics=5
    )
    
    # Save for later use
    experiment.to_file("math_eval_suite", "json")
    return experiment

# Run generation
experiment = asyncio.run(create_math_eval())
print(f"Generated {len(experiment.cases)} test cases")
```

--------------------------------

### Avoided Log Formatting in Python

Source: https://github.com/strands-agents/evals/blob/main/STYLE_GUIDE.md

These examples demonstrate incorrect log formatting practices to avoid, such as direct variable interpolation in messages and inconsistent field/message separation.

```python
# Avoid: No structured fields, direct variable interpolation in message
logger.debug(f"User {user_id} performed action {action}")
```

```python
# Avoid: Inconsistent formatting, punctuation
logger.info("Request completed in %d ms.", duration)
```

```python
# Avoid: No separation between fields and message
logger.warning("Retry limit approaching! attempt=%d max_attempts=%d", attempt, max_attempts)
```

--------------------------------

### Saving and Loading Experiments

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Demonstrates how to persist an Experiment object to a file and subsequently load it back for later use or analysis.

```python
experiment.to_file("my_eval", "json")
loaded = Experiment.from_file("./experiment_files/my_eval.json", "json")
```

--------------------------------

### Running Integration Tests

Source: https://github.com/strands-agents/evals/blob/main/AGENTS.md

Execute integration tests using Hatch. This command requires credentials to be set via environment variables.

```bash
hatch run test-integ                 # Integration tests
```

--------------------------------

### Run Full Readiness Check

Source: https://github.com/strands-agents/evals/blob/main/AGENTS.md

Execute the full readiness check command to ensure code quality, including linting, formatting, and testing. This command should be run before opening a pull request.

```bash
hatch run prepare   # lint + format + test-lint + test --all
```

--------------------------------

### Get Failed Cases from Evaluation Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Retrieves a list of all cases that failed evaluation (where `test_pass` is False). This helps in identifying specific instances that require attention.

```python
failed = report.get_failed_cases()
for case in failed:
    print(f"Failed: {case['name']}")
    print(f"  Reason: {case['reason']}")
```

--------------------------------

### Run Display Evaluation Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Renders evaluation results in the terminal using Rich formatting. Use this to get a quick overview of results, metrics, and recommendations.

```python
report = experiment.run_evaluations(task_function)
report.run_display()
```

--------------------------------

### Multi-Turn Conversation with Telemetry

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

Manages multi-turn conversations by capturing spans for each turn and accumulating them. Spans are cleared at the start of each turn and then mapped to a session at the end of the conversation.

```python
from strands import Agent
from strands_evals import Case, Experiment, ActorSimulator
from strands_evals.evaluators import GoalSuccessRateEvaluator
from strands_evals.telemetry import StrandsEvalsTelemetry
from strands_evals.mappers import StrandsInMemorySessionMapper

telemetry = StrandsEvalsTelemetry().setup_in_memory_exporter()
mapper = StrandsInMemorySessionMapper()

def task_function(case: Case) -> dict:
    # Create simulator for multi-turn conversation
    simulator = ActorSimulator.from_case_for_user_simulator(
        case=case,
        max_turns=10
    )
    
    # Create agent
    agent = Agent(
        trace_attributes={
            "gen_ai.conversation.id": case.session_id,
            "session.id": case.session_id
        },
        callback_handler=None
    )
    
    # Run multi-turn conversation
    all_spans = []
    user_message = case.input
    
    while simulator.has_next():
        telemetry.clear()  # Reset for each turn
        
        agent_response = agent(user_message)
        turn_spans = telemetry.get_spans()
        all_spans.extend(turn_spans)
        
        user_result = simulator.act(str(agent_response))
        user_message = str(user_result.structured_output.message)
    
    # Map all spans to session
    session = mapper.map_to_session(all_spans, session_id=case.session_id)
    
    return {"output": str(agent_response), "trajectory": session}

```

--------------------------------

### ToolSimulator: Initialize

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_simulators.md

Initializes the ToolSimulator. This simulator is used for testing agent interactions with tools that require schema-validated responses and can optionally maintain shared state.

```python
simulator = ToolSimulator()
```

--------------------------------

### Custom Evaluator Implementation Example

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluators.md

Provides a template for creating a custom evaluator by extending the base Evaluator class. It includes a basic implementation of the evaluate and _calculate_match methods.

```python
from strands_evals.evaluators import Evaluator
from strands_evals.types.evaluation import EvaluationData, EvaluationOutput

class CustomEvaluator(Evaluator[str, str]):
    def __init__(self, threshold: float = 0.8, name: str | None = None):
        super().__init__(name=name)
        self.threshold = threshold
    
    def evaluate(self, evaluation_case: EvaluationData[str, str]) -> list[EvaluationOutput]:
        actual = evaluation_case.actual_output or ""
        expected = evaluation_case.expected_output or ""
        
        # Custom logic
        match_score = self._calculate_match(actual, expected)
        
        return [EvaluationOutput(
            score=match_score,
            test_pass=match_score >= self.threshold,
            reason=f"Match score: {match_score}",
            label="pass" if match_score >= self.threshold else "fail"
        )]
    
    def _calculate_match(self, actual: str, expected: str) -> float:
        # Implementation
        return 1.0 if actual == expected else 0.0
```

--------------------------------

### Experiment Initialization

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_experiment.md

Initializes an Experiment with optional test cases, evaluators, and diagnosis configuration. Use this to set up a new evaluation workflow.

```python
class Experiment(Generic[InputT, OutputT]):
    def __init__(
        self,
        cases: list[Case[InputT, OutputT]] | None = None,
        evaluators: list[Evaluator[InputT, OutputT]] | None = None,
        diagnosis_config: DiagnosisConfig | None = None,
    ) -> None
```

--------------------------------

### Initialize CloudWatchProvider with Agent Name

Source: https://github.com/strands-agents/evals/blob/main/src/strands_evals/providers/README.md

Instantiate CloudWatchProvider by discovering the log group from the agent name and specifying the region.

```python
from strands_evals.providers import CloudWatchProvider

# Option 2: Discover the log group from the agent name
provider = CloudWatchProvider(agent_name="my-agent", region="us-east-1")
```

--------------------------------

### Get OpenTelemetry Tracer

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_telemetry.md

Retrieves an OpenTelemetry tracer instance by name, allowing for manual span creation. Use this to instrument custom operations within your evaluation logic.

```python
from strands_evals.telemetry import get_tracer

tracer = get_tracer(__name__)

with tracer.start_as_current_span("custom_operation") as span:
    span.set_attribute("operation_type", "evaluation")
    # Do work
```

--------------------------------

### Experiment Serialization and Loading

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_experiment.md

Save an experiment configuration to a file (e.g., JSON) using `to_file` and later load it using `from_file` to rerun the experiment.

```python
# Save experiment with metadata
experiment.to_file("customer_service_eval", "json")

# Later, load and run
loaded_experiment = Experiment.from_file(
    "./experiment_files/customer_service_eval.json", "json"
)
report = loaded_experiment.run_evaluations(task_function)
```

--------------------------------

### Initialize and Use ActorSimulator

Source: https://github.com/strands-agents/evals/blob/main/SKILL.md

Simulates multi-turn user interactions for testing conversational agents. Pairs with GoalSuccessRateEvaluator for objective verification.

```python
from strands_evals import ActorSimulator  # UserSimulator is an alias

simulator = ActorSimulator.from_case_for_user_simulator(case=case, max_turns=10)

while simulator.has_next():
    memory_exporter.clear()
    agent_response = agent(user_message)
    turn_spans = list(memory_exporter.get_finished_spans())
    all_spans.extend(turn_spans)
    user_message = str(simulator.act(str(agent_response)).structured_output.message)
```

--------------------------------

### Handle RuntimeError: Model unavailable for diagnosis

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/errors.md

Provides examples of configuring DiagnosisConfig with valid and invalid model IDs. A RuntimeError can occur if the specified model is unavailable or credentials are invalid.

```python
# ❌ Fails if model service is down
diagnosis_config = DiagnosisConfig(
    model="invalid-model-id"
)

# ✅ Use valid model ID
diagnosis_config = DiagnosisConfig(
    model="global.anthropic.claude-haiku-4-5"
)
```

--------------------------------

### Configure Experiment for Failure Detection

Source: https://github.com/strands-agents/evals/blob/main/README.md

Set up an experiment to automatically detect failures and perform root cause analysis. Configure the diagnosis trigger and confidence threshold.

```python
from strands_evals import Case, DiagnosisConfig, Experiment
from strands_evals.evaluators import HelpfulnessEvaluator
from strands_evals.types.detector import ConfidenceLevel, DiagnosisTrigger

# Enable diagnosis on failing cases — runs failure detection then root cause analysis
experiment = Experiment(
    cases=test_cases,
    evaluators=[HelpfulnessEvaluator()],
    diagnosis_config=DiagnosisConfig(
        trigger=DiagnosisTrigger.ON_FAILURE,           # ON_FAILURE or ALWAYS
        confidence_threshold=ConfidenceLevel.MEDIUM,   # LOW, MEDIUM, or HIGH
    ),
)

report = experiment.run_evaluations(task_function)

# Display results with recommendations
report.display(include_recommendations=True)
```

--------------------------------

### Get Metrics Summary from Evaluation Report

Source: https://github.com/strands-agents/evals/blob/main/_autodocs/api_reference_evaluation_report.md

Retrieves a dictionary containing aggregate metrics such as pass rate and average score. Use this to quickly access key performance indicators.

```python
summary = report.get_metrics_summary()
print(f"Pass rate: {summary['pass_rate']}")
print(f"Average score: {summary['avg_score']}")
```