### EvalTournament Setup and Strategies

Source: https://context7.com/lars20070/assays/llms.txt

Demonstrates setting up an EvalTournament with multiple EvalPlayer objects and an EvalGame. It shows how to run the tournament using different strategies: adaptive uncertainty (default), random sampling, and round robin, with specific parameters for each.

```python
from assays.evaluators.bradleyterry import (
    EvalPlayer, EvalGame, EvalTournament,
    random_sampling_strategy,
    round_robin_strategy,
    adaptive_uncertainty_strategy,
)
from pydantic_ai import Agent
from pydantic_ai.settings import ModelSettings

# Create players from response items
players = [
    EvalPlayer(idx=0, item="Vanilla ice cream"),
    EvalPlayer(idx=1, item="Chocolate swirl with hazelnut"),
    EvalPlayer(idx=2, item="Pickled citrus ribbon"),
    EvalPlayer(idx=3, item="Classic strawberry"),
]

# Define evaluation game with criterion
game = EvalGame(criterion="Which ice cream flavor is more creative and unusual?")

# Create tournament
tournament = EvalTournament(players=players, game=game)

# Setup evaluation agent
eval_agent = Agent(
    model="openai:gpt-4o",
    output_type=Literal["A", "B"],
    system_prompt="Evaluate which answer is better.",
    retries=5,
)
model_settings = ModelSettings(temperature=0.0, timeout=300)

# Run tournament with different strategies
async def run_tournaments():
    # Adaptive uncertainty strategy (default, most efficient)
    scored_players = await tournament.run(
        agent=eval_agent,
        model_settings=model_settings,
        strategy=adaptive_uncertainty_strategy,
        max_standard_deviation=2.0,  # Convergence threshold
        alpha=0.1,  # Prior strength for Bradley-Terry
    )

    # Random sampling strategy
    scored_players = await tournament.run(
        agent=eval_agent,
        model_settings=model_settings,
        strategy=random_sampling_strategy,
        fraction_of_games=0.5,  # Play 50% of all possible games
    )

    # Round robin strategy
    scored_players = await tournament.run(
        agent=eval_agent,
        model_settings=model_settings,
        strategy=round_robin_strategy,
        number_of_rounds=3,  # Each player plays 3 rounds
    )

    # Access results
    for player in sorted(scored_players, key=lambda p: p.score or 0, reverse=True):
        print(f"Player {player.idx}: {player.item} (score: {player.score:.4f})")

```

--------------------------------

### Configure BradleyTerryEvaluator with OpenAI

Source: https://context7.com/lars20070/assays/llms.txt

Demonstrates configuring the BradleyTerryEvaluator with an OpenAI model and setting a convergence threshold based on standard deviation. This setup is useful for automated evaluation tasks where a specific model and stopping criterion are desired.

```python
evaluator = BradleyTerryEvaluator(
    model="openai:gpt-4o",
    criterion="Which response is more informative and well-structured?",
    max_standard_deviation=2.0,  # Stop when uncertainty drops below this threshold
)
```

--------------------------------

### Marking Tests with @pytest.mark.assay Decorator

Source: https://context7.com/lars20070/assays/llms.txt

The `@pytest.mark.assay` decorator marks a pytest function for AI agent evaluation. It accepts a `generator` for test cases and an `evaluator` for the comparison strategy. The decorator injects an `AssayContext` into the test function, providing the dataset, file path, and current mode. This example demonstrates setting up a test for search query generation.

```python
import pytest
from pydantic_ai import Agent
from pydantic_evals import Case, Dataset
from assays.plugin import PairwiseEvaluator
from assays.models import AssayContext

def generate_cases() -> Dataset:
    """Generator function that creates test cases."""
    cases = [
        Case(name="case_001", inputs={"topic": "quantum computing"}, expected_output=""),
        Case(name="case_002", inputs={"topic": "machine learning"}, expected_output=""),
        Case(name="case_003", inputs={"topic": "climate change"}, expected_output=""),
    ]
    return Dataset(cases=cases)

@pytest.mark.assay(
    generator=generate_cases,
    evaluator=PairwiseEvaluator(
        model="openai:gpt-4o",
        criterion="Which search query is more creative and insightful?",
    ),
)
@pytest.mark.asyncio
async def test_search_query_generation(context: AssayContext):
    """Test that evaluates search query generation quality."""
    agent = Agent(
        model="openai:gpt-4o",
        system_prompt="Generate a concise web search query for the given topic.",
    )

    for case in context.dataset.cases:
        async with agent:
            result = await agent.run(
                user_prompt=f"Generate a search query for: {case.inputs['topic']}"
            )
        # Agent responses are automatically captured by the plugin

# Run modes:
# uv run pytest test_file.py --assay-mode=new_baseline  # Record baseline responses
# uv run pytest test_file.py --assay-mode=evaluate      # Compare against baseline

```

--------------------------------

### Configuring PairwiseEvaluator for LLM Comparisons

Source: https://context7.com/lars20070/assays/llms.txt

The `PairwiseEvaluator` facilitates A/B comparisons of agent responses using an LLM judge. It can be configured with different LLM providers, including OpenAI models and local Ollama models. The `criterion` parameter guides the LLM in its comparison. This snippet shows instantiation with both OpenAI and Ollama.

```python
from assays.plugin import PairwiseEvaluator
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Using OpenAI model
evaluator_openai = PairwiseEvaluator(
    model="openai:gpt-4o",
    criterion="Which response demonstrates deeper understanding of the topic?",
)

# Using local Ollama model
ollama_model = OpenAIChatModel(
    model_name="qwen2.5:14b",
    provider=OpenAIProvider(base_url="http://localhost:11434/v1"),
)
evaluator_ollama = PairwiseEvaluator(
    model=ollama_model,
    criterion="Which of the two answers is more accurate and helpful?",
)

# Use in test decorator
@pytest.mark.assay(
    generator=generate_evaluation_cases, # Assuming generate_evaluation_cases is defined elsewhere
    evaluator=PairwiseEvaluator(
        model="openai:gpt-4o",
        criterion="Which search query shows more genuine curiosity and creativity?",
    ),
)
@pytest.mark.asyncio
async def test_query_creativity(context: AssayContext):
    # Test implementation...
    pass

# Readout output format (tests/assays/<module>/<test>.readout.json):
# {
#   "passed": true,
#   "details": {
#     "test_cases_count": 10,
#     "wins_baseline": [false, false, true, false, ...],
#     "wins_novel": [true, true, false, true, ...]
#   }
# }

```

--------------------------------

### Assays Framework Configuration with Pydantic-Settings

Source: https://context7.com/lars20070/assays/llms.txt

Shows how to configure the Assays framework using pydantic-settings, including accessing default values and overriding them via environment variables or .env files. It demonstrates setting up OpenAI-compatible models and providers.

```python
from assays.config import config
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
from assays.evaluator import PairwiseEvaluator # Assuming PairwiseEvaluator is imported correctly

# Access default configuration
print(config.ollama_base_url)  # "http://localhost:11434"
print(config.ollama_model)     # "qwen2.5:14b"
print(config.logfire_token)    # None (optional observability)

# Environment variables (.env file):
# OLLAMA_BASE_URL=http://localhost:11434
# OLLAMA_MODEL=qwen3:8b
# LOGFIRE_TOKEN=your-token-here

# Using configuration in custom evaluator setup
model = OpenAIChatModel(
    model_name=config.ollama_model,
    provider=OpenAIProvider(base_url=f"{config.ollama_base_url}/v1"),
)

# Or use string model identifiers for hosted providers
evaluator = PairwiseEvaluator(
    model="openai:gpt-4o",  # Uses OPENAI_API_KEY env var
    criterion="Which response is better?",
)

```

--------------------------------

### Generate Assays Baseline Data and Readout Reports

Source: https://github.com/lars20070/assays/blob/master/README.md

These bash commands illustrate how to use pytest with the Assays framework to generate baseline data and readout reports. The `new_baseline` mode creates `.json` files with baseline responses, while the `evaluate` mode generates `.readout.json` files containing the evaluation results.

```bash
uv run pytest tests/test_evaluation.py --assay-mode=new_baseline  # Generates baseline data test_query_generation.json
```

```bash
uv run pytest tests/test_evaluation.py --assay-mode=evaluate      # Generates readout report test_query_generation.readout.json
```

--------------------------------

### Configure BradleyTerryEvaluator with Local Model

Source: https://context7.com/lars20070/assays/llms.txt

Shows how to configure the BradleyTerryEvaluator using a local OpenAI-compatible model. This involves specifying the model name and the base URL for the local provider, along with a custom criterion and a potentially different convergence threshold.

```python
local_model = OpenAIChatModel(
    model_name="qwen2.5:14b",
    provider=OpenAIProvider(base_url="http://localhost:11434/v1"),
)
evaluator_local = BradleyTerryEvaluator(
    model=local_model,
    criterion="Which answer better addresses the user's question?",
    max_standard_deviation=2.1,  # Higher threshold for faster convergence
)
```

--------------------------------

### Pytest Assay Test with Context and Dataset Access

Source: https://context7.com/lars20070/assays/llms.txt

Demonstrates how to use the AssayContext injected by the pytest plugin to access datasets, iterate through test cases, and check the assay mode. It shows how to retrieve inputs, expected outputs, case names, and the file path for the dataset.

```python
import pytest
from pathlib import Path
from assays.context import AssayContext
from assays.dataset import Dataset

# Assuming my_generator and my_evaluator are defined elsewhere
# from your_module import my_generator, my_evaluator

@pytest.mark.assay(generator=my_generator, evaluator=my_evaluator)
@pytest.mark.asyncio
async def test_example(context: AssayContext):
    # Access the evaluation dataset
    dataset: Dataset = context.dataset

    # Iterate over test cases
    for case in dataset.cases:
        inputs = case.inputs          # Dict with input parameters
        expected = case.expected_output  # Baseline response (if exists)
        name = case.name              # Case identifier

        print(f"Processing {name}: {inputs}")

    # Check current mode
    if context.assay_mode == "new_baseline":
        print("Recording new baseline responses")
    elif context.assay_mode == "evaluate":
        print("Evaluating against baseline")

    # File path where dataset is stored
    path: Path = context.path  # e.g., tests/assays/test_module/test_name.json

```

--------------------------------

### Evaluate Pydantic AI Agent Responses with Assays

Source: https://github.com/lars20070/assays/blob/master/README.md

This Python code snippet demonstrates how to use the Assays framework to evaluate an AI agent. It defines a test function decorated with `@pytest.mark.assay` which specifies a data generator and an evaluator. The agent is then used to process inputs from the dataset, and its responses are captured for evaluation against baseline data.

```python
def generate_evaluation_cases():
    ...
    return Dataset(cases)


@pytest.mark.assay(
    generator=generate_evaluation_cases,
    evaluator=PairwiseEvaluator(
        model="openai:gpt-4o",
        criterion="Which of the two search queries shows more genuine curiosity and creativity?",
    ),
)
@pytest.வதும்.mark.asyncio
async def test_query_generation(context: AssayContext):

    agent = Agent(
        model="openai:gpt-4o",
        system_prompt="Generate a concise web search query for the given research topic.",
    )

    for case in context.dataset.cases:
        async with agent:
            result = await agent.run(user_prompt=f"Generate a search query for the following topic: <TOPIC>{case.inputs['topic']}</TOPIC>")
```

--------------------------------

### Create and Serialize Custom Readout Report

Source: https://context7.com/lars20070/assays/llms.txt

Illustrates how to create a custom Readout object with specific test results and details, and then serialize it to a JSON file. This is useful for custom evaluation metrics or reporting.

```python
from pathlib import Path
from assays.readout import Readout

def create_custom_readout() -> Readout:
    readout = Readout(
        passed=True,
        details={
            "test_cases_count": 10,
            "accuracy": 0.95,
            "average_score": 3.7,
            "failed_cases": ["case_003", "case_007"],
        },
    )

    # Serialize to file
    readout.to_file(Path("tests/assays/module/test.readout.json"))

    return readout

# Example of the JSON output:
# {
#   "passed": true,
#   "details": {
#     "test_cases_count": 10,
#     "accuracy": 0.95,
#     "average_score": 3.7,
#     "failed_cases": ["case_003", "case_007"]
#   }
# }

```

--------------------------------

### AssayContext and Readout Models

Source: https://context7.com/lars20070/assays/llms.txt

Introduces the AssayContext and Readout models. AssayContext provides test functions with evaluation data and file paths, while Readout structures the evaluation results, including pass/fail status and detailed metrics, for JSON serialization.

```python
from assays.models import AssayContext, Readout
from pydantic_evals import Dataset, Case
from pathlib import Path
```

--------------------------------

### Initializing BradleyTerryEvaluator for Ranked Comparisons

Source: https://context7.com/lars20070/assays/llms.txt

The `BradleyTerryEvaluator` employs the Bradley-Terry statistical model for tournament-style pairwise comparisons, enabling scoring and ranking of responses. It utilizes an adaptive uncertainty strategy to optimize the selection of pairs for comparison, balancing efficiency and statistical rigor. This snippet shows the import statement for its initialization.

```python
from assays.plugin import BradleyTerryEvaluator
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider


```

--------------------------------

### Pytest Assay with BradleyTerryEvaluator

Source: https://context7.com/lars20070/assays/llms.txt

Illustrates how to use the `@pytest.mark.assay` decorator to integrate a BradleyTerryEvaluator into a pytest test function. This allows for automated evaluation of test cases generated by a specified generator, using a configured evaluator.

```python
@pytest.mark.assay(
    generator=generate_evaluation_cases,
    evaluator=BradleyTerryEvaluator(
        model="openai:gpt-4o",
        criterion="Which search query is less formulaic and more creative?",
        max_standard deviation=2.0,
    ),
)
@pytest.mark.asyncio
async def test_query_ranking(context: AssayContext):
    # Test implementation...
    pass
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.