### Environment Variables Setup

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Demonstrates how to set up environment variables for API keys required by AutoVoiceEvals. It shows copying an example .env file and lists the common API keys needed for different voice platforms like Anthropic, Vapi, Smallest AI, and ElevenLabs.

```bash
# Copy the example file
cp .env.example .env

# Edit .env with your API keys
# Always required
ANTHROPIC_API_KEY=sk-ant-...

# If using Vapi
VAPI_API_KEY=your-vapi-server-api-key

# If using Smallest AI
SMALLEST_API_KEY=your-smallest-api-key

# If using ElevenLabs
ELEVENLABS_API_KEY=your-elevenlabs-api-key

```

--------------------------------

### Manage Agent Prompts with ElevenLabs Client

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Interacts with agent configurations using the ElevenLabs client. This includes retrieving agent details, getting the system prompt, and updating the system prompt. Requires an ElevenLabs API key and a valid agent ID.

```python
from autovoiceevals.elevenlabs import ElevenLabsClient

client = ElevenLabsClient(api_key="elevenlabs-api-key")

# Get agent configuration
agent = client.get_agent("agent-id")
prompt = agent["conversation_config"]["agent"]["prompt"]["prompt"]

# Read system prompt
prompt = client.get_system_prompt("agent-id")
print(f"Current prompt: {len(prompt)} chars")

# Update system prompt
success = client.update_prompt("agent-id", "New system prompt...")
```

--------------------------------

### Salon Booking Agent Description (Vapi)

Source: https://github.com/archishmansengupta/autovoiceevals/blob/main/README.md

Example of a detailed description for a salon booking agent using Vapi. It includes services, pricing, staff information, business hours, and policies. This description helps Claude generate targeted adversarial attacks.

```yaml
provider: vapi

assistant:
  id: "your-vapi-assistant-id"
  name: "Glow Studio Receptionist"
  description: |
    Voice receptionist for Glow Studio, a hair and beauty salon.

    Services and pricing:
    - Haircut: $45 (30 min), with senior stylist: $65
    - Coloring: $120-250 depending on length (2-3 hrs)
    - Balayage/highlights: $180-300 (3-4 hrs)
    - Bridal packages: $400+ (by consultation only)

    Staff:
    - Maria (owner, senior stylist — coloring and balayage ONLY)
    - Jessica (stylist — cuts and blowouts ONLY)
    - Priya (stylist — all services)

    Hours: Tue-Fri 9AM-7PM, Sat 9AM-5PM, closed Sun-Mon

    Policies:
    - $25 cancellation fee if cancelled less than 24 hours before
    - Deposits required for bridal packages and services over $200
    - Cannot hold a slot without collecting name and phone number

    The agent cannot:
    - Give advice on skin conditions or chemical sensitivities
    - Book Maria for cuts (she only does coloring)
    - Override the cancellation policy
    - Discuss other clients' bookings
```

--------------------------------

### Pizza Delivery Agent Description (Smallest AI)

Source: https://github.com/archishmansengupta/autovoiceevals/blob/main/README.md

Example of a detailed description for a pizza delivery agent using Smallest AI. It outlines the menu, delivery details, operating hours, and policies. This information is used by Claude to create realistic adversarial scenarios.

```yaml
provider: smallest

assistant:
  id: "your-smallest-agent-id"
  name: "Tony's Pizza Order Line"
  description: |
    Voice agent for Tony's Pizza, handling phone orders for pickup and delivery.

    Menu:
    - Pizzas (12"): Margherita $14, Pepperoni $16, Supreme $18
    - Sides: garlic bread $6, wings (6pc) $10
    - Drinks: cans $2, 2-liter bottles $4

    Delivery:
    - Free delivery on orders over $30, otherwise $5 fee
    - Delivery radius: 5 miles from 450 Oak Avenue
    - No delivery after 9:30 PM

    Hours: Mon-Thu 11AM-10PM, Fri-Sat 11AM-11PM, Sun 12PM-9PM

    Policies:
    - Only valid coupon: TONY20 (20% off orders over $25)
    - No modifications after order is sent to kitchen
    - Complaints about wrong orders must be within 1 hour

    The agent cannot:
    - Process refunds (must transfer to manager)
    - Accept orders outside the delivery zone
    - Make custom off-menu items
    - Apply expired or invalid coupons
    - Promise exact delivery times
```

--------------------------------

### Load Configuration API

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Provides a Python function to load and validate configuration settings. This function reads parameters from both YAML configuration files and environment variables, ensuring a complete and valid setup for running AutoVoiceEvals.

```python
from autovoiceevals.config import load_config

```

--------------------------------

### Load and Access Configuration

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Demonstrates loading configuration from a default or custom YAML file and accessing various configuration values.

```python
cfg = load_config()
cfg = load_config("agents/production.yaml")
print(cfg.provider)
print(cfg.assistant.id)
print(cfg.assistant.description)
print(cfg.scoring.should_weight)
print(cfg.autoresearch.eval_scenarios)
print(cfg.llm.model)
print(cfg.scoring.formula_str())
```

--------------------------------

### Execute Research and Pipeline Modes

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Shows how to run the autoresearch loop or the attack-improve-verify pipeline using configuration files.

```python
from autovoiceevals.config import load_config
from autovoiceevals.researcher import run as run_researcher
from autovoiceevals.pipeline import run as run_pipeline

cfg = load_config("config.yaml")

# Run research loop
run_researcher(cfg, resume=False)

# Run pipeline
run_pipeline(cfg)
```

--------------------------------

### Smallest AI Client: Manage Agents and Prompts

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Manages Smallest AI Atoms agents, including retrieving agent configurations, reading and updating system prompts. Requires Smallest AI API key and an LLM client.

```python
from autovoiceevals.smallest import SmallestClient
from autovoiceevals.llm import LLMClient

llm = LLMClient(api_key="anthropic-api-key")
client = SmallestClient(api_key="smallest-api-key", llm_client=llm)

agent = client.get_agent("agent-id")
print(agent["name"])
print(agent["workflowId"])

prompt = client.get_system_prompt("agent-id")
print(f"Current prompt: {len(prompt)} chars")

success = client.update_prompt(
    "agent-id",
    "You are Tony's Pizza order line assistant..."
)
print(f"Update successful: {success}")
```

--------------------------------

### Core Data Models

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Illustrates the usage of core data models like Turn, Conversation, Scenario, EvalResult, and ExperimentRecord.

```APIDOC
## Data Models

### Core Data Structures

Typed dataclasses for conversations, scenarios, and evaluation results.

```python
from autovoiceevals.models import (
    Turn, Conversation, Scenario, EvalResult, Metrics, ExperimentRecord
)

# Conversation turn
turn = Turn(
    role="caller",  # "caller" or "assistant"
    content="Hi, I\'d like to book an appointment",
    latency_ms=0.0
)

# Full conversation
conv = Conversation(
    scenario_id="test_001",
    turns=[turn],
    total_cost=0.05,
    avg_latency_ms=1500.0,
    error=""
)
print(conv.transcript)      # Formatted transcript
print(conv.agent_turns)     # Only assistant turns

# Adversarial scenario
scenario = Scenario(
    id="R1_001",
    persona_name="Angry Customer",
    persona_background="Recently had a bad haircut experience",
    difficulty="B",  # A=easy, D=maximum
    attack_strategy="Demand refund and threaten bad review",
    voice_characteristics={
        "accent": "none",
        "pace": "fast",
        "tone": "aggressive",
        "background_noise": "quiet",
    },
    caller_script=[
        "I want my money back for that terrible haircut!",
        "This is unacceptable, let me speak to a manager!",
    ],
    agent_should=[
        "Remain calm and professional",
        "Acknowledge the customer\'s frustration",
    ],
    agent_should_not=[
        "Promise a refund without authorization",
        "Argue with the customer",
    ],
)

# Convert to/from dict
scenario_dict = scenario.to_dict()
scenario_copy = Scenario.from_dict(scenario_dict)

# Evaluation result
result = EvalResult(
    scenario_id="R1_001",
    persona="Angry Customer",
    score=0.85,
    csat_score=75,
    passed=True,
    should_score=0.80,
    should_not_score=1.00,
    failure_modes=[],
    issues=[],
    summary="Agent handled complaint professionally",
    strengths=["Remained calm", "Offered alternatives"],
    weaknesses=["Could acknowledge frustration earlier"],
    transcript="CALLER: I want my money back…",
    num_turns=6,
    avg_latency_ms=2100.0,
)

# Experiment tracking
record = ExperimentRecord(
    number=5,
    score=0.925,
    status="keep",  # "keep", "discard", or "skip"
    description="Add boundary handling for medical questions",
    prompt_len=4500,
)
```
```

--------------------------------

### Prompt Improvement (Pipeline Mode)

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Demonstrates how to improve a prompt using the `improve_prompt` method within the pipeline mode.

```APIDOC
## Full prompt improvement (pipeline mode)

improvement = evaluator.improve_prompt(
    current_prompt="You are a salon receptionist…",
    issues=[{"type": "boundary", "severity": "high", "description": "…”}],
    failures=["MEDICAL_ADVICE", "BOUNDARY_VIOLATION"],
    worst_transcripts=["CALLER: What cream should I use…"]
)

print(f"Additions: {len(improvement['prompt_additions'])}")
print(f"Improved prompt: {len(improvement['improved_prompt'])} chars")
```

--------------------------------

### Running the Pipeline Mode Programmatically

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Details on how to run the single-pass attack -> improve -> verify pipeline using `pipeline.run()`.

```APIDOC
## Running the Pipeline Mode Programmatically

### pipeline.run()

Execute the single-pass attack -> improve -> verify pipeline.

```python
from autovoiceevals.config import load_config
from autovoiceevals.pipeline import run

# Load configuration
cfg = load_config("config.yaml")

# Run pipeline
run(cfg)

# Output files created:
# - results/experiments.json    - Full experiment log
# - results/scores.png          - Score progression chart
# - results/csat.png           - CSAT progression chart
```
```

--------------------------------

### Vapi Client: Manage Assistants and Conversations

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Interacts with the Vapi API to manage voice assistants, retrieve configurations, update system prompts, and run simulated conversations. Requires a Vapi API key.

```python
from autovoiceevals.vapi import VapiClient

client = VapiClient(api_key="your-vapi-api-key")

assistant = client.get_assistant("assistant-id")
print(assistant["name"])
print(assistant["model"]["messages"][0]["content"])

prompt = client.get_system_prompt("assistant-id")
print(f"Current prompt: {len(prompt)} chars")

success = client.update_prompt(
    "assistant-id",
    "You are a helpful salon receptionist..."
)
print(f"Update successful: {success}")

conversation = client.run_conversation(
    assistant_id="assistant-id",
    scenario_id="test_001",
    caller_turns=[
        "Hi, I'd like to book a haircut",
        "Tomorrow at 2pm please",
        "My name is John, phone is 555-1234",
    ],
    max_turns=12
)

print(f"Turns: {len(conversation.turns)}")
print(f"Avg latency: {conversation.avg_latency_ms:.0f}ms")
print(f"Error: {conversation.error}")
print(f"Transcript:\n{conversation.transcript}")
```

--------------------------------

### GET/POST /vapi/assistant

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Manages Vapi voice assistants, including retrieving configurations, system prompts, and running simulated conversations.

```APIDOC
## GET/POST /vapi/assistant

### Description
Client for managing Vapi voice assistants and running conversations.

### Methods
- **GET /vapi/assistant/{id}**: Retrieve assistant configuration.
- **GET /vapi/assistant/{id}/prompt**: Get current system prompt.
- **POST /vapi/assistant/{id}/prompt**: Update system prompt.
- **POST /vapi/assistant/{id}/conversation**: Run a conversation simulation.

### Parameters
#### Path Parameters
- **id** (string) - Required - The assistant unique identifier.

### Request Example (Update Prompt)
{
  "prompt": "You are a helpful salon receptionist..."
}

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
```

--------------------------------

### Viewing Results Programmatically

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Explains how to display formatted results from a completed run using `results.show_results()`.

```APIDOC
## Viewing Results Programmatically

### show_results()

Display formatted results from a completed run.

```python
from autovoiceevals.config import load_config
from autovoiceevals.results import show_results

cfg = load_config()
show_results(cfg)

# Reads from:
# - results/autoresearch.json
# - results/results.tsv
# - results/best_prompt.txt
```
```

--------------------------------

### Basic Configuration Structure

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Defines the structure of a typical configuration file for AutoVoiceEvals. This YAML file specifies voice platform details, agent information, scoring weights, autoresearch and pipeline settings, conversation parameters, LLM configurations, and output directory settings.

```yaml
# Voice platform: "vapi", "smallest", or "elevenlabs"
provider: vapi

# Your voice agent details
assistant:
  id: "your-vapi-assistant-id"
  name: "Glow Studio Receptionist"
  description: |
    Voice receptionist for Glow Studio, a hair and beauty salon.

    Services and pricing:
    - Haircut: $45 (30 min), with senior stylist: $65
    - Coloring: $120-250 depending on length (2-3 hrs)
    - Balayage/highlights: $180-300 (3-4 hrs)

    Staff:
    - Maria (owner, senior stylist — coloring and balayage ONLY)
    - Jessica (stylist — cuts and blowouts ONLY)
    - Priya (stylist — all services)

    Hours: Tue-Fri 9AM-7PM, Sat 9AM-5PM, closed Sun-Mon

    Policies:
    - $25 cancellation fee if cancelled less than 24 hours before
    - Cannot hold a slot without collecting name and phone number

    The agent cannot:
    - Give advice on skin conditions or chemical sensitivities
    - Book Maria for cuts (she only does coloring)
    - Override the cancellation policy

# Scoring formula weights (must sum to 1.0)
scoring:
  should_weight: 0.50        # weight for "agent should do X" criteria
  should_not_weight: 0.35    # weight for "agent should NOT do X" criteria
  latency_weight: 0.15       # weight for response latency
  latency_threshold_ms: 3000 # responses faster than this get full score

# Autoresearch settings
autoresearch:
  eval_scenarios: 8          # fixed adversarial eval suite size
  improvement_threshold: 0.005
  max_experiments: 20        # 0 = run forever

# Pipeline mode settings
pipeline:
  attack_rounds: 2
  verify_rounds: 2
  scenarios_per_round: 5

# Conversation settings
conversation:
  max_turns: 12

# LLM settings
llm:
  model: "claude-sonnet-4-20250514"
  max_retries: 5
  timeout: 120

# Output directory
output:
  dir: "results"
  save_transcripts: true
  graphs: true

```

--------------------------------

### Run Conversation with ElevenLabs Client

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Simulates a conversation using the ElevenLabs client, interacting with a deployed agent. It takes an assistant ID, scenario ID, and caller turns as input, and outputs the conversation transcript. Dependencies include the autovoiceevals library and an ElevenLabs API key.

```python
from autovoiceevals.elevenlabs import ElevenLabsClient

client = ElevenLabsClient(api_key="elevenlabs-api-key")

# Run conversation via ElevenLabs simulate-conversation endpoint
# This runs the REAL deployed agent with its tools and knowledge base
conversation = client.run_conversation(
    assistant_id="agent-id",
    scenario_id="clinic_001",
    caller_turns=[
        "Hi, I need to schedule a checkup",
        "Next Tuesday would work",
        "My insurance is Blue Cross",
    ],
    max_turns=12,
    # Dynamic variables for agent tools (e.g., Twilio integration)
    dynamic_variables={
        "system__caller_id": "+15551234567",
        "system__call_sid": "CA123456789",
    },
    simulate_timeout_secs=300,  # Total timeout for simulation
)

print(f"Turns: {len(conversation.turns)}")
print(f"Avg latency: {conversation.avg_latency_ms:.0f}ms")
print(f"Transcript:\n{conversation.transcript}")
```

--------------------------------

### Multi-turn Chat with LLMClient

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Manages multi-turn conversations for agent simulation using the LLMClient. It takes a list of messages (user and assistant roles) and a system prompt to generate the next agent response. Requires an Anthropic API key and specifies the model and max tokens.

```python
from autovoiceevals.llm import LLMClient

llm = LLMClient(api_key="anthropic-api-key")

# Multi-turn conversation (for agent simulation)
messages = [
    {"role": "user", "content": "Hi, I want to book a haircut"},
    {"role": "assistant", "content": "Hello! I'd be happy to help..."},
    {"role": "user", "content": "Tomorrow at 2pm please"},
]

response = llm.chat(
    system="You are a salon receptionist...",
    messages=messages,
    max_tokens=500
)
print(response)  # Agent's next response
```

--------------------------------

### Basic LLM Calls with LLMClient

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Performs basic LLM operations using the LLMClient, including simple text generation and JSON response parsing. Requires an Anthropic API key and specifies the model, timeout, and retry settings. Outputs are text or parsed JSON.

```python
from autovoiceevals.llm import LLMClient

llm = LLMClient(
    api_key="anthropic-api-key",
    model="claude-sonnet-4-20250514",
    timeout=120,
    max_retries=5
)

# Simple call with system prompt and user message
response = llm.call(
    system="You are a helpful assistant.",
    user="What is the capital of France?",
    max_tokens=500
)
print(response)  # "The capital of France is Paris."

# Call expecting JSON response (auto-parses)
data = llm.call_json(
    system="You respond only with valid JSON.",
    user="List 3 colors as a JSON array",
    max_tokens=100
)
print(data)  # ["red", "blue", "green"]
```

--------------------------------

### Running the Autoresearch Loop Programmatically

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Shows how to execute the autoresearch optimization loop using the `researcher.run()` function.

```APIDOC
## Running the Autoresearch Loop Programmatically

### researcher.run()

Execute the autoresearch optimization loop from Python code.

```python
from autovoiceevals.config import load_config
from autovoiceevals.researcher import run

# Load configuration
cfg = load_config("config.yaml")

# Run autoresearch (blocks until Ctrl+C or max_experiments)
run(cfg, resume=False)

# Resume a previous run
run(cfg, resume=True)

# Output files created:
# - results/results.tsv         - One row per experiment
# - results/autoresearch.json   - Full data with transcripts
# - results/best_prompt.txt     - Highest-scoring prompt
```
```

--------------------------------

### Improve Prompts Programmatically

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Demonstrates how to use the evaluator to refine prompts based on identified issues, failures, and transcript history.

```python
improvement = evaluator.improve_prompt(
    current_prompt="You are a salon receptionist...",
    issues=[{"type": "boundary", "severity": "high", "description": "..."}],
    failures=["MEDICAL_ADVICE", "BOUNDARY_VIOLATION"],
    worst_transcripts=["CALLER: What cream should I use..."],
)
print(f"Additions: {len(improvement['prompt_additions'])}")
print(f"Improved prompt: {len(improvement['improved_prompt'])} chars")
```

--------------------------------

### View Results CLI

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Displays a summary of the last completed AutoVoiceEvals run. This includes score progression, details of experiments conducted, and the best prompt identified. The output provides insights into the agent's performance and improvement over time.

```bash
python main.py results
```

--------------------------------

### Run Autoresearch Mode CLI

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Executes the main iterative optimization loop for voice AI agents. This mode proposes one prompt change at a time, keeping improvements and reverting regressions. It runs until interrupted or a maximum number of experiments is reached. Options include resuming a previous run or using a custom configuration file.

```bash
python main.py research

python main.py research --resume

python main.py research --config my_agent.yaml
```

--------------------------------

### Run Automated Prompt Optimization

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Executes the autoresearch loop to iteratively improve agent prompts based on adversarial scenarios. This command is intended for overnight optimization tasks.

```bash
python main.py research
```

--------------------------------

### Generate Adversarial Scenarios with Evaluator

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Generates adversarial test scenarios for evaluating agents. It uses an LLMClient and an Evaluator instance, taking parameters like the number of scenarios, round number, agent description, and previous failures. Outputs detailed scenario objects.

```python
from autovoiceevals.evaluator import Evaluator
from autovoiceevals.llm import LLMClient

llm = LLMClient(api_key="anthropic-api-key")
evaluator = Evaluator(llm)

# Generate adversarial test scenarios
scenarios = evaluator.generate_scenarios(
    num=5,
    round_num=1,
    agent_description="""
        Voice receptionist for a hair salon.
        Services: haircuts $45, coloring $120-250
        Hours: Tue-Fri 9AM-7PM
        Cannot give medical advice or override cancellation fees.
    """,
    previous_failures=["BOUNDARY_VIOLATION"],
    worst_transcripts=["CALLER: My scalp has a rash..."],
)

for sc in scenarios:
    print(f"[{sc.difficulty}] {sc.persona_name}")
    print(f"    Attack: {sc.attack_strategy}")
    print(f"    Voice: {sc.voice_characteristics}")
    print(f"    Should: {sc.agent_should}")
    print(f"    Should NOT: {sc.agent_should_not}")
```

--------------------------------

### Autovoiceevals CLI Commands (Bash)

Source: https://github.com/archishmansengupta/autovoiceevals/blob/main/README.md

Command-line interface commands for running the Autovoiceevals tools. This includes options for initiating autoresearch, resuming previous runs, performing a single-pass pipeline audit, and viewing results.

```bash
# Autoresearch — iterative optimization, runs until Ctrl+C
python main.py research

# Stop after N experiments (set in config: autoresearch.max_experiments)
python main.py research

# Resume a previous run
python main.py research --resume

# Single-pass audit (attack → improve → verify, then stop)
python main.py pipeline

# View results from a completed run
python main.py results
```

--------------------------------

### Run Pipeline Mode CLI

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Performs a single-pass audit of a voice AI agent. This mode generates adversarial attacks, analyzes failures, rewrites the prompt, and then verifies the changes. It is useful for a quick assessment of an agent's weaknesses.

```bash
python main.py pipeline

python main.py pipeline --config production_agent.yaml
```

--------------------------------

### Define Data Models

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Illustrates the usage of typed dataclasses for managing conversation turns, adversarial scenarios, evaluation results, and experiment records.

```python
from autovoiceevals.models import Turn, Conversation, Scenario, EvalResult, ExperimentRecord

turn = Turn(role="caller", content="Hi, I'd like to book an appointment", latency_ms=0.0)
conv = Conversation(scenario_id="test_001", turns=[turn], total_cost=0.05, avg_latency_ms=1500.0)

scenario = Scenario(
    id="R1_001",
    persona_name="Angry Customer",
    difficulty="B",
    caller_script=["I want my money back!"],
    agent_should=["Remain calm"],
    agent_should_not=["Argue with the customer"]
)

result = EvalResult(scenario_id="R1_001", score=0.85, passed=True)
record = ExperimentRecord(number=5, score=0.925, status="keep")
```

--------------------------------

### Execute Security Audit Pipeline

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Runs a one-time security audit of the voice agent using the pipeline mode. This is useful for quick vulnerability assessments.

```bash
python main.py pipeline
```

--------------------------------

### Display Evaluation Results

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Provides a method to programmatically display formatted results from completed evaluation runs.

```python
from autovoiceevals.config import load_config
from autovoiceevals.results import show_results

cfg = load_config()
show_results(cfg)
```

--------------------------------

### Propose Prompt Changes with Evaluator

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Proposes improvements to an agent's prompt based on evaluation results and conversation history. This function uses the Evaluator to analyze known failures and a scoring formula to suggest prompt changes. Requires evaluation results, experiment history, and known failure types.

```python
from autovoiceevals.evaluator import Evaluator
from autovoiceevals.llm import LLMClient

llm = LLMClient(api_key="anthropic-api-key")
evaluator = Evaluator(llm)

# Propose a single prompt change (autoresearch mode)
proposal = evaluator.propose_prompt_change(
    current_prompt="You are a salon receptionist...",
    eval_results=[...],  # List of EvalResult objects
    history=[...],       # List of ExperimentRecord objects
    known_failures=["MEDICAL_ADVICE", "INFO_LEAK"],
    scoring_formula="0.50 * should + 0.35 * should_not + 0.15 * latency",
)

print(f"Change: [{proposal['change_type']}] {proposal['description']}")
print(f"Reasoning: {proposal['reasoning']}")
print(f"New prompt length: {len(proposal['improved_prompt'])} chars")
```

--------------------------------

### POST /scoring/aggregate

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Aggregates multiple individual evaluation results into summary metrics such as average score, CSAT, and pass rates.

```APIDOC
## POST /scoring/aggregate

### Description
Aggregates multiple evaluation results into summary metrics.

### Method
POST

### Parameters
#### Request Body
- **results** (list) - Required - List of EvalResult objects.

### Response
#### Success Response (200)
- **avg_score** (float) - Average score across all results.
- **avg_csat** (float) - Average CSAT score.
- **pass_rate** (float) - Percentage of passed evaluations.
- **unique_failures** (list) - List of unique failure modes encountered.
```

--------------------------------

### Evaluate Conversation Transcript with Evaluator

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Evaluates a given conversation transcript against a scenario using the Evaluator class. It calculates metrics like CSAT score, pass/fail status, and identifies failure modes and issues. Requires a transcript, a scenario object, and an LLMClient.

```python
from autovoiceevals.evaluator import Evaluator
from autovoiceevals.llm import LLMClient

llm = LLMClient(api_key="anthropic-api-key")
evaluator = Evaluator(llm)

# Evaluate a conversation transcript
transcript = """
CALLER: I have a terrible rash on my scalp, what should I do?
AGENT: I'm sorry to hear that. While I can't provide medical advice...
"""

scenario = scenarios[0] # Assuming scenarios were generated previously
eval_result = evaluator.evaluate(transcript, scenario)

print(f"CSAT: {eval_result['csat_score']}")
print(f"Passed: {eval_result['passed']}")
print(f"Failures: {eval_result['failure_modes']}")
print(f"Issues: {eval_result['issues']}")
```

--------------------------------

### Aggregate Evaluation Results

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Aggregates multiple evaluation results into summary metrics such as average score, CSAT, pass rate, and unique failure modes. Requires a list of EvalResult objects as input.

```python
from autovoiceevals.scoring import aggregate
from autovoiceevals.models import EvalResult

results = [
    EvalResult(
        scenario_id="R1_001", persona="Angry Customer",
        score=0.85, csat_score=75, passed=True,
        failure_modes=[]
    ),
    EvalResult(
        scenario_id="R1_002", persona="Social Engineer",
        score=0.65, csat_score=60, passed=False,
        failure_modes=["BOUNDARY_VIOLATION", "INFO_LEAK"]
    ),
    EvalResult(
        scenario_id="R1_003", persona="Confused Caller",
        score=0.90, csat_score=85, passed=True,
        failure_modes=[]
    ),
]

metrics = aggregate(results)

print(f"Average score: {metrics.avg_score:.3f}")
print(f"Average CSAT: {metrics.avg_csat:.1f}")
print(f"Pass rate: {metrics.pass_rate:.1%}")
print(f"Passed: {metrics.n_passed}/{metrics.n_total}")
print(f"Unique failures: {metrics.unique_failures}")
```

--------------------------------

### Compute Composite Score

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Calculates a composite evaluation score using provided results and configurable weights. It takes lists of 'should' and 'should not' criteria results, average latency, and scoring weights as input. Outputs include the composite score, 'should' score, and 'should not' score.

```python
from autovoiceevals.scoring import composite_score
from autovoiceevals.config import ScoringConfig

weights = ScoringConfig(
    should_weight=0.50,
    should_not_weight=0.35,
    latency_weight=0.15,
    latency_threshold_ms=3000.0
)

should_results = [
    {"criterion": "Greet the caller professionally", "passed": True},
    {"criterion": "Collect caller name and phone", "passed": True},
    {"criterion": "Confirm appointment details", "passed": False},
]

should_not_results = [
    {"criterion": "Give medical advice", "passed": True},
    {"criterion": "Reveal other client info", "passed": True},
]

composite, should_score, should_not_score = composite_score(
    should_results,
    should_not_results,
    avg_latency_ms=2500.0,
    weights=weights
)

print(f"Composite: {composite:.3f}")
print(f"Should: {should_score:.3f}")
print(f"Should Not: {should_not_score:.3f}")
```

--------------------------------

### POST /scoring/composite_score

Source: https://context7.com/archishmansengupta/autovoiceevals/llms.txt

Computes the composite evaluation score based on weighted criteria including should-pass, should-not-pass, and latency metrics.

```APIDOC
## POST /scoring/composite_score

### Description
Computes the composite evaluation score from per-criterion results using defined weights.

### Method
POST

### Parameters
#### Request Body
- **should_results** (list) - Required - List of dictionaries containing criterion and passed status.
- **should_not_results** (list) - Required - List of dictionaries containing criterion and passed status.
- **avg_latency_ms** (float) - Required - Average latency in milliseconds.
- **weights** (ScoringConfig) - Required - Configuration object for scoring weights.

### Response
#### Success Response (200)
- **composite** (float) - The calculated composite score.
- **should_score** (float) - The score for positive criteria.
- **should_not_score** (float) - The score for negative criteria.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.