### Install Autoevals (Bash)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Installs the autoevals library for either TypeScript/Node.js or Python environments.

```bash
# TypeScript/Node
npm install autoevals

# Python
pip install autoevals
```

--------------------------------

### Set Up Python Development Environment

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Sets up a Python virtual environment (.venv) with development and scipy extras, and installs pre-commit hooks.

```bash
make develop
```

--------------------------------

### Install Autoevals with pip

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

Install the Autoevals package using pip for Python projects.

```bash
pip install autoevals
```

--------------------------------

### Init and Configuration

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/INDEX.md

Documentation for initializing the autoevals library, including the `init()` function, `getDefaultModel()`, environment variable resolution, and multi-provider setup.

```APIDOC
## Init and Configuration

### Description
This section details how to initialize the autoevals library and configure its behavior. It covers the primary `init()` function, retrieving default models, managing environment variables, and setting up integrations with multiple providers through the Braintrust Gateway.

### Functions
- **`init(config)`**: Initializes the autoevals library with the provided configuration.
- **`getDefaultModel()`**: Retrieves the default model configured for the library.

### Configuration
- **Environment Variables**: Explains how environment variables are used to configure settings.
- **Multi-provider Setup**: Describes how to configure the library to use multiple providers via the Braintrust Gateway.
```

--------------------------------

### Install Autoevals with npm

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

Install the Autoevals package using npm for TypeScript projects.

```bash
npm install autoevals
```

--------------------------------

### Run Linters After Development Setup

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Runs all configured linters using pre-commit after the development environment has been set up.

```bash
pre-commit run --all-files
```

--------------------------------

### ContextRelevancy Evaluator Example (Python)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/ragas-evaluators.md

Shows how to use the ContextRelevancy evaluator to assess if the provided context is relevant to the input question. Ensure the Ragas library is installed.

```python
from autoevals.ragas import ContextRelevancy

evaluator = ContextRelevancy()
result = evaluator(
    output="The answer about photosynthesis",
    input="What is photosynthesis?",
    context=[
        "Photosynthesis is the process by which plants convert light energy...",
        "Chloroplasts are the organelles where photosynthesis occurs..."
    ]
)
```

--------------------------------

### Python Scorer Instance Usage Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/score.md

Shows how to instantiate a Python scorer and call it synchronously with output, expected, and input, then prints the results.

```python
from autoevals import Factuality

# Create and call a scorer instance
evaluator = Factuality()
result = evaluator(
    output="Paris is the capital of France",
    expected="Paris is the capital of France",
    input="What is the capital of France?"
)

print(result.name)           # "Factuality"
print(result.score)          # 1.0
print(result.metadata)       # {"rationale": "...", "choice": "A" }
```

--------------------------------

### Basic Usage Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/00-START-HERE.txt

Demonstrates how to initialize Autoevals and use the Factuality evaluator for basic checks. Ensure you have the necessary client (e.g., OpenAI) initialized.

```typescript
import { Factuality, init } from "autoevals";
init({ client: new OpenAI() });
const result = await Factuality({
  output: "Paris",
  expected: "Paris",
  input: "What is the capital of France?"
});
```

```python
from autoevals.llm import Factuality
from autoevals import init
from openai import OpenAI

init(OpenAI())
result = Factuality()(
  output="Paris",
  expected="Paris",
  input="What is the capital of France?"
)
```

--------------------------------

### Battle Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Shows how to use the Battle classifier to compare two outputs and determine which better follows instructions. Prints the score.

```python
evaluator = Battle()
result = evaluator(
    instructions="Write a greeting for a professional email",
    output="Dear Colleague,ளையும்",
    expected="Hello there,"
)

print(result.score)  # 1.0 if output is better
```

--------------------------------

### Install TypeScript Dependencies

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Installs project dependencies for TypeScript. Use --frozen-lockfile to ensure reproducible builds.

```bash
pnpm install --frozen-lockfile
```

--------------------------------

### Example Mustache Template with Logic

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/templates-and-partial.md

A comprehensive Mustache template demonstrating variable substitution, conditionals, and structured output for evaluating a request.

```mustache
Evaluate this request:

User Question: {{input}}
Model Response: {{output}}

{{#if context}}
Supporting Context:
{{context}}
{{/if}}

Expected Answer: {{expected}}

Does the response correctly answer the question?
A. Yes, completely accurate
B. Partially correct
C. Incorrect

```

--------------------------------

### TypeScript Code Quality Evaluator Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/templates-and-partial.md

Demonstrates how to create and use a 'CodeQuality' evaluator using LLMClassifierFromTemplate. This example shows setting up a prompt, choice scores, model, and chain-of-thought, then applying it to a code snippet.

```typescript
import { LLMClassifierFromTemplate } from "autoevals";

const CodeQuality = LLMClassifierFromTemplate({
  name: "CodeQuality",
  promptTemplate: "
    Review this code for quality:    
    Code:
    {{output}}
    
    Is the code well-structured, readable, and maintainable?    
    A. Excellent - Clean, well-documented, easy to understand
    B. Good - Mostly clear with minor improvements possible
    C. Fair - Works but has readability or structure issues
    D. Poor - Difficult to understand or maintain
  ",
  choiceScores: {
    "A": 1,
    "B": 0.67,
    "C": 0.33,
    "D": 0
  },
  model: "gpt-4-turbo",
  useCoT: true
});

// Use it
const result = await CodeQuality({
  output: "
    function fib(n) {
      return n <= 1 ? n : fib(n-1) + fib(n-2);
    }
  "
});

console.log(result.score);
```

--------------------------------

### Example Usage of LLMClassifierFromTemplate in Python

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Define a Python class 'CodeQualityScorer' inheriting from `LLMClassifierFromTemplate`. This example shows how to initialize the scorer with a prompt and choice scores, and then use an instance to evaluate code.

```python
from autoevals.llm import LLMClassifierFromTemplate

class CodeQualityScorer(LLMClassifierFromTemplate):
    def __init__(self):
        super().__init__(
            name="CodeQuality",
            prompt_template="""
            Review this code:
            {{output}}
            
            Is it well-structured?
            A. Yes
            B. No
            """,
            choice_scores={"A": 1, "B": 0}
        )

evaluator = CodeQualityScorer()
result = evaluator(output="def fibonacci(n): return n <= 1 and n or fibonacci(n-1) + fibonacci(n-2)")
```

--------------------------------

### Python Autoevals Setup

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

Use this Python code to define an evaluation with Braintrust and Autoevals. Ensure the file is named `eval_*.py`.

```python
import braintrust
from autoevals.llm import Factuality

Eval(
    "Autoevals",
    data=lambda:
        [
            dict(
                input="Which country has the highest population?",
                expected="China",
            ),
        ],
    task=lambda *args: "People's Republic of China",
    scores=[Factuality],
)
```

--------------------------------

### Example Usage of Scorer.partial (TypeScript)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/types.md

Demonstrates how to use the `.partial()` method to create a scorer with pre-bound input arguments. The resulting scorer can then be called with the remaining arguments.

```typescript
const FactualityWithInput = Factuality.partial({
  input: "What is the capital of France?"
});

const result = await FactualityWithInput({
  output: "Paris",
  expected: "Paris"
});
```

--------------------------------

### ClosedQA Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Provides an example of using the ClosedQA classifier to test if an output answers a question using model knowledge, with a specified criteria.

```python
evaluator = ClosedQA()
result = evaluator(
    input="What is the capital of France?",
    output="Paris",
    expected="Paris",
    criteria="Must be city name only"
)
```

--------------------------------

### Factuality Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Demonstrates how to instantiate and use the Factuality classifier to check if an output is factual compared to an expected value. Prints the score and rationale.

```python
from autoevals.llm import Factuality

evaluator = Factuality()
result = evaluator.eval(
    output="Paris is the capital of France",
    expected="Paris is the capital of France",
    input="What is the capital of France?"
)

print(result.score)  # 1.0
print(result.metadata["rationale"])  # Explanation
```

--------------------------------

### TypeScript: Implement and Use Custom Cache

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Demonstrates how to create a custom cache class with get and set methods and then use it with an Autoevals evaluator. Ensure your custom cache class implements asynchronous get and set operations.

```typescript
import { Factuality } from "autoevals";

class CustomCache {
  private cache = new Map();

  async get(params) {
    return this.cache.get(JSON.stringify(params)) || null;
  }

  async set(params, response) {
    this.cache.set(JSON.stringify(params), response);
  }
}

const cache = new CustomCache();

const result = await Factuality({
  output: "...",
  expected: "...",
  cache  // Use custom cache for this evaluator
});
```

--------------------------------

### Faithfulness Evaluator Example (Python)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/ragas-evaluators.md

Demonstrates how to use the Faithfulness evaluator to check if the generated output is supported by the provided context. Requires Ragas library.

```python
from autoevals.ragas import Faithfulness

evaluator = Faithfulness()
result = evaluator(
    output="Paris is the capital of France",
    input="What is the capital of France?",
    context=["Paris is the capital and largest city of France."]
)

print(result.score)  # 1.0
```

--------------------------------

### Security Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Demonstrates the usage of the Security classifier to evaluate if a response is safe or malicious. Prints the score.

```python
from autoevals.llm import Security

evaluator = Security()
result = evaluator(output="Safe and helpful response")
print(result.score)  # 1 if safe, 0 if malicious
```

--------------------------------

### Python: Async Evaluation with Custom Client

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Configure a custom asynchronous OpenAI client for Braintrust initialization, enabling async evaluations and specifying models. This example demonstrates using a custom base URL and API key.

```python
import asyncio
from openai import AsyncOpenAI
from autoevals import init
from autoevals.llm import Factuality

async def main():
    # Set up custom client with specific configuration
    client = AsyncOpenAI(
        base_url="https://custom-api.example.com/v1/",
        api_key="sk-..."
    )
    
    init(
        client=client,
        default_model="gpt-4-turbo"
    )
    
    evaluator = Factuality()
    
    # Use async API for batch evaluations
    result = await evaluator.eval_async(
        output="Paris is the capital of France",
        expected="Paris is the capital of France",
        input="What is the capital of France?"
    )
    
    print(f"Score: {result.score}")
    print(f"Rationale: {result.metadata.get('rationale')}")

asyncio.run(main())
```

--------------------------------

### TypeScript Scorer Usage Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/score.md

Demonstrates how to import and call a TypeScript scorer function with output, expected, and input arguments, logging the results.

```typescript
import { Factuality, Score } from "autoevals";

// Call a scorer function
const result: Score = await Factuality({
  output: "Paris is the capital of France",
  expected: "Paris is the capital of France",
  input: "What is the capital of France?"
});

console.log(result.name);        // "Factuality"
console.log(result.score);       // 1
console.log(result.metadata);    // { rationale: "...", choice: "A" }
```

--------------------------------

### Multi-Provider via Braintrust Gateway

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Configure Autoevals to use different LLM providers through the Braintrust Gateway. This example shows how to initialize the client with a custom baseURL and API key for a specific provider.

```typescript
import OpenAI from "openai";
import { init } from "autoevals";

const client = new OpenAI({
  baseURL: "https://gateway.braintrust.dev",
  apiKey: process.env.BRAINTRUST_API_KEY
});

init({
  client,
  defaultModel: "claude-3-5-sonnet-20241022"  // Use Anthropic
});
```

--------------------------------

### Combining Levenshtein and Embedding Similarity (Python)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/string-evaluators.md

This Python example demonstrates a strategy to first perform a fast Levenshtein check and fall back to semantic comparison if the strings are not sufficiently similar. It requires importing both evaluators and using asyncio for asynchronous operations.

```python
from autoevals.string import Levenshtein, EmbeddingSimilarity
import asyncio

async def smart_compare(output, expected):
    # Fast check first
    levenshtein = Levenshtein()
    exact_match = levenshtein(output, expected)
    
    if exact_match.score > 0.95:
        return exact_match  # Close enough
    
    # If not close, do semantic comparison
    embedding = EmbeddingSimilarity()
    semantic = await embedding.eval_async(output, expected)
    
    return semantic

result = asyncio.run(smart_compare(
    "Paris is France's capital",
    "France's capital is Paris"
))
```

--------------------------------

### Local Validation Commands

Source: https://github.com/braintrustdata/autoevals/blob/main/docs/PUBLISHING.md

Run these commands locally to validate your release before publishing. They check version synchronization, install dependencies, build the project, perform a dry run of the npm publish, sync development dependencies with uv, build the Python package, and check the distribution files.

```bash
python3 .github/scripts/check_version_sync.py
```

```bash
pnpm install --frozen-lockfile
```

```bash
pnpm run build
```

```bash
npm publish --dry-run --access public
```

```bash
uv sync --extra dev
```

```bash
uv run python -m build
```

```bash
uv run python -m twine check dist/*
```

--------------------------------

### ListContains Scorer Example

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Demonstrates how to use the ListContains scorer to check if all expected items are present in an output list. The scorer returns a score between 0.0 and 1.0.

```python
from autoevals.list import ListContains

scorer = ListContains()
result = scorer.eval(
    output=["apple", "banana", "cherry"],
    expected=["apple", "banana"]
)
# Score: 1.0 (both expected items present)
```

--------------------------------

### AnswerSimilarity Evaluator (Async Python)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/ragas-evaluators.md

Provides an example of using the AnswerSimilarity evaluator asynchronously to compare a generated answer against an expected answer. Requires the Ragas library and asyncio.

```python
from autoevals.ragas import AnswerSimilarity
import asyncio

async def compare_answers():
    evaluator = AnswerSimilarity()
    result = await evaluator.eval_async(
        output="Paris is the capital of France",
        expected="The capital of France is Paris"
    )
    return result.score

asyncio.run(compare_answers())
```

--------------------------------

### TypeScript Autoevals Setup

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

Use this TypeScript code to define an evaluation with Braintrust and Autoevals. Ensure the file is named `*.eval.ts` or `*.eval.js` and run with `npx braintrust run`.

```typescript
import { Eval } from "braintrust";
import { Factuality } from "autoevals";

Eval("Autoevals", {
  data: () => [
    {
      input: "Which country has the highest population?",
      expected: "China",
    },
  ],
  task: () => "People's Republic of China",
  scores: [Factuality],
});
```

```bash
npx braintrust run example.eval.js
```

--------------------------------

### Battle Scorer Example

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Employ the Battle scorer to compare two outputs and determine which is superior. This scorer is useful for A/B testing or when a subjective comparison is needed. The 'input', 'output', and 'expected' parameters are required.

```python
from autoevals.llm import Battle

evaluator = Battle()
result = evaluator.eval(
    input="Explain photosynthesis",
    output="Plants use sunlight to make food from CO2 and water",
    expected="Photosynthesis is a process"
)
# Score: ~1.0 (first answer is more detailed)
```

--------------------------------

### Factuality Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Use this classifier to test if a model's output is factually correct against an expected value. Ensure you have the 'autoevals' package installed.

```typescript
import { Factuality } from "autoevals";

const result = await Factuality({
  output: "Paris is the capital of France",
  expected: "Paris is the capital of France",
  input: "What is the capital of France?"
});

console.log(`Score: ${result.score}`);
console.log(`Rationale: ${result.metadata?.rationale}`);
```

--------------------------------

### Handle OpenAI API Errors

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/errors-and-exceptions.md

Catch and handle specific OpenAI SDK errors like RateLimitError. Ensure you have the OpenAI SDK installed. This example shows basic error logging.

```typescript
import { Factuality } from "autoevals";
import { RateLimitError } from "openai";

try {
  const result = await Factuality({
    output: "...",
    expected: "...",
    openAiApiKey: "invalid-key"
  });
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log("Rate limit exceeded, retry after delay");
  } else if (error instanceof Error) {
    console.log(`Error: ${error.message}`);
  }
}
```

--------------------------------

### Python: init() Function

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Sets global configuration for all autoevals scorers. Call once at application startup.

```APIDOC
## init()

### Description
Sets global configuration for all autoevals scorers. Call once at application startup.

### Parameters
#### client
- **Type**: Client | None
- **Required**: No
- **Default**: None
- **Description**: OpenAI v1 client or compatible API client

#### default_model
- **Type**: str | DefaultModelConfig | None
- **Required**: No
- **Default**: None
- **Description**: Default model(s) as a string or config object with `completion` and `embedding` keys

#### api_key
- **Type**: str | None
- **Required**: No
- **Default**: None
- **Description**: API key (deprecated; use client instead)

#### base_url
- **Type**: str | None
- **Required**: No
- **Default**: None
- **Description**: Base URL (deprecated; use client instead)

### DefaultModelConfig Type
```python
class DefaultModelConfig(TypedDict, total=False):
    completion: str    # Default LLM model (e.g., "gpt-5-mini", "claude-3-5-sonnet-20241022")
    embedding: str     # Default embedding model (e.g., "text-embedding-ada-002", "text-embedding-3-large")
```
```

--------------------------------

### Configure AsyncOpenAI Client in Python

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Initialize an asynchronous OpenAI client in Python with API key and organization for Braintrust.

```python
from openai import AsyncOpenAI
from autoevals import init

client = AsyncOpenAI(
    api_key="sk-வுகளை",
    organization="org-abc123"
)

init(client=client)
```

--------------------------------

### Example Usage of LLMClassifierFromTemplate in TypeScript

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Instantiate a custom scorer like 'CodeQuality' using `LLMClassifierFromTemplate`. This example demonstrates setting up the scorer with a prompt and choice scores, then using it to evaluate an output.

```typescript
import { LLMClassifierFromTemplate } from "autoevals";

const CodeQualityScorer = LLMClassifierFromTemplate({
  name: "CodeQuality",
  promptTemplate: `
    Review this code:
    {{output}}
    
    Is it well-structured and maintainable?
    A. Excellent
    B. Good
    C. Needs improvement
  `,
  choiceScores: { "A": 1, "B": 0.5, "C": 0 }
});

const result = await CodeQualityScorer({
  output: "function fibonacci(n) { return n <= 1 ? n : fibonacci(n-1) + fibonacci(n-2); }"
});

console.log(result.score);
```

--------------------------------

### Handle Client Configuration Errors

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/errors-and-exceptions.md

Catch `ValueError` during `init` if essential client configurations, like an API key, are missing.

```python
from autoevals import init
from openai import OpenAI

try:
    # Missing API key
    client = OpenAI(api_key=None)
    init(client=client)
except ValueError as e:
    print(f"Configuration error: {e}")
    # "The 'api_key' field is required"
```

--------------------------------

### TypeScript: init() Function

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Sets global configuration for all autoevals evaluators. Call once at application startup.

```APIDOC
## init()

### Description
Sets global configuration for all autoevals evaluators. Call once at application startup.

### Parameters
#### options.client
- **Type**: OpenAI
- **Required**: No
- **Description**: OpenAI-compatible client instance (e.g., OpenAI, Azure OpenAI, or Braintrust Gateway client)

#### options.defaultModel
- **Type**: string | object
- **Required**: No
- **Description**: Default model(s) to use for evaluations. Can be a string (backward compat) or object with `completion` and/or `embedding` properties.

### InitOptions Type
```typescript
export interface InitOptions {
  client?: OpenAI;
  defaultModel?: 
    | string
    | {
        completion?: string;    // e.g., "gpt-4-turbo", "claude-3-5-sonnet-20241022"
        embedding?: string;     // e.g., "text-embedding-3-large"
      };
}
```
```

--------------------------------

### Configure Default Scorers with Python

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Use init() to set default client and model for all scorers. Ensure the OpenAI client is properly initialized with an API key.

```python
from autoevals import init
from openai import OpenAI

init(OpenAI(api_key="..."), default_model="gpt-5-mini")
```

--------------------------------

### Set Custom OpenAI Endpoint and API Key via Environment

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Configure a custom OpenAI-compatible API endpoint and authentication using OPENAI_BASE_URL and OPENAI_API_KEY.

```bash
# Custom OpenAI-compatible API
export OPENAI_BASE_URL=https://custom-api.example.com/v1
export OPENAI_API_KEY=sk-...
```

--------------------------------

### Security Classifier

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

An example of a specific LLM classifier, Security, which checks if a response is safe. It follows the general pattern of inheriting from OpenAILLMScorer.

```APIDOC
## Security Classifier

### Description
Example classifier for security evaluation.

### Class Definition
```python
from autoevals.llm import Security

class Security(OpenAILLMScorer):
    """Example classifier for security evaluation."""
```

### Example Usage
```python
from autoevals.llm import Security

evaluator = Security()
result = evaluator(output="Safe and helpful response")
print(result.score)  # 1 if safe, 0 if malicious
```
```

--------------------------------

### Configure Azure OpenAI Client in Python

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Set up an Azure OpenAI client in Python with API key, version, and endpoint for Braintrust.

```python
from openai import AzureOpenAI
from autoevals import init

client = AzureOpenAI(
    api_key="...",
    api_version="2024-02-15-preview",
    azure_endpoint="https://your-resource.openai.azure.com/"
)

init(client=client)
```

--------------------------------

### LLM Evaluator Template

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Example of an LLM evaluator defined using a YAML template with Mustache variable substitution for prompt construction.

```yaml
prompt: |
  Evaluate: {{input}}
  Response: {{output}}
  Expected: {{expected}}

choice_scores:
  "A": 1
  "B": 0.5
  "C": 0
```

--------------------------------

### Run All Linters

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Runs all configured linters (black, ruff, prettier, codespell) using pre-commit.

```bash
uv run --extra dev pre-commit run --all-files
```

--------------------------------

### Python init() Function Signature

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Sets global configuration for all autoevals scorers. Call once at application startup. Supports client, default model, API key, and base URL.

```python
def init(
    client: Client | None = None,
    default_model: str | DefaultModelConfig | None = None,
    api_key: str | None = None,
    base_url: str | None = None,
) -> None
```

--------------------------------

### API Response Validation

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/evaluators-summary.md

Validate API responses for both structure and content. This example uses 'ValidJSON' for schema validation and 'JSONDiff' for content comparison.

```typescript
const [structure, similarity] = await Promise.all([
  ValidJSON({ output: response_json, schema }),
  JSONDiff({ output: response_json, expected: expected_json })
])
```

--------------------------------

### Humor Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Use the Humor classifier to evaluate if a given output is funny. It takes the output and an optional expected description of humor.

```typescript
const result = await Humor({
  output: "Why did the programmer quit his job? Because he didn't get arrays.",
  expected: "A joke about programming"
});
```

--------------------------------

### Initialize Autoevals Client

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Set up the global client and default model for Autoevals. This configuration is used by all subsequent evaluators.

```typescript
import { init } from "autoevals";
import OpenAI from "openai";

// Set up global client and default model
init({
  client: new OpenAI(),
  defaultModel: "gpt-4-turbo"
});

// All evaluators use this configuration
```

--------------------------------

### TypeScript init() Function Signature

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Sets global configuration for all autoevals evaluators. Call once at application startup.

```typescript
export function init(options: InitOptions): void
```

--------------------------------

### Activate Python Virtual Environment

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Activates the Python virtual environment created by 'make develop'.

```bash
source env.sh
```

--------------------------------

### Security Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

The Security classifier checks if an output contains malicious or harmful content. It returns a score of 1 if safe and 0 if malicious.

```typescript
const result = await Security({
  output: "This is a helpful response to a user question"
});

console.log(result.score); // 1 if safe, 0 if malicious
```

--------------------------------

### Use ListContains Scorer

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/moderation-and-lists.md

Example usage of the ListContains class for array comparison, partial matching, and string parsing. Demonstrates how the score reflects the match.

```python
from autoevals.list import ListContains

evaluator = ListContains()

# Array comparison
result = evaluator(
    output=["apple", "banana", "orange"],
    expected=["apple", "banana"]
)

print(result.score)  # 1.0

# Partial match
result2 = evaluator(
    output=["apple", "banana"],
    expected=["apple", "banana", "grape"]
)

print(result2.score)  # 0.667 (2 out of 3)

# String parsing
result3 = evaluator(
    output="apple, banana, orange",
    expected="apple, banana"
)

print(result3.score)  # 1.0

```

--------------------------------

### Python: Bind Client

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/templates-and-partial.md

Creates a Factuality evaluator with a pre-bound OpenAI client and specific model/temperature settings. Ensures consistent API interaction.

```python
from autoevals.llm import Factuality
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-...")

# Create a factuality evaluator with bound client
StrictFactuality = Factuality(
    client=client,
    model="gpt-4-turbo",
    temperature=0
)

# Use it
result = StrictFactuality(
    output="...",
    expected="..."
)
```

--------------------------------

### Use Moderation Scorer

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/moderation-and-lists.md

Example of how to use the Moderation class for evaluating text safety. Demonstrates both synchronous and asynchronous usage with score and metadata output.

```python
from autoevals.moderation import Moderation

evaluator = Moderation()
result = evaluator(output="This is helpful and appropriate content")

print(result.score)         # 1 (safe)
print(result.metadata)      # {"category_scores": {...}, "threshold": None}

# With async
import asyncio

async def check_safety():
    evaluator = Moderation(threshold=0.3)
    result = await evaluator.eval_async(
        output="Potentially sensitive content"
    )
    return result

asyncio.run(check_safety())

```

--------------------------------

### Basic Factuality Evaluation (Python)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Demonstrates basic usage of the Factuality evaluator in Python. Ensure the OpenAI client is initialized before use.

```python
from autoevals.llm import Factuality
from openai import OpenAI

from autoevals import init
init(OpenAI())

result = Factuality()(
    input="What is the capital of France?",
    output="Paris",
    expected="Paris"
)

print(result.score)  # 1.0
```

--------------------------------

### Set Braintrust Gateway API Key and URL via Environment

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Configure authentication and endpoint for the Braintrust Gateway using BRAINTRUST_API_KEY and BRAINTRUST_AI_GATEWAY_URL.

```bash
# Braintrust Gateway
export BRAINTRUST_API_KEY=token_...
export BRAINTRUST_AI_GATEWAY_URL=https://gateway.braintrust.dev
```

--------------------------------

### Factuality Scorer Example

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Use the Factuality scorer to evaluate if an output is factually consistent with the expected answer. Ensure 'input', 'output', and 'expected' parameters are provided.

```typescript
import { Factuality } from "autoevals";

const result = await Factuality({
  input: "What is the capital of France?",
  output: "Paris",
  expected: "The capital of France is Paris",
});
// Score: 1.0 (factually correct)
```

--------------------------------

### Handle Template Variable Errors

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/errors-and-exceptions.md

Catch errors related to missing template variables when using LLMClassifierFromTemplate. This example shows handling a missing 'input' variable.

```typescript
import { LLMClassifierFromTemplate } from "autoevals";

const scorer = LLMClassifierFromTemplate({
  name: "CustomScorer",
  promptTemplate: "Evaluate this: {{input}}",
  choiceScores: { A: 1, B: 0 }
});

try {
  const result = await scorer({
    output: "response",
    // Missing: input
  });
} catch (error) {
  // Mustache template rendering error
  console.log("Template variable missing: input");
}
```

--------------------------------

### Score Interface

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/INDEX.md

Reference for the Score interface, ScorerArgs, and Scorer types in TypeScript, and the Score class and Scorer base class in Python. Includes usage examples.

```APIDOC
## Score Interface

### Description
Provides details on the `Score` interface and related types (`ScorerArgs`, `Scorer`) for TypeScript, and the `Score` class and `Scorer` base class for Python. This section is intended for developers integrating custom scoring logic.

### Usage
Refer to the specific language examples provided within the documentation for detailed implementation guidance.
```

--------------------------------

### Global Python Client Configuration

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

Initialize a global client for all evaluators in Python. This client will be used by default for all autoevals.

```python
import openai
import asyncio
from autoevals import init
from autoevals.llm import Factuality

client = init(openai.AsyncOpenAI(base_url="https://api.openai.com/v1/"))

async def main():
    evaluator = Factuality()
    result = await evaluator.eval_async(
        input="What is the speed of light in a vacuum?",
        output="The speed of light in a vacuum is 299,792,458 meters per second.",
        expected="The speed of light in a vacuum is approximately 300,000 kilometers per second."
    )
    print(f"Factuality score: {result.score}")

asyncio.run(main())
```

--------------------------------

### Build Python Package

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Builds the Python package using the 'build' module.

```bash
uv run --all-extras python -m build
```

--------------------------------

### ClosedQA Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Utilize the ClosedQA classifier to verify if a model's answer to a question relies solely on its internal knowledge. You can specify additional criteria for the answer.

```typescript
const result = await ClosedQA({
  input: "What is the capital of France?",
  output: "Paris",
  expected: "Paris",
  criteria: "Must be city name only"
});
```

--------------------------------

### Handle JSON Parsing Errors

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/errors-and-exceptions.md

Catch SyntaxError when using JSONDiff or ValidJSON with malformed JSON input. This example shows handling invalid JSON in the 'output' parameter.

```typescript
import { JSONDiff, ValidJSON } from "autoevals";

try {
  const result = await JSONDiff({
    output: "{ invalid json",  // Malformed JSON
    expected: '{"valid": true}'
  });
} catch (error) {
  // SyntaxError from JSON.parse
  console.log("Invalid JSON in output");
}
```

--------------------------------

### Python: Global Configuration

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Initialize the Braintrust SDK globally in your Python application with an AsyncOpenAI client and specify default models for completion and embedding.

```python
from openai import AsyncOpenAI
from autoevals import init, Factuality

# Initialize once at application startup
client = AsyncOpenAI(api_key="sk-...")
init(
    client=client,
    default_model={
        "completion": "gpt-4-turbo",
        "embedding": "text-embedding-3-large"
    }
)

# All evaluators use the configured client and models
evaluator = Factuality()
result = evaluator(
    output="The capital of France is Paris",
    expected="Paris is the capital of France"
)
print(result.score)
```

--------------------------------

### Handle Validation Errors

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/errors-and-exceptions.md

Catch and handle validation errors thrown by scorers when required parameters are missing. This example demonstrates catching a missing 'expected' parameter for LevenshteinScorer.

```typescript
import { Levenshtein } from "autoevals";

try {
  const result = await Levenshtein({
    output: "hello world"
    // Missing: expected
  });
} catch (error) {
  if (error instanceof Error) {
    console.log(error.message); // "LevenshteinScorer requires an expected value"
  }
}
```

--------------------------------

### TypeScript: Global Configuration

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/init-and-configuration.md

Initialize the Braintrust SDK once at application startup with a specific OpenAI client and default models for completion and embedding.

```typescript
import OpenAI from "openai";
import { init, Factuality } from "autoevals";

// Initialize once at app startup with a specific model
init({
  client: new OpenAI({ apiKey: process.env.OPENAI_API_KEY }),
  defaultModel: {
    completion: "gpt-4-turbo",
    embedding: "text-embedding-3-large"
  }
});

// All evaluators will now use the configured client and models
const result = await Factuality({
  output: "The capital of France is Paris",
  expected: "Paris is the capital of France"
});
```

--------------------------------

### Build TypeScript Project

Source: https://github.com/braintrustdata/autoevals/blob/main/CLAUDE.md

Builds the TypeScript project, outputting JavaScript files to the jsdist/ directory.

```bash
pnpm run build
```

--------------------------------

### Import OpenAI Client (TypeScript)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/types.md

Import the standard OpenAI client from the 'openai' package. This client can be configured for various OpenAI-compatible APIs.

```typescript
import { OpenAI, AzureOpenAI } from "openai";
```

--------------------------------

### TypeScript Custom Variables in Templates

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/templates-and-partial.md

Illustrates using custom variables within a prompt template for an LLMClassifier. This example defines a 'CustomEvaluator' that accepts 'domain' and 'style' as runtime parameters.

```typescript
const CustomEvaluator = LLMClassifierFromTemplate({
  name: "CustomEvaluator",
  promptTemplate: "
    Evaluate this {{domain}} response for {{style}} style.
    Response: {{output}}
    Expected: {{expected}}
  ",
  choiceScores: { "A": 1, "B": 0 }
});

const result = await CustomEvaluator({
  output: "The capital of France is Paris",
  expected: "Paris",
  domain: "geography",
  style: "formal"
});
```

--------------------------------

### Python Scorer Async API Usage Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/score.md

Illustrates how to use the asynchronous evaluation method (`eval_async`) of a Python scorer within an async function and run it using asyncio.

```python
# Using async API
import asyncio

async def evaluate():
    result = await evaluator.eval_async(
        output="Paris is the capital of France",
        expected="Paris is the capital of France",
        input="What is the capital of France?"
    )
    return result

asyncio.run(evaluate())
```

--------------------------------

### Battle Classifier Example

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/llm-classifiers.md

Employ the Battle classifier to compare two outputs and determine which one better adheres to given instructions. This is useful for A/B testing or comparing model versions.

```typescript
const result = await Battle({
  instructions: "Write a greeting for a professional email",
  output: "Dear Colleague,",
  expected: "Hello there,"
});

console.log(result.score); // 1 if "Dear Colleague" is better
```

--------------------------------

### Configure OpenAI Client in TypeScript

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/configuration.md

Initialize the OpenAI client with an API key and organization ID for use with Braintrust.

```typescript
import OpenAI from "openai";
import { init } from "autoevals";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  organization: "org-abc123"
});

init({ client });
```

--------------------------------

### InitOptions Interface (TypeScript)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/types.md

Defines the configuration options for initializing the Autoevals client, including optional OpenAI client and default model settings for completion and embedding.

```typescript
export interface InitOptions {
  client?: OpenAI;
  defaultModel?:
    | string
    | {
        completion?: string;
        embedding?: string;
      };
}
```

--------------------------------

### Python LLMClassifierFromTemplate Class Definition

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/templates-and-partial.md

Shows how to define a custom scorer class in Python by inheriting from LLMClassifierFromTemplate. This example outlines the structure for setting up the prompt template and choice scores within the class constructor.

```python
from autoevals.llm import LLMClassifierFromTemplate

class CustomScorer(LLMClassifierFromTemplate):
    def __init__(self, **kwargs):
        super().__init__(
            name="CustomScorer",
            prompt_template="""
            [Mustache template here with {{output}}, {{expected}}, etc.]
            """,
            choice_scores={"A": 1, "B": 0.5, "C": 0},
            **kwargs
        )

evaluator = CustomScorer()
result = evaluator(output="...", expected="...")
```

--------------------------------

### Configure API Access with Environment Variables

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Set environment variables to configure API access for OpenAI, custom endpoints, or the Braintrust Gateway without modifying code.

```bash
# OpenAI API
export OPENAI_API_KEY=sk-...

# Custom endpoint
export OPENAI_BASE_URL=https://custom-api.example.com/v1

# Braintrust Gateway
export BRAINTRUST_API_KEY=token_...
export BRAINTRUST_AI_GATEWAY_URL=https://gateway.braintrust.dev
```

--------------------------------

### TypeScript Factuality Evaluator with Claude 3.5 Sonnet

Source: https://github.com/braintrustdata/autoevals/blob/main/README.md

This TypeScript example demonstrates how to run an LLM-based Factuality evaluator using the Claude 3.5 Sonnet model. It requires the BRAINTRUST_API_KEY environment variable to be set.

```typescript
// NOTE: ensure BRAINTRUST_API_KEY is set in your environment
import { Factuality } from "autoevals";

(async () => {
  const input = "Which country has the highest population?";
  const output = "People's Republic of China";
  const expected = "China";

  // Run an LLM-based evaluator using the Claude 3.5 Sonnet model from Anthropic
  const result = await Factuality({
    model: "claude-3-5-sonnet-latest",
    output,
    expected,
    input,
  });

  // The evaluator returns a score from [0,1] and includes the raw outputs from the evaluator
  console.log(`Factuality score: ${result.score}`);
  console.log(`Factuality metadata: ${result.metadata?.rationale}`);
})();
```

--------------------------------

### Templates and Partial Evaluation

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/INDEX.md

Documentation on using YAML templates to create custom evaluators, function currying with `.partial()`, and thread variables for multi-turn evaluation.

```APIDOC
## Templates and Partial Evaluation

### Description
This section explains how to leverage YAML templates for creating custom evaluators within the autoevals framework. It also covers function currying using the `.partial()` method and the use of thread variables for managing state in multi-turn evaluation scenarios.

### Features
- **YAML Template System**: Describes the system for defining custom evaluators using YAML.
- **Custom Evaluators**: Guidance on creating evaluators from templates.
- **`.partial()` Method**: Explains function currying for creating specialized evaluator instances.
- **Thread Variables**: Details how to use thread variables for multi-turn evaluation contexts.
```

--------------------------------

### Configure Default Scorers with TypeScript

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Use init() to set default client and model for all scorers. Ensure the OpenAI client is properly initialized with an API key.

```typescript
import { init } from "autoevals";
import OpenAI from "openai";

init({
  client: new OpenAI({ apiKey: "..." }),
  defaultModel: "gpt-5-mini",
});
```

--------------------------------

### ListContains Constructor Parameters

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/moderation-and-lists.md

Defines the parameters for initializing the ListContains class, including item scorer, semantic similarity threshold, and OpenAI client details.

```python
def __init__(
    self,
    item_scorer=None,
    semantic_similarity_threshold: float = 0.8,
    client: Client | None = None,
    api_key: str | None = None,
    base_url: str | None = None,
) -> None

```

--------------------------------

### Numeric Difference Scorer Examples

Source: https://github.com/braintrustdata/autoevals/blob/main/SCORERS.md

Evaluates numeric differences between generated and expected values. Supports both absolute and relative difference calculations with configurable thresholds. Scores range from 0 (significant difference) to 1 (within threshold or equal).

```typescript
import { NumericDiff } from "autoevals";

// Absolute difference
const result1 = await NumericDiff({
  output: 10.5,
  expected: 10.0,
  maxDiff: 1.0,
});
// Score: 0.5 (difference of 0.5 out of max 1.0)
```

```typescript
// Relative difference
const result2 = await NumericDiff({
  output: 100,
  expected: 110,
  relative: true,
});
// Score: ~0.91 (10% difference)
```

--------------------------------

### Python RAG Evaluation Pipeline

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/api-reference/ragas-evaluators.md

This Python snippet demonstrates how to evaluate a RAG system using Ragas metrics. It initializes the autoevals client with an OpenAI instance and uses asyncio for concurrent evaluation.

```python
from autoevals import init
from autoevals.ragas import (
    Faithfulness,
    ContextRelevancy,
    ContextPrecision,
    AnswerCorrectness
)
from openai import OpenAI
import asyncio

init(OpenAI())

async def evaluate_rag_system(question, answer, context):
    evaluators = [
        Faithfulness(),
        ContextRelevancy(),
        ContextPrecision(),
        AnswerCorrectness()
    ]
    
    results = await asyncio.gather(*[
        evaluator.eval_async(
            output=answer,
            input=question,
            context=context,
            expected=None  # If available
        )
        for evaluator in evaluators
    ])
    
    scores = {
        "faithfulness": results[0].score,
        "context_relevancy": results[1].score,
        "context_precision": results[2].score,
        "answer_correctness": results[3].score,
    }
    scores["average"] = sum(scores.values()) / len(scores)
    
    return scores

# Run evaluation
result = asyncio.run(evaluate_rag_system(
    question="What is photosynthesis?",
    answer="Photosynthesis is the process by which plants convert light energy into chemical energy.",
    context=["Photosynthesis occurs in chloroplasts...", "The process requires sunlight..."]
))

print(result)
```

--------------------------------

### Basic Factuality Evaluation (TypeScript)

Source: https://github.com/braintrustdata/autoevals/blob/main/_autodocs/README.md

Demonstrates basic usage of the Factuality evaluator in TypeScript. Ensure the OpenAI client is initialized before use.

```typescript
import { Factuality, init } from "autoevals";
import OpenAI from "openai";

init({ client: new OpenAI() });

const result = await Factuality({
  input: "What is the capital of France?",
  output: "Paris",
  expected: "Paris"
});

console.log(result.score);  // 1.0
```