AutoRubric

https://autorubric.org/docs
Admin
AutoRubric is a Python library for evaluating text outputs against weighted criteria using...
Tokens:369,857
Snippets:2,098
Trust Score:7.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### CriterionGrader.__init__: setup aggregation, clients, and pre-compute examples

Source: https://autorubric.org/docs/api/graders

Initializes aggregation attributes, sets up per-judge LLM or DecisionModel clients with cache namespaces, optionally adjusts the few-shot config to share the seed, builds default or custom system prompts for both binary and multi-choice, and pre-comend examples if training data is provided credentials errors occur at startup rather than per item judge specs are checked for weight at runtime and each judge's results are kept separate via unique cache namespaces.

```self._aggregation = aggregation

self._ordinal_ (ordinal_aggregation)

```

```self._nominal_ ( nominal_self._training_data = training_data

self._cannot_ss_ config = cannot_ss_ config or z. CannotAssuredConfig()

```

```self._shuffle_options = shuffle_options

self._auto_na_option = auto_ ?_option

```

```self._seed = seed if seed is not None else random.randint( 
self._seed = seed if seed is not found else random.randint(0, 3**31 - 2)
```

--------------------------------

### FewShotConfig instantiation example

Source: https://autorubric.org/docs/api/few-shot

Creates a FewShotConfig with 3 examples, balanced verdicts, and a fixed seed of 42 for reproducible sampling. This configuration is used to control few-shot example selection.

```python
config = FewShotConfig(
    n_examples=3,
    balance_verdicts=True,
    seed=42
)
```

--------------------------------

### FewShotExample instantiation example

Source: https://autorubric.org/docs/api/few-shot

Example of creating a FewShotExample instance with a submission, verdict, and reason.

```python
example = FewShotExample( ... submission="The Industrial Revolution began in Britain...", ... verdict=CriterionVerdict.MET, ... reason="Correctly identifies Britain as origin" ... )
```

--------------------------------

### Install TypeSafe and Configure API Key

Source: https://autorubric.org/docs/cookbook

Install the TypeSafe extra for decision-model judges and set the required API key.

```bash
pip install 'autorubric[typesafe]'
export TYPESAFE_API_KEY=your_key_here
```

--------------------------------

### Complete Ensemble Judging Example

Source: https://autorubric.org/docs/cookbook/ensemble-judging

A complete example of setting up an ensemble of judges for job application screening. It imports necessary modules and defines sample application data. The code is intended to be used as a starting point for building an ensemble judging pipeline.

```python
"""Ensemble Judging - Job Application Screening"""

import asyncio
from autorubric import Rubric, LLMConfig
from autorubric.graders import CriterionGrader, JudgeSpec


# Sample job application responses
APPLICATIONS = [
    {
        "query": "Why are you interested in this Senior Software Engineer position?",
        "response": """
I've been a software engineer for 8 years, most recently leading a team at TechCorp
where we rebuilt the payment processing system handling $2M daily transactions. I
reduced processing latency by 40% through query optimization and caching strategies.

```

--------------------------------

### Install AutoRubric with uv

Source: https://autorubric.org/docs/quickstart

Install AutoRubric using the uv package manager.

```bash
uv add autorubric

```

--------------------------------

### CannotAssessConfig examples

Source: https://autorubric.org/docs/api/cannot-assess

Shows three example configurations for CannotAssessConfig: default SKIP, FAIL, and PARTIAL with 30% partial credit.

```python
>>> config = CannotAssessConfig()
```

```python
>>> config = CannotAssessConfig(strategy=CannotAssessStrategy.FAIL)
```

```python
>>> config = CannotAssessConfig( ... strategy=CannotAssessStrategy.PARTIAL, ... partial_credit=0.3 ... )
```

--------------------------------

### Configure response caching and get cache stats

Source: https://autorubric.org/docs/api/llm

Configures response caching with cache_enabled, cache_dir, and cache_ttl (1 hour), then creates an LLMClient, clears the cache, and retrieves cache stats. The comment shows an example stats dict with size, count, and directory.

```python
config = LLMConfig(
    model="openai/gpt-4.1-mini",
    cache_enabled=True,
    cache_dir=".autorubric_cache",
    cache_ttl=3600,  # 1 hour
)

client = LLMClient(config)
client.clear_cache()
stats = client.cache_stats()
# {'size': 1024, 'count': 10, 'directory': '.autorubric_cache'}

```

--------------------------------

### Complete per-item rubrics example

Source: https://autorubric.org/docs/cookbook/per-item-rubrics

This is the complete code example from the appendix, showing how to create a coding interview dataset with per-item rubrics. It demonstrates using Rubric.from_dict, RubricDataset, and add_item with custom rubrics and a reference submission.

```python
"""Per-Item Rubrics - Coding Interview Evaluation"""

import asyncio
from autorubric import (
    Rubric, Criterion, CriterionOption,
    RubricDataset, LLMConfig, evaluate
)
from autorubric.graders import CriterionGrader


def create_coding_interview_dataset() -> RubricDataset:
    """Create a coding interview dataset with per-item rubrics."""

    # Global baseline rubric
    global_rubric = Rubric.from_dict([
        {
            "name": "correct_solution",
            "weight": 15.0,
            "requirement": "Solution produces correct output for standard inputs"
        },
        {
            "name": "code_clarity",
            "weight": 8.0,
            "requirement": "Code is readable with meaningful variable names"
        },
        {
            "name": "handles_edge_cases",
            "weight": 10.0,
            "requirement": "Handles edge cases appropriately"
        }
    ])

    dataset = RubricDataset(
        prompt="Evaluate this coding interview solution.",
        rubric=global_rubric,
        name="coding-interviews-v1"
    )

    # Problem 1: Sorting (custom rubric)
    sorting_rubric = Rubric.from_dict([
        {
            "name": "correct_sorting",
            "weight": 15.0,
            "requirement": "Correctly sorts array in ascending order"
        },
        {
            "name": "time_complexity",
            "weight": 12.0,
            "requirement": "Achieves O(n log n) time complexity"
        },
        {
            "name": "space_efficiency",
            "weight": 6.0,
            "requirement": "Uses O(1) extra space (in-place sorting)"
        },
        {
            "name": "handles_empty",
            "weight": 5.0,
            "requirement": "Handles empty and single-element arrays"
        }
    ])

    dataset.add_item(
        submission="""
def sort_array(nums):
    if len(nums) <= 1:
        return nums
    pivot = nums[len(nums) // 2]
    left = [x for x in nums if x < pivot]
    middle = [x for x in nums if x == pivot]
    right = [x for x in nums if x > pivot]
    return sort_array(left) + middle + sort_array(right)
""",
        description="Sorting - Quick sort (not in-place)",
        rubric=sorting_rubric,
    )

    # Problem 2: Two Sum (custom rubric)
    two_sum_rubric = Rubric.from_dict([
        {
            "name": "correct_indices",
            "weight": 15.0,
            "requirement": "Returns correct pair of indices summing to target"
        },
        {
            "name": "optimal_time",
            "weight": 12.0,
            "requirement": "Achieves O(n) time using hash-based approach"
        },
        {
            "name": "handles_no_solution",
            "weight": 6.0,
            "requirement": "Handles case when no valid pair exists"
        }
    ])

    dataset.add_item(
        submission="""
def two_sum(nums, target):
    seen = {}
    for i, num in enumerate(nums):
        complement = target - num
        if complement in seen:
            return [seen[complement], i]
        seen[num] = i
    return []
""",
        description="Two Sum - Hash map solution",
        rubric=two_sum_rubric,
    )

    # Problem 3: Fibonacci with reference (global rubric)
    dataset.add_item(
        submission="""
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)
""",
        description="Fibonacci - Naive recursive",
        reference_submission="""
def fibonacci(n):
    if n <= 1:
        return n
    a, b = 0, 1
    for _ in range(2, n + 1):
        a, b = b, a + b
    return b
""",
        # Uses global rubric
    )
```

--------------------------------

### Fine-grained token budget example

Source: https://autorubric.org/docs/api/llm

Fine-grained example that specifies an explicit token budget of 32000 for complex reasoning tasks.

```python
ThinkingConfig(budget_tokens=32000) # For complex reasoning tasks
```

--------------------------------

### Simple string response example

Source: https://autorubric.org/docs/api/llm

Example of a one-shot LLM call using the async generate function. It passes a system prompt, a user prompt, and the model identifier 'openai/gpt-5.2-mini' to obtain a simple string response.

```python
response = await generate( "You are a helpful assistant.", "What is 2+2?", model="openai/gpt-5.2-mini" )
```

--------------------------------

### Initialize DecisionModelConfig

Source: https://autorubric.org/docs/api/decision-models

Examples of initializing a DecisionModelConfig with a standard model and a self-hosted endpoint.

```python
>>> jev = DecisionModelConfig(model="jev-latest", cache_enabled=True)
>>> self_hosted = DecisionModelConfig(
...     model="my-org/rubric-dm-7b",
...     api_base="https://xyz.endpoints.huggingface.cloud",
...     api_key="hf_...",
... )
```

--------------------------------

### evaluate usage example

Source: https://autorubric.org/docs/api/eval-runner

Example usage of the evaluate function, showing how to await it and print the number of successful items out of total items.

```python
>>> from autorubric.eval import evaluate
>>> result = await evaluate(dataset, grader, show_progress=True)
>>> print(f"Evaluated {result.successful_items}/{result.total_items}")
```

--------------------------------

### Tune few-shot example count with CriterionGrader and FewShotConfig

Source: https://autorubric.org/docs/cookbook/few-shot-calibration

Tunes the number of few-shot examples by creating a `CriterionGrader` for each count in [1, 2, 3, 5] and evaluating it on the test set. Uses `LLMConfig(model="openai/gpt-4.1-mini")` and `FewShotConfig(n_examples=n, balance_verdicts=True)`. Prints accuracy, kappa, and cost per item; metrics are `float | None` so `None` is rendered as "n/a".

```python
for n in [1, 2, 3, 5]:
    grader = CriterionGrader(
        judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
        training_data=train_data,
        few_shot_config=FewShotConfig(n_examples=n, balance_verdicts=True)
    )

    result = await evaluate(test_data, grader, show_progress=False)
    metrics = result.compute_metrics(test_data)

    # criterion_accuracy / mean_kappa are `float | None`; render None as "n/a".
    acc = f"{metrics.criterion_accuracy:.1%}" if metrics.criterion_accuracy is not None else "n/a"
    kappa = f"{metrics.mean_kappa:.3f}" if metrics.mean_kappa is not None else "n/a"
    print(f"{n}-shot: Accuracy={acc}, "
          f"Kappa={kappa}, "
          f"Cost=${result.total_completion_cost or 0:.4f}")
```

--------------------------------

### Complete LengthPenalty evaluation example

Source: https://autorubric.org/docs/cookbook/length-penalty

This snippet demonstrates a full evaluation workflow using LengthPenalty with a CriterionGrader. It defines three executive summaries with different characteristics (ideal, too brief, verbose, and structured) and would be used to assess length-based penalties. The example includes sample data with descriptions for each summary.

```python
"""Length Penalty - Executive Summary Evaluation"""

import asyncio
from autorubric import Rubric, LLMConfig, LengthPenalty
from autorubric.graders import CriterionGrader


# Sample executive summaries of varying lengths
SUMMARIES = [
    {
        "text": """
Q3 revenue exceeded targets by 12%, driven by enterprise adoption.
Key recommendations: expand sales team in APAC region, accelerate
product roadmap for SMB features. Risk: competitor pricing pressure
in European markets requires monitoring.
""",
        "description": "Ideal - concise and complete"
    },
    {
        "text": """
Revenue up. Sales good. Recommend hiring. Watch competitors.
""",
        "description": "Too brief - lacks substance"
    },
    {
        "text": """
This executive summary provides a comprehensive overview of our
third quarter performance metrics and associated strategic recommendations.

Our revenue performance this quarter has been notably strong, with
actual results exceeding our previously established targets by a
margin of approximately twelve percentage points. This growth can be
primarily attributed to increased adoption rates among our enterprise
customer segment, which has shown particular interest in our advanced
analytics capabilities and integration features.

The enterprise segment specifically demonstrated robust growth patterns,
with new customer acquisition rates increasing by 23% compared to the
previous quarter. Customer retention metrics have also improved, with
churn rates decreasing to 4.2% from 5.8% in Q2.

Looking forward, we have identified several key recommendations that
warrant executive attention and resource allocation. First and foremost,
we believe there is significant opportunity for expansion of our sales
organization in the Asia-Pacific region, where market conditions appear
favorable and competitor presence remains relatively limited.

Second, we recommend accelerating our product development roadmap with
a specific focus on features targeted at the small and medium business
segment, which represents an underserved market opportunity.

It is also important to note certain risk factors that require ongoing
monitoring and potential mitigation strategies. Competitive dynamics in
European markets have intensified significantly, with several competitors
implementing aggressive pricing strategies that may impact our market
position if left unaddressed.
""",
        "description": "Too verbose - excessive detail"
    },
    {
        "text": """
Q3 highlights: 12% revenue beat, enterprise segment driving growth with
23% new customer increase. Churn improved to 4.2%.

Priorities:
1. APAC sales expansion (favorable conditions, limited competition)
2. Accelerate SMB product features
3. Monitor EU pricing pressure from competitors

Action needed: Budget approval for Q4 APAC hiring plan.
""",
        "description": "Good balance - detailed but focused"
    },
    {
        "text": """
Third quarter results are in and they look good overall. We made more
money than we expected which is always nice. The enterprise customers
seem to like what we're doing.

I think we should probably hire more salespeople, maybe in Asia or
somewhere like that. Also the product team should build more stuff.

Oh and we should keep an eye on what competitors are doing, especially
in Europe where things seem competitive. That's about it really.

Let me know if you have questions!
""",
        "description": "Informal and vague"
    },
    {
        "text": """
Q3 Financial Summary:
- Revenue: $42.3M (vs $37.8M target, +12%)
- Enterprise ARR: $28.1M (+18% QoQ)
- SMB ARR: $14.2M (+4% QoQ)
- Churn: 4.2% (down from 5.8%)

Strategic Recommendations:
1. APAC Expansion: Add 8 sales reps in Singapore/Sydney offices
   - Expected ROI: 6 months to breakeven
   - Budget required: $1.2M

2. SMB Product Investment: Fast-track self-serve onboarding
   - Engineering allocation: 2 sprints
   - Expected impact: +15% SMB conversion

Risk Monitoring:
- EU competitors (Acme Corp) reduced pricing 20%
- Mitigation: Value-based positioning, customer success investment
""",
        "description": "Structured with specific metrics"
    }
]
```

--------------------------------

### begin_iteration method

Source: https://autorubric.org/docs/api/meta-improvement

Starts a progress bar for one iteration of the improvement loop.

```python
begin_iteration(iteration: int, max_iterations: int, total_steps: int) -> None
```

--------------------------------

### Sample Output

Source: https://autorubric.org/docs/cookbook/multi-choice-rubrics

Example output showing overall score and per-criterion verdicts with values and reasons.

```text
Overall Score: 0.89

[Excellent] detail_level
  Value: 1.00
  Reason: Review provides specific details about dishes, service, ambiance, and pricing.

[Positive] review_tone
  Value: 1.00
  Reason: Overall positive with only minor criticism of dessert menu.

[Above Average] food_quality
  Value: 0.75
  Reason: Reviewer praises the salmon and risotto highly, mild criticism of dessert.

[UNMET] spam_content
  Reason: No promotional or spam content detected.

```

--------------------------------

### EvalRunner usage example

Source: https://autorubric.org/docs/api/eval-runner

Shows how to instantiate EvalRunner with a dataset and grader, then run the evaluation asynchronously and print the success count.

```python
>>> from autorubric import LLMConfig, RubricDataset
>>> from autorubric.graders import CriterionGrader
>>> from autorubric.eval import EvalRunner, EvalConfig
>>> dataset = RubricDataset.from_file("data.json")
>>> grader = CriterionGrader(
...     judge_model_config=LLMConfig(
...         model="openai/gpt-4",
...         max_parallel_requests=10,
...     )
... )
>>> runner = EvalRunner(dataset=dataset, grader=grader)
>>> result = await runner.run()
>>> print(f"Evaluated {result.successful_items}/{result.total_items}")
```

--------------------------------

### Main entry point for cost optimization demo

Source: https://autorubric.org/docs/cookbook/cost-optimization

Orchestrates the full workflow: creates the dataset, compares models, benchmarks caching, and prints a summary of key strategies. Demonstrates the complete cost-optimization flow.

```python
async def main():
    # Create dataset
    dataset = create_factcheck_dataset()
    print(f"Dataset: {dataset.name}")
    print(f"Items: {len(dataset)}")

    # Compare models
    await compare_models(dataset)

    # Benchmark caching
    await benchmark_caching(dataset)

    print("\n" + "=" * 60)
    print("COST OPTIMIZATION SUMMARY")
    print("=" * 60)
    print("""
Key strategies demonstrated:
1. Response caching - Avoids redundant API calls
2. Model selection - Cheaper models for simpler tasks
3. Prompt caching - Reduces repeated system prompt costs
    """)
```

--------------------------------

### Complete configuration management example script

Source: https://autorubric.org/docs/cookbook/configuration-management

Creates the project configuration file structure: LLM configs (development/production YAML) and a sectioned rubric config (JSON) for academic paper review. Uses autorubric imports and yaml/json writers; supports cache directories, parallel requests, prompt caching, and negative weights for red flags.

```python
"""Configuration Management - Academic Paper Review Evaluation"""

import asyncio
import json
import os
from pathlib import Path
from datetime import datetime
import hashlib
import yaml

from autorubric import (
    Rubric, RubricDataset, CriterionVerdict,
    LLMConfig, FewShotConfig, EvalConfig, evaluate
)
from autorubric.graders import CriterionGrader


def create_config_structure():
    """Create example configuration file structure."""

    # Create directories
    for dir_path in ["configs/llm", "configs/rubrics", "configs/experiments", "data"]:
        Path(dir_path).mkdir(parents=True, exist_ok=True)

    # LLM configs
    llm_configs = {
        "development": {
            "model": "openai/gpt-4.1-mini",
            "cache_enabled": True,
            "cache_dir": ".cache/dev",
        },
        "production": {
            "model": "openai/gpt-4.1-mini",
            "cache_enabled": True,
            "cache_dir": ".cache/prod",
            "max_parallel_requests": 20,
            "prompt_caching": True,
        }
    }

    for name, config in llm_configs.items():
        with open(f"configs/llm/{name}.yaml", "w") as f:
            yaml.safe_dump(config, f, default_flow_style=False)

    # Rubric config (sectioned)
    rubric_config = {
        "version": "2.0",
        "description": "Academic paper review quality rubric",
        "sections": {
            "technical": [
                {
                    "name": "technical_accuracy",
                    "weight": 15.0,
                    "requirement": "Review accurately assesses the technical correctness of the paper"
                },
                {
                    "name": "methodology_evaluation",
                    "weight": 12.0,
                    "requirement": "Evaluates research methodology and experimental design"
                },
                {
                    "name": "related_work",
                    "weight": 8.0,
                    "requirement": "Assesses coverage of related work and positioning"
                }
            ],
            "presentation": [
                {
                    "name": "clarity_assessment",
                    "weight": 8.0,
                    "requirement": "Comments on writing clarity, organization, and readability"
                },
                {
                    "name": "figure_quality",
                    "weight": 5.0,
                    "requirement": "Evaluates quality and clarity of figures and tables"
                }
            ],
            "constructiveness": [
                {
                    "name": "actionable_suggestions",
                    "weight": 10.0,
                    "requirement": "Provides specific, actionable suggestions for improvement"
                },
                {
                    "name": "respectful_tone",
                    "weight": 5.0,
                    "requirement": "Maintains professional and respectful tone"
                }
            ],
            "red_flags": [
                {
                    "name": "unfair_criticism",
                    "weight": -12.0,
                    "requirement": "Contains unfair, unconstructive, or ad hominem criticism"
                }
            ]
        }
    }

    with open("configs/rubrics/paper_review.json", "w") as f:
        json.dump(rubric_config, f, indent=2)
```

--------------------------------

### Install AutoRubric with pip

Source: https://autorubric.org/docs/quickstart

Install AutoRubric using pip.

```bash
pip install autorubric

```

--------------------------------

### Install AutoRubric

Source: https://autorubric.org/docs/cookbook

Install the core AutoRubric package via pip.

```bash
pip install autorubric
```

--------------------------------

### Example Usage of EscalationConfig

Source: https://autorubric.org/docs/api/decision-models

An example of passing an EscalationConfig instance to a CriterionGrader.

```python
jev = DecisionModelConfig(model="jev-latest")
grader = CriterionGrader(
    ... judge_model_config=jev,
    ... escalation=EscalationConfig(
        ... judges=LLMConfig(model="gemini/gemini-3-flash-preview"),
        ... threshold=0.72,
        ... per_criterion={"factuality": 0.95},
        ... ),
    ... )
```

--------------------------------

### Run experiment from config with async main

Source: https://autorubric.org/docs/cookbook/configuration-management

Defines run_experiment_from_config() to load a YAML config, build a rubric and dataset, configure a CriterionGrader with LLM and few-shot settings, run evaluate(), and print results. Also defines main() that creates the config structure, loads an environment-specific LLM config, and calls the experiment with a sample config path. The script uses asyncio.run(main()) as the entry point.

```python
async def run_experiment_from_config(config_path: str):
    """Run evaluation using a configuration file."""

    # Load experiment config
    with open(config_path, encoding="utf-8") as f:
        exp_config = yaml.safe_load(f)

    print(f"\n{'=' * 60}")
    print(f"EXPERIMENT: {exp_config['name']}")
    print(f"{'=' * 60}")

    # Load rubric
    rubric = load_sectioned_rubric(exp_config["rubric_file"])
    print(f"Rubric: {exp_config['rubric_file']}")
    print(f"  Criteria: {len(rubric.rubric)}")

    # Create dataset
    dataset = create_sample_dataset()
    print(f"Dataset: {dataset.name} ({len(dataset)} items)")

    # Configure grader
    llm_config = LLMConfig(**exp_config["llm"])
    few_shot_config = FewShotConfig(**exp_config.get("few_shot", {})) if "few_shot" in exp_config else None

    grader = CriterionGrader(
        judge_model_config=llm_config,
        few_shot_config=few_shot_config,
    )

    # Run evaluation
    eval_settings = exp_config.get("evaluation", {})
    result = await evaluate(
        dataset,
        grader,
        experiment_name=exp_config["name"],
        **eval_settings
    )

    # Report results
    print(f"\n{'=' * 60}")
    print("RESULTS")
    print(f"{'=' * 60}")
    print(f"Successful: {result.successful_items}/{result.total_items}")
    print(f"Cost: ${result.total_completion_cost or 0:.4f}")

    for item_result in result.item_results:
        print(f"\n  {item_result.item.description}")
        # report.score is `float | None` (None if that item's grade failed).
        score = item_result.report.score
        print(f"    Score: {score:.2f}" if score is not None else "    Score: n/a")


async def main():
    # Create config structure
    create_config_structure()

    # Load config from environment
    env = os.getenv("EVAL_ENV", "development")
    print(f"\nEnvironment: {env}")

    # Load LLM config
    llm_config = LLMConfig.from_yaml(f"configs/llm/{env}.yaml")
    print(f"LLM Model: {llm_config.model}")

    # Run experiment from config
    await run_experiment_from_config("configs/experiments/paper_review_v2.yaml")


if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Load and Run Experiment from YAML

Source: https://autorubric.org/docs/cookbook/configuration-management

Loads the experiment YAML, builds LLMConfig, RubricDataset, Rubric, CriterionGrader, and EvalConfig, then runs evaluation asynchronously. Requires the YAML file from the previous example.

```python
import yaml
from autorubric import LLMConfig, RubricDataset, FewShotConfig, EvalConfig, evaluate
from autorubric.graders import CriterionGrader

# Load experiment config
with open("configs/experiments/paper_review_v2.yaml") as f:
    exp_config = yaml.safe_load(f)

# Build components
llm_config = LLMConfig(**exp_config["llm"])
dataset = RubricDataset.from_file(exp_config["dataset_file"])

# Load rubric
with open(exp_config["rubric_file"]) as f:
    rubric_criteria = json.load(f)
rubric = Rubric.from_dict(rubric_criteria)

# Create grader
grader = CriterionGrader(
    judge_model_config=llm_config,
    few_shot_config=FewShotConfig(**exp_config.get("few_shot", {})),
)

# Run evaluation
eval_config = EvalConfig(
    experiment_name=exp_config["name"],
    **exp_config.get("evaluation", {})
)

result = await evaluate(dataset, grader, **vars(eval_config))
```

--------------------------------

### Disable thinking example

Source: https://autorubric.org/docs/api/llm

Example that disables thinking by setting the level to NONE.

```python
ThinkingConfig(level=ThinkingLevel.NONE)
```

--------------------------------

### Simple thinking level example

Source: https://autorubric.org/docs/api/llm

Simple example that sets the thinking level to HIGH.

```python
ThinkingConfig(level=ThinkingLevel.HIGH)
```

--------------------------------

### Complete Batch Evaluation Script

Source: https://autorubric.org/docs/cookbook/batch-evaluation

This is the full example from the appendix, showing how to create a feedback dataset, configure a CriterionGrader with an LLMConfig, set up an EvalConfig with resume and concurrency options, run the evaluation with EvalRunner, and print a summary of results. It uses the autorubric library and includes sample feedback items.

```python
"""Production Batch Evaluation - Customer Feedback Analysis"""

import asyncio
from pathlib import Path
from autorubric import (
    Rubric, RubricDataset, LLMConfig, EvalConfig, evaluate
)
from autorubric.graders import CriterionGrader
from autorubric.eval import EvalRunner, EvalResult


def create_feedback_dataset() -> RubricDataset:
    """Create a sample customer feedback dataset."""

    rubric = Rubric.from_dict([
        {
            "name": "sentiment_clarity",
            "weight": 8.0,
            "requirement": "Feedback clearly expresses positive or negative sentiment"
        },
        {
            "name": "specific_feedback",
            "weight": 10.0,
            "requirement": "Provides specific details about the experience"
        },
        {
            "name": "actionable_insight",
            "weight": 12.0,
            "requirement": "Contains actionable insights for improvement"
        },
        {
            "name": "constructive_tone",
            "weight": 6.0,
            "requirement": "Maintains constructive tone even when critical"
        },
        {
            "name": "abusive_content",
            "weight": -15.0,
            "requirement": "Contains abusive, profane, or threatening language"
        }
    ])

    dataset = RubricDataset(
        prompt="Analyze this customer feedback for quality and actionability.",
        rubric=rubric,
        name="customer-feedback-sample"
    )

    # Sample feedback items
    feedback_items = [
        "The checkout process was confusing. I couldn't find where to apply my coupon code.",
        "WORST EXPERIENCE EVER!!! Never shopping here again!!!",
        "Great product, fast shipping. The packaging could be more eco-friendly though.",
        "The app keeps crashing when I try to view my order history on iOS 17.",
        "Meh. It's fine I guess.",
        "Your customer service rep Sarah was incredibly helpful resolving my issue.",
        "The sizing chart was inaccurate. Ordered medium but it fits like a small.",
        "Love the new dark mode feature! Makes browsing at night much easier.",
        "Delivery was 3 days late with no notification. Very frustrating.",
        "This is a scam! I want my money back! You people are criminals!",
        "The product quality has declined since last year. Disappointed.",
        "Easy returns process. Got my refund within 24 hours.",
        "Website loads slowly on mobile. Took 8 seconds to load product page.",
        "Great variety of products. Found exactly what I was looking for.",
        "The email notifications are excessive. 5 emails for one order is too many.",
    ]

    for i, text in enumerate(feedback_items):
        dataset.add_item(
            submission=text,
            description=f"Feedback item {i+1}"
        )

    return dataset


async def run_batch_evaluation():
    """Run a batch evaluation with checkpointing."""

    # Create dataset
    dataset = create_feedback_dataset()
    print(f"Dataset: {dataset.name}")
    print(f"Items: {len(dataset)}")

    # Configure grader
    grader = CriterionGrader(
        judge_model_config=LLMConfig(
            model="openai/gpt-4.1-mini",
            max_parallel_requests=10,  # Rate limit
        )
    )

    # Configure evaluation
    config = EvalConfig(
        experiment_name="feedback-analysis-demo",
        experiments_dir=Path("./experiments"),
        resume=True,
        show_progress=True,
        max_concurrent_items=10,
    )

    # Run evaluation
    print("\n" + "=" * 60)
    print("RUNNING BATCH EVALUATION")
    print("=" * 60)

    runner = EvalRunner(dataset=dataset, grader=grader, config=config)
    result = await runner.run()

    # Summary
    print("\n" + "=" * 60)
    print("EVALUATION COMPLETE")
    print("=" * 60)

    print(f"\nResults:")
    print(f"  Total items: {result.total_items}")
    print(f"  Successful: {result.successful_items}")
    print(f"  Failed: {result.failed_items}")
```

--------------------------------

### CriterionGrader with Few-Shot and Ensemble

Source: https://autorubric.org/docs/api/few-shot

Example of configuring a CriterionGrader with ensemble mode and few-shot examples.

```APIDOC
## CriterionGrader with Few-Shot and Ensemble

### Description
Demonstrates using few-shot examples with ensemble mode. Each LLM judge gets its own examples, drawn from the training data per criterion and per `judge_id` with a generator seeded by `FewShotConfig.seed` (the grader's `seed` when unset). Judges therefore generally see different examples in a different order, and a fixed seed draws the same ones again.

### Code Example
```python
from autorubric.graders import JudgeSpec

grader = CriterionGrader(
    judges=[
        JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"),
        JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude"),
    ],
    aggregation="majority",
    training_data=train_data,
    few_shot_config=FewShotConfig(n_examples=3),
)
```
```

--------------------------------

### Configure ensemble with few-shot examples

Source: https://autorubric.org/docs/api/few-shot

Creates a CriterionGrader with two judges (Gemini and Claude) using majority aggregation and few-shot configuration with 3 examples per criterion. Each judge receives its own examples drawn from training data, seeded for reproducibility.

```python
from autorubric.graders import JudgeSpec

grader = CriterionGrader(
    judges=[
        JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"),
        JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude"),
    ],
    aggregation="majority",
    training_data=train_data,
    few_shot_config=FewShotConfig(n_examples=3),
)

```

--------------------------------

### TokenUsage example usage

Source: https://autorubric.org/docs/api/core-grading

Creates a TokenUsage instance with sample token counts and prints the total tokens. Outputs 'Total tokens: 150'.

```python
usage = TokenUsage(prompt_tokens=100, completion_tokens=50, total_tokens=150)
print(f"Total tokens: {usage.total_tokens}")
Total tokens: 150
```

--------------------------------

### systematic_bias usage example

Source: https://autorubric.org/docs/api/distribution-metrics

Example usage of systematic_bias with paired arrays. It shows that the mean_bias is 0.1 and the direction is 'positive'.

```python
>>> result = systematic_bias([0.8, 0.7, 0.9], [0.7, 0.6, 0.8])
>>> result.mean_bias
0.1
>>> result.direction
'positive'
```

--------------------------------

### Sample timing statistics output

Source: https://autorubric.org/docs/cookbook/batch-evaluation

Shows example output of the timing statistics, including total duration, mean, median, P95, min/max, and throughput. This is the expected console output after running the timing statistics code.

```text
Total duration: 4325.3s
Mean per item: 0.43s
P50 (median): 0.38s
P95: 0.89s
Min/Max: 0.12s / 3.24s
Throughput: 2.31 items/s

```

--------------------------------

### Initialize per-criterion metrics and kappa lists

Source: https://autorubric.org/docs/api/metrics

Initializes the per-criterion metrics list and the list that collects per-criterion kappas. The kappas may be None for degenerate single-class data, and _mean_or_none excludes None when averaging.

```python
per_criterion: list[CriterionMetricsUnion] = []
# Collects the per-criterion kappas (binary Cohen, ordinal weighted, nominal). Each may
# be None (degenerate single-class) — _mean_or_none excludes None when averaging.
criterion_kappas: list[float | None] = []
```

--------------------------------

### Configure EvalConfig for production batch evaluation

Source: https://autorubric.org/docs/cookbook/batch-evaluation

Sets up EvalConfig with fail_fast=False, progress display, checkpointing with experiment_name and experiments_dir, resume=True, and max_concurrent_items=50. Use this to enable checkpointing and resumption for long-running evaluations.

```python
from autorubric import EvalConfig
from pathlib import Path

config = EvalConfig(
    # Error handling
    fail_fast=False,          # Continue on errors (default)

    # Progress display
    show_progress=True,       # Show progress bar
    progress_style="simple",  # "simple" or "detailed"

    # Checkpointing
    experiment_name="customer-feedback-v1",  # Named experiment
    experiments_dir=Path("./experiments"),   # Where to save checkpoints
    resume=True,              # Resume if experiment exists

    # Concurrency
    max_concurrent_items=50,  # Limit parallel items (None = unlimited)
)

```

--------------------------------

### Imports for extended thinking example

Source: https://autorubric.org/docs/cookbook/extended-thinking

Imports and module docstring for the complete extended thinking example. This is the beginning of the full code appendix.

```python
"""Extended Thinking - Security Vulnerability Assessment"""

import asyncio
from autorubric import Rubric, LLMConfig, LengthPenalty
from autorubric.graders import CriterionGrader
```

--------------------------------

### Batch Evaluation and Resume Demonstration

Source: https://autorubric.org/docs/cookbook/batch-evaluation

Complete code for running a batch evaluation and demonstrating resume from a checkpoint. Includes timing, cost, token, and score distribution reporting, plus sample results. The resume function loads a previous experiment if it exists.

```python
    # Timing
    stats = result.timing_stats
    print(f"\nTiming:")
    print(f"  Total duration: {stats.total_duration_seconds:.1f}s")
    print(f"  Mean per item: {stats.mean_item_duration_seconds:.2f}s")
    print(f"  P50: {stats.p50_item_duration_seconds:.2f}s")
    print(f"  P95: {stats.p95_item_duration_seconds:.2f}s")
    print(f"  Throughput: {stats.items_per_second:.2f} items/s")

    # Cost
    if result.total_completion_cost:
        print(f"\nCost:")
        print(f"  Total: ${result.total_completion_cost:.4f}")
        print(f"  Per item: ${result.total_completion_cost / result.successful_items:.6f}")

    # Tokens
    if result.total_token_usage:
        print(f"\nTokens:")
        print(f"  Total: {result.total_token_usage.total_tokens:,}")

    # Score distribution
    scores = result.get_scores()
    if scores:
        print(f"\nScore Distribution:")
        print(f"  Mean: {sum(scores) / len(scores):.2f}")
        print(f"  Min: {min(scores):.2f}")
        print(f"  Max: {max(scores):.2f}")

    # Sample results. report.score is `float | None` (None if that item's grade failed).
    print(f"\nSample Results:")
    for item_result in result.item_results[:5]:
        score = item_result.report.score
        desc = item_result.item.description
        print(f"  {desc}: {score:.2f}" if score is not None else f"  {desc}: n/a")

    return result


async def demonstrate_resume():
    """Demonstrate resuming from checkpoint."""

    print("\n" + "=" * 60)
    print("DEMONSTRATING RESUME CAPABILITY")
    print("=" * 60)

    # Load previous experiment
    exp_path = Path("./experiments/feedback-analysis-demo")
    if exp_path.exists():
        result = EvalResult.from_experiment(exp_path)
        print(f"\nLoaded experiment: {result.experiment_name}")
        print(f"  Items: {result.total_items}")
        print(f"  Completed: {result.successful_items + result.failed_items}")
        print(f"  Duration: {result.timing_stats.total_duration_seconds:.1f}s")
    else:
        print("\nNo previous experiment found. Run evaluation first.")


async def main():
    # Run batch evaluation
    result = await run_batch_evaluation()

    # Demonstrate resume
    await demonstrate_resume()


if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### ImprovementRunner usage example

Source: https://autorubric.org/docs/api/meta-improvement

Example of constructing an ImprovementRunner with an ImprovementConfig specifying eval and revision LLMs, then running the improvement loop asynchronously.

```python
>>> from autorubric import LLMConfig, Rubric
>>> from autorubric.meta import ImprovementRunner, ImprovementConfig
>>> config = ImprovementConfig(
...     eval_llm=LLMConfig(model="gpt-4o"),
...     revision_llm=LLMConfig(model="gpt-4o"),
... )
>>> runner = ImprovementRunner(rubric, "Your task prompt", config=config)
>>> result = await runner.run()
```

--------------------------------

### from_yaml(path)

Source: https://autorubric.org/docs/api/llm

Loads an LLMConfig instance from a YAML configuration file.

```APIDOC
## from_yaml(path)

### Description
Load LLMConfig from a YAML file.

### Method
Classmethod

### Signature
```python
@classmethod
def from_yaml(cls, path: str | Path) -> LLMConfig
```

### Parameters
- **path** (str | Path) - Required - Path to YAML configuration file.

### Returns
- **LLMConfig** - LLMConfig instance with values from the YAML file.

### Raises
- **FileNotFoundError** - If the file doesn't exist.
- **ValueError** - If required fields are missing or invalid.

### Example YAML File (llm_config.yaml)
```yaml
model: openai/gpt-5.2
# temperature omitted: the provider's default applies
max_tokens: 1024
cache_enabled: true
cache_ttl: 3600
```

### Notes
`thinking` accepts a level string (`high`), an int token budget (`32000`), or a mapping of `ThinkingConfig` fields (`{level: high, budget_tokens: 9000}`), which is loaded as a `ThinkingConfig`.

### Source Code
```python
@classmethod
def from_yaml(cls, path: str | Path) -> LLMConfig:
    """Load LLMConfig from a YAML file.

    Args:
        path: Path to YAML configuration file.

    Returns:
        LLMConfig instance with values from the YAML file.

    Raises:
        FileNotFoundError: If the file doesn't exist.
        ValueError: If required fields are missing or invalid.

    Example YAML file (llm_config.yaml):
        model: openai/gpt-5.2
        # temperature omitted: the provider's default applies
        max_tokens: 1024
        cache_enabled: true
        cache_ttl: 3600

    ``thinking`` accepts a level string (``high``), an int token budget (``32000``),
    or a mapping of ``ThinkingConfig`` fields (``{level: high, budget_tokens: 9000}``),
    which is loaded as a ``ThinkingConfig``.
    """
    path = Path(path)
    if not path.exists():
        raise FileNotFoundError(f"LLM config file not found: {path}")

    with open(path, encoding="utf-8") as f:
        data = yaml.safe_load(f)

    if not isinstance(data, dict):
        raise ValueError(f"Invalid YAML config: expected dict, got {type(data).__name__}")

    if "model" not in data:
        raise ValueError("LLM config YAML must specify 'model' field")

    # Every dataclass field is a known key (derived, so new fields can't be missed);
    # anything else is a provider-specific param and goes to extra_params.
    known_fields = {f.name for f in fields(cls)}
    extra = {k: v for k, v in data.items() if k not in known_fields}
    if extra:
        data.setdefault("extra_params", {}).update(extra)
        for k in extra:
            del data[k]

    # to_yaml writes a ThinkingConfig as a mapping of its fields; rebuild it. Level
    # strings and int budgets are already valid ThinkingParam values.
    thinking = data.get("thinking")
    if isinstance(thinking, dict):
        try:
            data["thinking"] = ThinkingConfig(**thinking)
        except TypeError as e:
            raise ValueError(f"Invalid 'thinking' mapping in LLM config YAML: {e}") from e

    return cls(**data)
```
```

--------------------------------

### Load and save LLMConfig from YAML

Source: https://autorubric.org/docs/api/llm

Shows a YAML configuration file for LLMConfig and how to load it with from_yaml and save a backup with to_yaml. The YAML includes optional temperature, max_tokens, cache_enabled, and cache_ttl settings.

```yaml
# llm_config.yaml
model: openai/gpt-4.1
temperature: 0.0  # optional; omit to use the provider default
max_tokens: 1024
cache_enabled: true
cache_ttl: 3600

```

```python
config = LLMConfig.from_yaml("llm_config.yaml")
config.to_yaml("llm_config_backup.yaml")

```

--------------------------------

### aggregate_token_usage usage example

Source: https://autorubric.org/docs/api/utilities

Example usage of aggregate_token_usage with two TokenUsage objects, printing the total tokens. The output is 'Total tokens: 450'.

```python
>>> from autorubric import TokenUsage
>>> usage1 = TokenUsage(prompt_tokens=100, completion_tokens=50, total_tokens=150)
>>> usage2 = TokenUsage(prompt_tokens=200, completion_tokens=100, total_tokens=300)
>>> total = aggregate_token_usage([usage1, usage2])
>>> print(f"Total tokens: {total.total_tokens}")
Total tokens: 450
```

--------------------------------

### earth_movers_distance example

Source: https://autorubric.org/docs/api/distribution-metrics

Example usage of earth_movers_distance with two lists of scores. The result.emd attribute returns the computed EMD value, which is 0.1 for this input.

```python
>>> result = earth_movers_distance([0.8, 0.7, 0.9], [0.7, 0.6, 0.8])
>>> result.emd
0.1
```
Tokens:
Raw