### Install Rubric

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Install the Rubric library using pip or uv.

```bash
uv add rubric
# or
pip install rubric
```

--------------------------------

### Install Rubric with uv

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Use this command to add the Rubric library to your project using the uv package manager.

```bash
uv add rubric
```

--------------------------------

### PerCriterionGenerateFn Example

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Example implementation of the PerCriterionGenerateFn protocol for the PerCriterionGrader. This function should make an LLM call and return a validated PerCriterionOutput.

```python
from rubric import PerCriterionGenerateFn, PerCriterionOutput

async def my_generate_fn(
    system_prompt: str,
    user_prompt: str,
    **kwargs
) -> PerCriterionOutput:
    # Your LLM call here
    return PerCriterionOutput(
        criterion_status="MET",
        explanation="The criterion is satisfied."
    )
```

--------------------------------

### Implement a Custom Rule-Based Grader

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Shows how to create a custom rule-based grader by subclassing `Autograder`. This example checks for keyword presence without using LLM calls.

```python
import asyncio
from rubric import Rubric, EvaluationReport
from rubric.autograders import Autograder
from rubric.types import Criterion, CriterionReport
from pydantic import BaseModel

class KeywordOutput(BaseModel):
    matched_keywords: list[str]

class KeywordGrader(Autograder):
    """Simple rule-based grader: checks for keyword presence without LLM calls."""

    def __init__(self, *, normalize: bool = True):
        super().__init__(normalize=normalize)

    async def judge(
        self, to_grade: str, rubric: list[Criterion], query: str | None = None
    ) -> list[CriterionReport]:
        reports = []
        for criterion in rubric:
            keyword = criterion.requirement.split("keyword:")[-1].strip().lower()
            verdict = "MET" if keyword in to_grade.lower() else "UNMET"
            reports.append(CriterionReport(
                weight=criterion.weight,
                requirement=criterion.requirement,
                verdict=verdict,
                reason=f"Keyword '{keyword}' {'found' if verdict == 'MET' else 'not found'}.",
            ))
        return reports

    async def aggregate(
        self, judge_results: list[CriterionReport], *, normalize: bool = True
    ) -> EvaluationReport:
        total_positive = sum(max(0.0, r.weight) for r in judge_results)
        raw = sum((1.0 if r.verdict == "MET" else 0.0) * r.weight for r in judge_results)
        score = max(0.0, min(1.0, raw / total_positive)) if normalize and total_positive else raw
        return EvaluationReport(score=score, raw_score=raw, llm_raw_score=None, report=judge_results)

async def main():
    rubric = Rubric.from_dict([
        {"weight": 3.0, "requirement": "keyword: paris"},
        {"weight": 2.0, "requirement": "keyword: eiffel"},
        {"weight": -1.0,"requirement": "keyword: london"},
    ])
    grader = KeywordGrader()
    result = await rubric.grade("Paris is home to the Eiffel Tower.", autograder=grader)
    print(f"Score: {result.score:.2f}")  # Score: 1.00
    for c in result.report:
        print(f"  [{c.verdict}] {c.requirement}")
    # [MET] keyword: paris
    # [MET] keyword: eiffel
    # [UNMET] keyword: london

asyncio.run(main())
```

--------------------------------

### Quick Start with Default Gemini Generate Functions

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Set your Gemini API key and run this Python script for quick testing with default generation functions. It defines a simple rubric and grades a given text.

```bash
export GEMINI_API_KEY=your_api_key_here
```

```python
import asyncio
from rubric import Rubric, default_per_criterion_generate_fn
from rubric.autograders import PerCriterionGrader

async def main():
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "Response mentions Paris"},
        {"weight": 5.0, "requirement": "Response is concise"}
    ])

    grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)
    result = await rubric.grade("Paris is the capital of France.", autograder=grader)
    print(f"Score: {result.score}")

asyncio.run(main())
```

--------------------------------

### Rubric YAML Format Example

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Shows the YAML representation for a rubric, mirroring the JSON structure with a list of criteria, each having a weight and a requirement.

```yaml
- weight: 10.0
  requirement: "States Q4 2023 base margin as 17.2%"
- weight: 8.0
  requirement: "Explicitly uses Shapley attribution for decomposition"
- weight: -15.0
  requirement: "Uses total deliveries instead of cash-only deliveries"
```

--------------------------------

### Load Rubrics from Various Sources

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Provides examples for constructing a `Rubric` object directly, from a list of dictionaries, from a JSON string, from a YAML string, and from external files (JSON and YAML).

```python
# Direct construction
rubric = Rubric([
    Criterion(weight=10.0, requirement="States Q4 2023 base margin as 17.2%"),
    Criterion(weight=8.0, requirement="Explicitly uses Shapley attribution for decomposition"),
    Criterion(weight=-15.0, requirement="Uses total deliveries instead of cash-only deliveries")
])

# From list of dictionaries
rubric = Rubric.from_dict([
    {"weight": 10.0, "requirement": "States Q4 2023 base margin as 17.2%"},
    {"weight": 8.0, "requirement": "Explicitly uses Shapley attribution for decomposition"},
    {"weight": -15.0, "requirement": "Uses total deliveries instead of cash-only deliveries"}
])

# From JSON string
rubric = Rubric.from_json('[{"weight": 10.0, "requirement": "Example requirement"}]')

# From YAML string
yaml_data = '''
- weight: 10.0
  requirement: "Example requirement"
'''
rubric = Rubric.from_yaml(yaml_data)

# From files
rubric = Rubric.from_file('rubric.json')
rubric = Rubric.from_file('rubric.yaml')
```

--------------------------------

### Rubric JSON Format Example

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Illustrates the expected JSON structure for defining a rubric, where each object in the array represents a criterion with its weight and requirement.

```json
[
  {
    "weight": 10.0,
    "requirement": "States Q4 2023 base margin as 17.2%"
  },
  {
    "weight": 8.0,
    "requirement": "Explicitly uses Shapley attribution for decomposition"
  },
  {
    "weight": -15.0,
    "requirement": "Uses total deliveries instead of cash-only deliveries"
  }
]
```

--------------------------------

### Custom Generate Function with OpenAI

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Implement a custom `generate_fn` using OpenAI's API for production use. This example shows how to define a function that returns structured outputs for rubric evaluation.

```python
import asyncio
import os
from openai import AsyncOpenAI
from rubric import Rubric, PerCriterionOutput
from rubric.autograders import PerCriterionGrader

# Declare custom generate function with any model and inference provider
async def generate_with_openai(system_prompt: str, user_prompt: str) -> PerCriterionOutput:
    client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = await client.chat.completions.create(
        model="gpt-5-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt},
        ],
        response_format={"type": "json_schema", "json_schema": {
            "name": "criterion_output",
            "schema": PerCriterionOutput.model_json_schema()
        }},
        max_tokens=400,
        temperature=0.0,
    )
    content = response.choices[0].message.content or "{}"
    return PerCriterionOutput.model_validate_json(content)

async def main():
    # Build rubric
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "States Q4 2023 base margin as 17.2%"},
        {"weight": 8.0, "requirement": "Explicitly uses Shapley attribution for decomposition"},
        {"weight": -15.0, "requirement": "Uses total deliveries instead of cash-only deliveries"}
    ])

    # Select autograder strategy
    grader = PerCriterionGrader(
        generate_fn=generate_with_openai,
        normalize=False,  # Raw weighted sums
        system_prompt="This overrides the default grader system prompt",
    )

    # Grade output
    result = await rubric.grade(
        query="Input query...",
        to_grade="Output to evaluate...",
        autograder=grader
    )

    print(f"Score: {result.score:.2f}")  # Raw weighted sum
    for criterion in result.report:
        print(f"  [{criterion.verdict}] {criterion.requirement}")
        print(f"    → {criterion.reason}")

asyncio.run(main())
```

--------------------------------

### Compute RL Rewards with `normalize=False`

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Pass `normalize=False` to `PerCriterionGrader` to obtain raw weighted sums for RL reward signals. The `raw_score` field is always populated. This example demonstrates computing rewards for a batch of responses.

```python
import asyncio
from rubric import Rubric
from rubric.autograders import PerCriterionGrader
from rubric.utils import default_per_criterion_generate_fn

rubric = Rubric.from_dict([
    {"weight": 10.0, "requirement": "Correct diagnosis"},
    {"weight": 5.0,  "requirement": "Correct treatment"},
    {"weight": -8.0, "requirement": "Dangerous contraindication recommended"},
])

async def compute_rewards_batch(responses: list[str]) -> list[float]:
    grader = PerCriterionGrader(
        generate_fn=default_per_criterion_generate_fn,
        normalize=False,  # Raw weighted sums for RL
    )
    import asyncio as aio
    results = await aio.gather(*[
        rubric.grade(resp, autograder=grader) for resp in responses
    ])
    return [r.score for r in results]
    # Possible values: 15.0 (perfect), 10.0 (diagnosis only), -8.0 (dangerous error), etc.

async def main():
    rewards = await compute_rewards_batch([
        "Correct diagnosis and treatment.",
        "Wrong diagnosis, no treatment.",
    ])
    print(rewards)  # e.g. [15.0, 0.0]

asyncio.run(main())
```

--------------------------------

### Implement Retries for LLM Calls with Validation

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Implement retries within your `generate_fn` to handle `ValidationError` from Pydantic models. This example shows how to retry an LLM call up to `max_retries` times, continuing on validation errors and raising the exception only on the last attempt.

```python
from pydantic import ValidationError
from rubric import PerCriterionOutput

async def generate_with_retries(system_prompt: str, user_prompt: str, max_retries: int = 3) -> PerCriterionOutput:
    for attempt in range(max_retries):
        try:
            response = await your_llm_call(system_prompt, user_prompt)
            return PerCriterionOutput.model_validate_json(response)
        except ValidationError as e:
            if attempt == max_retries - 1:
                raise
            continue  # Retry on validation error
```

--------------------------------

### Override System Prompt for PerCriterionGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Demonstrates how to provide a custom system prompt when initializing a PerCriterionGrader. Ensure your custom prompt is compatible with the expected response structure.

```python
grader = PerCriterionGrader(
    generate_fn=your_function,
    system_prompt="Your custom system prompt here"
)
```

--------------------------------

### Use Built-in Gemini Graders

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Demonstrates using default Gemini-backed graders for quick testing. Requires setting the GEMINI_API_KEY environment variable. Not recommended for production due to lack of retry logic and single model usage.

```python
import asyncio
import os
from rubric import Rubric, default_per_criterion_generate_fn
from rubric.utils import default_oneshot_generate_fn, default_rubric_as_judge_generate_fn
from rubric.autograders import PerCriterionGrader, PerCriterionOneShotGrader, RubricAsJudgeGrader

os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"

rubric = Rubric.from_dict([
    {"weight": 2.0, "requirement": "Response mentions Paris"},
    {"weight": 1.0, "requirement": "Response mentions France"},
    {"weight": -0.5,"requirement": "Response contains profanity"},
])
response_text = "Paris is the capital of France. It has a rich history."

async def main():
    for name, grader in [
        ("PerCriterion",        PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)),
        ("OneShotGrader",       PerCriterionOneShotGrader(generate_fn=default_oneshot_generate_fn)),
        ("RubricAsJudgeGrader", RubricAsJudgeGrader(generate_fn=default_rubric_as_judge_generate_fn)),
    ]:
        result = await rubric.grade(response_text, autograder=grader)
        print(f"{name}: score={result.score:.2f}, raw={result.raw_score:.2f}")
        # PerCriterion:        score=1.00, raw=3.00
        # OneShotGrader:       score=1.00, raw=3.00
        # RubricAsJudgeGrader: score=0.95, raw=2.85

asyncio.run(main())
```

--------------------------------

### Common Autograder Constructor Parameters

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

All autograders accept a TypedGenerateFn and optional system prompt and normalization settings. The generate_fn is required.

```python
def __init__(
    self,
    generate_fn: TypedGenerateFn,                   # Typed LLM generation function (required)
    *,
    system_prompt: str = DEFAULT_SYSTEM_PROMPT,     # Customizable system prompt
    normalize: bool = True,                          # If False, return raw weighted sums
):
```

--------------------------------

### Instantiate PerCriterionOneShotGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Use PerCriterionOneShotGrader for evaluating all criteria in a single LLM call, which is cost-efficient for fewer criteria. Requires a OneShotGenerateFn returning OneShotOutput.

```python
from rubric import OneShotGenerateFn, OneShotOutput
from rubric.autograders import PerCriterionOneShotGrader

async def my_generate_fn(system_prompt: str, user_prompt: str) -> OneShotOutput:
    # Your LLM call here
    ...

grader = PerCriterionOneShotGrader(generate_fn=my_generate_fn)
```

--------------------------------

### Using PerCriterionGrader with normalize=False

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Instantiate the grader with normalize=False to obtain raw weighted sums instead of normalized 0-1 scores. This is useful for RL training scenarios where normalized scores can complicate optimization.

```python
grader = PerCriterionGrader(generate_fn=your_generate_fn, normalize=False)
result = await rubric.grade(response, autograder=grader)
# result.score = raw weighted sum (can be negative if many negative criteria are MET)
# result.raw_score = same as score when normalize=False
```

--------------------------------

### Instantiate RubricAsJudgeGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Use RubricAsJudgeGrader for a single holistic score (0-100) from the LLM. This is the fastest option but provides no per-criterion breakdown. Requires a RubricAsJudgeGenerateFn returning RubricAsJudgeOutput.

```python
from rubric import RubricAsJudgeGenerateFn, RubricAsJudgeOutput
from rubric.autograders import RubricAsJudgeGrader

async def my_generate_fn(system_prompt: str, user_prompt: str) -> RubricAsJudgeOutput:
    # Your LLM call here
    ...

grader = RubricAsJudgeGrader(generate_fn=my_generate_fn)
```

--------------------------------

### Custom generate_fn for PerCriterionGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Shows how to integrate a custom asynchronous function as the generate_fn for PerCriterionGrader. This function should accept system and user prompts and return a Pydantic model matching the autograder's expected output type.

```python
from rubric import PerCriterionOutput

async def your_custom_function(system_prompt: str, user_prompt: str) -> PerCriterionOutput:
    # Your LLM call here with structured outputs
    ...
    return PerCriterionOutput(criterion_status="MET", explanation="...")

grader = PerCriterionGrader(generate_fn=your_custom_function)
```

--------------------------------

### Accessing JSON Schema for Constrained Decoding

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Demonstrates how to access the JSON schema of a Pydantic model for use with LLM clients that support structured outputs, like OpenAI.

```python
from rubric import PerCriterionOutput

# Get JSON schema for your LLM client
schema = PerCriterionOutput.model_json_schema()

# Example with OpenAI structured outputs
response = await openai_client.chat.completions.create(
    model="gpt-4",
    messages=[...],
    response_format={"type": "json_schema", "json_schema": {
        "name": "criterion_output",
        "schema": schema
    }}
)

# Parse response into validated Pydantic model
output = PerCriterionOutput.model_validate_json(response.choices[0].message.content)
```

--------------------------------

### PerCriterionGrader with Normalized and Raw Scores

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Demonstrates how to use the `PerCriterionGrader` with and without score normalization. When `normalize=True` (default), scores are between 0 and 1. When `normalize=False`, scores represent the raw weighted sum.

```python
# Default: normalized 0-1 scores
grader = PerCriterionGrader(generate_fn=your_function)
result = await rubric.grade(text, autograder=grader)
print(result.score)      # 0.85 (normalized)
print(result.raw_score)  # 12.75 (raw weighted sum)

# Raw scores
grader = PerCriterionGrader(generate_fn=your_function, normalize=False)
result = await rubric.grade(text, autograder=grader)
print(result.score)      # 12.75 (raw, can be negative)
print(result.raw_score)  # 12.75 (same as score)
```

--------------------------------

### Instantiate PerCriterionGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Use PerCriterionGrader for evaluating each criterion in parallel LLM calls. Requires a PerCriterionGenerateFn returning PerCriterionOutput.

```python
from rubric import PerCriterionGenerateFn, PerCriterionOutput
from rubric.autograders import PerCriterionGrader

async def my_generate_fn(system_prompt: str, user_prompt: str) -> PerCriterionOutput:
    # Your LLM call here
    ...

grader = PerCriterionGrader(generate_fn=my_generate_fn)
```

--------------------------------

### Single-Call Per-Criterion Evaluation with PerCriterionOneShotGrader

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Use PerCriterionOneShotGrader for cost-efficiency with many criteria. It evaluates all criteria in a single LLM call and requires a `generate_fn` returning `OneShotOutput`. Ensure the `generate_fn` correctly parses the LLM response into the expected format.

```python
import asyncio
import os
from anthropic import AsyncAnthropic
from rubric import Rubric, OneShotOutput, CriterionEvaluation
from rubric.autograders import PerCriterionOneShotGrader
import json

async def claude_oneshot_fn(system_prompt: str, user_prompt: str) -> OneShotOutput:
    client = AsyncAnthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))
    response = await client.messages.create(
        model="claude-3-5-haiku-latest",
        max_tokens=1024,
        system=system_prompt,
        messages=[{"role": "user", "content": user_prompt}],
    )
    data = json.loads(response.content[0].text)
    return OneShotOutput.model_validate(data)

async def main():
    rubric = Rubric.from_dict([
        {"weight": 5.0, "requirement": "Mentions Paris"},
        {"weight": 3.0, "requirement": "Mentions the Eiffel Tower"},
        {"weight": 2.0, "requirement": "Response is under 100 words"},
        {"weight": -4.0,"requirement": "Contains incorrect population figures"},
    ])

    grader = PerCriterionOneShotGrader(generate_fn=claude_oneshot_fn)
    result = await rubric.grade(
        to_grade="Paris, home to the iconic Eiffel Tower, is the capital of France.",
        autograder=grader,
    )
    print(f"Score: {result.score:.2f}")  # Score: 1.00
    for c in result.report:
        print(f"  [{c.verdict}] {c.requirement}")

asyncio.run(main())

```

--------------------------------

### Implement Custom Autograder

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Extend the Autograder base class to create a custom autograder. Implement the `judge` method to process the response and rubric, and the `aggregate` method to format the results. The `generate_fn` pattern is optional.

```python
class MyAutograder(Autograder):
    def __init__(
        self,
        generate_fn: MyCustomGenerateFn,  # Optional - just an implementation choice
        *,
        system_prompt: str = "Grade this response.",
        normalize: bool = True,
    ):
        super().__init__(normalize=normalize)
        self.generate_fn = generate_fn
        self.system_prompt = system_prompt

    async def judge(
        self,
        to_grade: str,
        rubric: list[Criterion],
        query: str | None = None,
    ) -> MyCustomOutput:
        # Build prompt with rubric and response
        user_prompt = f"Response: {to_grade}\nCriteria: {rubric}"

        # Call LLM with your chosen approach
        result = await self.generate_fn(self.system_prompt, user_prompt)
        return result

    async def aggregate(self, judge_results: MyCustomOutput) -> EvaluationReport:
        # Convert judge results to EvaluationReport
        return EvaluationReport(
            score=judge_results.score,
            raw_score=judge_results.score,
            llm_raw_score=None,
            report=None,
        )
```

--------------------------------

### Initialize Rubric and Grade Text

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Construct a Rubric object with a list of Criterion objects and use it to grade text with a PerCriterionGrader. Requires setting an API key for the default generate function.

```python
import asyncio
from rubric import Rubric, Criterion
from rubric.autograders import PerCriterionGrader
from rubric.utils import default_per_criterion_generate_fn
import os

os.environ["GEMINI_API_KEY"] = "your-gemini-api-key"

rubric = Rubric([
    Criterion(weight=10.0, requirement="Response mentions Paris as the capital"),
    Criterion(weight=5.0,  requirement="Response is concise (under 50 words)"),
    Criterion(weight=-3.0, requirement="Response contains factual errors"),
])

async def main():
    grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)
    result = await rubric.grade(
        to_grade="Paris is the capital of France.",
        autograder=grader,
        query="What is the capital of France?",
    )
    print(f"Score: {result.score:.2f}")        # Score: 1.00
    print(f"Raw score: {result.raw_score}")    # Raw score: 15.0
    for c in result.report:
        print(f"  [{c.verdict}] {c.requirement}")

asyncio.run(main())
```

--------------------------------

### Import Core Rubric Types

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Import essential classes and protocols for defining rubrics, criteria, and generation functions. Includes Pydantic output schemas and default generate functions.

```python
from rubric import (
    # Core classes
    Rubric,
    Criterion,
    CriterionReport,
    EvaluationReport,

    # Pydantic output schemas
    PerCriterionOutput,
    OneShotOutput,
    RubricAsJudgeOutput,
    CriterionEvaluation,

    # Typed protocols
    PerCriterionGenerateFn,
    OneShotGenerateFn,
    RubricAsJudgeGenerateFn,

    # Default generate functions
    default_per_criterion_generate_fn,
    default_oneshot_generate_fn,
    default_rubric_as_judge_generate_fn,
)
```

--------------------------------

### Holistic Scoring with RubricAsJudgeGrader

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Demonstrates grading a text using RubricAsJudgeGrader, which passes the entire rubric to the LLM for a single 0-100 score. Requires a generate_fn returning RubricAsJudgeOutput. The raw score is converted for consistency.

```python
import asyncio
import os
from openai import AsyncOpenAI
from rubric import Rubric, RubricAsJudgeOutput
from rubric.autograders import RubricAsJudgeGrader
import json

async def openai_judge_fn(system_prompt: str, user_prompt: str) -> RubricAsJudgeOutput:
    client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user",   "content": user_prompt},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "rubric_judge_output",
                "schema": RubricAsJudgeOutput.model_json_schema(),
            },
        },
        temperature=0.0,
    )
    return RubricAsJudgeOutput.model_validate_json(response.choices[0].message.content)

async def main():
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "Correctly identifies the primary diagnosis"},
        {"weight": 8.0,  "requirement": "Lists appropriate first-line treatments"},
        {"weight": -10.0,"requirement": "Recommends a contraindicated treatment"},
    ])

    grader = RubricAsJudgeGrader(generate_fn=openai_judge_fn)
    result = await rubric.grade(
        to_grade="This is anaphylaxis. Administer epinephrine IM immediately.",
        autograder=grader,
    )
    print(f"Score (normalized): {result.score:.2f}")      # e.g. 0.90
    print(f"LLM raw score (0-100): {result.llm_raw_score}") # e.g. 90.0
    print(f"Weighted-sum raw_score: {result.raw_score:.2f}") # e.g. 16.2
    print(result.report)  # None — no per-criterion breakdown

asyncio.run(main())
```

--------------------------------

### Import Autograder Base Classes

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Import the base Autograder class and its specialized implementations for different grading strategies. These classes provide the foundation for building autograding logic.

```python
from rubric.autograders import (
    Autograder,              # Base class
    PerCriterionGrader,      # Parallel per-criterion grading
    PerCriterionOneShotGrader,  # Single-call all-criteria grading
    RubricAsJudgeGrader,     # Holistic scoring
)
```

--------------------------------

### Override System Prompt for RubricAsJudgeGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/README.md

Illustrates providing a custom system prompt to the RubricAsJudgeGrader. The custom prompt must be designed to elicit the desired 0-100 score and handle the expected response format.

```python
grader = RubricAsJudgeGrader(
    generate_fn=your_function,
    system_prompt="Your custom system prompt here"
)
```

--------------------------------

### Create Rubric from YAML String

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Construct a Rubric object by parsing a YAML formatted string. Ensure the YAML string is correctly formatted with 'weight' and 'requirement' keys.

```python
from rubric import Rubric

yaml_data = """
- weight: 10.0
  requirement: \"States Q4 2023 base margin as 17.2%\"
- weight: 8.0
  requirement: \"Explicitly uses Shapley attribution for decomposition\"
- weight: -15.0
  requirement: \"Uses total deliveries instead of cash-only deliveries\"
"""

rubric = Rubric.from_yaml(yaml_data)
print(rubric.rubric[0].weight)      # 10.0
print(rubric.rubric[2].weight)      # -15.0  (negative = error criterion)
```

--------------------------------

### Create Rubric from File

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Construct a Rubric object from a YAML or JSON file. The function also accepts file-like objects.

```python
from rubric import Rubric

# rubric.yaml:
# - weight: 10.0
#   requirement: "Mentions the patient's chief complaint"
# - weight: -5.0
#   requirement: "Recommends contraindicated medication"

rubric = Rubric.from_file("rubric.yaml")
# also accepts .json files and file-like objects
with open("rubric.json") as f:
    rubric2 = Rubric.from_file(f)

print(rubric.rubric[0].requirement)  # "Mentions the patient's chief complaint"
```

--------------------------------

### Parallel Per-Criterion Evaluation with PerCriterionGrader

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Use PerCriterionGrader for rubrics with many criteria where maximum accuracy is desired. It requires a `generate_fn` that returns `PerCriterionOutput` and utilizes `asyncio.gather` for concurrent LLM calls.

```python
import asyncio
import os
from openai import AsyncOpenAI
from rubric import Rubric, PerCriterionOutput
from rubric.autograders import PerCriterionGrader

async def openai_generate_fn(system_prompt: str, user_prompt: str) -> PerCriterionOutput:
    client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user",   "content": user_prompt},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "criterion_output",
                "schema": PerCriterionOutput.model_json_schema(),
            },
        },
        temperature=0.0,
    )
    return PerCriterionOutput.model_validate_json(response.choices[0].message.content)

async def main():
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "States Q4 2023 base margin as 17.2%"},
        {"weight": 8.0,  "requirement": "Uses Shapley attribution for decomposition"},
        {"weight": -15.0,"requirement": "Uses total deliveries instead of cash-only deliveries"},
    ])

    grader = PerCriterionGrader(
        generate_fn=openai_generate_fn,
        normalize=True,   # Default: normalized 0-1 score
    )

    result = await rubric.grade(
        to_grade="The Q4 2023 base margin was 17.2%. We used Shapley values to attribute changes.",
        autograder=grader,
    )
    print(f"Score: {result.score:.2f}")  # Score: 1.00
    for c in result.report:
        print(f"  [{c.verdict}] {c.requirement}")
        # [MET] States Q4 2023 base margin as 17.2%
        # [MET] Uses Shapley attribution for decomposition
        # [UNMET] Uses total deliveries instead of cash-only deliveries

asyncio.run(main())

```

--------------------------------

### Grade Text with Rubric

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Evaluate a given text against a Rubric using an autograder. Requires asyncio for asynchronous execution. The result includes a score, raw score, and a detailed report.

```python
import asyncio
from rubric import Rubric
from rubric.autograders import PerCriterionGrader
from rubric.utils import default_per_criterion_generate_fn

rubric = Rubric.from_dict([
    {"weight": 10.0, "requirement": "Identifies anaphylaxis as the diagnosis"},
    {"weight": 8.0,  "requirement": "Recommends epinephrine as first-line treatment"},
    {"weight": -10.0, "requirement": "Recommends antihistamine as first-line treatment"},
])

async def main():
    grader = PerCriterionGrader(generate_fn=default_per_criterion_generate_fn)

    result = await rubric.grade(
        to_grade="This is anaphylaxis. Give epinephrine IM immediately.",
        autograder=grader,
        query="Patient presents with urticaria, hypotension, and throat swelling after bee sting.",
    )

    print(f"Score:     {result.score:.2f}")      # Score:     1.00
    print(f"Raw score: {result.raw_score:.2f}")  # Raw score: 18.00
    for c in result.report:
        print(f"  [{c.verdict}] (w={c.weight}) {c.requirement}")
        print(f"    → {c.reason}")

asyncio.run(main())
```

--------------------------------

### Define OneShotGenerateFn for PerCriterionOneShotGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Implement this function to handle LLM calls for PerCriterionOneShotGrader. It should return a OneShotOutput object containing criterion evaluations.

```python
from rubric import OneShotGenerateFn, OneShotOutput, CriterionEvaluation

async def my_generate_fn(
    system_prompt: str,
    user_prompt: str,
    **kwargs
) -> OneShotOutput:
    # Your LLM call here
    return OneShotOutput(
        criteria_evaluations=[
            CriterionEvaluation(
                criterion_idx=0,
                criterion_status="MET",
                explanation="First criterion satisfied"
            ),
            # ... more evaluations
        ]
    )
```

--------------------------------

### PerCriterionOutput Schema for Single Criterion Evaluation

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Defines the Pydantic model for per-criterion grader LLM output. Use `.model_json_schema()` to enable constrained decoding for LLM structured outputs.

```python
from rubric import PerCriterionOutput
from typing import Literal

# Manually construct (for testing/mocking)
output = PerCriterionOutput(
    criterion_status="MET",
    explanation="The response explicitly states the Q4 2023 base margin as 17.2%.",
)
print(output.criterion_status)  # "MET"
print(output.explanation)       # "The response explicitly states..."

# Get JSON schema for constrained decoding
schema = PerCriterionOutput.model_json_schema()
# Use with OpenAI structured outputs, Anthropic tool use, etc.
```

--------------------------------

### Define Criterion Objects

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Create Criterion objects to define evaluation rules with positive weights for desired traits and negative weights for errors. These are immutable Pydantic models.

```python
from rubric import Criterion

# Positive criterion: response should contain this
desired = Criterion(weight=10.0, requirement="States Q4 2023 base margin as 17.2%")

# Negative criterion: penalize if response makes this error
error = Criterion(weight=-5.0, requirement="Uses total deliveries instead of cash-only deliveries")

print(desired.weight)       # 10.0
print(desired.requirement)  # "States Q4 2023 base margin as 17.2%"
print(error.weight)         # -5.0
```

--------------------------------

### Construct Rubric from Dictionary List

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Create a Rubric object from a flat list of dictionaries, where each dictionary represents a criterion with 'weight' and 'requirement' keys.

```python
from rubric import Rubric

# Flat list format
rubric = Rubric.from_dict([
    {"weight": 10.0, "requirement": "Mentions Shapley attribution"},
    {"weight": 8.0,  "requirement": "States the 17.2% base margin figure"},
    {"weight": -15.0, "requirement": "Uses incorrect total deliveries metric"},
])
```

--------------------------------

### Create Rubric from Dictionary

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Construct a Rubric object from a Python dictionary. Supports nested sections and criteria with weights.

```python
rubric_sectioned = Rubric.from_dict({
    "sections": [
        {
            "criteria": [
                {"weight": 5.0, "requirement": "Cites primary sources"},
                {"weight": 3.0, "requirement": "Uses formal language"},
            ]
        },
        {
            "criteria": [
                {"weight": -2.0, "requirement": "Contains spelling errors"},
            ]
        },
    ]
})

print(len(rubric.rubric))           # 3
print(len(rubric_sectioned.rubric)) # 3
```

--------------------------------

### OneShotOutput Schema for Multi-Criterion LLM Response

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Pydantic model for one-shot grader LLM output containing evaluations for all criteria. The `.model_json_schema()` method is used for constrained decoding.

```python
from rubric import OneShotOutput, CriterionEvaluation

output = OneShotOutput(
    criteria_evaluations=[
        CriterionEvaluation(criterion_idx=0, criterion_status="MET",
                            explanation="Response mentions Paris."),
        CriterionEvaluation(criterion_idx=1, criterion_status="UNMET",
                            explanation="Response does not mention the Eiffel Tower."),
        CriterionEvaluation(criterion_idx=2, criterion_status="MET",
                            explanation="Response is under 100 words."),
    ]
)

schema = OneShotOutput.model_json_schema()  # For constrained decoding
for ev in output.criteria_evaluations:
    print(f"Criterion {ev.criterion_idx}: {ev.criterion_status}")
# Criterion 0: MET
# Criterion 1: UNMET
# Criterion 2: MET
```

--------------------------------

### Define RubricAsJudgeGenerateFn for RubricAsJudgeGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Implement this function for RubricAsJudgeGrader. It should return a RubricAsJudgeOutput object with an overall score and explanation.

```python
from rubric import RubricAsJudgeGenerateFn, RubricAsJudgeOutput

async def my_generate_fn(
    system_prompt: str,
    user_prompt: str,
    **kwargs
) -> RubricAsJudgeOutput:
    # Your LLM call here
    return RubricAsJudgeOutput(overall_score=85.0, explanation="Good quality overall")
```

--------------------------------

### EvaluationReport Structure

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Understand the structure of an EvaluationReport, which contains the normalized score, raw weighted score, original LLM output, and a per-criterion breakdown.

```python
from rubric.types import EvaluationReport, CriterionReport

# EvaluationReport structure
report = EvaluationReport(
    score=0.85,           # 0-1 normalized (or raw weighted sum if normalize=False)
    raw_score=12.75,      # Always the weighted sum
    llm_raw_score=12.75,  # Original LLM output (0-100 for RubricAsJudgeGrader)
    report=[
        CriterionReport(
            weight=10.0,
            requirement="States Q4 2023 base margin as 17.2%",
            verdict="MET",      # "MET" or "UNMET"
            reason="The response states Q4 2023 base margin as 17.2%.",
        ),
    ],
)

print(report.score)           # 0.85
print(report.raw_score)       # 12.75
for c in report.report:
    print(f"[{c.verdict}] {c.requirement}")  # [MET] States Q4 2023 base margin as 17.2%
```

--------------------------------

### PerCriterionOutput Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Schema for PerCriterionGrader, used for single-criterion evaluation. It includes the status and an explanation for the criterion.

```python
from typing import Literal
from pydantic import BaseModel

class PerCriterionOutput(BaseModel):
    criterion_status: Literal["MET", "UNMET"]
    explanation: str
```

--------------------------------

### OneShotOutput Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Schema for PerCriterionOneShotGrader, used for batch evaluation of multiple criteria in a single LLM call. It contains a list of criterion evaluations.

```python
from typing import Literal
from pydantic import BaseModel, Field

class CriterionEvaluation(BaseModel):
    criterion_idx: int  # 0-based index
    criterion_status: Literal["MET", "UNMET"]
    explanation: str

class OneShotOutput(BaseModel):
    criteria_evaluations: list[CriterionEvaluation] = Field(min_length=1)
```

--------------------------------

### Create Rubric from JSON String

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Construct a Rubric object from a JSON formatted string. The JSON should represent a list of criteria objects, each with 'weight' and 'requirement'.

```python
import json
from rubric import Rubric

json_data = json.dumps([
    {"weight": 10.0, "requirement": "Response is in English"},
    {"weight": 5.0,  "requirement": "Response answers the question directly"},
    {"weight": -3.0, "requirement": "Response contains hallucinated citations"},
])

rubric = Rubric.from_json(json_data)
print(len(rubric.rubric))  # 3
```

--------------------------------

### Grade Calculation Logic

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

This code illustrates the calculation of the final score based on positive and negative criteria weights and verdicts. The score is clamped to the [0, 1] range.

```python
total_positive_weight = sum(max(0.0, c.weight) for c in criteria)
weighted_sum = sum((1.0 if verdict == "MET" else 0.0) * c.weight for c in criteria)
score = max(0.0, min(1.0, weighted_sum / total_positive_weight))
```

--------------------------------

### RubricAsJudgeOutput Schema for Holistic Score LLM Response

Source: https://context7.com/the-llm-data-company/rubric/llms.txt

Pydantic model for holistic grader LLM output, providing a 0-100 score and an explanation. The `.model_json_schema()` method is available for constrained decoding.

```python
from rubric import RubricAsJudgeOutput

output = RubricAsJudgeOutput(
    overall_score=85.0,
    explanation="The response satisfies most criteria. The margin figure is correct and "
                "Shapley attribution is used, but one minor error criterion was triggered.",
)
print(output.overall_score)  # 85.0
print(output.explanation)    # "The response satisfies most criteria..."

schema = RubricAsJudgeOutput.model_json_schema()  # For constrained decoding
```

--------------------------------

### RubricAsJudgeOutput Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Schema for RubricAsJudgeGrader, used for holistic scoring. It provides an overall score and a brief explanation.

```python
from pydantic import BaseModel

class RubricAsJudgeOutput(BaseModel):
    overall_score: float  # 0-100 scale
    explanation: str      # Brief explanation of the score
```

--------------------------------

### Raw Score Conversion for RubricAsJudgeGrader

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

The LLM's 0-100 score is converted to weighted-sum semantics for consistency across grader types. This ensures comparability in training pipelines.

```python
raw_score = (llm_score / 100.0) * total_positive_weight
```

--------------------------------

### CriterionReport Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Extends Criterion to include the evaluation result for a single criterion, specifying the verdict and the reason.

```python
class CriterionReport(Criterion):
    verdict: Literal["MET", "UNMET"]
    reason: str
```

--------------------------------

### EvaluationReport Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Represents the final grading result, including a normalized score, raw score, optional LLM raw score, and per-criterion details.

```python
class EvaluationReport(BaseModel):
    score: float                              # Normalized 0-1 (or raw if normalize=False)
    raw_score: float | None                   # Always weighted-sum semantics (consistent across all graders)
    llm_raw_score: float | None               # Original LLM output before conversion
    report: list[CriterionReport] | None      # Per-criterion details (if available)
```

--------------------------------

### Criterion Pydantic Model

Source: https://github.com/the-llm-data-company/rubric/blob/main/CLAUDE.md

Defines a single evaluation criterion with its weight and requirement. Weight is positive for desired traits and negative for errors.

```python
class Criterion(BaseModel):
    weight: float      # Positive for desired traits, negative for errors
    requirement: str   # What to evaluate
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.