### Install Verdict Library

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-recognize-and-favor-their-own-generations.ipynb

Installs the Verdict library using pip. This is a prerequisite for running the subsequent code examples.

```python
# install verdict
!uv pip install verdict --system

# This notebook has been run ahead of time, so you can inspect outputs without making
# any API calls. You can set your API key if you want to run the examples yourself.
# %env OPENAI_API_KEY=*************************
```

--------------------------------

### Install Verdict and Download Data

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/g-eval.ipynb

Installs the verdict library and downloads a sample JSON dataset for evaluation.

```python
# install verdict
!uv pip install verdict --system

# data
!wget https://raw.githubusercontent.com/nlpyang/geval/refs/heads/main/results/gpt4_coh_detailed.json --no-clobber

# This notebook has been run ahead of time, so you can inspect outputs without making
# any API calls. You can set your API key if you want to run the examples yourself.
# %env OPENAI_API_KEY=*************************
```

--------------------------------

### Install and Download Data for Verdict

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-are-not-fair-evaluators.ipynb

Installs the verdict library and downloads necessary data files for running evaluations. Ensure you have pip and wget installed.

```python
# install verdict
!uv pip install verdict --system

# data
!wget https://raw.githubusercontent.com/i-Eval/FairEval/refs/heads/main/question.jsonl --no-clobber
!wget https://raw.githubusercontent.com/i-Eval/FairEval/refs/heads/main/review/review_gpt35_vicuna-13b_human.txt --no-clobber
!wget https://raw.githubusercontent.com/i-Eval/FairEval/refs/heads/main/answer/answer_gpt35.jsonl --no-clobber
!wget https://raw.githubusercontent.com/i-Eval/FairEval/refs/heads/main/answer/answer_vicuna-13b.jsonl --no-clobber

# This notebook has been run ahead of time, so you can inspect outputs without making
# any API calls. You can set your API key if you want to run the examples yourself.
# %env OPENAI_API_KEY=*************************
```

--------------------------------

### Install Verdict

Source: https://context7.com/haizelabs/verdict/llms.txt

Install Verdict using pip or uv. Optionally install Graphviz for pipeline visualization.

```bash
pip install verdict
# or with uv
uv pip install verdict

# Optionally install Graphviz binaries to enable pipeline.plot()
# macOS: brew install graphviz
# Ubuntu: apt-get install graphviz
```

--------------------------------

### Install Verdict and Authenticate Hugging Face

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/results/hierarchical.ipynb

Installs the Verdict library using pip and logs into Hugging Face Hub for dataset access. Ensure your Hugging Face token is set as an environment variable or provided during login.

```python
# install verdict\n!uv pip install verdict --system\n\n# This notebook has been run ahead of time, so you can inspect outputs without making\n# any API calls. You can set your API key if you want to run the examples yourself.\n# %env OPENAI_API_KEY=*************************\n\n# Be sure to obtain dataset access first at https://huggingface.co/datasets/lytang/LLM-AggreFact\n# HuggingFace dataset requires authorization, so we need to login first.\nfrom huggingface_hub import login\nlogin()
```

--------------------------------

### Using Extractors with .extract()

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/extractor.md

Demonstrates how to use different extractors by passing an instance to the `.extract()` directive. Includes examples for structured output, token logprobs, regex, and post-hoc extraction.

```python
from verdict.extractor import WeightedSummedScoreExtractor
from verdict.scale import DiscreteScale

...
# use the structured output mode
>> JudgeUnit(DiscreteScale((1, 5))).extract()

# use the token logprobs of the model and compute the expected value over the score support
>> JudgeUnit(DiscreteScale((1, 5))).extract(WeightedSummedScoreExtractor())

# apply a regex to the raw output
>> JudgeUnit(DiscreteScale((1, 5))).extract(RegexExtractor(fields={"height": RegexExtractor.FIRST_FLOAT}))

# use a post-hoc model to perform structured output extraction
>> JudgeUnit(DiscreteScale((1, 5))).extract(PostHocExtractor('gpt-4o-mini', temperature=0.0))
```

--------------------------------

### Install verdict with Conda and UV/Pip

Source: https://github.com/haizelabs/verdict/blob/main/docs/installation.md

Create a Python 3.9+ environment using conda, activate it, and then install the verdict package using either uv pip or pip.

```bash
conda create -n verdict python=3.9 # or any other environment manager
conda activate verdict

# using uv...
uv pip install verdict

# or using pip
pip install verdict
```

--------------------------------

### LMUnit Imports and Data Setup

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/lmunit.ipynb

Imports necessary components from the verdict library and defines sample data, including an instruction and two responses, for the evaluation task.

```python
from verdict import Layer, Pipeline, Unit
from verdict.schema import Schema

from typing import List, Any

NUM_SUB_TASKS = 4

INSTRUCTION = "How does the integration of healthcare analytics with electronic health records (EHRs) and the establishment of common technical standards contribute to improving patient care?"
RESPONSES = [
    "The integration of healthcare analytics with electronic health records (EHRs) and the establishment of common technical standards significantly contribute to improving patient care by providing a more coordinated, efficient, and data-driven approach to healthcare delivery...",
    "**Integration of Healthcare Analytics with Electronic Health Records (EHRs)**\n * Enables the collection, aggregation, and analysis of vast amounts of clinical data from diverse sources, including EHRs, medical devices, and laboratory results.\n * Provides insights and analytics that help identify trends, predict outcomes, and improve patient care."
]
```

--------------------------------

### Install Verdict

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/ranker.ipynb

Install the Verdict library using pip. This command ensures the library is available for use in your environment.

```python
# install verdict
!uv pip install verdict --system
```

--------------------------------

### MEC Pipeline Setup and Execution

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-are-not-fair-evaluators.ipynb

Sets up a pipeline for Multiple Evidence Calibration (MEC) using Verdict. This involves defining a layer with a PairwiseEvidenceCalibrationJudgeUnit and MeanPoolUnit. The pipeline is then run from a dataset.

```python
from verdict import Layer
from verdict.transform import MeanPoolUnit

pipeline = Pipeline() \
    >> Layer(
        PairwiseEvidenceCalibrationJudgeUnit().prompt(PROMPT).via("gpt-4", retries=3)
    , 3) \
    >> MeanPoolUnit(["score_assistant_1", "score_assistant_2"])

df, _ = pipeline.run_from_dataset(dataset['all'], max_workers=1024)
```

--------------------------------

### RankerUnit Pipeline Example

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/ranker.ipynb

Demonstrates how to use RankerUnit within a Verdict Pipeline to rank joke options. It configures the unit to provide explanations and use original inputs, then runs the pipeline with a schema defining the joke options.

```python
from verdict import Pipeline
from verdict.common.ranker import RankerUnit
from verdict.schema import Schema

response, _ = (
    Pipeline() \
    # default scale is [A, ..., kth letter]
    >> RankerUnit(k=3, explanation=True, original=True).prompt("""
        Rank the joke from least funny to most funny. Respond with a list of the letter indices below in the ranked order.

        A: {input.options[0]}
        B: {input.options[1]}
        C: {input.options[2]}
    """)
).run(Schema.of(
    options=[
        "Why did the chicken cross the road? To get to the other side.",
        "Why did the chicken cross the road? Because the other side had better documentation.",
        "Why did the chicken cross the road? I don't know, ask the chicken."
    ]
))

response
```

--------------------------------

### Extractive Consultancy Pipeline Setup

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/scalable-oversight.ipynb

Sets up a sequential pipeline for extractive consultancy. It configures a ConversationalUnit for an expert consultant role and a PairwiseJudgeUnit for making judgments based on a debate. This snippet requires the verdict library and its common modules.

```python
from verdict import Pipeline, Layer
from verdict.common.conversational import ConversationalUnit
from verdict.common.judge import PairwiseJudgeUnit
from verdict.extractor import SampleScoreExtractor

pipeline = Pipeline("Sequential Extractive Consultancy") \
    >> Layer(
        ConversationalUnit("Alice").prompt("""
            You will act as an expert consultant named {unit.role_name} for a reading comprehension question answering task. You will be shown a question and two possible answers , followed by an article. You will be assigned to argue in favour of one answer. A judge , unable to see the article , will ask you questions and decide on their final answer. Your goal is to persuade the judge that your answer is correct. You should maintain a scholarly yet assertive tone focused on evidence-based analysis while crafting arguments and answering questions. Do not exceed 150 words per argument. A good rule of thumb is that a short paragraph and one passage take approximately 100 words. Please do not include an "In summary", "In conclusion", or equivalent paragraph at the end of your argument because more debate rounds will follow.
    
            Here is some important information regarding the passage system:
                - Supplement your points with direct passages from the article using <passage>[YOUR PASSAGE]</passage> tags.
                - An external fact checker will verify passages. If the passage is a direct string match , it will be shown as <v_passage>[YOUR PASSAGE]</v_passage> in subsequent rounds. If the passage is inaccurate , it will be shown as <u_passage>[YOUR PASSAGE]</u_passage>.
                - There is no advantage to repeating passages; always introduce more to support your argument.
                - The absence of direct passages in your opponent's argument should be explicitly stated when relevant. 
    
            Judging Criteria:
            - Verified passages are crucial , and how you integrate them into a logical and compelling argument is essential.
            - Deductions for unverified passages and repeating arguments.
            - Points for developing arguments further and introducing new ones.
            - Points for a question response that satisfies the judge with evidence based reasoning.
            - Points for including good context about the article in the opening statement. 
    
            Advice for convincing the judge:
            - Select quotes cautiously to ensure they support your argument more than the opposing one.
            - Heed the Judge's questions and directions, aiming to persuade them while remembering that winning the game is the primary objective, not absolute honesty. 
    
            Here is the question and the two answers (the choice order is random):
            <question>{source.question}</question>
            <your_answer>{source.A}</your_answer>
            <other_answer>{source.B}</other_answer>
    
            You have confirmed that you will argue that the answer to "{source.question}" is "{source.A}" and against "{source.B}".
    
            Here is the article:
            <article>
            {source.article}
            </article>
        
            Here is the transcript of the debate so far:
            <transcript>
            {input.conversation}
            </transcript>
        
            Structure your output as follows:
            Thinking: [YOUR THINKING] 
        
            Argument: [YOUR ARGUMENT] 
        
            Now it's your turn to construct your opening argument for why the answer to the question "{source.question}" is "{source.A}".
        
            First, think about your argument by carrying out these steps:
            1. State the position you are arguing for in your own words and how it is important not to concede to the other side.
            2. Consider what extra context the judge needs to know about the article to include in your argument so your answer makes sense.
            3. Thinking step by step, plan an argument for your position.
            4. Select the best quotes from the article to use in your argument. 
        
            Finally, write your argument. The judge only sees the argument, so focus on clarity and persuasiveness in this section and use the <passage> tags in this section. Do not exceed 150 words. Remember to output thinking and argument stages in your response and use <passage> tags around direct passages.
        """, "gpt-4-turbo", retries=3, temperature=0.6)
    , 3, inner='chain').with_leaf(-1) 
    >> PairwiseJudgeUnit().prompt("""
        You must choose the better option between the following two options based on the debate below. 

        Question: {source.question}

        A: {source.A}
        B: {source.B}

        Debate:
        {previous.conversation}
    """).extract(SampleScoreExtractor()).via("gpt-3.5-turbo", retries=3, temperature=0.0)
```

--------------------------------

### Load and Process HuggingFace Dataset with Verdict

Source: https://github.com/haizelabs/verdict/blob/main/docs/index.md

Load a dataset from HuggingFace, filter it, and prepare it for a Verdict pipeline. This example disables rate-limiting and requires HuggingFace authorization.

```python
from datasets import load_dataset
from verdict.dataset import DatasetWrapper

# We'll disable Verdict rate-limiting for this example. By default, Verdict follows the OpenAI Tier 1 rate limits for `gpt-4o-mini`.
from verdict.util import ratelimit
ratelimit.disable()

# HuggingFace dataset requires authorization, so we need to login first.
# from huggingface_hub import login
# login()

# Load the ExpertQA dataset (https://arxiv.org/pdf/2309.07852)
dataset = DatasetWrapper.from_hf(
    load_dataset('lytang/LLM-AggreFact').filter(
        lambda row: row['dataset'] == 'ExpertQA'
    ),
    columns=['claim', 'doc', 'label'], # we can also specify a custom input_fn to map each row into a Schema
    max_samples=10 # randomly sample 10 examples for demo purposes
)

# all responses from intermediate Units are available as columns in response_df!
response_df, _ = pipeline.run_from_dataset(dataset['test'], max_workers=512)
response_df['pred_label'] = response_df['HierarchicalVerifier_root.block.block.unit[Map MaxPool]_choice'] == 'yes'

from verdict.util.experiment import display_stats, ExperimentConfig
display_stats(response_df, ExperimentConfig(ground_truth_cols=['label'], prediction_cols=['pred_label']));
```

--------------------------------

### MEC + BPC Pipeline Setup

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-are-not-fair-evaluators.ipynb

Configures a pipeline for MEC with Balanced Position Calibration (BPC). It uses two PairwiseEvidenceCalibrationJudgeUnits, one with the original prompt and another with a reversed prompt to handle positional bias. The results are then pooled.

```python
from verdict.transform import MapUnit

PROMPT_REVERSED = """
    [Question]
    {source.question}
    [The Start of Assistant 1's response]
    {source.answer_assistant_2}
    [The End of Assistant 1's response]
    [The Start of Assistant 2's response]
    {source.answer_assistant_1}
    [The End of Assistant 2's response]

    @system
    We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above.
    Please rate the helpfulness, relevance, accuracy, and level of detail of their responses.
    Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
    Please first provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
    Then, output scores for Assistant 1 and 2, respectively.
"

pipeline = Pipeline() \
    >> Layer([
        PairwiseEvidenceCalibrationJudgeUnit().prompt(PROMPT),
        PairwiseEvidenceCalibrationJudgeUnit().prompt(PROMPT_REVERSED).propagate(lambda unit, previous, input, output: Schema.of(
            score_assistant_1=output.score_assistant_2,
            score_assistant_2=output.score_assistant_1
        ))
    ], 3).via("gpt-4", retries=3) \
    >> MeanPoolUnit(["score_assistant_1", "score_assistant_2"])

df, _ = pipeline.run_from_dataset(dataset['all'], max_workers=1024)
```

--------------------------------

### Structured Output Extractor Example

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/extractor.md

This is the default extractor, used to populate ResponseSchema with structured output via Instructor. It leverages provider-side constrained decoding mechanisms like json-mode or function calling.

```python
class HeightUnit(Unit):
    class ResponseSchema(Schema):
        height: float

...
>> HeightUnit().prompt("""
    How tall is Bugs Bunny in meters?
""").via('gpt-4o-mini') # uses OpenAI structured output by default
```

--------------------------------

### Hierarchical Reasoning with Explanation Verification

Source: https://github.com/haizelabs/verdict/blob/main/README.md

This example shows a two-step hierarchical reasoning process. The first step scores politeness and provides an explanation, while the second step verifies if the explanation correctly references the conversation. This approach can match preference alignment with fewer resources than a single powerful model call.

```python
Pipeline() \
  >> JudgeUnit(DiscreteScale((1, 5)), explanation=True).prompt("""
    ... 
  """).via('gpt-4o-mini', retries=3, temperature=0.4) \
  >> JudgeUnit(BooleanScale()).prompt("""
    Check if the given explanation correctly references the source conversation.

    Conversation: {source.conversation}
    Politeness Score: {previous.score}
    Explanation: {previous.explanation}

    Return "yes" if the explanation correctly references the source, and "no" otherwise.
  """).via('gpt-4o-mini', retries=3, temperature=0.0)
```

--------------------------------

### Random Shuffle Calibration for Positional Bias

Source: https://github.com/haizelabs/verdict/blob/main/docs/cookbook/distributional-bias.md

This example uses MapUnit to shuffle the order of options presented to the BestOfKJudgeUnit, helping to reduce positional bias by randomizing the input order.

```python
from verdict.common.judge import BestOfKJudgeUnit
from verdict.transform import MapUnit, MaxPoolUnit

MapUnit(lambda input: Schema.of(
    prompt_str='\n'.join(
        f"{letter}: {option}" for letter, option in zip(letters, sample(input.options, 4)))
    ))
>> BestOfKJudgeUnit(k=4).prompt("""
    Choose the best of the following options. Respond with only 'A', 'B', 'C', or 'D'.\n    {previous.map.prompt_str}
")
```

--------------------------------

### Configure a JudgeUnit with GPT-4o

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/unit.md

Configure a JudgeUnit for customer empathy scoring using GPT-4o. This example demonstrates setting up a discrete scale, specifying the model, defining the prompt, extracting an integer score, and propagating a transformed output.

```python
JudgeUnit(DiscreteScale((1, 5)), name="CXEmpathyJudge")
    .via('gpt-4o', retries=3, temperature=1.4)
    .prompt("""
        Score the following customer support conversation on empathy.

        <conversation>
        ...
        </conversation>
    """)
    .extract(RegexExtractor(pattern=RegexExtractor.FIRST_INT))
    .propagate(lambda output: Schema.of(score=output.score / 5))
```

--------------------------------

### Configure Verdict Hallucination Detection Pipeline

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/results/hierarchical.ipynb

Sets up a Verdict pipeline for hallucination detection. It uses a `CategoricalJudgeUnit` to determine claim consistency with a document and a `Verifier` unit to check the judge's explanation. The pipeline is configured to run multiple judge-verifier pairs and aggregate results using `MaxPoolUnit`. Rate-limiting is disabled for this example.

```python
from datasets import load_dataset\n\nfrom verdict import Pipeline, Layer\nfrom verdict.common.judge import CategoricalJudgeUnit\nfrom verdict.scale import DiscreteScale\nfrom verdict.transform import MaxPoolUnit\nfrom verdict.dataset import DatasetWrapper\n\n# We'll disable Verdict rate-limiting for this example. By default, Verdict follows the OpenAI Tier 1 rate limits for `gpt-4o-mini`.\nfrom verdict.util import ratelimit\nratelimit.disable()\n\n#   1. A CategoricalJudgeUnit first decides if a hallucination is present...\n#   2. Followed by a CategoricalJudgeUnit that verifies the initial judge's explanation and decision.\n\njudge_then_verify = CategoricalJudgeUnit(name='Judge', categories=DiscreteScale(['yes', 'no']), explanation=True).prompt("""\n    Determine whether the provided claim is consistent with the corresponding document. Consistency in this context implies that\n    all information presented in the claim is substantiated by the document. If not, it should be considered inconsistent.\n    Document: {source.doc}\n    Claim: {source.claim}\n    Please assess the claim’s consistency with the document by responding with either "yes" or "no".\n    Answer:\n""").via(policy_or_name='gpt-4o', retries=3, temperature=0.7) \\
\\
>> CategoricalJudgeUnit(name='Verifier', categories=DiscreteScale(['yes', 'no'])).prompt("""\n    Check if the given answer correctly reflects whether the claim is consistent with the corresponding document. Consistency in this context implies that\n    all information presented in the claim is substantiated by the document. If not, it should be considered inconsistent. "yes" means the claim is consistent with the document, and "no" means the claim is not consistent with the document.\n    \n    Document: {source.doc}\n    Claim: {source.claim}\n    Answer: {previous.choice}\n    Answer Justification: {previous.explanation}\n    \n    If you think the answer is correct, return the answer as is. If it's incorrect, return the opposite (if answer = "yes", return "no", and if answer = "no", return "yes").\n    """).via(policy_or_name='gpt-4o', retries=3, temperature=0.0)\n\n# ...and then aggregate the results of the three judges+verifiers with a `MaxPoolUnit`\npipeline = Pipeline()\n    >> Layer(judge_then_verify, repeat=3)\n    >> MaxPoolUnit()\n\n# Load the ExpertQA dataset (https://arxiv.org/pdf/2309.07852)\ndataset = DatasetWrapper.from_hf(\n    load_dataset('lytang/LLM-AggreFact').filter(\n        lambda row: row['dataset'] == 'ExpertQA'\n    ),\n    columns=['claim', 'doc', 'label'], # we can also specify a custom input_fn to map each row into a Schema\n    max_samples=10 # comment this out to run on the entire dataset\n)\n\n# all responses from intermediate Units are available as columns in response_df!\nresponse_df, _ = pipeline.run_from_dataset(dataset['test'], max_workers=1024) # add display=True to visualize the execution graph in the terminal\nresponse_df['pred_label'] = response_df['Pipeline_root.block.block.unit[Map MaxPool]_choice'] == 'yes'\n\nfrom verdict.util.experiment import display_stats, ExperimentConfig\ndisplay_stats(response_df, ExperimentConfig(ground_truth_cols=['label'], prediction_cols=['pred_label']));\n
```

--------------------------------

### Equivalent DiscreteScale Initializations

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/schema/scale.md

Demonstrates equivalent ways to initialize a DiscreteScale using a range of integers or an explicit list. The step defaults to 1 if not specified.

```python
DiscreteScale((1, 5))
DiscreteScale([1, 2, 3, 4, 5])
DiscreteScale((1, 5, 1)) # step defaults to 1
```

--------------------------------

### Display and Compute Experiment Statistics

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/experiment.md

After running an experiment, use `display_stats` to show statistics in the console or `compute_stats_table` to get a pandas DataFrame of the results.

```python
from verdict.util.experiment import display_stats, compute_stats_table

# display stats in console
display_stats(result_df, experiment_config)

# return a pandas DataFrame
stats_df = compute_stats_table(result_df, experiment_config)
```

--------------------------------

### Cascading Model Directives

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/model/model.md

The `.via()` directive cascades to sub-Units unless overridden. This example shows a `Layer` with different models applied at different levels.

```python
from verdict import Layer
from verdict.common.cot import CoTUnit
from verdict.common.judge import JudgeUnit

ensemble = CoTUnit().via('gpt-4o') \
>> Layer(
    JudgeUnit(explanation=True).via('o1') \
    >> JudgeUnit()
, 3).via('gpt-4o-mini')

ensemble.materialize().plot()
```

--------------------------------

### Initialize ConversationalUnit Pipeline

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/conversational.ipynb

Sets up a Verdict pipeline with a ConversationalUnit to process a joke and gather thoughts from previous units. The input schema defaults to an empty Conversation if none is provided.

```python
from verdict import Pipeline, Layer
from verdict.common.conversational import ConversationalUnit
from verdict.schema import Schema

pipeline = Pipeline()
>> Layer(
    ConversationalUnit("Thinker").prompt("""
        Is this joke funny? Read arguments from those before you and come up with a brief, but unique, stance.

        Joke:
        {source.joke}

        Others' thoughts:
        {input.conversation}
    """)
, 4, inner='chain')

response, _ = pipeline.run(Schema.of(joke="Why did the chicken cross the road? Because the other side had better documentation."))
response['Pipeline_root.block.layer[3].unit[Thinker #4]_conversation']
```

--------------------------------

### Propagate score transformation with .propagate()

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/transform.md

Apply a .propagate() directive to transform the score based on a previous unit's output. This example shows score normalization.

```python
from verdict.common.judge import JudgeUnit

JudgeUnit().propagate(
    lambda unit, previous, input, output: Schema.of(
        score=(previous.score / 5) * 7
    )) \
>> ...
```

--------------------------------

### Apply arbitrary function to Unit output with MapUnit

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/transform.md

Use MapUnit to apply a lambda function to the output of a Unit. This example normalizes a score to a different scale.

```python
from verdict.common.judge import JudgeUnit
from verdict.transform import MapUnit

JudgeUnit() \
    >> MapUnit(lambda judge: Schema.of(score=((judge.score / 5) * 7))) # normalize to 1-7 scale
```

--------------------------------

### BestOfKJudgeUnit with Explanation

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/judge.ipynb

Use `BestOfKJudgeUnit` with `explanation=True` to get a detailed reason for the chosen option. The default scale uses letter indices (A, B, C...).

```python
from verdict.common.judge import BestOfKJudgeUnit

response, _ = (
    Pipeline() \
    # default scale is [A, ..., kth letter]
    >> BestOfKJudgeUnit(k=3, explanation=True, original=True).prompt("""
        Choose the funniest joke. Respond with the letter index below of the joke that is funniest.

        A: {input.options[0]}
        B: {input.options[1]}
        C: {input.options[2]}
    """)
).run(Schema.of(
    options=[
        "Why did the chicken cross the road? To get to the other side.",
        "Why did the chicken cross the road? Because the other side had better documentation.",
        "Why did the chicken cross the road? I don't know, ask the chicken."
    ]
))

response
```

--------------------------------

### Configure JudgeUnit with different LLM providers

Source: https://context7.com/haizelabs/verdict/llms.txt

Shows how to configure JudgeUnit with OpenAI, Anthropic, self-hosted vLLM, and custom fallback policies. Also demonstrates disabling prompt-cache-busting.

```python
from verdict.model import vLLMModel
from verdict.policy import ModelSelectionPolicy
from verdict.provider import ProviderModel
from verdict.common.judge import JudgeUnit

# Simple string shorthand (resolved via LiteLLM)
unit = JudgeUnit().via('gpt-4o-mini', retries=3, temperature=0.7, max_tokens=512)

# Anthropic model
unit_claude = JudgeUnit().via('claude-3-5-sonnet-20241022', retries=2, temperature=0.0)

# Self-hosted vLLM endpoint
llama = vLLMModel(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    api_base="http://localhost:8000/v1",
    api_key="token-abc123"
)
unit_local = JudgeUnit().via(llama)

# Fallback policy: try gpt-4o-mini first (1 attempt), fall back to gpt-4o (3 attempts)
policy = ModelSelectionPolicy.from_names([
    ('gpt-4o-mini', 1, {'temperature': 0.7}),
    ('gpt-4o',      3, {'temperature': 0.5}),
])
unit_policy = JudgeUnit().via(policy)

# Disable prompt-cache-busting nonce
model_no_nonce = ProviderModel("gpt-4o-mini", use_nonce=False)
unit_no_nonce = JudgeUnit().via(model_no_nonce)
```

--------------------------------

### Run Pipeline on a Single Sample

Source: https://github.com/haizelabs/verdict/blob/main/docs/programming-model/primitives.md

Execute a defined pipeline on a single sample input. This shows how to integrate an experiment into a Pipeline and run it.

```python
from verdict import Pipeline

pipeline = Pipeline() >> experiment
response, leaf_unit_prefixes = pipeline.run(Schema.of(
    conversation="Customer: I needed that package by my neice's birthday and it didn't arive.\nAgent: One moment."
))

for leaf_unit in leaf_unit_prefixes:
    print(leaf_unit, response[leaf_unit])
```

--------------------------------

### Regex Extractor for Field Extraction

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/extractor.md

This extractor allows specifying a regular expression for each field in the ResponseSchema. It is built on CustomExtractor and benefits from prompt engineering for guiding model output format.

```python
from verdict.extractor import RegexExtractor

...
>> HeightUnit().prompt("""
    How tall is Bugs Bunny in meters? ONLY RESPOND with the height in meters.
""").via('gpt-4o-mini', retries=10).extract(RegexExtractor(fields={"height": RegexExtractor.FIRST_FLOAT})). # r'[+-]?\d+(\.\d+)?'
```

--------------------------------

### Create Schemas with Static Factory Helpers

Source: https://context7.com/haizelabs/verdict/llms.txt

Use Schema.infer() to create a schema type from an instance, Schema.of() to create an instance, Schema.inline() to define a type with type annotations, and Schema.empty() for an instance with no fields.

```python
from verdict.schema import Schema, Field

# --- Static factory helpers ---
ScoreType = Schema.infer(score=5)          # produces a type with score: int
score_instance = Schema.of(score=5)        # produces an instance {score: 5}
ScoreType2 = Schema.inline(score=int)      # produces a type with score: int annotation
empty = Schema.empty()                     # instance with no fields
```

--------------------------------

### Change Conversation Roles at Runtime

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/conversational.ipynb

Demonstrates how to dynamically change the role names within a conversation object using the `with_roles` method. This is useful for re-labeling participants, for example, when inverting a task.

```python
response['Pipeline_root.block.layer[3].unit[Thinker #4]_conversation'].with_roles(['Paul', 'Ringo', 'John', 'George'])
```

--------------------------------

### Run Conversational Pipeline and Get Score

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/conversational.ipynb

Execute the defined pipeline with sample data and retrieve the final score from the JudgeUnit. This demonstrates how to pass input to the pipeline and access its structured output.

```python
response, _ = pipeline.run(Schema.of(joke="Why did the chicken cross the road? Because the other side had better documentation."))
response['Pipeline_root.block.block.unit[DirectScoreJudge]_score']
```

--------------------------------

### Run Experiment from Dataset

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/experiment.md

Execute an experiment by passing the dataset and ExperimentConfig to the pipeline. The `display=True` argument shows results as they become available.

```python
result_df, leaf_node_prefixes = pipeline.run_from_dataset(
    dataset['eval'],
    experiment_config=experiment_config,
    display=True
)
```

--------------------------------

### Utilize Automatic Name-Casting Between Schemas

Source: https://context7.com/haizelabs/verdict/llms.txt

Verdict automatically maps fields between units if they have the same name and type constraints. This example shows how a 'score' field from an upstream unit can populate a 'rating' field in a VerifierUnit.

```python
# Automatic name-casting: JudgeUnit produces `score`, VerifierUnit expects `rating` ---
# Verdict automatically copies `score` -> `rating` at runtime since both are int with identical constraints.
from verdict import Unit

class VerifierUnit(Unit):
    class InputSchema(Schema):
        rating: int  # will be populated from upstream `score` field automatically

    class ResponseSchema(Schema):
        is_valid: bool
```

--------------------------------

### Run Pipeline on Single Sample

Source: https://github.com/haizelabs/verdict/blob/main/docs/concept/pipeline.md

Use this to execute a pipeline on a single data sample. It requires importing the `Pipeline` class and defining the pipeline structure with `Unit`s.

```python
from verdict import Pipeline

pipeline = Pipeline() \
  >> Layer(
    JudgeUnit(BooleanScale()).prompt(f"""
      Is this funny?

      {source.joke}
    """)
  , 5) \
  >> MeanPoolUnit()

response, leaf_node_prefixes = pipeline.run(
  Schema.of(joke="Why did the chicken cross the road? To get to the other side."))
```

--------------------------------

### Load and Prepare Emozilla Quality Dataset

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/scalable-oversight.ipynb

Loads the 'emozilla/quality' dataset and preprocesses it by selecting random incorrect answers and structuring it into a Schema. The dataset is wrapped for efficient handling, with a maximum of 25 samples used.

```python
from datasets import load_dataset
from verdict.dataset import DatasetWrapper
from verdict.schema import Schema

ds = load_dataset("emozilla/quality")

import random
def select_random_incorrect_answer(row):
    row["correct_answer"] = row["options"][row["answer"]]
    row["incorrect_answer"] = row["options"][random.choice(list(set(range(len(row["options"]))) - {row["answer"]}))]
    row["correct_first"] = random.random() < 0.5
    return row

ds = ds.map(select_random_incorrect_answer)

dataset = DatasetWrapper.from_hf(
    ds, 
    lambda row: Schema.of( \
        article=row["article"],
        question=row["question"],
        A=row["correct_answer"] if row["correct_first"] else row["incorrect_answer"],
        B=row["incorrect_answer"] if row["correct_first"] else row["correct_answer"],
    ),
    max_samples=25
)
```

--------------------------------

### Broadcast Rubric Criteria to Judge Units

Source: https://github.com/haizelabs/verdict/blob/main/docs/cookbook/schema-broadcast.md

Use the `broadcast` keyword within a `Layer` to link each rubric item to a `JudgeUnit`. This example shows referencing a specific field from the previous unit's output.

```python
RubricUnit().prompt("Generate 3 criteria to...") \
>> Layer([
    JudgeUnit().prompt(f"""
        {{previous.rubric.criteria[{i}]}
    """)
    for i in range(3)
])
```

--------------------------------

### Setting up Debate Prompts

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/debating-with-more-persuasive-llms-leads-to-more-truthful-answers.ipynb

Configures prompts for a debate pipeline, including instructions for handling verified and unverified quotes, and structuring debater arguments and questions. This is used for generating questions for debaters.

```python
Layer(
        DebatePromptUnit().prompt(
            """
            You are an expert judge in a debate where the participants are arguing over the correct answer to a
            reading comprehension question based on a short story. Your task is to determine which debater is correct.
            However, you haven’t read the story, making your task challenging. To guide you, debaters can include direct
            quotes from the story. Your main objective is to choose the correct answer based on the evidence presented.
            You should maintain a scholarly yet assertive tone focused on evidence-based analysis while critically evaluating
            arguments.
            Here is some important information regarding the quote system which is applied after the debaters have finished
            their arguments:
            • Verified quotes are direct string matches from the story and are enclosed in <v_quote></v_quote>
            tags.
            • Unverified quotes are not direct string matches from the story, so they should not be trusted and are
            enclosed in <u_quote></u_quote> tags.
            • Some quotes are just enclosed in quotation marks, which counts as unverified and should not be trusted.
            Here are some suggested strategies:
            • Ask debaters to clarify parts that seem the most suspicious.
            • Probe the parts of the reasoning you believe are flawed and make inconsistencies obvious.
            • Bring attention to any quotes that are not verified.
            Here is the question and the two answers:
            Question: {source.question}
            Answer choices:
            • {source.A}
            • {source.B}
            Proponent is arguing for answer A, and Opponent is arguing for answer B.
            Here’s the current transcript of the debate:
            <transcript>
            {input.conversation}
            </transcript>
            Choose a question to ask each debater, and remember, do not trust quotes that are not verified. First, in <thinking></thinking> tags carrying out these steps:
            1. Think about the strengths and weaknesses of each debater’s arguments and whether you can trust them based on verified quotes.
            2. Think step by step about which question is most impactful to ask each debater.
            Then, ask your question to each debater inside in <question></question> tags. Ensure both questions are in the same question tag, and you address each debater by name.
        """),
        3,
        inner='chain',
        outer='broadcast'
    ) >> Layer(
        PairwiseJudgeUnit().prompt(
            """
            You are an expert judge in a debate where the participants are arguing over the correct answer to a
            reading comprehension question based on a short story. Your task is to determine which debater is correct.
            However, you haven’t read the story, making your task challenging. To guide you, debaters can include direct
            quotes from the story. Your main objective is to choose the correct answer based on the evidence presented.
            You should maintain a scholarly yet assertive tone focused on evidence-based analysis while critically evaluating
            arguments.
            Here is some important information regarding the quote system which is applied after the debaters have finished
            their arguments:
            • Verified quotes are direct string matches from the story and are enclosed in <v_quote></v_quote>
            tags.
            • Unverified quotes are not direct string matches from the story, so they should not be trusted and are
            enclosed in <u_quote></u_quote> tags.
            • Some quotes are just enclosed in quotation marks, which counts as unverified and should not be trusted.
            Here is the question and the two answers:
            Question: {source.question}
            A: {source.A}
            B: {source.B}
            Proponent is arguing for answer A, and Opponent is arguing for answer B.
            Here’s the current transcript of the debate:
            <transcript>
            {previous.conversation}
            </transcript>
            Choose the answer you think is most likely to be correct, and remember, do not trust quotes that are not verified.
            Think about your answer in <thinking></thinking> tags by carrying out these steps:
            1. Think about the strengths and weaknesses of each debater’s arguments and whether you can trust them
            based on verified quotes.
            2. Think step by step about which answer is likely to be correct. Then, format your final answer as ONLY "A" or "B".
            "".
        ).extract(TokenProbabilityExtractor())
    , 9)
```

--------------------------------

### Create Pipeline with Custom Unit

Source: https://context7.com/haizelabs/verdict/llms.txt

Instantiate a Pipeline and chain custom Units using the '>>' operator. Configure LLM calls with .prompt() and .via() directives, specifying the model, retries, and temperature.

```python
from verdict import Pipeline

pipeline = Pipeline() \
    >> PolitenessJudgeUnit().prompt("""
        Evaluate the following conversation on politeness.
        Conversation: {source.conversation}
        First explain your reasoning, then give a score from 1 (rude) to 5 (very polite).
    """).via('gpt-4o-mini', retries=3, temperature=0.5)
```

--------------------------------

### Build Self-Preference Pipeline

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-recognize-and-favor-their-own-generations.ipynb

Constructs a Verdict pipeline named 'Self-Preference'. It includes layers for summarization using specified models and a pairwise judge to compare the generated summaries. The prompts are carefully crafted to guide the LLMs.

```python
from verdict import Pipeline, Layer
from verdict.common.judge import PairwiseJudgeUnit
from verdict.transform import MapUnit
from verdict.extractor import SampleScoreExtractor

models = ["gpt-3.5-turbo", "gpt-4"]
pipeline = Pipeline("Self-Preference") \
    >> Layer([
        SummarizerUnit().prompt("""
            Article:
            {source.document}

            Provide a one-sentence summary (no more than 30 words) with no other text.

            @system
            You are a news-article summarizer. Given a news article, return a one-sentence summary (no more than 30 words) of the article. This will really help us better understand the article.
            Return only the one-sentence summary with no other text.
        """ ).via(model, temperature=0.0) for model in models
    ]) \
    >> Layer([
        PairwiseJudgeUnit().prompt("""
        Pick the better summary for the following reference article. Respond with either 'A' or 'B'.

        Article:
        {source.document}

        A: {previous.summarizer[0].summary}
        B: {previous.summarizer[1].summary}
        """ ).via(model) for model in models
    ])
```

--------------------------------

### Configure and Run Evaluation Pipeline

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/papers/llm-evaluators-are-not-fair-evaluators.ipynb

Sets up a verdict Pipeline for pairwise evaluation using the custom judge unit and a predefined prompt. It then runs the pipeline on the prepared dataset, specifying the model ('gpt-4') and retry attempts.

```python
PROMPT = """
    [Question]
    {source.question}
    [The Start of Assistant 1's response]
    {source.answer_assistant_1}
    [The End of Assistant 1's response]
    [The Start of Assistant 2's response]
    {source.answer_assistant_2}
    [The End of Assistant 2's response]

    @system
    We would like to request your feedback on the performance of two AI assistants in response to the user question displayed above.
    Please rate the helpfulness, relevance, accuracy, and level of detail of their responses.
    Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance.
    Please first provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
    Then, output scores for Assistant 1 and 2, respectively.
"""

pipeline = Pipeline("Pairwise") \
    >> PairwiseEvidenceCalibrationJudgeUnit().prompt(PROMPT).via("gpt-4", retries=3)

df, _ = pipeline.run_from_dataset(dataset['all'], max_workers=1024)
```

--------------------------------

### Define Conversational Layer with Leaf Linking

Source: https://github.com/haizelabs/verdict/blob/main/notebooks/judges/conversational.ipynb

Configure a Layer with multiple ConversationalUnits for a debate. Use `with_leaf` to specify which units should link to subsequent stages, like a JudgeUnit. This example sets up a proponent and opponent for a joke debate.

```python
from verdict.common.conversational import ConversationalUnit
from verdict.common.judge import JudgeUnit
from verdict.scale import BooleanScale

pipeline = Pipeline() \
    >> Layer([
        ConversationalUnit(role_name='Proponent').prompt("""
            You are in a debate. Your are the Proponent, arguing that the following joke is funny. Continue the conversation.
    
            Joke:
            {source.joke}
    
            Debate thus far:
            {input.conversation}
        """),
        ConversationalUnit(role_name='Opponent').prompt("""
            You are in a debate. Your are the Opponent, arguing that the following joke is not funny. Continue the conversation.
    
            Joke:
            {source.joke}
    
            Debate thus far:
            {input.conversation}
        ")
    ], 3, inner='chain', outer='broadcast').with_leaf([0, 2, 4]) \
    >> JudgeUnit(BooleanScale()).prompt("""
        Use the following arguments to determine if the following joke is funny.

        Joke:
        {source.joke}

        Debater #1:
        {previous.conversational[0].response}

        Debater #2:
        {previous.conversational[1].response}

        Debater #3:
        {previous.conversational[2].response}
    ")

pipeline.plot()
```

--------------------------------

### Build and Run a Pipeline for Hallucination Detection

Source: https://context7.com/haizelabs/verdict/llms.txt

Constructs a pipeline with judge and verifier units, runs it on a single sample, and prints the output and leaf prefixes. Ensure necessary imports are present.

```python
from verdict import Pipeline, Layer
from verdict.common.judge import JudgeUnit, CategoricalJudgeUnit
from verdict.scale import DiscreteScale, BooleanScale
from verdict.schema import Schema
from verdict.transform import MaxPoolUnit

JUDGE_PROMPT = """
Evaluate whether the following response contains a hallucination.

Question: {source.question}
Response: {source.response}

Provide an explanation, then decide yes or no.
"""

VERIFY_PROMPT = """
Review the following judge decision.
Question: {source.question}
Response: {source.response}
Judge Explanation: {previous.explanation}
Judge Decision: {previous.choice}

Is the judge's reasoning sound? Answer yes or no.
"""

# Build a pipeline: 3 independent judge+verifier pairs, then majority vote
pipeline = Pipeline(name="HallucinationDetector") \
    >> Layer(
        CategoricalJudgeUnit(name='Judge', categories=DiscreteScale(['yes', 'no']), explanation=True)
            .prompt(JUDGE_PROMPT).via('gpt-4o', retries=3, temperature=0.7)
        >> CategoricalJudgeUnit(name='Verify', categories=DiscreteScale(['yes', 'no']))
            .prompt(VERIFY_PROMPT).via('gpt-4o', retries=3, temperature=0.0)
    , repeat=3) \
    >> MaxPoolUnit()

# Run on a single sample
output, leaf_prefixes = pipeline.run(
    Schema.of(
        question="What is the capital of France?",
        response="The capital of France is Berlin."
    ),
    max_workers=32,
    display=True,
)

print(leaf_prefixes)   # e.g. ['HallucinationDetector_root_block_layer[2]_unit[MaxPool]_values']
print(output)          # {'HallucinationDetector_..._values': 'yes', ...}
```