### Prompt Rendering Example

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Demonstrates how to render a Jinja template for judge prompts, formatting the question and candidate answers.

```python
formatted_prompt = prompts.render_template("arena_hard_judge_prompt", prompt=question, answer_a=answer_A, answer_b=answer_B)
```

--------------------------------

### Run Arena-Hard Judge with GPT-4o-mini

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Example command to run the Arena-Hard judge using GPT-4o-mini, specifying the dataset and response model. Ensure your OPENAI_API_KEY is exported.

```bash
export OPENAI_API_KEY=your-api-key-here
python run_judge.py --judge_name arena_hard --judge_model gpt-4o-mini --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Load and Inspect JudgeBench Dataset

Source: https://context7.com/scalerlab/judgebench/llms.txt

Demonstrates how to load the JudgeBench dataset from HuggingFace and inspect individual records. Ensure the 'datasets' library is installed.

```python
import json

# Sample record from data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
record = {
    "pair_id": "81ec57f2-483f-515f-93ff-78a8910b2153",   # unique pair identifier
    "original_id": "10646",                               # source question id
    "source": "mmlu-pro-computer science",                # benchmark category
    "question": "Consider an additive white Gaussian noise channel ...",
    "response_model": "gpt-4o-2024-05-13",
    "response_A": "To determine the capacity ...",
    "response_B": "We are given an additive white Gaussian noise ...",
    "label": "B>A"                                        # ground-truth correctness label
}

# Load from HuggingFace (alternative to local files)
from datasets import load_dataset
gpt_data   = load_dataset("ScalerLab/JudgeBench", split="gpt")    # GPT-4o pairs
claude_data = load_dataset("ScalerLab/JudgeBench", split="claude") # Claude-3.5-Sonnet pairs

print(gpt_data[0]["source"])   # e.g. "mmlu-pro-computer science"
print(gpt_data[0]["label"])    # e.g. "A>B"
```

--------------------------------

### Serve Local Models with vLLM

Source: https://context7.com/scalerlab/judgebench/llms.txt

Instructions for starting a vLLM OpenAI-compatible server for local models and running JudgeBench against it. This is useful for fine-tuned or private models. Ensure you have sufficient GPU VRAM and the correct Hugging Face token.

```bash
# Start the vLLM server (requires sufficient GPU VRAM)
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-huggingface-token>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Skywork/Skywork-Critic-Llama-3.1-8B

# From the same machine (or via SSH tunnel on port 8000), run the judge
python run_judge.py \
    --judge_name skywork_critic \
    --judge_model Skywork/Skywork-Critic-Llama-3.1-8B \
    --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl \
    --concurrency_limit 4

# For reward models (GPU-only, no SSH tunnel support):
pip install -r requirements-cuda.txt
python run_judge.py \
    --judge_name reward_model \
    --judge_model internlm/internlm2-20b-reward \
    --single_game \
    --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Registering a New Judge

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Example of how to add a new judge implementation to the judge registry. Ensure your new judge is correctly mapped by its name.

```python
def get_judge_from_judge_name_and_model(judge_name: str, judge_model: str) -> Judge:
    if judge_name == "arena_hard":
        return ArenaHard(judge_model)
    elif judge_name == "my-new-judge":
        return MyNewJudge(judge_model)
    else:
        raise NotImplementedError(f"Judge with name {judge_name} is not yet implemented.")
```

--------------------------------

### Serve Model Locally with vLLM Docker

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Docker command to start a vLLM OpenAI-compatible server for serving models locally. Mounts Hugging Face cache and sets necessary environment variables.

```bash
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-huggingface-token>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model Skywork/Skywork-Critic-Llama-3.1-8B
```

--------------------------------

### JudgeBench CLI Execution Output

Source: https://context7.com/scalerlab/judgebench/llms.txt

Example of the expected output when running the `run_judge.py` script, showing progress and final accuracy metrics per category.

```bash
# Expected output:
# Judging pairs ...
# 100%|████████| 270/270
# Computing final metrics ...
# mmlu-pro: 72.50%.
# livebench-reasoning: 68.10%.
# livebench-math: 65.30%.
# livecodebench: 70.20%.
# Overall: 69.80%.
```

--------------------------------

### Run Judge with vLLM Local Server

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Example command to run a judge when the model is served via a locally-running vLLM OpenAI-compatible server. Ensure the vLLM server is running and accessible.

```bash
python run_judge.py --judge_name skywork_critic --judge_model Skywork/Skywork-Critic-Llama-3.1-8B --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Load JudgeBench Dataset from HuggingFace

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Access the JudgeBench dataset directly using the HuggingFace Datasets library. Ensure you have the library installed. The data format mirrors the structure described in the documentation.

```python
from datasets import load_dataset
gpt_data = load_dataset("ScalerLab/JudgeBench", split="gpt")
claude_data = load_dataset("ScalerLab/JudgeBench", split="claude")
```

--------------------------------

### Run Reward Model Locally

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Command to run a reward model on JudgeBench after installing required CUDA libraries. Uses the 'reward_model' judge name and specifies the HuggingFace model identifier.

```bash
python run_judge.py --judge_name reward_model --judge_model Skywork/Skywork-Reward-Llama-3.1-8B --single_game --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Implement a Custom Judge by Subclassing `Judge`

Source: https://context7.com/scalerlab/judgebench/llms.txt

Provides an example of creating a custom judge by subclassing `Judge` and implementing `get_judgment`. This custom judge uses a chat model for a simple chain-of-thought evaluation. Remember to register your custom judge in the factory function.

```python
# utils/judges.py — add your custom judge class

from utils.judges import Judge, get_judge_from_judge_name_and_model
import utils.models as models
import utils.prompts as prompts
from typing import Dict, Any

class MyCustomJudge(Judge):
    """Example: a simple chain-of-thought judge using any chat model."""

    def __init__(self, model_name: str):
        self.model_name = model_name
        self.api = models.get_chat_api_from_model(model_name)

    async def get_judgment(self, question: str, answer_A: str, answer_B: str) -> Dict[str, Any]:
        prompt = (
            f"Question: {question}\n\n"
            f"Answer A: {answer_A}\n\n"
            f"Answer B: {answer_B}\n\n"
            "Which answer is more accurate and complete? Reply with only 'A' or 'B'."
        )
        output = await self.api.chat(
            messages=[{"role": "user", "content": prompt}],
            temperature=0.0,
            max_tokens=16,
        )
        decision = "A>B" if output.strip().upper() == "A" else "B>A"
        return {
            "judgment": {"judge_model": self.model_name, "prompt": prompt, "response": output},
            "decision": decision
        }


# Register in get_judge_from_judge_name_and_model():
# elif judge_name == "my_custom_judge":
#     return MyCustomJudge(judge_model)

# Run from CLI:
# python run_judge.py --judge_name my_custom_judge --judge_model gpt-4o-mini \
#   --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Judge Factory for Instantiating Judge Models

Source: https://context7.com/scalerlab/judgebench/llms.txt

Factory function to get judge instances by name and model. Supports prompted judges and reward models. Requires API keys for prompted judges and GPU for reward models.

```python
from utils.judges import get_judge_from_judge_name_and_model

# Prompted judges (require API key env vars)
arena_judge     = get_judge_from_judge_name_and_model("arena_hard",    "gpt-4o-mini")
vanilla_judge   = get_judge_from_judge_name_and_model("vanilla",       "claude-3-5-sonnet-20240620")
auto_j_judge    = get_judge_from_judge_name_and_model("auto_j",        "GAIR/autoj-13b")
prometheus      = get_judge_from_judge_name_and_model("prometheus_2",  "prometheus-eval/prometheus-7b-v2.0")
skywork_critic  = get_judge_from_judge_name_and_model("skywork_critic","Skywork/Skywork-Critic-Llama-3.1-8B")
compass         = get_judge_from_judge_name_and_model("compass_judger","opencompass/CompassJudger-1-7B-Instruct")
panda           = get_judge_from_judge_name_and_model("panda_lm",      "WeOpenML/PandaLM-7B-v1")
judge_lm        = get_judge_from_judge_name_and_model("judge_lm",      "BAAI/JudgeLM-7B-v1.0")

# Reward models (require GPU, use --single_game)
rm_internlm = get_judge_from_judge_name_and_model("reward_model", "internlm/internlm2-7b-reward")
rm_grm      = get_judge_from_judge_name_and_model("reward_model", "Ray2333/GRM-Gemma-2B-rewardmodel-ft")
rm_skywork  = get_judge_from_judge_name_and_model("reward_model", "Skywork/Skywork-Reward-Llama-3.1-8B")

# Use any judge identically via the abstract interface
import asyncio
result = asyncio.run(
    arena_judge.get_judgment(
        question="What is the time complexity of quicksort?",
        answer_A="O(n log n) on average.",
        answer_B="O(n^2) in the worst case, O(n log n) average."
    )
)
print(result["decision"])           # "B>A"
print(result["judgment"]["response"])  # raw judge output text
```

--------------------------------

### Load, Filter, and Save Dataset with JSONL

Source: https://context7.com/scalerlab/judgebench/llms.txt

Demonstrates reading a JSONL dataset, filtering it based on a keyword, and writing the subset to a new JSONL file. Includes verification of the round-trip process.

```python
pairs = read_jsonl("data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl")
print(len(pairs))          # 350
print(pairs[0]["source"])  # "mmlu-pro-computer science"
print(pairs[0]["label"])   # "A>B" or "B>A"

# Filter to a subset and write out
math_pairs = [p for p in pairs if "math" in p["source"]]
write_to_jsonl("outputs/math_subset.jsonl", math_pairs)

# Verify round-trip
reloaded = read_jsonl("outputs/math_subset.jsonl")
assert len(reloaded) == len(math_pairs)
```

--------------------------------

### Render Jinja2 Prompts with `prompts.render_template()`

Source: https://context7.com/scalerlab/judgebench/llms.txt

Shows how to use `prompts.render_template` to construct various prompts for different judge types, including pairwise comparison, vanilla, and Prometheus-2 prompts. Ensure the correct template name and keyword arguments are provided.

```python
from utils import prompts

# Arena-Hard judge prompt (pairwise comparison)
user_prompt = prompts.render_template(
    "arena_hard_judge_prompt",
    prompt="What is the time complexity of merge sort?",
    answer_a="O(n log n) in all cases.",
    answer_b="O(n log n) average, O(n^2) worst case."
)
# Returns:
# <|User Prompt|>
# What is the time complexity of merge sort?
# <|The Start of Assistant A's Aside>
# O(n log n) in all cases.
# ...

system_prompt = prompts.render_template("arena_hard_judge_system")

# Vanilla judge prompt
vanilla_prompt = prompts.render_template(
    "vanilla_prompt",
    question="Explain recursion.",
    answer_a="A function calling itself.",
    answer_b="A technique where a function solves a problem by solving smaller instances."
)

# Prometheus-2 prompt (requires rubric)
p2_prompt = prompts.render_template(
    "prometheus2_prompt",
    instruction="Explain recursion.",
    response_A="A function calling itself.",
    response_B="A technique where a function solves a problem by solving smaller instances.",
    rubric="[Are the model's responses factually correct and well-supported by evidence?]"
)

print(user_prompt[:80])  # preview
```

--------------------------------

### Demonstrate Judge.get_judgment() with multiple judge types

Source: https://context7.com/scalerlab/judgebench/llms.txt

Shows how to use different judge implementations (ArenaHard, Vanilla, AutoJ) to evaluate model responses. Ensure the correct judge model is initialized for each type.

```python
import asyncio
from utils.judges import ArenaHard, Vanilla, AutoJ, Prometheus2, SkyworkCritic

async def demo():
    question = "Explain the difference between TCP and UDP."
    answer_A = "TCP is connection-oriented and guarantees delivery. UDP is connectionless and faster."
    answer_B = "TCP uses handshake and acks; UDP just fires packets without confirmation."

    # ArenaHard: multi-attempt judgment with [[A>B]] style verdict parsing
    judge = ArenaHard("gpt-4o-mini")
    result = await judge.get_judgment(question, answer_A, answer_B)
    # result = {
    #   "decision": "A>B",
    #   "judgment": {
    #     "judge_model": "gpt-4o-mini",
    #     "prompt": "<|User Prompt|>
Explain the difference ...",
    #     "response": "... [[A>B]]"
    #   }
    # }

    # Vanilla: returns exactly "Output (a)" or "Output (b)"
    judge2 = Vanilla("gpt-4o-mini")
    result2 = await judge2.get_judgment(question, answer_A, answer_B)
    # result2["decision"] => "A>B" or "B>A"

    # AutoJ: parses "final decision is Response 1/2/tie"
    judge3 = AutoJ("GAIR/autoj-13b")   # served via vLLM locally
    result3 = await judge3.get_judgment(question, answer_A, answer_B)

    print(result["decision"], result2["decision"], result3["decision"])

asyncio.run(demo())
```

--------------------------------

### Export API Keys for Cloud Providers

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Set environment variables for API keys to authenticate with OpenAI, Anthropic, Together, and Google Cloud VertexAI. This is required before running judges with models hosted by these providers.

```bash
export OPENAI_API_KEY=your-openai-api-key         # (GPT-4o, GPT-4o-mini, o1-preview, o1-mini, etc.)
export ANTHROPIC_API_KEY=your-anthropic-api-key   # (Claude-3.5-Sonnet, Claude-3-Haiku, etc.)
export TOGETHER_API_KEY=your-together-api-key     # (Llama-3.1-405B-Instruct, etc.)
```

--------------------------------

### Factory for Chat API backends based on model name

Source: https://context7.com/scalerlab/judgebench/llms.txt

Automatically selects the correct ChatAPI backend (OpenAI, Anthropic, Google Gemini, Together AI, local vLLM) using get_chat_api_from_model(). Ensure API keys are set in environment variables for respective services.

```python
from utils.models import get_chat_api_from_model
import asyncio, os

os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."
os.environ["TOGETHER_API_KEY"] = "..."

async def demo():
    # OpenAI (gpt-* or o1-*)
    oai = get_chat_api_from_model("gpt-4o-mini")
    r1 = await oai.chat([{"role": "user", "content": "Hello"}], temperature=0, max_tokens=64)

    # Anthropic (claude-*)
    ant = get_chat_api_from_model("claude-3-haiku-20240307")
    r2 = await ant.chat(
        [{"role": "system", "content": "Be concise."}, {"role": "user", "content": "Hello"}],
        temperature=0, max_tokens=64
    )

    # Together AI (Llama etc.)
    tog = get_chat_api_from_model("meta-llama/Meta-Llama-3.1-70B-Instruct")
    r3 = await tog.chat([{"role": "user", "content": "Hello"}], temperature=0.7, max_tokens=128)

    # Local vLLM server on localhost:8000 (any other model name)
    local = get_chat_api_from_model("Skywork/Skywork-Critic-Llama-3.1-8B")
    r4 = await local.chat([{"role": "user", "content": "Hello"}], temperature=0, max_tokens=256)
    # Also supports raw completion (for PandaLM, JudgeLM):
    r5 = await local.complete(prompt="Once upon a time", temperature=0, max_tokens=50)

    print(r1, r2, r3, r4, r5)

asyncio.run(demo())
```

--------------------------------

### Read and write newline-delimited JSON files

Source: https://context7.com/scalerlab/judgebench/llms.txt

Utility functions for reading and writing JSONL files, essential for dataset I/O throughout the pipeline. Used for both input datasets and judged output files.

```python
from utils.file_operations import read_jsonl, write_to_jsonl
```

--------------------------------

### Run Reward Model Judge

Source: https://context7.com/scalerlab/judgebench/llms.txt

Executes a reward model judge. Use the `--single_game` flag to avoid duplicate passes for order-independent judges. Ensure the specified judge model is compatible.

```bash
# Run a reward model (order-independent; use --single_game to avoid duplicate passes)
python run_judge.py \
  --judge_name reward_model \
  --judge_model Skywork/Skywork-Reward-Llama-3.1-8B \
  --single_game \
  --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### Run Locally-Served vLLM Model Judge

Source: https://context7.com/scalerlab/judgebench/llms.txt

Executes a judge model served locally via vLLM, such as Skywork-Critic. Ensure the vLLM server is running and accessible.

```bash
# Run a locally-served vLLM model (Skywork-Critic)
python run_judge.py \
  --judge_name skywork_critic \
  --judge_model Skywork/Skywork-Critic-Llama-3.1-8B \
  --pairs data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl
```

--------------------------------

### get_chat_api_from_model()

Source: https://context7.com/scalerlab/judgebench/llms.txt

A factory function that auto-selects the correct async ChatAPI backend based on the model name prefix. It supports various providers including OpenAI, Anthropic, Google Gemini, Together AI, and local vLLM servers.

```APIDOC
## `get_chat_api_from_model()` — Model Backend Factory

Auto-selects the correct async ChatAPI backend from the model name prefix. Supports OpenAI, Anthropic, Google Gemini (VertexAI), Together AI, and a local vLLM OpenAI-compatible server.

### Method
`def get_chat_api_from_model(model_name: str) -> ChatAPI`

### Parameters
- **model_name** (str) - The name or identifier of the model to get the API for.

### Returns
- **ChatAPI** - An instance of the appropriate ChatAPI backend.

### Supported Providers
- OpenAI (`gpt-*` or `o1-*` prefixes)
- Anthropic (`claude-*` prefixes)
- Google Gemini (VertexAI)
- Together AI (e.g., `meta-llama/*`)
- Local vLLM server (OpenAI-compatible, defaults to `localhost:8000`)

### Example
```python
from utils.models import get_chat_api_from_model
import asyncio, os

os.environ["OPENAI_API_KEY"] = "sk-..."
os.environ["ANTHROPIC_API_KEY"] = "sk-ant-..."
os.environ["TOGETHER_API_KEY"] = "..."

async def demo():
    # OpenAI
    oai = get_chat_api_from_model("gpt-4o-mini")
    r1 = await oai.chat([{"role": "user", "content": "Hello"}], temperature=0, max_tokens=64)

    # Anthropic
    ant = get_chat_api_from_model("claude-3-haiku-20240307")
    r2 = await ant.chat(
        [{"role": "system", "content": "Be concise."}, {"role": "user", "content": "Hello"}],
        temperature=0, max_tokens=64
    )

    # Together AI
    tog = get_chat_api_from_model("meta-llama/Meta-Llama-3.1-70B-Instruct")
    r3 = await tog.chat([{"role": "user", "content": "Hello"}], temperature=0.7, max_tokens=128)

    # Local vLLM server
    local = get_chat_api_from_model("Skywork/Skywork-Critic-Llama-3.1-8B")
    r4 = await local.chat([{"role": "user", "content": "Hello"}], temperature=0, max_tokens=256)
    r5 = await local.complete(prompt="Once upon a time", temperature=0, max_tokens=50)

    print(r1, r2, r3, r4, r5)

asyncio.run(demo())
```
```

--------------------------------

### read_jsonl() / write_to_jsonl()

Source: https://context7.com/scalerlab/judgebench/llms.txt

Utility functions for reading from and writing to newline-delimited JSON (JSONL) files. These are essential for handling input datasets and storing judged outputs throughout the pipeline.

```APIDOC
## `read_jsonl()` / `write_to_jsonl()` — Dataset I/O

Utility functions for reading and writing newline-delimited JSON files, used throughout the pipeline for both the input datasets and the judged output files.

### Methods
- `read_jsonl(filepath: str) -> list`
- `write_to_jsonl(filepath: str, data: list)`

### Parameters
- **filepath** (str) - The path to the JSONL file.
- **data** (list) - The data to be written to the JSONL file (for `write_to_jsonl`).

### Returns
- **list** - The data read from the JSONL file (for `read_jsonl`).

### Example
```python
from utils.file_operations import read_jsonl, write_to_jsonl

# Writing data
output_data = [{"id": 1, "text": "example"}, {"id": 2, "text": "another example"}]
write_to_jsonl("output.jsonl", output_data)

# Reading data
input_data = read_jsonl("output.jsonl")
print(input_data)
```
```

--------------------------------

### Async Batch Judging with Concurrency Limit

Source: https://context7.com/scalerlab/judgebench/llms.txt

Runs judge over pairs asynchronously, respecting a concurrency limit. Results are streamed as they complete. Requires 'utils.file_operations' and 'utils.judges'.

```python
import asyncio
import utils.file_operations as file_operations
import utils.judges as judges

async def run():
    pairs = file_operations.read_jsonl(
        "data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl"
    )

    judge = judges.get_judge_from_judge_name_and_model("arena_hard", "gpt-4o-mini")

    semaphore = asyncio.Semaphore(5)  # up to 5 concurrent requests
    import asyncio, json
    from tqdm.asyncio import tqdm_asyncio

    async def judge_one(pair):
        async with semaphore:
            j1 = await judge.get_judgment(pair["question"], pair["response_A"], pair["response_B"])
            j2 = await judge.get_judgment(pair["question"], pair["response_B"], pair["response_A"])
            pair["judgments"] = [j1, j2]
            return pair

    tasks = [asyncio.create_task(judge_one(p)) for p in pairs[:10]]
    judged = []
    for future in tqdm_asyncio.as_completed(tasks):
        judged.append(await future)

    # Each judged pair looks like:
    # {
    #   "pair_id": "...", "question": "...", "label": "A>B",
    #   "judgments": [
    #     {"decision": "A>B", "judgment": {"judge_model": "gpt-4o-mini", "prompt": "...", "response": "..."}},
    #     {"decision": "A>B", "judgment": {"judge_model": "gpt-4o-mini", "prompt": "...", "response": "..."}}
    #   ]
    # }
    print(judged[0]["judgments"][0]["decision"])  # e.g. "A>B"

asyncio.run(run())
```

--------------------------------

### judge_pairs()

Source: https://context7.com/scalerlab/judgebench/llms.txt

Asynchronously runs a judge over a list of pairs, respecting a concurrency limit. Each pair can be judged once or twice (reversed order). Results are streamed to a JSONL output file as they complete.

```APIDOC
## `judge_pairs()` — Async Batch Judging

Asynchronously runs a judge over a list of pairs, respecting a concurrency limit via `asyncio.Semaphore`. Each pair can be judged once or twice (reversed order). Results are streamed to a JSONL output file as they complete.

```python
import asyncio
import utils.file_operations as file_operations
import utils.judges as judges

async def run():
    pairs = file_operations.read_jsonl(
        "data/dataset=judgebench,response_model=gpt-4o-2024-05-13.jsonl"
    )

    judge = judges.get_judge_from_judge_name_and_model("arena_hard", "gpt-4o-mini")

    semaphore = asyncio.Semaphore(5)  # up to 5 concurrent requests
    import asyncio, json
    from tqdm.asyncio import tqdm_asyncio

    async def judge_one(pair):
        async with semaphore:
            j1 = await judge.get_judgment(pair["question"], pair["response_A"], pair["response_B"])
            j2 = await judge.get_judgment(pair["question"], pair["response_B"], pair["response_A"])
            pair["judgments"] = [j1, j2]
            return pair

    tasks = [asyncio.create_task(judge_one(p)) for p in pairs[:10]]
    judged = []
    for future in tqdm_asyncio.as_completed(tasks):
        judged.append(await future)

    # Each judged pair looks like:
    # {
    #   "pair_id": "...", "question": "...", "label": "A>B",
    #   "judgments": [
    #     {"decision": "A>B", "judgment": {"judge_model": "gpt-4o-mini", "prompt": "...", "response": "..."}},
    #     {"decision": "A>B", "judgment": {"judge_model": "gpt-4o-mini", "prompt": "...", "response": "..."}}
    #   ]
    # }
    print(judged[0]["judgments"][0]["decision"])

asyncio.run(run())
```
```

--------------------------------

### JudgeBench Citation

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

BibTeX entry for citing the JudgeBench paper in academic work.

```bibtex
@misc{judgebench2024,
  title={JudgeBench: A Benchmark for Evaluating LLM-Based Judges},
  author={Sijun Tan and Siyuan Zhuang and Kyle Montgomery and Willian Y. Tang and Alejandro Cuadron and Chenguang Wang and Raluca Ada Popa and Ion Stoica},
  year={2024},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2410.12784}
}
```

--------------------------------

### Judge.get_judgment()

Source: https://context7.com/scalerlab/judgebench/llms.txt

The core abstract interface for all judge classes. It takes a question and two candidate responses and returns a dictionary containing the decision and judgment details.

```APIDOC
## `Judge.get_judgment()` — Core Abstract Interface

All judge classes implement this single async method. It takes a question and two candidate responses and returns a dict with a `decision` key (`"A>B"`, `"B>A"`, `"A=B"`, or `None`) and a `judgment` key with model/prompt/response details.

### Method
`async def get_judgment(question: str, answer_A: str, answer_B: str) -> dict`

### Parameters
- **question** (str) - The question posed to the models.
- **answer_A** (str) - The first candidate response.
- **answer_B** (str) - The second candidate response.

### Returns
- **dict** - A dictionary containing:
  - **decision** (str): The judgment outcome, one of `"A>B"`, `"B>A"`, `"A=B"`, or `None`.
  - **judgment** (dict): Details about the judgment, including judge model, prompt, and response.

### Example
```python
import asyncio
from utils.judges import ArenaHard

async def demo():
    question = "Explain the difference between TCP and UDP."
    answer_A = "TCP is connection-oriented and guarantees delivery. UDP is connectionless and faster."
    answer_B = "TCP uses handshake and acks; UDP just fires packets without confirmation."

    judge = ArenaHard("gpt-4o-mini")
    result = await judge.get_judgment(question, answer_A, answer_B)
    print(result["decision"])

asyncio.run(demo())
```
```

--------------------------------

### Compute judge accuracy metrics

Source: https://context7.com/scalerlab/judgebench/llms.txt

Calculates judge accuracy from judged pairs, supporting single-game and double-game modes. Double-game mode uses consensus scoring to mitigate position bias. Use `include_fn` for category-specific scores.

```python
from utils.metrics import compute_final_metrics
from utils.file_operations import read_jsonl

# Load previously-judged output file
pairs = read_jsonl(
    "outputs/dataset=judgebench,response_model=gpt-4o-2024-05-13,"
    "judge_name=arena_hard,judge_model=gpt-4o-mini.jsonl"
)

# Overall accuracy (double-game mode: each pair judged twice with swapped order)
overall = compute_final_metrics(pairs, reverse_order=True)
print(f"Overall: {overall:.2f}%")   # e.g. "Overall: 71.30%"

# Per-category breakdown
for source in ["mmlu-pro", "livebench-reasoning", "livebench-math", "livecodebench"]:
    score = compute_final_metrics(
        pairs,
        reverse_order=True,
        include_fn=lambda x, s=source: x["source"].startswith(s)
    )
    print(f"{source}: {score:.2f}%")

# Single-game mode (for reward models run with --single_game)
rm_pairs = read_jsonl(
    "outputs/dataset=judgebench,response_model=gpt-4o-2024-05-13,"
    "judge_name=reward_model,judge_model=Skywork_Skywork-Reward-Llama-3.1-8B.jsonl"
)
single_score = compute_final_metrics(rm_pairs, reverse_order=False)
print(f"Reward model accuracy: {single_score:.2f}%")
```

--------------------------------

### compute_final_metrics()

Source: https://context7.com/scalerlab/judgebench/llms.txt

Computes the accuracy of judges by evaluating the percentage of correctly labeled pairs. It supports both single-game and double-game modes, with the latter employing a consensus scoring scheme to mitigate position bias.

```APIDOC
## `compute_final_metrics()` — Accuracy Evaluation

Computes judge accuracy (percentage of correctly labeled pairs) from a list of judged pairs. Supports single-game and double-game (reversed order) modes; in double-game mode uses a consensus scoring scheme to handle position bias.

### Method
`def compute_final_metrics(pairs: list, reverse_order: bool = False, include_fn: Optional[Callable] = None) -> float`

### Parameters
- **pairs** (list) - A list of judged pairs, typically loaded from a JSONL file.
- **reverse_order** (bool, optional) - If True, enables double-game mode, assuming each pair was judged twice with swapped order. Defaults to `False`.
- **include_fn** (Callable, optional) - A function that takes a pair and a source string, returning `True` if the pair should be included in the metric calculation. Useful for filtering by category.

### Returns
- **float** - The computed accuracy as a percentage.

### Example
```python
from utils.metrics import compute_final_metrics
from utils.file_operations import read_jsonl

# Load previously-judged output file
pairs = read_jsonl(
    "outputs/dataset=judgebench,response_model=gpt-4o-2024-05-13,"
    "judge_name=arena_hard,judge_model=gpt-4o-mini.jsonl"
)

# Overall accuracy (double-game mode)
overall = compute_final_metrics(pairs, reverse_order=True)
print(f"Overall: {overall:.2f}%")

# Per-category breakdown
for source in ["mmlu-pro", "livebench-reasoning", "livebench-math", "livecodebench"]:
    score = compute_final_metrics(
        pairs,
        reverse_order=True,
        include_fn=lambda x, s=source: x["source"].startswith(s)
    )
    print(f"{source}: {score:.2f}%")

# Single-game mode
rm_pairs = read_jsonl(
    "outputs/dataset=judgebench,response_model=gpt-4o-2024-05-13,"
    "judge_name=reward_model,judge_model=Skywork_Skywork-Reward-Llama-3.1-8B.jsonl"
)
single_score = compute_final_metrics(rm_pairs, reverse_order=False)
print(f"Reward model accuracy: {single_score:.2f}%")
```
```

--------------------------------

### get_judge_from_judge_name_and_model()

Source: https://context7.com/scalerlab/judgebench/llms.txt

Returns the appropriate `Judge` subclass instance given a judge name and model identifier. This is the single dispatch point for all judge types.

```APIDOC
## `get_judge_from_judge_name_and_model()` — Judge Factory

Returns the appropriate `Judge` subclass instance given a judge name and model identifier. This is the single dispatch point for all judge types.

```python
from utils.judges import get_judge_from_judge_name_and_model

# Prompted judges (require API key env vars)
arena_judge     = get_judge_from_judge_name_and_model("arena_hard",    "gpt-4o-mini")
vanilla_judge   = get_judge_from_judge_name_and_model("vanilla",       "claude-3-5-sonnet-20240620")
auto_j_judge    = get_judge_from_judge_name_and_model("auto_j",        "GAIR/autoj-13b")
prometheus      = get_judge_from_judge_name_and_model("prometheus_2",  "prometheus-eval/prometheus-7b-v2.0")
skywork_critic  = get_judge_from_judge_name_and_model("skywork_critic","Skywork/Skywork-Critic-Llama-3.1-8B")
compass         = get_judge_from_judge_name_and_model("compass_judger","opencompass/CompassJudger-1-7B-Instruct")
panda           = get_judge_from_judge_name_and_model("panda_lm",      "WeOpenML/PandaLM-7B-v1")
judge_lm        = get_judge_from_judge_name_and_model("judge_lm",      "BAAI/JudgeLM-7B-v1.0")

# Reward models (require GPU, use --single_game)
rm_internlm = get_judge_from_judge_name_and_model("reward_model", "internlm/internlm2-7b-reward")
rm_grm      = get_judge_from_judge_name_and_model("reward_model", "Ray2333/GRM-Gemma-2B-rewardmodel-ft")
rm_skywork  = get_judge_from_judge_name_and_model("reward_model", "Skywork/Skywork-Reward-Llama-3.1-8B")

# Use any judge identically via the abstract interface
import asyncio
result = asyncio.run(
    arena_judge.get_judgment(
        question="What is the time complexity of quicksort?",
        answer_A="O(n log n) on average.",
        answer_B="O(n^2) in the worst case, O(n log n) average."
    )
)
print(result["decision"])
print(result["judgment"]["response"])
```
```

--------------------------------

### Abstract Judge Class Definition

Source: https://github.com/scalerlab/judgebench/blob/main/README.md

Defines the abstract base class for all judges. Implement the get_judgment method to provide custom judging logic.

```python
class Judge(ABC):
    @abstractmethod
    async def get_judgment(self, question: str, answer_A: str, answer_B: str) -> Dict[str, Any]:
        pass
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.