### Example Data Format for Mathematics Category

Source: https://github.com/thudm/alignbench/blob/master/README-en.md

This JSON object illustrates the structure of a sample within the AlignBench dataset, specifically for the 'Mathematics' category. It includes a question ID, category, subcategory, the user query, and a reference answer.

```json
{
    "question_id": 1,
    "category": "数学计算",
    "subcategory": "初等数学",
    "question": "有一串彩珠，按“2红3绿4黄”的顺序依次排列。第600颗是什么颜色?",
    "reference": "一组\"2红3绿4黄\"共有9颗珠子。600除以9的商是66，余数是6。因此，第600颗珠子是在第67组的第6颗，即\"2红3绿4黄\"中的第6颗，也就是黄色。所以，第600颗珠子是黄色。"
}
```

--------------------------------

### Run Target LLM Inference

Source: https://github.com/thudm/alignbench/blob/master/README-en.md

Modify and run this script to get answers from your target LLM. Ensure your API calling class is correctly implemented and specified.

```bash
MODEL=do_nothing # TODO modify the model name(the same as your API calling class)

python get_answers.py \
    --model do_nothing \
    --workers 2 \
    --question-file data/data_release.jsonl \
    --save-dir data/model_answer
```

--------------------------------

### Get GPT-4 Judgments

Source: https://github.com/thudm/alignbench/blob/master/README-en.md

Run this script after filling in your GPT-4 API key in the configuration file. It obtains judgments from GPT-4 for the target LLM's responses.

```bash
MODEL=do_nothing # TODO modify the model name(the same as your API calling class)

python judge.py \
    --config-path config/multi-dimension.json \
    --model-name $MODEL \
    --parallel 2 \

```

--------------------------------

### Load and Inspect AlignBench Dataset

Source: https://context7.com/thudm/alignbench/llms.txt

Loads the AlignBench dataset using jsonlines and prints the total number of samples. Shows the structure of samples for different categories, including the optional 'evidences' field.

```python
import jsonlines

# Load and inspect the benchmark dataset
with jsonlines.open("data/data_v1.1_release.jsonl", "r") as f:
    samples = list(f)

print(f"Total samples: {len(samples)}")  # 683

# Sample structure (Professional Knowledge category with evidence)
sample = {
    "question_id": 8,
    "category": "专业能力",        # Primary category
    "subcategory": "历史",          # Subcategory used for dimension mapping
    "question": "麦哲伦航队在全球旅行时使用了六分仪测量经纬度么？",
    "reference": "不，麦哲伦航队在全球旅行时没有使用六分仪来测量经纬度...",
    "evidences": [
        {
            "url": "https://baike.baidu.com/item/வுகளை",
            "quote": "1519年，率领船队开始环球航行..."
        }
    ]
}

# Mathematics category sample (no evidences field)
math_sample = {
    "question_id": 1,
    "category": "数学计算",
    "subcategory": "初等数学",
    "question": "有一串彩珠，按\"2红3绿4黄\"的顺序依次排列。第600颗是什么颜色?",
    "reference": "一组\"2红3绿4黄\"共有9颗珠子。600除以9的商是66，余数是6..."
}

# Count samples per category
from collections import Counter
cats = Counter(s["category"] for s in samples)
# {'专业能力': 124, '角色扮演': 116, '数学计算': 112, '逻辑推理': 92,
#  '文本写作': 75, '基本任务': 68, '中文理解': 58, '综合问答': 38}
```

--------------------------------

### Load Model API and Generate Text

Source: https://context7.com/thudm/alignbench/llms.txt

Dynamically loads a model API using `get_model_api` and generates text responses for a batch of sample questions. Requires the model's API key to be set as an environment variable.

```python
from inference.models import get_model_api

# Load built-in GPT-4 model (requires OPENAI_API_KEY env var)
model = get_model_api("gpt_4", workers=5)

# Generate responses for a batch of samples
samples = [
    {"question": "什么是机器学习？", "temperature": 0.1},
    {"question": "写一首关于春天的诗", "temperature": 0.7},
]
outputs = model.generate_text(samples)
# outputs: ["机器学习是...", "春风轻抚大地..."]
```

--------------------------------

### get_answers.py

Source: https://context7.com/thudm/alignbench/llms.txt

Script for Stage 1 of the evaluation pipeline: collecting model responses.

```APIDOC
## `get_answers.py` — Stage 1: Collect Model Responses

Runs inference on the target LLM against all benchmark questions. Applies per-category temperatures from `config/temperature.json`. Automatically repredicts up to 3 times for empty/failed responses. Saves results to `data/model_answer/{model}.jsonl`.

```bash

```

--------------------------------

### Run Inference Stage 1: Collect Model Responses

Source: https://context7.com/thudm/alignbench/llms.txt

Executes the inference stage to collect responses from the target LLM for all benchmark questions. It applies per-category temperatures and retries failed responses, saving results to a specified file.

```bash
# get_model_api.py — Stage 1: Collect Model Responses

# Runs inference on the target LLM against all benchmark questions. Applies per-category temperatures from `config/temperature.json`. Automatically repredicts up to 3 times for empty/failed responses. Saves results to `data/model_answer/{model}.jsonl`.
```

--------------------------------

### Dataset Format

Source: https://context7.com/thudm/alignbench/llms.txt

Details the structure of the AlignBench dataset, including sample format and category distribution.

```APIDOC
## Dataset Format — `data/data_v1.1_release.jsonl`

Each line is a JSON object representing one benchmark sample. The dataset contains 683 samples across 8 categories, with optional `evidences` fields added in v1.1 for factual questions.

```python
import jsonlines

# Load and inspect the benchmark dataset
with jsonlines.open("data/data_v1.1_release.jsonl", "r") as f:
    samples = list(f)

print(f"Total samples: {len(samples)}")  # 683

# Sample structure (Professional Knowledge category with evidence)
sample = {
    "question_id": 8,
    "category": "专业能力",        # Primary category
    "subcategory": "历史",          # Subcategory used for dimension mapping
    "question": "麦哲伦航队在全球旅行时使用了六分仪测量经纬度么？",
    "reference": "不，麦哲伦航队在全球旅行时没有使用六分仪来测量经纬度...",
    "evidences": [
        {
            "url": "https://baike.baidu.com/item/",
            "quote": "1519年，率领船队开始环球航行..."
        }
    ]
}

# Mathematics category sample (no evidences field)
math_sample = {
    "question_id": 1,
    "category": "数学计算",
    "subcategory": "初等数学",
    "question": "有一串彩珠，按\"2红3绿4黄\"的顺序依次排列。第600颗是什么颜色?",
    "reference": "一组\"2红3绿4黄\"共有9颗珠子。600除以9的商是66，余数是6..."
}

# Count samples per category
from collections import Counter
cats = Counter(s["category"] for s in samples)
# {'专业能力': 124, '角色扮演': 116, '数学计算': 112, '逻辑推理': 92,
#  '文本写作': 75, '基本任务': 68, '中文理解': 58, '综合问答': 38}
```
```

--------------------------------

### Display Evaluation Results

Source: https://github.com/thudm/alignbench/blob/master/README-en.md

Execute this script to generate an Excel file containing the results of all LLM judgments. The results are stored in the specified output directory.

```bash
python show_result.py \
    --input-dir data/judgment \
    --ques-file data/data_release.jsonl \
    --save-file data/results/results.xlsx
```

--------------------------------

### get_model_api

Source: https://context7.com/thudm/alignbench/llms.txt

Dynamically imports and instantiates a model API class for inference.

```APIDOC
## `get_model_api(model_name, workers)` — Load a Model API

Dynamically imports and instantiates a model API class from `inference/api_models/{model_name}.py`. The class name must match the filename. Returns an `api_model` instance with a `generate_text()` method.

```python
from inference.models import get_model_api

# Load built-in GPT-4 model (requires OPENAI_API_KEY env var)
model = get_model_api("gpt_4", workers=5)

# Generate responses for a batch of samples
samples = [
    {"question": "什么是机器学习？", "temperature": 0.1},
    {"question": "写一首关于春天的诗", "temperature": 0.7},
]
outputs = model.generate_text(samples)
# outputs: ["机器学习是...", "春风轻抚大地..."]
```
```

--------------------------------

### Integrate Custom LLM API

Source: https://context7.com/thudm/alignbench/llms.txt

Subclasses `api_model` to integrate a custom LLM API. Implement `get_api_result` to handle requests and responses. Place the file in `inference/api_models/` and test its availability.

```python
# inference/api_models/my_model.py
from inference.models import api_model
import requests

class my_model(api_model):
    def __init__(self, workers=10):
        self.workers = workers
        self.api_url = "https://api.example.com/chat"
        self.api_key = "YOUR_API_KEY"
        self.temperature = 0.7

    def get_api_result(self, sample):
        question = sample["question"]
        temperature = sample.get("temperature", self.temperature)

        response = requests.post(
            self.api_url,
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"message": question, "temperature": temperature}
        )
        return response.json()["reply"]

# Test the custom model integration
from inference.utils import test_api_alive
test_api_alive("my_model")
# User: {'question': '你是谁？', 'temperature': 0}
# Assistant: 我是...
# True
```

--------------------------------

### api_model Base Class

Source: https://context7.com/thudm/alignbench/llms.txt

Provides a base class for integrating custom LLM APIs into the benchmark framework.

```APIDOC
## `api_model` Base Class — Custom Model Integration

Subclass `api_model` and implement `get_api_result(sample)` to integrate any LLM API. Place the file in `inference/api_models/` with the filename matching the class name. The base class handles parallel execution with retry logic.

```python
# inference/api_models/my_model.py
from inference.models import api_model
import requests

class my_model(api_model):
    def __init__(self, workers=10):
        self.workers = workers
        self.api_url = "https://api.example.com/chat"
        self.api_key = "YOUR_API_KEY"
        self.temperature = 0.7

    def get_api_result(self, sample):
        question = sample["question"]
        temperature = sample.get("temperature", self.temperature)

        response = requests.post(
            self.api_url,
            headers={"Authorization": f"Bearer {self.api_key}"},
            json={"message": question, "temperature": temperature}
        )
        return response.json()["reply"]

# Test the custom model integration
from inference.utils import test_api_alive
test_api_alive("my_model")
# User: {'question': '你是谁？', 'temperature': 0}
# Assistant: 我是...
# True
```
```

--------------------------------

### Calculate Per-Dimension Scores

Source: https://context7.com/thudm/alignbench/llms.txt

Reads judgment files to compute average scores for individual evaluation dimensions across models. Saves a model x dimension matrix to a file. Requires jsonlines and pandas libraries.

```python
# Run directly as a script (hardcoded to read from data/judgment/)
# python dimension_calculate.py

# Equivalent programmatic usage:
import jsonlines
import pandas as pd
from collections import defaultdict

model_ratings = defaultdict(lambda: defaultdict(int))
model_counts = defaultdict(lambda: defaultdict(int))

# Load judgments
with jsonlines.open("data/judgment/my_model.jsonl") as f:
    for line in f:
        if line["score"] != -1:
            for dim, score in line["rating"].items():
                if isinstance(score, (int, float)) and "," not in dim:
                    model_ratings["my_model"][dim] += score
                    model_counts["my_model"][dim] += 1

# Compute averages
avg = {dim: model_ratings["my_model"][dim] / model_counts["my_model"][dim]
       for dim in model_ratings["my_model"]}
print(avg)
# {'事实正确性': 8.2, '满足用户需求': 7.8, '逻辑连贯性': 8.0,
#  '完备性': 7.5, '综合得分': 7.9}

# Saves all models to dimensional_scores.xlsx
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.