OpenCompass

https://opencompass.readthedocs.io/en/latest/
Admin
OpenCompass is a comprehensive LLM and VLM evaluation platform that supports 189+ datasets for...
Tokens:40,134
Snippets:483
Trust Score:9.6
Update:6 months ago
Show doc for...
Tokens:
Raw
### Install OpenCompass from Source

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Clones the repository and installs it in editable mode to access the latest features.

```bash
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
```

--------------------------------

### Install vLLM

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/accelerator_intro.html

Install the vLLM library via pip.

```bash
pip install vllm
```

--------------------------------

### Install and Initialize Pre-commit Hooks

Source: https://opencompass.readthedocs.io/en/latest/notes/contribution_guide.html

Install pre-commit and initialize hooks in the repository root directory for first-time contributors.

```bash
pip install -U pre-commit
pre-commit install
```

--------------------------------

### Configure CompassVerifier-32B with OpenAISDK

Source: https://opencompass.readthedocs.io/en/latest/notes/academic.html

Configuration example for the CompassVerifier-32B model using OpenAISDK. This setup includes parameters for API access, model path, and evaluation-specific settings like `temperature` and `max_out_len`.

```python
judge_cfg = dict(
    abbr='CompassVerifier',
    type=OpenAISDK,
    path='opencompass/CompassVerifier-32B',
    key='YOUR_API_KEY',
    openai_api_base='YOUR_API_BASE',
    meta_template=dict(
        round=[
            dict(role='HUMAN', api_role='HUMAN'),
            dict(role='BOT', api_role='BOT', generate=True),
        ]),
    query_per_second=1,
    batch_size=8,
    temperature=0.001,
    max_out_len=8192,
    max_seq_len=32768,
    mode='mid',
)
```

--------------------------------

### Monitor Inference Process

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Example log output indicating the start of the inference process.

```text
[2023-07-12 18:23:55,076] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
```

--------------------------------

### Install Inference Backends

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Installs optional inference backends. Separate virtual environments are recommended to avoid dependency conflicts.

```bash
 # Model inference backends. Since these backends often have dependency conflicts,
 # we recommend using separate virtual environments to manage them.
 pip install "opencompass[lmdeploy]"
 # pip install "opencompass[vllm]"
```

--------------------------------

### Example Prompt Structure

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

A basic example of a prompt containing instructions, in-context examples, and a target question.

```text
Solve the following questions.
1+1=?
2
3+9=?
12
5+6=?
```

--------------------------------

### Example Conversation with System Instructions

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

Example of a conversation string including meta instructions and system role tags.

```text
Meta instruction: You are now a helpful and harmless AI assistant.
<SYSTEM>: Solve the following math questions<eosys>\n
<HUMAN>: 1+1=?<eoh>\n
<BOT>: 2<eob>\n
<HUMAN>: 2+2=?<eoh>\n
<BOT>: 4<eob>\n
end of conversation
```

--------------------------------

### Install API Dependencies

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Installs dependencies required for calling commercial model APIs.

```bash
pip install "opencompass[api]"

# pip install openai # GPT-3.5-Turbo / GPT-4-Turbo / GPT-4 / GPT-4o (API)
# pip install anthropic # Claude (API)
# pip install dashscope # Qwen (API)
# pip install volcengine-python-sdk # ByteDance Volcano Engine (API)
# ...
```

--------------------------------

### Install Dataset Dependencies

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Installs full dataset support or specific agent evaluation dependencies.

```bash
pip install "opencompass[full]"
```

```bash
git clone --recurse-submodules git@github.com:open-compass/human-eval.git
cd human-eval
pip install -e .
pip install -e evalplus
```

```bash
# T-Eval
pip install lagent==0.1.2
# CIBench
pip install -r requirements/agent.txt
```

--------------------------------

### Initiate Evaluation

Source: https://opencompass.readthedocs.io/en/latest/user_guides/interns1.html

Command to start the evaluation process using the specified configuration file.

```bash
opencompass examples/eval_bench_intern_s1.py
```

--------------------------------

### Complete Math Evaluation Setup

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/math_verify.html

Full configuration script for setting up math evaluation in OpenCompass.

```python
from mmengine.config import read_base
from opencompass.models import TurboMindModelwithChatTemplate
from opencompass.datasets import CustomDataset
from opencompass.openicl.icl_evaluator.math_evaluator import MATHVerifyEvaluator
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer

# Dataset reader configuration
math_reader_cfg = dict(input_columns=['problem'], output_column='solution')

# Inference configuration
math_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(
            round=[
                dict(
                    role='HUMAN',
                    prompt='{problem}\nPlease reason step by step, and put your final answer within \\boxed{}.',
                ),
            ]
        ),
    ),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer),
)
```

--------------------------------

### Example Training and Testing Data

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

Sample data structures for training examples and testing inputs.

```python
ex1 = {
    'question': '2+2=?',
    'answer': '4',
    'irrelavent_infos': 'blabla',
}
ex2 = {
    'question': '3+3=?',
    'answer': '6',
    'irrelavent_infos': 'blabla',
}
ex3 = {
    'question': '1+1=?',
    'answer': '2',  # Assume the answer is in the reader_cfg.output_column
    'irrelavent_infos': 'blabla',
}
```

--------------------------------

### Prepare DLC Environment

Source: https://opencompass.readthedocs.io/en/latest/user_guides/evaluation.html

Install the dlc CLI tool and configure it before using DLCRunner.

```bash
cd ~
wget https://dlc-cli.oss-cn-zhangjiakou.aliyuncs.com/light/binary/linux/amd64/dlc
chmod +x ./dlc
sudo ln -rs dlc /usr/local/bin
./dlc config
```

--------------------------------

### Configure Few-Shot CoT

Source: https://opencompass.readthedocs.io/en/latest/prompt/chain_of_thought.html

Create a one-shot prompt by embedding a full reasoning example within the PromptTemplate configuration.

```python
qa_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=
'''Question: Mark's basketball team scores 25 2 pointers, 8 3 pointers and 10 free throws.  Their opponents score double the 2 pointers but half the 3 pointers and free throws.  What's the total number of points scored by both teams added together?
Let's think step by step
Answer:
Mark's team scores 25 2 pointers, meaning they scored 25*2= 50 points in 2 pointers.
His team also scores 6 3 pointers, meaning they scored 8*3= 24 points in 3 pointers
They scored 10 free throws, and free throws count as one point so they scored 10*1=10 points in free throws.
All together his team scored 50+24+10= 84 points
Mark's opponents scored double his team's number of 2 pointers, meaning they scored 50*2=100 points in 2 pointers.
His opponents scored half his team's number of 3 pointers, meaning they scored 24/2= 12 points in 3 pointers.
They also scored half Mark's team's points in free throws, meaning they scored 10/2=5 points in free throws.
All together Mark's opponents scored 100+12+5=117 points
The total score for the game is both team's scores added together, so it is 84+117=201 points
The answer is 201

Question: {question}\nLet's think step by step:\n{answer}
'''),
    retriever=dict(type=ZeroRetriever)
)
```

--------------------------------

### Install libGL dependency

Source: https://opencompass.readthedocs.io/en/latest/get_started/faq.html

Install the libGL dependency if you encounter 'libGL.so.1' not found errors and prefer not to use opencv-python-headless.

```bash
sudo apt-get update
sudo apt-get install -y libgl1 libglib2.0-0
```

--------------------------------

### Zero-shot Configuration

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

Example configuration for 0-shot learning using ZeroRetriver.

```python
datasets = [
    dict(
        infer_cfg=dict(
            ice_template=dict(
                type=PromptTemplate,
                template="Q: {question}\nA: {answer}",
            ),
            retriever=dict(type=ZeroRetriever),
            inferencer=dict(type=GenInferencer),
        )
    ),
]
```

--------------------------------

### Install OpenCompass via Pip

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Installs the OpenCompass package. Use the full or api extras for additional functionality.

```bash
# For support of most datasets and models
pip install -U opencompass

# Complete installation (supports more datasets)
# pip install "opencompass[full]"

# API Testing (e.g., OpenAI, Qwen)
# pip install "opencompass[api]"
```

--------------------------------

### Full Evaluation Configuration Example

Source: https://opencompass.readthedocs.io/en/latest/user_guides/config.html

This configuration sets up evaluation for multiple datasets and defines a HuggingFaceCausalLM model. It uses `read_base` to import dataset configurations.

```python
from mmengine.config import read_base

with read_base():
    # Read the required dataset configurations directly from the preset dataset configurations
    from .datasets.piqa.piqa_ppl import piqa_datasets
    from .datasets.siqa.siqa_gen import siqa_datasets

# Concatenate the datasets to be evaluated into the datasets field
datasets = [*piqa_datasets, *siqa_datasets]

# Evaluate models supported by HuggingFace's `AutoModelForCausalLM` using `HuggingFaceCausalLM`
from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        # Initialization parameters for `HuggingFaceCausalLM`
        path='huggyllama/llama-7b',
        tokenizer_path='huggyllama/llama-7b',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        # Common parameters for all models, not specific to HuggingFaceCausalLM's initialization parameters
        abbr='llama-7b',            # Model abbreviation for result display
        max_out_len=100,            # Maximum number of generated tokens
        batch_size=16,
        run_cfg=dict(num_gpus=1),   # Run configuration for specifying resource requirements
    )
]

```

--------------------------------

### List API-based model configuration files

Source: https://opencompass.readthedocs.io/en/latest/user_guides/models.html

Directory structure showing where additional API-based model examples are located.

```text
configs
├── eval_zhipu.py
├── eval_xunfei.py
└── eval_minimax.py
```

--------------------------------

### Install opencv-python-headless

Source: https://opencompass.readthedocs.io/en/latest/get_started/faq.html

Use this command to uninstall opencv-python and install opencv-python-headless if you encounter 'libGL.so.1' not found errors.

```bash
pip uninstall opencv-python
pip install opencv-python-headless
```

--------------------------------

### Configure an OpenAI API-based model

Source: https://opencompass.readthedocs.io/en/latest/user_guides/models.html

Example configuration for integrating an OpenAI model via API.

```python
from opencompass.models import OpenAI

models = [
    dict(
        type=OpenAI,                             # Using the OpenAI model
        # Parameters for `OpenAI` initialization
        path='gpt-4',                            # Specify the model type
        key='YOUR_OPENAI_KEY',                   # OpenAI API Key
        max_seq_len=2048,                        # The max input number of tokens
        # Common parameters shared by various models, not specific to `OpenAI` initialization.
        abbr='GPT-4',                            # Model abbreviation used for result display.
        max_out_len=512,                         # Maximum number of generated tokens.
        batch_size=1,                            # The size of a batch during inference.
        run_cfg=dict(num_gpus=0),                # Resource requirements (no GPU needed)
    ),
]
```

--------------------------------

### Install LMDeploy

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/evaluation_lmdeploy.html

Install lmdeploy via pip. Use this command for Python 3.8+.

```bash
pip install lmdeploy

```

--------------------------------

### Verify Inference Backends

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Commands to verify successful installation of LMDeploy or vLLM backends.

```bash
lmdeploy chat internlm/internlm2_5-1_8b-chat --backend turbomind
```

```bash
vllm serve facebook/opt-125m
```

--------------------------------

### Configure advanced HuggingFace model evaluation

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Detailed command line example specifying tokenizer, model arguments, and resource constraints.

```bash
python run.py --datasets siqa_gen winograd_ppl \
--hf-type base \  # HuggingFace model type, base or chat
--hf-path facebook/opt-125m \  # HuggingFace model path
--tokenizer-path facebook/opt-125m \  # HuggingFace tokenizer path (if the same as the model path, can be omitted)
--tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \  # Arguments to construct the tokenizer
--model-kwargs device_map='auto' \  # Arguments to construct the model
--max-seq-len 2048 \  # Maximum sequence length the model can accept
--max-out-len 100 \  # Maximum number of tokens to generate
--min-out-len 100 \  # Minimum number of tokens to generate
--batch-size 64  \  # Batch size
--hf-num-gpus 1  # Number of GPUs required to run the model
```

--------------------------------

### Configure Qwen3-235B-A22B Model with OpenAISDK

Source: https://opencompass.readthedocs.io/en/latest/notes/academic.html

Example configuration for a Qwen3-235B-A22B model deployed via OpenAISDK. Includes parameters like `abbr`, `path`, `temperature`, and `meta_template`. The `pred_postprocessor` can extract non-reasoning content from models that output with a think tag.

```python
from opencompass.models import OpenAISDK
from opencompass.utils.text_postprocessors import extract_non_reasoning_content

qwen3_235b_a22b_model = dict(
    abbr="qwen_3_235b_a22b_thinking", # Used to identify the model configuration
    key="YOUR_SERVE_API_KEY",
    openai_api_base="YOUR_SERVE_API_URL",
    type=OpenAISDK, # The model configuration types, commonly used such as OpenAISDK, TurboMindModelwithChatTemplate, HuggingFacewithChatTemplate
    path="Qwen/Qwen3-235B-A22B",
    temperature=0.6,
    meta_template=dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ],
),
    query_per_second=1,
    max_out_len=32000,
    max_seq_len=32768,
    batch_size=8,
    retry=10,
    extra_body={
        'chat_template_kwargs': {'enable_thinking': True},
    }, # Additional configurations of the model, such as the option in Qwen3 series to control whether they thinks or not
    pred_postprocessor=dict(type=extract_non_reasoning_content), # adding this pred_postprocessor can extract the non-reasoning content from models that output with a think tag
)

models = [
    qwen3_235b_a22b_model,
]
```

--------------------------------

### Install OpenAI Package

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/accelerator_intro.html

Install the necessary openai Python package using pip. This is a prerequisite for interacting with OpenAI-compatible APIs.

```bash
pip install openai

```

--------------------------------

### Define QA dataset in JSONL format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Example of a JSONL file for question and answer pairs.

```json
{"question": "752+361+181+933+235+986=", "answer": "3448"}
{"question": "712+165+223+711=", "answer": "1811"}
{"question": "921+975+888+539=", "answer": "3323"}
{"question": "752+321+388+643+568+982+468+397=", "answer": "4519"}
```

--------------------------------

### Import Dataset Configurations using mmengine

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Use mmengine's read_base function to import preset dataset configurations for evaluation. This example shows importing configurations for Winograd and SIQA datasets, combining them into a final 'datasets' list.

```python
from mmengine.config import read_base  # Use mmengine.read_base() to read the base configuration

with read_base():
    # Directly read the required dataset configurations from the preset dataset configurations
    from .datasets.winograd.winograd_ppl import winograd_datasets  # Read Winograd configuration, evaluated based on PPL (perplexity)
    from .datasets.siqa.siqa_gen import siqa_datasets  # Read SIQA configuration, evaluated based on generation

datasets = [*siqa_datasets, *winograd_datasets]       # The final config needs to contain the required evaluation dataset list 'datasets'
```

--------------------------------

### Example .meta.json with template and input/output columns

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

This configuration is used when a custom question template is defined and specific input and output columns need to be specified. The `template` field overrides `human_prompt` and `bot_prompt`. `input_columns` lists the fields to be used as input, and `output_column` specifies the field containing the answer.

```json
{
    "template": "Question: {my_question}\nX. {X}\nY. {Y}\nZ. {Z}\nW. {W}\nAnswer:",
    "input_columns": ["my_question", "X", "Y", "Z", "W"],
    "output_column": "my_answer"
}
```

--------------------------------

### Setup CustomDataset for Math

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/math_verify.html

Integration of reader, inference, and evaluation configurations into a CustomDataset object.

```python
from mmengine.config import read_base
from opencompass.models import TurboMindModelwithChatTemplate
from opencompass.datasets import CustomDataset

math_datasets = [
    dict(
        type=CustomDataset,
        abbr='my-math-dataset',              # Dataset abbreviation
        path='path/to/your/dataset',         # Path to your dataset file
        reader_cfg=math_reader_cfg,
        infer_cfg=math_infer_cfg,
        eval_cfg=math_eval_cfg,
    )
]
```

--------------------------------

### Run Evaluation with Command-Line Arguments

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Execute evaluations by specifying models and datasets via command-line arguments. This example demonstrates how to use --models and --datasets to run evaluations with a specified Hugging Face model and the 'base_medium' dataset collection.

```bash
python run.py --models hf_llama_7b --datasets base_medium
```

--------------------------------

### Define QA dataset in CSV format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Example of a CSV file for question and answer pairs.

```csv
question,answer
123+147+874+850+915+163+291+604=,3967
149+646+241+898+822+386=,3142
332+424+582+962+735+798+653+214=,4700
649+215+412+495+220+738+989+452=,4170
```

--------------------------------

### Configure custom datasets in Python

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Example of adding custom datasets to the datasets variable in an OpenCompass configuration file.

```python
datasets = [
    {"path": "xxx/test_mcq.csv", "data_type": "mcq", "infer_method": "ppl"},
    {"path": "xxx/test_qa.jsonl", "data_type": "qa", "infer_method": "gen"},
]
```

--------------------------------

### Run OpenCompass Evaluation

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/evaluation_lmdeploy.html

Command to start the evaluation process in OpenCompass using a specified configuration file and output directory.

```bash
python run.py configs/eval_internlm2_lmdeploy.py -w outputs

```

--------------------------------

### Configure ToT for Game24 in OpenCompass

Source: https://opencompass.readthedocs.io/en/latest/prompt/chain_of_thought.html

Example configuration for the 24-Point game using ToTInferencer. Requires a custom task wrapper to handle step-specific prompts and evaluation logic.

```python
# This ToT Game24 config can be found at: opencompass/configs/datasets/game24/game24_gen_8dfde3.py.
from opencompass.datasets import (Game24Dataset, game24_postprocess,
                                  Game24Evaluator, Game24PromptWrapper)

generation_kwargs = dict(temperature=0.7)

game24_infer_cfg = dict(
        prompt_template=dict(
        type=PromptTemplate,
        template='{input}'), # Directly pass the input content, as the Prompt needs to be specified in steps
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=ToTInferencer, # Replace GenInferencer with ToTInferencer
                    generation_kwargs=generation_kwargs,
                    method_generate='propose',  # Method for generating reasoning content, can be independent sampling (sample) or sequential generation (propose)
                    method_evaluate='value', # Method for evaluating reasoning content, can be voting (vote) or scoring (value)
                    method_select='greedy', # Method for selecting reasoning content, can be greedy (greedy) or random (sample)
                    n_evaluate_sample=3,
                    n_select_sample=5,
                    task_wrapper=dict(type=Game24PromptWrapper) # This Wrapper class includes the prompts for each step and methods for generating and evaluating reasoning content, needs customization according to the task
                    ))
```

--------------------------------

### Example Conversation Output

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

The resulting string format received by the model when the SYSTEM role is not configured.

```text
<HUMAN>: Solve the following math questions<eoh>\n
<HUMAN>: 1+1=?<eoh>\n
<BOT>: 2<eob>\n
<HUMAN>: 2+2=?<eoh>\n
<BOT>: 4<eob>\n
end of conversation
```

--------------------------------

### Define Basic Model Meta Template

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

Initial configuration for defining HUMAN and BOT roles with specific start and end tokens.

```python
meta_template = dict(
    round=[
          dict(role='HUMAN', begin='<HUMAN>: ', end='<eoh>\n'),
          dict(role='BOT', begin='<BOT>: ', end='<eob>\n'),
    ],
    reserved_roles=[dict(role='SYSTEM', begin='<SYSTEM>: ', end='<eosys>\n'),],
 ),
```

--------------------------------

### Define a Full Meta Template

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

A comprehensive configuration example for a model's meta template, including system instructions, dialogue rounds, and EOS token settings.

```python
models = [
    dict(meta_template = dict(
            begin="Meta instruction: You are now a helpful and harmless AI assistant.",
            round=[
                    dict(role='HUMAN', begin='HUMAN: ', end='<eoh>\n'),  # begin and end can be a list of strings or integers.
                    dict(role='THOUGHTS', begin='THOUGHTS: ', end='<eot>\n', prompt='None'), # Here we can set the default prompt, which may be overridden by the specific dataset
                    dict(role='BOT', begin='BOT: ', generate=True, end='<eob>\n'),
            ],
            end="end of conversion",
            reserved_roles=[dict(role='SYSTEM', begin='SYSTEM: ', end='\n'),],
            eos_token_id=10000,
         ),
     )
]
```

--------------------------------

### Define Data Example Dictionary

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

A sample data dictionary used for testing template field replacement.

```python
example = {
    'question': '1+1=?',
    'answer': '2',  # Assume the answer is in the reader_cfg.output_column
    'irrelevant_infos': 'blabla',
}
```

--------------------------------

### Define MCQ dataset in JSONL format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Example of a JSONL file for multiple-choice questions where each line contains the question, options, and the correct answer.

```json
{"question": "165+833+650+615=", "A": "2258", "B": "2263", "C": "2281", "answer": "B"}
{"question": "368+959+918+653+978=", "A": "3876", "B": "3878", "C": "3880", "answer": "A"}
{"question": "776+208+589+882+571+996+515+726=", "A": "5213", "B": "5263", "C": "5383", "answer": "B"}
{"question": "803+862+815+100+409+758+262+169=", "A": "4098", "B": "4128", "C": "4178", "answer": "C"}
```

--------------------------------

### Configure Meta Template with Begin and End Strings

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

Extending the meta template to include global conversation start and end strings.

```python
meta_template = dict(
    round=[
          dict(role='HUMAN', begin='<HUMAN>: ', end='<eoh>\n'),
          dict(role='BOT', begin='<BOT>: ', end='<eob>\n'),
    ],
    reserved_roles=[dict(role='SYSTEM', begin='<SYSTEM>: ', end='<eosys>\n'),],
    begin="Meta instruction: You are now a helpful and harmless AI assistant.",
    end="end of conversation",
 ),
```

--------------------------------

### Example .meta.json with human_prompt and bot_prompt

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Use this configuration when custom human and bot prompts are required. The `human_prompt` field defines the template for user input, and `bot_prompt` defines the template for the model's response. Variables within curly braces `{}` are placeholders.

```json
{
    "human_prompt": "Question: 127 + 545 + 588 + 620 + 556 + 199 =\nA. 2632\nB. 2635\nC. 2645\nAnswer: Let's think step by step, 127 + 545 + 588 + 620 + 556 + 199 = 672 + 588 + 620 + 556 + 199 = 1260 + 620 + 556 + 199 = 1880 + 556 + 199 = 2436 + 199 = 2635. So the answer is B.\nQuestion: {question}\nA. {A}\nB. {B}\nC. {C}\nAnswer: ",
    "bot_prompt": "{answer}"
}
```

--------------------------------

### List configurations for models and datasets

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Use this command to identify available model and dataset configuration paths.

```bash
python tools/list_configs.py llama mmlu
```

--------------------------------

### Configure Model Backend

Source: https://opencompass.readthedocs.io/en/latest/user_guides/deepseek_r1.html

Sets up the model configuration using LMDeploy as the reasoning model backend.

```python
# LMDeploy model configuration example
models = [
    dict(
        type=TurboMindModelwithChatTemplate,
        abbr='deepseek-r1-distill-qwen-7b-turbomind',
        path='deepseek-ai/DeepSeek-R1-Distill-Qwen-7B',
        engine_config=dict(session_len=32768, max_batch_size=128, tp=1),
        gen_config=dict(
            do_sample=True,
            temperature=0.6,
            top_p=0.95,
            max_new_tokens=32768
        ),
        max_seq_len=32768,
        batch_size=64,
        run_cfg=dict(num_gpus=1),
        pred_postprocessor=dict(type=extract_non_reasoning_content)
    ),
    # Extendable 14B/32B configurations...
]
```

--------------------------------

### Launch evaluation task with run.py

Source: https://opencompass.readthedocs.io/en/latest/user_guides/experimentation.html

General command-line syntax for initiating an evaluation task.

```bash
python run.py $EXP {--slurm | --dlc | None} [-p PARTITION] [-q QUOTATYPE] [--debug] [-m MODE] [-r [REUSE]] [-w WORKDIR] [-l] [--dry-run] [--dump-eval-details]
```

--------------------------------

### Install LMDeploy with CUDA 11+

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/evaluation_lmdeploy.html

Install lmdeploy with specific versions for CUDA 11+ compatibility. Ensure LMDEPLOY_VERSION and PYTHON_VERSION are set.

```bash
export LMDEPLOY_VERSION=0.6.0
export PYTHON_VERSION=310
pip install https://github.com/InternLM/lmdeploy/releases/download/v${LMDEPLOY_VERSION}/lmdeploy-${LMDEPLOY_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux2014_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118

```

--------------------------------

### Example CSV Dataset Format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/llm_judge.html

This is an example of the CSV format for datasets used in LLM judge evaluation. Each entry requires a 'problem' and an 'answer'.

```csv
problem,answer
"What is the capital of France?","Paris"
```

--------------------------------

### Example JSONL Dataset Format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/llm_judge.html

This is an example of the JSON Lines format for datasets used in LLM judge evaluation. Each entry requires a 'problem' and an 'answer'.

```json
{"problem": "What is the capital of France?", "answer": "Paris"}
```

--------------------------------

### Complete LLM Judge Evaluation Setup

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/llm_judge.html

This Python code demonstrates a complete setup for LLM judge evaluation using CustomDataset and GenericLLMEvaluator. It includes necessary imports, judge template definition, and configuration for dataset reader.

```python
from mmengine.config import read_base
from opencompass.models import TurboMindModelwithChatTemplate
from opencompass.datasets import CustomDataset
from opencompass.evaluator import GenericLLMEvaluator
from opencompass.datasets import generic_llmjudge_postprocess
from opencompass.openicl.icl_prompt_template import PromptTemplate
from opencompass.openicl.icl_retriever import ZeroRetriever
from opencompass.openicl.icl_inferencer import GenInferencer

# Import your judge model configuration
with read_base():
    from opencompass.configs.models.qwen2_5.lmdeploy_qwen2_5_14b_instruct import \
        models as judge_model

# Define your judge template
JUDGE_TEMPLATE = """
Please evaluate whether the following response correctly answers the question.
Question: {problem}
Reference Answer: {answer}
Model Response: {prediction}

Is the model response correct? If correct, answer "A"; if incorrect, answer "B".
""".strip()

# Dataset reader configuration
reader_cfg = dict(input_columns=['problem'], output_column='answer')
```

--------------------------------

### Example MATHVerifyEvaluator Output

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/math_verify.html

Structure of the evaluation results returned by the MATHVerifyEvaluator.

```json
{
    'accuracy': 85.0,  # Percentage of correct answers
    'details': [
        {
            'predictions': 'x = 2',           # Parsed prediction
            'references': 'x = 2',         # Parsed reference
            'correct': True            # Whether they match
        },
        # ... more results
    ]
}
```

--------------------------------

### Define a data sample

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

Example of a data sample structure used for testing.

```python
example = {
    'question': '1+1=?',
    'answer': '2',  # Assume the answer is in the reader_cfg.output_column
    'irrelavent_infos': 'blabla',
}
```

--------------------------------

### Humanevalx Evaluation Output

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/code_eval_service.html

Example of the JSON response returned by the evaluation service.

```json
"{\"pass@1\": 37.19512195121951%}"
```

--------------------------------

### List All Model and Dataset Configurations

Source: https://opencompass.readthedocs.io/en/latest/tools.html

Lists all available model and dataset configurations. Supports fuzzy search and wildcard characters. Execute without parameters to list all configurations.

```bash
python tools/list_configs.py [PATTERN1] [PATTERN2] [...]
```

```bash
python tools/list_configs.py mmlu llama
```

--------------------------------

### Configure CascadeEvaluator

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/llm_judge.html

Initializes a rule-based evaluator for use within a CascadeEvaluator setup.

```python
# Define a rule-based evaluator
rule_evaluator = dict(type=MATHVerifyEvaluator)
```

--------------------------------

### Display Evaluation Results

Source: https://opencompass.readthedocs.io/en/latest/user_guides/deepseek_r1.html

Example output format for evaluation metrics on specific datasets.

```text
dataset                             version    metric         mode    deepseek-r1-distill-qwen-7b-turbomind                                                                                                       ----------------------------------  ---------  -------------  ------  ---------------------------------------                                                                                                     MATH                                -          -              -                                         AIME2024-Aveage8                    -          naive_average  gen     56.25     
```

--------------------------------

### List available configurations

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Utility script to view or filter available model and dataset configurations.

```bash
# List all configurations
python tools/list_configs.py
```

--------------------------------

### Configure Zero-Shot CoT

Source: https://opencompass.readthedocs.io/en/latest/prompt/chain_of_thought.html

Implement Zero-Shot CoT by modifying the PromptTemplate to include the 'Let's think step by step' instruction.

```python
qa_infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template="Answer the question:\nQ: {question}?\nLet's think step by step:\n"
    ),
    retriever=dict(type=ZeroRetriever)
)
```

--------------------------------

### Generative Inference Prompt Truncation

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

Example of a truncated prompt used during generative task evaluation.

```text
Meta instruction: You are now a helpful and harmless AI assistant.
<SYSTEM>: Solve the following math questions<eosys>\n
<HUMAN>: 1+1=?<eoh>\n
<BOT>: 2<eob>\n
<HUMAN>: 2+2=?<eoh>\n
<BOT>:
```

--------------------------------

### Configure evaluation via command line

Source: https://opencompass.readthedocs.io/en/latest/user_guides/experimentation.html

Specify models and datasets directly through command-line arguments instead of a configuration file.

```bash
python run.py --models hf_opt_350m hf_opt_125m --datasets siqa_gen winograd_ppl
```

--------------------------------

### Example Evaluation Output Structure

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/llm_judge.html

Shows the JSON structure returned when evaluation details are dumped.

```json
{
    'accuracy': 75.0,  # Percentage of responses judged as correct
    'details': [
        {
            'origin_prompt': """
            Please evaluate whether the following response correctly answers the question.
            Question: What is the capital of France?
            Reference Answer: Paris
            Model Response: Paris
            Is the model response correct? If correct, answer "A"; if incorrect, answer "B".
""",
            'gold': 'Paris',
            'prediction': 'A',
        },
        # ... more results
    ]
}
```

--------------------------------

### Configure HuggingFace models via command line

Source: https://opencompass.readthedocs.io/en/latest/user_guides/experimentation.html

Define a HuggingFace model and specify datasets directly in the command line.

```bash
python run.py --datasets siqa_gen winograd_ppl --hf-type base --hf-path huggyllama/llama-7b
```

--------------------------------

### Configure Dataset Inference with PromptTemplate

Source: https://opencompass.readthedocs.io/en/latest/notes/rawprompttemplate.html

The original inference configuration format using the PromptTemplate class.

```python
infer_cfg = dict(
    prompt_template=dict(
        type=PromptTemplate,
        template=dict(
            begin=[
                dict(
                    role='SYSTEM',
                    fallback_role='HUMAN',
                    prompt="You are a helpful assistant.",
                )
           ],
            round=[
                dict(
                    role='HUMAN',
                    prompt='{problem}\nRemember to put your final answer within \\boxed{}.',
                ),
            ],
        ),
    ),
    retriever=dict(type=ZeroRetriever),
    inferencer=dict(type=GenInferencer),
)
```

--------------------------------

### Resulting Prompt Output

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

The output generated after applying the template to the data example, showing field masking.

```text
{anything}
Question: 1+1=?
Answer:
```

--------------------------------

### Create Conda Environment

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Initializes the runtime environment for OpenCompass using Conda.

```bash
# conda create --name opencompass_lmdeploy python=3.10 -y

conda activate opencompass
```

--------------------------------

### Define experiment configuration

Source: https://opencompass.readthedocs.io/en/latest/get_started/quick_start.html

Create a Python configuration file to define datasets and models using the inheritance mechanism.

```python
from mmengine.config import read_base

with read_base():
    from .datasets.siqa.siqa_gen import siqa_datasets
    from .datasets.winograd.winograd_ppl import winograd_datasets
    from .models.opt.hf_opt_125m import opt125m
    from .models.opt.hf_opt_350m import opt350m

datasets = [*siqa_datasets, *winograd_datasets]
models = [opt125m, opt350m]
```

--------------------------------

### Configure evaluation via file

Source: https://opencompass.readthedocs.io/en/latest/user_guides/experimentation.html

Execute an evaluation task using a Python configuration file containing datasets and models.

```bash
python run.py configs/eval_demo.py
```

--------------------------------

### Debug Code Evaluation Service

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/code_eval_service.html

Commands to mount the code-evaluator for local development and start the evaluation server.

```bash
docker run -it -p 5000:5000 -v /local/path/of/code-evaluator:/workspace/code-evaluator code-eval:latest bash
```

```bash
cd code-evaluator && pip install -r requirements.txt
python server.py
```

--------------------------------

### Clone OpenCompass Repository

Source: https://opencompass.readthedocs.io/en/latest/notes/contribution_guide.html

Clone the forked OpenCompass repository to your local machine when starting your first PR.

```git
git clone git@github.com:XXX/opencompass.git
```

--------------------------------

### Import and Combine Dataset Configurations

Source: https://opencompass.readthedocs.io/en/latest/user_guides/datasets.html

Use the mmengine read_base mechanism to import multiple dataset configurations into a single evaluation script.

```python
from mmengine.config import read_base

with read_base():
    from .datasets.CLUE_afqmc.CLUE_afqmc_gen_db509b import afqmc_datasets
    from .datasets.CLUE_cmnli.CLUE_cmnli_ppl_b78ad4 import cmnli_datasets

datasets = []
datasets += afqmc_datasets
datasets += cmnli_datasets
```

--------------------------------

### Run Code Evaluation Container with Go Module Support

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/code_eval_service.html

Use this command when creating the container if you encounter timeouts during Go evaluation, enabling Go modules and a specific proxy.

```bash
docker run -it -p 5000:5000 -e GO111MODULE=on -e GOPROXY=https://goproxy.io code-eval-{your-dataset}:latest python server.py
```

--------------------------------

### Download and Unzip Datasets with wget

Source: https://opencompass.readthedocs.io/en/latest/get_started/installation.html

Download dataset archives using wget. This is a standard command-line utility for downloading files from the web.

```bash
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-complete-20240207.zip
```

--------------------------------

### Define HuggingFaceCausalLM Model Configuration

Source: https://opencompass.readthedocs.io/en/latest/user_guides/config.html

Use this snippet to define a model configuration for HuggingFaceCausalLM. Ensure MMEngine is installed.

```python
from opencompass.models import HuggingFaceCausalLM

models = [
    dict(
        type=HuggingFaceCausalLM,
        path='huggyllama/llama-7b',
        model_kwargs=dict(device_map='auto'),
        tokenizer_path='huggyllama/llama-7b',
        tokenizer_kwargs=dict(padding_side='left', truncation_side='left'),
        max_seq_len=2048,
        max_out_len=50,
        run_cfg=dict(num_gpus=8, num_procs=1),
    )
]

```

--------------------------------

### Define infer_cfg Configuration

Source: https://opencompass.readthedocs.io/en/latest/prompt/prompt_template.html

A typical configuration block for defining prompt templates, in-context example retrieval, and inference methods.

```python
infer_cfg = dict(
    ice_template=dict(  # Template used to construct In Context Examples (ice).
        type=PromptTemplate,
        template='{question}\n{answer}'
    ),
    prompt_template=dict(  # Template used to construct the main prompt.
        type=PromptTemplate,
        template='Solve the following questions.\n</E>{question}\n{answer}',
        ice_token="</E>"
    ),
    retriever=dict(type=FixKRetriever, fix_id_list=[0, 1]),  # Definition of how to retrieve in-context examples.
    inferencer=dict(type=GenInferencer),  # Method used to generate predictions.
)
```

--------------------------------

### Read Configuration File with MMEngine

Source: https://opencompass.readthedocs.io/en/latest/user_guides/config.html

Parse a configuration file using `Config.fromfile` from MMEngine. This is used to load configurations defined in Python files.

```python
from mmengine.config import Config
>>> cfg = Config.fromfile('./model_cfg.py')
>>> print(cfg.models[0])
{'type': HuggingFaceCausalLM, 'path': 'huggyllama/llama-7b', 'model_kwargs': {'device_map': 'auto'}, ...}

```

--------------------------------

### Define MCQ dataset in CSV format

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/custom_dataset.html

Example of a CSV file for multiple-choice questions with headers defining the question, options, and answer.

```csv
question,A,B,C,answer
127+545+588+620+556+199=,2632,2635,2645,B
735+603+102+335+605=,2376,2380,2410,B
506+346+920+451+910+142+659+850=,4766,4774,4784,C
504+811+870+445=,2615,2630,2750,B
```

--------------------------------

### Configure Execution Strategies

Source: https://opencompass.readthedocs.io/en/latest/user_guides/evaluation.html

Define inference and evaluation workflows by specifying partitioners and runners in the configuration file.

```python
from opencompass.partitioners import SizePartitioner, NaivePartitioner
from opencompass.runners import SlurmRunner
from opencompass.tasks import OpenICLInferTask, OpenICLEvalTask

infer = dict(
    partitioner=dict(type=SizePartitioner, max_task_size=5000),
    runner=dict(
        type=SlurmRunner,
        max_num_workers=64,
        task=dict(type=OpenICLInferTask),
        retry=5),
)

eval = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(
        type=LocalRunner,
        max_num_workers=32,
        task=dict(type=OpenICLEvalTask)),
)
```

--------------------------------

### Configure API Model Meta Templates

Source: https://opencompass.readthedocs.io/en/latest/prompt/meta_template.html

Simplified meta template configurations for API models, supporting both configurations with and without system instructions.

```python
# If the API model does not support system instructions
meta_template=dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True)
    ],
)

# If the API model supports system instructions
meta_template=dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True)
    ],
    reserved_roles=[
        dict(role='SYSTEM', api_role='SYSTEM'),
    ],
)
```

--------------------------------

### Deploy LMDeploy Inference Service

Source: https://opencompass.readthedocs.io/en/latest/advanced_guides/accelerator_intro.html

Deploy an inference acceleration service using LMDeploy. This command starts an API server for the specified model.

```bash
lmdeploy serve api_server meta-llama/Meta-Llama-3-8B-Instruct --model-name Meta-Llama-3-8B-Instruct --server-port 23333

```
Tokens:
Raw