EvalScope

https://evalscope.readthedocs.io/zh-cn/latest/
Admin
EvalScope is a comprehensive benchmarking and evaluation framework for testing and comparing large...
Tokens:343,246
Snippets:2,671
Trust Score:9.6
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/longwriter.html

Install the required EvalScope framework and vLLM packages.

```bash
pip install evalscope[framework] -U
pip install vllm -U
```

--------------------------------

### Install SWE-bench package

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_lite_agentic.html

Required installation command before running evaluations.

```bash
pip install swebench==4.1.0
```

--------------------------------

### Local Sandbox Manager Log Output

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/sandbox.html

Example console output indicating the local sandbox manager has successfully started.

```text
[INFO:ms_enclave] Local sandbox manager started
...
```

--------------------------------

### Install τ²-bench

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/tau2_bench.html

Use pip to install the benchmark package from the official repository.

```bash
pip install git+https://github.com/sierra-research/tau2-bench@v0.2.0
```

--------------------------------

### Install OpenCompass Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/opencompass_backend.html

Install the necessary dependencies for the OpenCompass backend.

```bash
# 安装opencompass依赖
pip install evalscope[opencompass] -U
```

--------------------------------

### Install document parsing dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html

Install unstructured and system-level utilities for PDF processing.

```bash
pip install "unstructured[all-docs]"
apt install poppler-utils tesseract-ocr -y
```

--------------------------------

### Install Flash Attention

Source: https://evalscope.readthedocs.io/zh-cn/latest/best_practice/llm_full_stack.html

Installs the flash-attention library to optimize inference speed.

```bash
pip install flash-attn --no-build-isolation
```

--------------------------------

### Install EvalScope Performance Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/experiments/speed_benchmark/QwQ-32B-Preview.html

Install the necessary packages to run performance benchmarks.

```bash
pip install evalscope[perf] -U
```

--------------------------------

### Install τ-bench

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/tau_bench.html

Command to install the τ-bench package via pip.

```bash
pip install git+https://github.com/sierra-research/tau-bench
```

--------------------------------

### Install MCP Support

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/agent/native.html

Install the necessary dependencies to enable MCP server integration.

```bash
pip install evalscope[mcp]
```

--------------------------------

### Install SWE-bench_Pro dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench_pro.html

Install the necessary Python package to enable sandbox support for SWE-bench_Pro evaluations.

```bash
pip install 'evalscope[sandbox]'
```

--------------------------------

### Performance Overview Output

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/stress_test/multi_turn.html

Example output showing performance metrics and per-request latency statistics.

```text
             Performance Overview
┏━━━━━━┳━━━━━━┳━━━━━┳━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃Conc. ┃ Rate ┃ Num ┃  RPS ┃  Gen/s ┃ Success ┃
┡━━━━━━╇━━━━━━╇━━━━━╇━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│   10 │    - │ 300 │ 3.87 │ 985.21 │  100.0% │
└──────┴──────┴─────┴──────┴────────┴─────────┘


                    Per-Request Metrics
┏━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃Conc. ┃ Rate ┃ Metric        ┃    avg ┃   p50 ┃   p99 ┃
┡━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━╇━━━━━━━┩
│   10 │    - │ Latency (s)   │  2.571 │ 2.340 │ 4.103 │
│      │      │ TTFT (ms)     │   55.0 │  50.0 │  98.0 │
│      │      │ TPOT (ms)     │   10.0 │   9.5 │  13.0 │
│      │      │ Input Tokens  │  428.7 │ 390.0 │ 912.0 │
│      │      │ Output Tokens │  186.3 │ 160.0 │ 384.0 │
│      │      │ Turns/Req     │   1.98 │     - │     - │
│      │      │ Cache Hit (%) │  53.7% │     - │     - │
└──────┴──────┴───────────────┴────────┴───────┴───────┘
```

--------------------------------

### 使用 CLI 运行 MVBench 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/mvbench.html

通过命令行接口快速启动 MVBench 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets mvbench \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### 使用 CLI 运行 TVBench 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/tvbench.html

通过命令行接口执行 TVBench 基准测试评估。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets tvbench \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Install EvalScope Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/clip_benchmark.html

Install the necessary dependencies for RAG evaluation.

```bash
pip install evalscope[rag] -U
```

--------------------------------

### 通过 CLI 运行 GeneralArena 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/general_arena.html

使用命令行接口执行 GeneralArena 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets general_arena \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### 使用 CLI 运行 SIQA 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/siqa.html

通过命令行接口运行 SIQA 基准测试评估。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets siqa \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Install Terminal-Bench dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/terminal_bench.html

Install the necessary packages to run Terminal-Bench benchmarks.

```bash
pip install evalscope[terminal_bench]
```

--------------------------------

### 使用 CLI 运行 Winogrande 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/winogrande.html

通过命令行接口调用 evalscope 执行 Winogrande 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets winogrande \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Install Toolathlon Integration

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/toolathlon.html

Install the necessary dependencies for the Toolathlon integration in EvalScope.

```bash
pip install evalscope[toolathlon]
```

--------------------------------

### 使用 CLI 运行 MMLU-Pro 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/mmlu_pro.html

通过命令行接口调用 evalscope 进行 MMLU-Pro 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets mmlu_pro \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### 启动 vLLM 模型服务

Source: https://evalscope.readthedocs.io/zh-cn/latest/best_practice/qwen3_next.html

在本地启动 Qwen3-Next-80B-A3B-Instruct 服务，配置张量并行和上下文长度限制。

```bash
VLLM_USE_MODELSCOPE=true VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct --served-model-name Qwen3-Next-80B-A3B-Instruct --port 8801 --tensor-parallel-size 4 --max-model-len 8000 --gpu-memory-utilization 0.9 --max-num-seqs 32
```

--------------------------------

### Install BFCL-v3 Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/bfcl_v3.html

Commands to install the required EvalScope and bfcl-eval packages.

```bash
pip install evalscope # 安装 evalscope
pip install bfcl-eval==2025.10.27.1 # 安装 bfcl-eval
```

--------------------------------

### Install RAG dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/blog/RAG/multimodal_RAG.html

Install the necessary packages to support RAG evaluation.

```bash
pip install evalscope[rag] # 提供RAG评测依赖
```

--------------------------------

### 使用 CLI 运行 MiniWoB 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/miniwob.html

通过命令行接口配置并启动 MiniWoB 评估任务。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets miniwob \
    --agent-config '{"mode":"native","strategy":"function_calling","max_steps":10}' \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### 使用 CLI 运行评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/wenet_speech.html

通过命令行接口调用 evalscope 进行 WenetSpeech 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets wenet_speech \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Evaluate BrowseComp via CLI

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/browsecomp.html

Run the benchmark using the EvalScope command-line interface.

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets browsecomp \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Start vLLM Service

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/longwriter.html

Deploy the model using vLLM with specific memory and length configurations.

```bash
CUDA_VISIBLE_DEVICES=0 VLLM_USE_MODELSCOPE=True vllm serve --max-model-len=65536 --gpu_memory_utilization=0.95 --trust-remote-code ZhipuAI/LongWriter-glm4-9b
```

--------------------------------

### CLI 评估命令

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/bigcodebench.html

通过命令行接口运行 BigCodeBench 评估任务。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets bigcodebench \
    --sandbox '{"enabled": true}' \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### 使用 CLI 运行 MMMU-PRO 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/mmmu_pro.html

通过命令行界面调用 EvalScope 执行 MMMU-PRO 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets mmmu_pro \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Install VLMEvalKit dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/vlmevalkit_backend.html

Install the required extra dependencies for VLMEvalKit support using pip.

```bash
# 安装额外依赖
pip install evalscope[vlmeval] -U
```

--------------------------------

### ArxivRollBench Data Subset Example

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/arxivrollbench.html

Example JSON structure for a single entry in the ArxivRollBench dataset.

```json
{
  "input": [
    {
      "id": "7b220fd6",
      "content": "Answer the following ArxivRollBench multiple choice question. The entire content of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D.\n\nSelect the option that correctly compl ... [TRUNCATED 381 chars] ... m a diagonal matrix into the identity, allows us to write the input matrix as a product of transvections. **C**: Note that row and column operations are effected by left- and right multiplications by transvections\n\nA) BAC\nB) ABC\nC) ACB\nD) BCA"
    }
  ],
  "choices": [
    "BAC",
    "ABC",
    "ACB",
    "BCA"
  ],
  "target": "C",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "original_label": "Selection 3",
    "task_type": "s/c"
  }
}
```

--------------------------------

### Install EvalScope Sandbox Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/swe_bench_pro.html

Install the necessary dependencies to enable Docker sandbox support for evaluations.

```bash
pip install evalscope[sandbox]
```

--------------------------------

### 启动模型服务

Source: https://evalscope.readthedocs.io/zh-cn/latest/best_practice/gpt_oss.html

使用 vLLM 启动 GPT-OSS 模型服务，支持 ModelScope 或 HuggingFace 下载源。

```bash
VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 VLLM_USE_MODELSCOPE=true vllm serve openai-mirror/gpt-oss-20b --served-model-name gpt-oss-20b --trust_remote_code --port 8801
```

```bash
VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 vllm serve openai/gpt-oss-20b --served-model-name gpt-oss-20b --trust_remote_code --port 8801
```

--------------------------------

### 使用 CLI 运行评测

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/automation_bench.html

通过命令行接口启动 AutomationBench 评测任务。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets automation_bench \
    --limit 10  # 正式评测时请删除此行
```

--------------------------------

### Install PerspectiveGap Scorer

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/perspective_gap_role_assignment.html

Install the required scoring module from the official repository using pip or uv.

```bash
pip install 'perspective-gap @ git+https://github.com/WhymustIhaveaname/PerspectiveGap.git@9c6921b3337ff3e6a6a453f68d117a8c1663135e'
```

```bash
uv pip install 'perspective-gap @ git+https://github.com/WhymustIhaveaname/PerspectiveGap.git@9c6921b3337ff3e6a6a453f68d117a8c1663135e'
```

--------------------------------

### Configure NativeAgent with MCP Servers

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/agent/native.html

Example of setting up a task with both local Stdio and remote HTTP MCP servers.

```python
import sys
from evalscope import TaskConfig, run_task
from evalscope.api.agent import NativeAgentConfig
from evalscope.api.agent.mcp import MCPServerConfigStdio, MCPServerConfigHTTP

task_config = TaskConfig(
    model='qwen3-max',
    api_url='https://dashscope.aliyuncs.com/compatible-mode/v1',
    eval_type='openai_api',
    datasets=['gaia'],
    agent_config=NativeAgentConfig(
        mcp_servers=[
            # 本地启动 MCP 服务器进程
            MCPServerConfigStdio(
                command=sys.executable,
                args=['-m', 'mcp_server_fetch', '--ignore-robots-txt'],
                name='fetch',
            ),
            # 连接远端 MCP endpoint
            MCPServerConfigHTTP(
                url='https://mcp.api-inference.modelscope.net/<server-id>/mcp',
                headers={'Authorization': 'Bearer <TOKEN>'},
                name='modelscope_tool',
            ),
        ],
    ),
    dataset_args={'gaia': {'subset_list': ['2023_level1']}},
    limit=5,
)
run_task(task_config)
```

--------------------------------

### Verify Docker SDK and ms-enclave Installation

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/swe_bench_pro.html

Check if the required Docker SDK and ms-enclave packages are correctly installed in the environment.

```python
python -c "import docker, ms_enclave; print(docker.__version__, ms_enclave.__version__)"
```

--------------------------------

### Install WideSearch Dependencies

Source: https://evalscope.readthedocs.io/zh-cn/latest/third_party/wide_search.html

Install the base WideSearch package or include optional dependencies for MCP or Docker sandbox support.

```bash
pip install 'evalscope[wide_search]'
```

```bash
pip install 'evalscope[wide_search,mcp]'      # Fetch MCP 及其他 MCP server
pip install 'evalscope[wide_search,sandbox]'  # Docker sandbox
```

--------------------------------

### 使用 CLI 运行 JobBench 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/job_bench.html

通过命令行接口启动 JobBench 评估任务，需配置模型参数及智能体策略。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets job_bench \
    --agent-config '{"mode":"native","strategy":"function_calling","max_steps":250}' \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Sample Example

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/chinese_simpleqa.html

This is an example of a sample data point from the 'Chinese Culture' subset, showing the input question and the target answer.

```json
{
  "input": [
    {
      "id": "b6b48177",
      "content": "请回答问题：\n\n伏兔穴所属的经脉是什么？"
    }
  ],
  "target": "足阳明胃经",
  "id": 0,
  "group_id": 0,
  "subset_key": "中华文化",
  "metadata": {
    "id": "97e7f58a3b154facaa3a5c64d678c7bf",
    "primary_category": "中华文化",
    "secondary_category": "中医"
  }
}
```

--------------------------------

### Example Sample

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/ceval.html

This snippet shows an example of a question from the 'computer_network' subset, including the input, choices, target answer, and metadata.

```json
{
  "input": [
    {
      "id": "73073a35",
      "content": "以下是一些示例问题：\n\n问题：下列设备属于资源子网的是____。\n选项：\nA. 计算机软件\nB. 网桥\nC. 交换机\nD. 路由器\n解析：1. 首先，资源子网是指提供共享资源的网络，如打印机、文件服务器等。\r\n2. 其次，我们需要了解选项中设备的功能。网桥、交换机和路由器的主要功能是实现不同网络之间的通信和数据传输，是通信子网设备。而计算机软件可以提供共享资源的功能。\n答案：A\n\n问题：滑动窗口的作用是____。\n选项：\nA. 流量控制\nB. 拥塞控制\nC. 路由控制\nD. 差错 ... [TRUNCATED] ... Mbps，所以答案为min{80Mbps, 100Mbps}=80Mbps，选C。\n答案：C\n\n\n以下是中国关于计算机网络的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____\n选项：\nA. 1\nB. 2\nC. 3\nD. 4\n"
    }
  ],
  "choices": [
    "1",
    "2",
    "3",
    "4"
  ],
  "target": "C",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "id": 0,
    "explanation": "",
    "subject": "computer_network"
  }
}
```

--------------------------------

### Start Remote Sandbox Server

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/sandbox.html

Launch the sandbox server on a remote machine to accept evaluation requests.

```bash
ms-enclave server --host 0.0.0.0 --port 1234
```

--------------------------------

### Sample Example

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/broad_twitter_corpus.html

This is a sample example of the Broad Twitter Corpus dataset, showing the input, target, and metadata for a tweet.

```json
{
  "input": [
    {
      "id": "62394bfa",
      "content": "Here are some examples of named entity recognition:\n\nInput:\nI hate the words chunder , vomit and puke . BUUH .\n\nOutput:\n<response>I hate the words chunder , vomit and puke . BUUH .</response>\n\nInput:\n♥ . . ) ) ( ♫ . ( ړײ ) ♫ . ♥ . « ▓ » ♥ . ♫ ... [TRUNCATED] ... planations, just the tagged text.\n6. If entity spans overlap, choose the most specific entity type.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\n@ colgo hey , congrats to you and the team ! Always worth a read :)
"
    }
  ],
  "target": "<response><person>@</person> <person>colgo</person> hey , congrats to you and the team ! Always worth a read :)</response>",
  "id": 0,
  "group_id": 0,
  "metadata": {
    "tokens": [
      "@",
      "colgo",
      "hey",
      ",",
      "congrats",
      "to",
      "you",
      "and",
      "the",
      "team",
      "!",
      "Always",
      "worth",
      "a",
      "read",
      ":)"
    ],
    "ner_tags": [
      "B-PER",
      "B-PER",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O",
      "O"
    ]
  }
}
```

--------------------------------

### 使用命令行运行 MMBench 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/mm_bench.html

通过 evalscope CLI 工具执行 MMBench 基准测试的命令示例。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets mm_bench \
    --limit 10  # 正式评估时请删除此行
```

--------------------------------

### Example Sample for Art_Style Subset

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/blink.html

This is an example of a data sample from the 'Art_Style' subset, illustrating the input format with text and base64 encoded images, the choices, and the target answer.

```json
{
  "input": [
    {
      "id": "a522940e",
      "content": [
        {
          "text": "Answer the following multiple choice question. The last line of your response should be of the following format:\n'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B.\n\nSome most common art painting styles include Realism, Impressi ... [TRUNCATED] ...  of art paintings, use the first image as the reference image, and determine which one of the second or the third image shares the same style as the reference image?\nSelect from the following choices.\n(A) the second image\n(B) the third image\n"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~477.8KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~876.1KB]"
        },
        {
          "image": "[BASE64_IMAGE: jpeg, ~329.2KB]"
        }
      ]
    }
  ],
  "choices": [
    "the second image",
    "the third image"
  ],
  "target": "A",
  "id": 0,
  "group_id": 0
}
Copy code
```

--------------------------------

### Two-Stage Evaluation Results

Source: https://evalscope.readthedocs.io/zh-cn/latest/user_guides/backend/rageval_backend/mteb.html

Example output for the retrieval stage and the reranking stage.

```json
{
  "task_name": "T2Retrieval",
  "scores": {"dev": [{"main_score": 0.73143, "ndcg_at_10": 0.73143}]}
}
```

```json
{
  "task_name": "T2Retrieval",
  "scores": {"dev": [{"main_score": 0.661, "ndcg_at_10": 0.661}]}
}
```

--------------------------------

### 使用 CLI 运行 CharXiv 评估

Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/charxiv.html

通过命令行界面调用 evalscope 对模型进行 CharXiv 基准测试。

```bash
evalscope eval \
    --model YOUR_MODEL \
    --api-url OPENAI_API_COMPAT_URL \
    --api-key EMPTY_TOKEN \
    --datasets charxiv \
    --limit 10  # 正式评估时请删除此行
```
Tokens:
Raw