### Configuration File Example Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/embedding.html Example configuration for running a custom retrieval evaluation task. ```python task_cfg = { "work_dir": "outputs", "eval_backend": "RAGEval", "eval_config": { "tool": "MTEB", "model": [ { "model_name_or_path": "AI-ModelScope/m3e-base", "pooling_mode": None, # load from model config "max_seq_length": 512, "prompt": "", "model_kwargs": {"torch_dtype": "auto"}, "encode_kwargs": { "batch_size": 128, }, } ], "eval": { "tasks": ["CustomRetrieval"], "dataset_path": "custom_eval/text/retrieval", "verbosity": 2, "overwrite_results": True, "limits": 500, }, }, } ``` -------------------------------- ### Configuration File Example (YAML) Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/vlm.html Example of a configuration file in YAML format for running evaluations with VLMEvalKit. ```yaml eval_backend: VLMEvalKit eval_config: model: - type: qwen-vl-chat # 部署的模型名称 name: CustomAPIModel # 固定值 api_base: http://localhost:8000/v1/chat/completions key: EMPTY temperature: 0.0 img_size: -1 data: - custom_mcq # 自定义数据集名称，放在`~/LMUData`路径中 mode: all limit: 10 reuse: false work_dir: outputs nproc: 1 ``` -------------------------------- ### Example Sample Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/air_bench_foundation.html Example of a sample from the dataset, showing input, target, and metadata. ```json { "input": [ { "id": "544443c0", "content": [ { "audio": "[BASE64_AUDIO: mp3, ~25.9KB]", "format": "mp3" }, { "text": "Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\nWhich age range do you believe best matches the speaker's voice?\nA. teens to twenties\nB. thirties to fourties\nC. fifties to sixties\nD. seventies to eighties" } ] } ], "target": "B", "id": 0, "group_id": 0, "subset_key": "Speaker_Age_Prediction_common_voice_13.0_en", "metadata": { "uniq_id": 5973, "task_name": "Speaker_Age_Prediction", "dataset_name": "common_voice_13.0_en", "category": "speech", "answer_gt_text": "thirties to fourties", "choices": { "A": "teens to twenties", "B": "thirties to fourties", "C": "fifties to sixties", "D": "seventies to eighties" } } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/chartqa.html This is an example of a sample input and target for the human_test subset of ChartQA. ```json { "input": [ { "id": "c75439e0", "content": [ { "text": "\nHow many food item is shown in the bar graph?\n\nThe last line of your response should be of the form \"ANSWER: [ANSWER]\" (without quotes) where [ANSWER] is the a single word answer or number to the problem.\n" }, { "image": "[BASE64_IMAGE: png, ~42.9KB]" } ] } ], "target": "14", "id": 0, "group_id": 0, "subset_key": "human_test" } ``` -------------------------------- ### Queries JSONL Example Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/embedding.html Example format for each line in the queries.jsonl file. ```json {"_id": "query1", "text": "气候变化的影响是什么？"} {"_id": "query2", "text": "今天股市上涨的原因是什么？"} {"_id": "query3", "text": "人工智能如何改变行业？"} {"_id": "query4", "text": "可再生能源有哪些进展？"} {"_id": "query5", "text": "均衡饮食如何改善心理健康？"} {"_id": "query6", "text": "虚拟现实创造了哪些新机会？"} {"_id": "query7", "text": "为什么电动汽车越来越受欢迎？"} {"_id": "query8", "text": "太空探索任务揭示了哪些新信息？"} {"_id": "query9", "text": "区块链技术在加密货币之外有哪些应用？"} {"_id": "query10", "text": "远程工作的好处是什么？"} ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/air_bench_chat.html This is a sample JSON object representing an example from the speech_QA subset. ```json { "input": [ { "id": "5781ee73", "content": [ { "audio": "/root/.cache/modelscope/hub/datasets/evalscope/AIR-Bench-Dataset/Chat/speech_QA_iemocap/Ses01F_script01_1_M025.wav", "format": "wav" }, { "text": "Who is the speaker addressing at the end of the speech?" } ] } ], "target": "The speaker is addressing Mom at the end of the speech.", "id": 0, "group_id": 0, "subset_key": "speech_QA", "metadata": { "uniq_id": 400, "task_name": "speech_QA", "dataset_name": "iemocap", "category": "speech", "meta_info": "{'emotion': 'neutral', 'gender': 'male', 'transcription': \"And then we'll thrash it out with father. Okay Mom? Don't avoid me.\"}", "question": "Who is the speaker addressing at the end of the speech?" } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/drop.html This is a sample example from the DROP benchmark, illustrating the input passage, question, and expected target answer format. ```json { "input": [ { "id": "d4ab7ff6", "content": "You will be asked to read a passage and answer a question. Some examples of passages and Q&A are provided below.\n\n# Examples\n---\nPassage: Trunajaya rebellion or Trunajaya War was the ultimately unsuccessful rebellion waged by the Madurese pr ... [TRUNCATED] ... iled a 40-yard field goal, yet the Raiders' defense would shut down any possible attempt.\nQuestion: Who scored the first touchdown of the game?\n\nThink step by step, then write a line of the form \"Answer: [ANSWER]" at the end of your response." } ], "target": "[('Chaz Schilens',), ('JaMarcus Russell',)]", "id": 0, "group_id": 0, "metadata": { "passage": " Hoping to rebound from their loss to the Patriots, the Raiders stayed at home for a Week 16 duel with the Houston Texans. Oakland would get the early lead in the first quarter as quarterback JaMarcus Russell completed a 20-yard touchdown pa ... [TRUNCATED] ... 29-yard touchdown pass from Russell, followed up by an 80-yard punt return for a touchdown. The Texans tried to rally in the fourth quarter as Brown nailed a 40-yard field goal, yet the Raiders' defense would shut down any possible attempt.", "answer": { "number": "", "date": { "day": "", "month": "", "year": "" }, "spans": [ "Chaz Schilens" ], "worker_id": "", "hit_id": "" }, "validated_answers": { "number": [ "", "" ], "date": [ { "day": "", "month": "", "year": "" }, { "day": "", "month": "", "year": "" } ], "spans": [ [ "Chaz Schilens" ], [ "JaMarcus Russell" ] ], "worker_id": [ "", "" ], "hit_id": [ "", "" ] } } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/eq_bench.html This is a sample example of the EQ-Bench dataset, showing the input dialogue, target emotion scores, and metadata. ```json { "input": [ { "id": "97129bc9", "content": "Your task is to predict the likely emotional responses of a character in this dialogue:\n\nRobert: Claudia, you've always been the idealist. But let's be practical for once, shall we?\n Claudia: Practicality, according to you, means bulldozing ev ... [TRUNCATED] ... ary:\n\nRemorseful: \nIndifferent: \nAffectionate: \nAnnoyed: \n\n\n[End of answer]\n\nRemember: zero is a valid score, meaning they are likely not feeling that emotion. You must score at least one emotion > 0.\n\nYour answer:" } ], "target": "{'emotion1': 'Remorseful', 'emotion2': 'Indifferent', 'emotion3': 'Affectionate', 'emotion4': 'Annoyed', 'emotion1_score': 2, 'emotion2_score': 3, 'emotion3_score': 0, 'emotion4_score': 5}", "id": 0, "group_id": 0, "metadata": { "reference_answer": { "emotion1": "Remorseful", "emotion2": "Indifferent", "emotion3": "Affectionate", "emotion4": "Annoyed", "emotion1_score": 2, "emotion2_score": 3, "emotion3_score": 0, "emotion4_score": 5 }, "reference_answer_fullscale": { "emotion1": "Remorseful", "emotion2": "Indifferent", "emotion3": "Affectionate", "emotion4": "Annoyed", "emotion1_score": 0, "emotion2_score": "6", "emotion3_score": 0, "emotion4_score": "7" } } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/a_okvqa.html This is a sample example of the A-OKVQA dataset, showing the input format, choices, target answer, and metadata. ```json { "input": [ { "id": "3d2b0351", "content": [ { "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nWhat is in the motorcyclist\'s mouth?\n\nA) toothpick\nB) food\nC) popsicle stick\nD) cigarette" }, { "image": "[BASE64_IMAGE: jpeg, ~53.4KB]" } ] } ], "choices": [ "toothpick", "food", " popsicle stick", "cigarette" ], "target": "D", "id": 0, "group_id": 0, "metadata": { "question_id": "22jbM6gDxdaMaunuzgrsBB", "direct_answers": "['cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette', 'cigarette']", "difficult_direct_answer": false, "rationales": [ "He\'s smoking while riding.", "The motorcyclist has a lit cigarette in his mouth while he rides on the street.", "The man is smoking." ] } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/competition_math.html This snippet shows an example of a problem from the 'Level 1' subset, including the input, target answer, and metadata with reasoning. ```json { "input": [ { "id": "3848cf6a", "content": "Here are some examples of how to solve similar problems:\n\nProblem:\nWhen Joyce counts the pennies in her bank by fives, she has one left over. When she counts them by threes, there are two left over. What is the least possible number of pennie ... [TRUNCATED] ... of 12, how many eggs will be left over if all cartons are sold?\nSolution:\n4\nProblem:\nHow many of the six integers 1 through 6 are divisors of the four-digit number 1452?\n\nPlease reason step by step, and put your final answer within \boxed{}." } ], "target": "5", "id": 0, "group_id": 0, "subset_key": "Level 1", "metadata": { "reasoning": "All numbers are divisible by $1$. The last two digits, $52$, form a multiple of 4, so the number is divisible by $4$, and thus $2$. $1+4+5+2=12$, which is a multiple of $3$, so $1452$ is divisible by $3$. Since it is divisible by $2$ and $3$, it is divisible by $6$. But it is not divisible by $5$ as it does not end in $5$ or $0$. So the total is $\boxed{5}$.", "type": "Number Theory" } } ``` -------------------------------- ### Install VLMEvalKit Dependency Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/vlm.html Command to install EvalScope with VLMEvalKit support, for legacy formats. ```bash pip install evalscope[vlmeval] ``` -------------------------------- ### Example Sample Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/coin_flip.html This is an example of a single data point from the CoinFlip dataset, showing the input question and the expected target output. ```json { "input": [ { "id": "05503706", "content": [ { "text": "\nSolve the following coin flip problem step by step. The last line of your response should be of the form \"ANSWER: [ANSWER]\" (without quotes) where [ANSWER] is the answer to the problem.\n\nQ: A coin is heads up. rushawn flips the coin. yerania ... [TRUNCATED] ... the coin. jostin does not flip the coin. Is the coin still heads up?\n\nRemember to put your answer on its own line at the end in the form \"ANSWER: [ANSWER]\" (without quotes) where [ANSWER] is the answer YES or NO to the problem.\n\nReasoning:\n" } ] } ], "target": "NO", "id": 0, "group_id": 0, "metadata": { "answer": "NO" } } Copy code ``` -------------------------------- ### Example for the 'simple' subset Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/bfcl_v3.html This example demonstrates a simple function call scenario within the BFCL-v3 benchmark, showing the input query, the target function call with parameters, and metadata related to the test case. ```json { "input": [ { "id": "15e84989", "content": "[[{\"role\": \"user\", \"content\": \"Find the area of a triangle with a base of 10 units and height of 5 units.\"}]]" } ], "target": "[{\"calculate_triangle_area\": {\"base\": [10], \"height\": [5], \"unit\": [\"units\", \"\"], \"unit\": [\"units\", \"\"]}}]", "id": 0, "group_id": 0, "subset_key": "simple", "metadata": { "id": "simple_0", "multi_turn": false, "functions": [ { "name": "calculate_triangle_area", "description": "Calculate the area of a triangle given its base and height.", "parameters": { "type": "dict", "properties": { "base": { "type": "integer", "description": "The base of the triangle." }, "height": { "type": "integer", "description": "The height of the triangle." }, "unit": { "type": "string", "description": "The unit of measure (defaults to 'units' if not specified)" } }, "required": [ "base", "height" ] } } ], "tools": [ { "type": "function", "function": { "name": "calculate_triangle_area", "description": "Calculate the area of a triangle given its base and height. Note that the provided function is in Python 3 syntax.", "parameters": { "type": "object", "properties": { "base": { "type": "integer", "description": "The base of the triangle." }, "height": { "type": "integer", "description": "The height of the triangle." }, "unit": { "type": "string", "description": "The unit of measure (defaults to 'units' if not specified)" } }, "required": [ "base", "height" ] } } } ], "missed_functions": "{}", "initial_config": {}, "involved_classes": [], "turns": [ [ { "role": "user", "content": "Find the area of a triangle with a base of 10 units and height of 5 units." } ] ], "language": "Python", "test_category": "simple", "subset": "simple", "ground_truth": [ { "calculate_triangle_area": { "base": [ 10 ], "height": [ 5 ], "unit": [ "units", "" ] } } ], "should_execute_tool_calls": false, "missing_functions": {}, "is_fc_model": true } } ``` -------------------------------- ### Example Sample Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/aime24.html An example of a problem from the AIME-2024 benchmark, including the input question and the expected integer answer in a boxed format. ```json { "input": [ { "id": "32187d6f", "content": "\nSolve the following math problem step by step. Put your answer inside \\boxed{}.\n\nEvery morning Aya goes for a $9$-kilometer-long walk and stops at a coffee shop afterwards. When she walks at a constant speed of $s$ kilometers per hour, the w ... [TRUNCATED 164 chars] ... g $t$ minutes spent in the coffee shop. Suppose Aya walks at $s+\\frac{1}{2}$ kilometers per hour. Find the number of minutes the walk takes her, including the $t$ minutes spent in the coffee shop.\n\nRemember to put your answer inside \\boxed{}." } ], "target": "\\boxed{204}", "id": 0, "group_id": 0 } ``` -------------------------------- ### Example Sample Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/ceval.html This snippet shows an example of a question from the 'computer_network' subset, including the input, choices, target answer, and metadata. ```json { "input": [ { "id": "73073a35", "content": "以下是一些示例问题：\n\n问题：下列设备属于资源子网的是____。\n选项：\nA. 计算机软件\nB. 网桥\nC. 交换机\nD. 路由器\n解析：1. 首先，资源子网是指提供共享资源的网络，如打印机、文件服务器等。\r\n2. 其次，我们需要了解选项中设备的功能。网桥、交换机和路由器的主要功能是实现不同网络之间的通信和数据传输，是通信子网设备。而计算机软件可以提供共享资源的功能。\n答案：A\n\n问题：滑动窗口的作用是____。\n选项：\nA. 流量控制\nB. 拥塞控制\nC. 路由控制\nD. 差错 ... [TRUNCATED] ... Mbps，所以答案为min{80Mbps, 100Mbps}=80Mbps，选C。\n答案：C\n\n\n以下是中国关于计算机网络的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]"（不带引号），其中 [LETTER] 是 A、B、C、D 中的一个。\n\n问题：使用位填充方法，以01111110为位首flag，数据为011011111111111111110010，求问传送时要添加几个0____\n选项：\nA. 1\nB. 2\nC. 3\nD. 4\n" } ], "choices": [ "1", "2", "3", "4" ], "target": "C", "id": 0, "group_id": 0, "metadata": { "id": 0, "explanation": "", "subject": "computer_network" } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/anat_em.html This is a sample example of a JSON object representing an input and its target output for named entity recognition. ```json { "input": [ { "id": "64b20a23", "content": "Here are some examples of named entity recognition:\n\nInput:\nImmunostaining and confocal analysis\n\nOutput:\nImmunostaining and confocal analysis\n\nInput:\nDNA labelling and staining with 5 - bromo - 2 ' - deoxyuridine ( BrdU ... [TRUNCATED] ... Do not include explanations, just the tagged text.\n6. If entity spans overlap, choose the most specific entity type.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\n( a ) Schematic drawing of the magnetic tweezers ." } ], "target": "( a ) Schematic drawing of the magnetic tweezers .", "id": 0, "group_id": 0, "metadata": { "tokens": [ "(", "a", ")", "Schematic", "drawing", "of", "the", "magnetic", "tweezers", "." ], "ner_tags": [ "O", "O", "O", "O", "O", "O", "O", "O", "O", "O" ] } } ``` -------------------------------- ### Sample Example Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/ai2d.html This is a sample example of the AI2D dataset, showing the input format with text and image, the choices, and the target answer. ```json { "input": [ { "id": "789e28fa", "content": [ { "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nwhich of these define dairy item\n\nA) c\nB) D\nC) b\nD) a" }, { "image": "[BASE64_IMAGE: png, ~226.2KB]" } ] } ], "choices": [ "c", "D", "b", "a" ], "target": "B", "id": 0, "group_id": 0 } ``` -------------------------------- ### Example Sample for Art_Style Subset Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/blink.html This is an example of a data sample from the 'Art_Style' subset, illustrating the input format with text and base64 encoded images, the choices, and the target answer. ```json { "input": [ { "id": "a522940e", "content": [ { "text": "Answer the following multiple choice question. The last line of your response should be of the following format:\n'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B.\n\nSome most common art painting styles include Realism, Impressi ... [TRUNCATED] ... of art paintings, use the first image as the reference image, and determine which one of the second or the third image shares the same style as the reference image?\nSelect from the following choices.\n(A) the second image\n(B) the third image\n" }, { "image": "[BASE64_IMAGE: jpeg, ~477.8KB]" }, { "image": "[BASE64_IMAGE: jpeg, ~876.1KB]" }, { "image": "[BASE64_IMAGE: jpeg, ~329.2KB]" } ] } ], "choices": [ "the second image", "the third image" ], "target": "A", "id": 0, "group_id": 0 } Copy code ``` -------------------------------- ### GSM8K Adapter Implementation Example Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/add_benchmark.html Example of registering a Benchmark and implementing the GSM8KAdapter class for a math reasoning task. ```python from typing import Any, Dict from evalscope.api.benchmark import BenchmarkMeta, DefaultDataAdapter from evalscope.api.dataset import Sample from evalscope.api.evaluator import TaskState from evalscope.api.registry import register_benchmark from evalscope.constants import Tags # 定义提示模板 PROMPT_TEMPLATE = """ Solve the following math problem step by step. The last line of your response should be of the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem. {question} Remember to put your answer on its own line at the end in the form "ANSWER: $ANSWER" (without quotes) where $ANSWER is the answer to the problem, and you do not need to use a \boxed command. Reasoning: """.lstrip() ``` -------------------------------- ### Custom Local Dataset Example (example.jsonl) Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/llm.html A simple example of a custom local dataset with three samples in JSONL format. ```json {"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"请把 2 和 3 相加"}],"tools":[{"type":"function","function":{"name":"add","description":"将两个数字相加","parameters":{"type":"object","properties":{"a":{"type":"number","description":"第一个数字"},"b":{"type":"number","description":"第二个数字"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":true} {"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"今天天气不错，我们聊聊天"}],"tools":[{"type":"function","function":{"name":"add","description":"将两个数字相加","parameters":{"type":"object","properties":{"a":{"type":"number","description":"第一个数字"},"b":{"type":"number","description":"第二个数字"}},"required":["a","b"],"additionalProperties":false}}}],"should_call_tool":false} {"messages":[{"role":"system","content":"你是助手"},{"role":"user","content":"把 37 摄氏度转换为华氏度"}],"tools":[{"type":"function","function":{"name":"convert_temperature","description":"将摄氏度转换为华氏度","parameters":{"type":"object","properties":{"celsius":{"type":"number","description":"摄氏温度值"}},"required":["celsius"],"additionalProperties":false}}}],"should_call_tool":true} ``` -------------------------------- ### Example Data Format Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/bc5cdr.html This JSON structure shows an example of the input and target format for the BC5CDR dataset, including entity tagging. ```json { "input": [ { "id": "154a621d", "content": "Here are some examples of named entity recognition:\n\nInput:\nSelegiline - induced postural hypotension in Parkinson ' s disease : a longitudinal study on the effects of drug withdrawal .\n\nOutput:\nSelegiline - ind ... [TRUNCATED] ... e.\n7. Ensure every opening tag has a matching closing tag.\n\nText to process:\nTorsade de pointes ventricular tachycardia during low dose intermittent dobutamine treatment in a patient with dilated cardiomyopathy and congestive heart failure .\n" } ], "target": "Torsade de pointes ventricular tachycardia during low dose intermittent dobutamine treatment in a patient with dilated cardiomyopathy and congestive heart failure .", "id": 0, "group_id": 0, "metadata": { "tokens": [ "Torsade", "de", "pointes", "ventricular", "tachycardia", "during", "low", "dose", "intermittent", "dobutamine", "treatment", "in", "a", "patient", "with", "dilated", "cardiomyopathy", "and", "congestive", "heart", "failure", "." ], "ner_tags": [ "B-DISEASE", "I-DISEASE", "I-DISEASE", "I-DISEASE", "I-DISEASE", "O", "O", "O", "O", "B-CHEMICAL", "O", "O", "O", "O", "O", "B-DISEASE", "I-DISEASE", "O", "B-DISEASE", "I-DISEASE", "I-DISEASE", "O" ] } } ``` -------------------------------- ### Example for agronomy subset Source: https://evalscope.readthedocs.io/zh-cn/latest/benchmarks/cmmlu.html This snippet shows an example of a question from the agronomy subset, including the input, choices, and the correct target answer. ```json { "input": [ { "id": "4e04de48", "content": "回答下面的单项选择题，请选出其中的正确答案。你的回答的最后一行应该是这样的格式：\"答案：[LETTER]\"（不带引号），其中 [LETTER] 是 A,B,C,D 中的一个。请在回答前进行一步步思考。\n\n问题：在农业生产中被当作极其重要的劳动对象发挥作用，最主要的不可替代的基本生产资料是\n选项：\nA) 农业生产工具\nB) 土地\nC) 劳动力\nD) 资金\n" } ], "choices": [ "农业生产工具", "土地", "劳动力", "资金" ], "target": "B", "id": 0, "group_id": 0, "subset_key": "agronomy", "metadata": { "subject": "agronomy" } } Copy code ``` -------------------------------- ### General-VMCQ JSONL Example Source: https://evalscope.readthedocs.io/zh-cn/latest/advanced_guides/custom_dataset/vlm.html Example of a JSONL file for the General-VMCQ dataset, demonstrating how to include text, image placeholders, and media file paths. ```json {"question": "Which image shows a dog?", "options": ["", "", "", ""], "image_1": "custom_eval/multimodal/images/dog.jpg", "image_2": "custom_eval/multimodal/images/AMNH.jpg", "image_3": "custom_eval/multimodal/images/tesla.jpg", "image_4": "custom_eval/multimodal/images/tokyo.jpg", "answer": "A"} {"question": " What building is this?", "options": ["School", "Hospital", "Park", "Museum"], "image_1": "custom_eval/multimodal/images/AMNH.jpg", "answer": "D"} {"question": "