### Install Unitxt for LM-Eval Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/unitxt/README.md To use Unitxt datasets with the LM-eval harness, you need to install the unitxt library first. This command installs the library using pip. ```bash pip install unitxt ``` -------------------------------- ### Install RWKV-X Package Source: https://github.com/howard-hou/rwkv-x/blob/main/README.md Installs the RWKV-X library either from PyPI for easy access or from the source code for development purposes. ```Bash # Install from PyPI pip install rwkv-x # Or install from source cd RWKV-X/package pip install -e . ``` -------------------------------- ### Install lm-eval Package Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/leaderboard/README.md Installs the 'lm-eval' package with specific support for leaderboard evaluations, including dependencies for math, ifeval, and sentencepiece. This involves cloning the repository and then installing the package in editable mode. ```Bash git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness cd lm-evaluation-harness pip install -e ".[math,ifeval,sentencepiece]" ``` -------------------------------- ### Install tinyBenchmarks Package Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/tinyBenchmarks/README.md Installs the tinyBenchmarks package from its GitHub repository using pip. This package is essential for applying post-processing to evaluation results and obtaining accurate scores. ```sh pip install git+https://github.com/felipemaiapolo/tinyBenchmarks ``` -------------------------------- ### Install jsonschema Library Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/jsonschema_bench/README.md Installs the 'jsonschema' library with the 'format' extra, which is necessary for supporting the 'format' keyword in JSON Schema used in the tasks. ```bash pip install jsonschema[format] ``` -------------------------------- ### Install Math Dependencies Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/minerva_math/README.md Installs the necessary dependencies for evaluating mathematical tasks, including those used by the Minerva model and for exact match equivalence calculations with the sympy library. ```shell pip install lm-eval[math] ``` -------------------------------- ### RWKV-X Inference Example Source: https://github.com/howard-hou/rwkv-x/blob/main/README.md Demonstrates how to load a RWKV-X model and generate text based on a given prompt. It includes setting environment variables, model loading, and defining generation parameters. ```Python # !!! set these before import RWKV !!! import os os.environ["RWKV_CUDA_ON"] = '0' # '1' to compile CUDA kernel (10x faster), requires c++ compiler & cuda libraries from rwkv_x.model import RWKV_X from rwkv_x.utils import PIPELINE, PIPELINE_ARGS # Load model # You can get the model weights from: https://huggingface.co/howard-hou/RWKV-X/ model = RWKV_X(model_path='RWKV-X-0.2B-64k-Base.pth', strategy='cuda fp16') pipeline = PIPELINE(model) # Set generation parameters args = PIPELINE_ARGS( temperature=1.0, top_p=0.7, top_k=100, alpha_frequency=0.25, alpha_presence=0.25, token_ban=[], token_stop=[], chunk_len=256 ) # Generate text ctx = "This is a sample prompt." output = pipeline.generate(ctx, token_count=200, args=args) print(output) ``` -------------------------------- ### Checklist for Adding Novel Benchmarks/Datasets Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/bigbench/README.md This checklist guides users on the requirements for adding new benchmarks or datasets to the library. It emphasizes referencing original papers, checking against reference implementations, and clearly denoting main variants. ```Markdown [ ] Is the task an existing benchmark in the literature? [ ] Have you referenced the original paper that introduced the task? [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test? If other tasks on this dataset are already supported: * [ ] Is the "Main" variant of this task clearly denoted? * [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates? * [ ] Have you noted which, if any, published evaluation setups are matched by this variant? ``` -------------------------------- ### Running Paloma Benchmark with HuggingFace Model Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/paloma/README.md This example demonstrates how to run the Paloma benchmark, specifically the 'paloma_dolma_100_programing_languages' task, using a HuggingFace model. It highlights the necessity of passing `logits_cache=False` to the `--model_args` for correct execution. ```Shell lm_eval --model hf --model_args pretrained=EleutherAI/pythia-160m,logits_cache=False --tasks paloma ``` -------------------------------- ### Run JSONSchema Bench Task Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/jsonschema_bench/README.md Example command to run 10 instances of the 'jsonschema_bench_medium' task using the 'lm_eval' tool. It specifies the model, generation parameters, task, batch size, limit, and chat template application. ```bash lm_eval \ --model hf --gen_kwargs max_new_tokens=1024 \ --model_args pretrained=meta-llama/Llama-3.2-1B-Instruct,parallelize=True\ --tasks jsonschema_bench_medium \ --batch_size auto \ --limit 10 \ --apply_chat_template \ --fewshot_as_multiturn ``` -------------------------------- ### Evaluate LLM Performance with tinyBenchmarks Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/tinyBenchmarks/README.md Demonstrates how to use the lm_eval framework to evaluate LLM performance on a specific tiny benchmark (tinyHellaswag). It includes arguments for specifying the model, tasks, batch size, output path, and logging samples for detailed analysis. ```bash lm_eval --model hf \ --model_args pretrained="mistralai/Mistral-7B-Instruct-v0.2" \ --tasks tinyHellaswag \ --batch_size 4 \ --output_path '' \ --log_samples ``` -------------------------------- ### Evaluate FrenchBench with Hugging Face Models (GPT-2, Llama-2) Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/french_bench/README.md This section provides examples for evaluating FrenchBench using Hugging Face models via the `lm_eval` tool. It includes configurations for GPT-2 and Llama-2 7B, specifying tasks, device (e.g., cuda:0), few-shot settings, batch size, and output paths. ```Bash lm_eval --model hf --model_args pretrained=gpt2 --tasks french_bench --device cuda:0 --limit 100 --num_fewshot 3 --batch_size 8 --output_path data/french_bench/gpt2/results_french_bench_3shot.json ``` ```Bash lm_eval --model hf --model_args pretrained=gpt2 --tasks french_bench_opus_perplexity,crows_pairs_french --device cuda:0 --limit 100 --batch_size auto --output_path data/french_bench/gpt2/results_french_bench2_0shot.json ``` ```Bash lm_eval --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf --tasks french_bench --device cuda:0 --limit 100 --num_fewshot 3 --batch_size 4 --output_path data/french_bench/llama-2-7b-hf/results_french_bench_3shot.json ``` ```Bash lm_eval --model hf --model_args pretrained=meta-llama/Llama-2-7b-hf --tasks french_bench_opus_perplexity,crows_pairs_french --device cuda:0 --limit 100 --batch_size auto --output_path data/french_bench/llama-2-7b-hf/results_french_bench2_0shot.json ``` -------------------------------- ### RWKV-X Model Loading and Generation Source: https://github.com/howard-hou/rwkv-x/blob/main/package/README.md Loads the RWKV-X model with a specified path and strategy (e.g., 'cpu fp32'). Initializes a PIPELINE for text generation and demonstrates generating text with custom arguments like temperature, top_p, and frequency/presence penalties. ```Python # download models: https://huggingface.co/howard-hou/RWKV-X/tree/main model = RWKV_X(model_path='RWKV-X-0.2B-64k-Base.pth', strategy='cpu fp32') pipeline = PIPELINE(model) # for "world" models ctx = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese." print(ctx, end='') def my_print(s): print(s, end='', flush=True) # For alpha_frequency and alpha_presence, see "Frequency and presence penalties": # https://platform.openai.com/docs/api-reference/parameter-details args = PIPELINE_ARGS(temperature = 1.0, top_p = 0.7, top_k = 100, # top_k = 0 then ignore alpha_frequency = 0.25, alpha_presence = 0.25, alpha_decay = 0.996, # gradually decay the penalty token_ban = [], # ban the generation of some tokens token_stop = [], # stop generation whenever you see any token here chunk_len = 256) # split input into chunks to save VRAM (shorter -> slower) pipeline.generate(ctx, token_count=200, args=args, callback=my_print) print('\n') ``` -------------------------------- ### GSM8k Calculator Implementation Example Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/gsm8k/README.md This snippet shows an example implementation of a calculator for the GSM8k dataset, as referenced in the official documentation. It is intended to be used with language model generation functions to handle mathematical operations within the word problems. ```Python from grade_school_math.calculator import Calculator calculator = Calculator() result = calculator.evaluate("What is 5 + 3?") print(result) ``` -------------------------------- ### Run KorMedMCQA Dentist Benchmark Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/kormedmcqa/README.md This snippet provides an example of running the KorMedMCQA benchmark for the Korean Dentist Examination. It is part of a suite of tools for evaluating language models on specialized datasets. ```Python from lm_harness.tasks.kormedmcqa import kormedmcqa_dentist # Example usage (assuming necessary setup and data loading) # result = kormedmcqa_dentist.run_evaluation() # print(result) ``` -------------------------------- ### Winogrande Benchmark Tasks Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/metabench/README.md Offers subsets of the Winogrande benchmark, featuring original, permuted answer key, and secondary reduced versions. ```text metabench_winogrande metabench_winogrande_permute metabench_winogrande_secondary ``` -------------------------------- ### Qwen2-VL-2B MMMU Benchmark Configuration Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/mmmu/README.md Configuration for running the MMMU benchmark with the Qwen2-VL-2B model. It specifies the pretrained model, attention implementation, data type, and image format conversion. The output shows accuracy scores across various MMMU categories. ```text hf-multimodal (pretrained=Qwen/Qwen2-VL-2B-Instruct,attn_implementation=flash_attention_2,dtype=bfloat16,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2 | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |--------------------------------|------:|------|------|------|---|-----:|---|-----:| |mmmu_val | 0|none | |acc |↑ |0.3778|± |0.0155| | - Art and Design | 0|none | |acc |↑ |0.5500|± |0.0415| | - Business | 0|none | |acc |↑ |0.3600|± |0.0389| | - Health and Medicine | 0|none | |acc |↑ |0.3667|± |0.0394| | - Humanities and Social Science| 0|none | |acc |↑ |0.5167|± |0.0438| | - Science | 0|none | |acc |↑ |0.2467|± |0.0352| | - Tech and Engineering | 0|none | |acc |↑ |0.3143|± |0.0317| Author-reported score: 41.1% ``` -------------------------------- ### Set Max Length for RULER Model Arguments Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/ruler/README.md This example shows how to set the `max_length` parameter within the `model_args` to prevent truncation of longer sequences when using RULER. It specifies a `pretrained` model and a `max_length` of 32768. ```bash --model_args=pretrained=...,max_length=32768 ``` -------------------------------- ### Qwen2-VL-7B MMMU Benchmark Configuration Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/mmmu/README.md Configuration for running the MMMU benchmark with the Qwen2-VL-7B model. It specifies the pretrained model, attention implementation, data type, and image format conversion. The output shows accuracy scores across various MMMU categories. ```text hf-multimodal (pretrained=Qwen/Qwen2-VL-7B-Instruct,attn_implementation=flash_attention_2,dtype=bfloat16,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 2 | Groups |Version|Filter|n-shot|Metric| |Value | |Stderr| |--------------------------------|------:|------|------|------|---|-----:|---|-----:| |mmmu_val | 0|none | |acc |↑ |0.5056|± |0.0160| | - Art and Design | 0|none | |acc |↑ |0.6917|± |0.0398| | - Business | 0|none | |acc |↑ |0.4333|± |0.0406| | - Health and Medicine | 0|none | |acc |↑ |0.5667|± |0.0401| | - Humanities and Social Science| 0|none | |acc |↑ |0.6750|± |0.0426| | - Science | 0|none | |acc |↑ |0.3800|± |0.0392| | - Tech and Engineering | 0|none | |acc |↑ |0.4000|± |0.0341| Author-reported score: 54.1% ``` -------------------------------- ### Benchmark Addition Checklist Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/japanese_leaderboard/README.md This section outlines a checklist for adding new benchmarks or datasets to the library. It emphasizes verifying if the task is an existing benchmark, referencing the original paper, and checking for reference implementations. ```markdown For adding novel benchmarks/datasets to the library: * [x] Is the task an existing benchmark in the literature? * [x] Have you referenced the original paper that introduced the task? * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test? If other tasks on this dataset are already supported: * [ ] Is the "Main" variant of this task clearly denoted? * [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates? * [ ] Have you noted which, if any, published evaluation setups are matched by this variant? ``` -------------------------------- ### MMLU-Pro Evaluation Script Reference Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/mmlu_pro/README.md References a specific line in the MMLU-Pro GitHub repository's evaluation script, indicating a change related to adding a newline to task descriptions. This is relevant for users who need to ensure their local setup matches the reference implementation. ```python https://github.com/TIGER-AI-Lab/MMLU-Pro/blob/47b9891aacb8bd7cda29d5c5ba17b9434dd333bc/evaluate_from_local.py#L93 ``` -------------------------------- ### Program Induction Citation Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/agieval/README.md This BibTeX entry cites a paper on program induction by rationale generation for solving algebraic word problems, a dataset from which is included in AGIEval. ```BibTeX @inproceedings{ling-etal-2017-program, title = "Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems", author = "Ling, Wang and Yogatama, Dani and Dyer, Chris and Blunsom, Phil", booktitle = "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)", month = jul, year = "2017", address = "Vancouver, Canada", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/P17-1015", doi = "10.18653/v1/P17-1015", pages = "158--167", abstract = "Solving algebraic word problems requires executing a series of arithmetic operations{---}a program{---}to obtain a final answer. However, since programs can be arbitrarily complicated, inducing them directly from question-answer pairs is a formidable challenge. To make this task more feasible, we solve these problems by generating answer rationales, sequences of natural language and human-readable mathematical expressions that derive the final answer through a series of small steps. Although rationales do not explicitly specify programs, they provide a scaffolding for their structure via intermediate milestones. To evaluate our approach, we have created a new 100,000-sample dataset of questions, answers and rationales. Experimental results show that indirect supervision of program learning via answer rationales is a promising strategy for inducing arithmetic programs." } ``` -------------------------------- ### Evaluate FrenchBench with OpenAI Models Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/french_bench/README.md This snippet demonstrates how to use the `lm_eval` tool to evaluate FrenchBench tasks using OpenAI's text-davinci-003 model. It shows examples for both general FrenchBench tasks and specific perplexity/exclusion tasks, with configurations for few-shot learning and zero-shot learning. ```Bash lm_eval --model openai-completions --model_args engine=text-davinci-003 --tasks french_bench --limit 100 --num_fewshot 3 --batch_size auto --output_path data/french_bench/davinci-003/results_french_bench_3shot.json ``` ```Bash lm_eval --model openai-completions --model_args engine=text-davinci-003 --tasks french_bench_opus_perplexity,crows_pairs_french --limit 100 --batch_size auto --output_path data/french_bench/davinci-003/results_french_bench2_0shot.json ``` -------------------------------- ### HellaSwag Benchmark Tasks Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/metabench/README.md Offers subsets of the HellaSwag benchmark, featuring original, permuted answer key, and secondary reduced versions. ```text metabench_hellaswag metabench_hellaswag_permute metabench_hellaswag_secondary metabench_hellaswag_secondary_permute ``` -------------------------------- ### Train RWKV Model Source: https://github.com/howard-hou/rwkv-x/blob/main/sft/README.md This script demonstrates how to train an RWKV model using the `train.py` script. It includes arguments for loading a pre-trained model, specifying data files, setting training epochs and batch sizes, and configuring optimization parameters. ```bash python train.py --load_model ../RWKV-x070-World-1.5B-v3-20250127-ctx4096.pth \ --wandb "rwkv1b5-sft" --proj_dir out/rwkv1b5-sft \ --data_file ../output.json \ --data_type "json" --vocab_size 65536 \ --ctx_len 2048 --epoch_steps 1000 --epoch_count 10 --epoch_begin 0 --epoch_save 1 \ --micro_bsz 32 --accumulate_grad_batches 4 --n_layer 24 --n_embd 2048 --pre_ffn 0 \ --lr_init 6e-5 --lr_final 1.5e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.99 --adam_eps 1e-8 \ --accelerator gpu --devices 2 --precision bf16 --strategy deepspeed_stage_1 --grad_cp 1 \ --num_layers_to_freeze 0 \ --enable_progress_bar True ``` -------------------------------- ### Process and Evaluate LLM Outputs with tinyBenchmarks Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/tinyBenchmarks/README.md Provides a Python script to load and process evaluation results from a JSONL file generated by lm_eval. It extracts specific metrics (e.g., 'acc_norm'), sorts the outputs, and then uses the tinyBenchmarks package to evaluate the LLM's performance based on these processed results. ```python import json import tinyBenchmarks as tb import numpy as np # Choose benchmark (e.g. hellaswag) benchmark = 'hellaswag' # possible benchmarks: # ['mmlu','truthfulqa', 'gsm8k', # 'winogrande', 'arc', 'hellaswag'] # Get score vector from output-file (the metric [here `acc_norm`] depends on the benchmark) file_path = '/' with open(file_path, 'r') as file: outputs = json.load(file) # Ensuring correct order of outputs outputs = sorted(outputs, key=lambda x: x['doc_id']) y = np.array([float(item['acc_norm']) for item in outputs]) ### Evaluation tb.evaluate(y, benchmark) ``` -------------------------------- ### MMLU Benchmark Tasks Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/metabench/README.md Provides subsets of the MMLU benchmark, including original, permuted answer key, and secondary reduced versions. ```text metabench_mmlu metabench_mmlu_permute metabench_mmlu_secondary ``` -------------------------------- ### MMLU-Pro+ Benchmark Citation Source: https://github.com/howard-hou/rwkv-x/blob/main/evaluation/lm_eval/tasks/mmlu-pro-plus/README.md This snippet provides the BibTeX citation for the MMLU-Pro+ paper, which introduces an enhanced benchmark for evaluating higher-order reasoning and shortcut learning in LLMs. ```BibTeX @article{taghanaki2024mmlu, title={MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning in LLMs}, author={Taghanaki, Saeid Asgari and Khani, Aliasgahr and Khasahmadi, Amir}, journal={arXiv preprint arXiv:2409.02257}, year={2024} } ```