### Install BenchBuilder Dependencies Source: https://github.com/lmarena/arena-hard-auto/blob/main/BenchBuilder/README.md Installs the necessary dependencies for the BenchBuilder project by running pip install with the requirements.txt file. This is typically done after navigating to the BenchBuilder directory. ```bash cd BenchBuilder pip install -r requirements.txt ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Clones the Arena Hard Auto project repository and installs necessary dependencies, including optional ones like the Anthropic SDK. ```bash git clone https://github.com/lmarena/arena-hard-auto.git cd arena-hard pip install -r requirements.txt pip install -r requirements-optional.txt # Optional dependencies (e.g., anthropic sdk) ``` -------------------------------- ### Download Arena Hard Dataset Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Downloads the Arena Hard dataset using git-lfs and copies the data into the project's data directory for use with the evaluation scripts. ```bash > git lfs install > git clone git@hf.co:datasets/lmarena-ai/arena-hard-auto arena-hard-data // copy answers/judgments to the data directory > cp -r arena-hard-data/data . ``` -------------------------------- ### Benchmark Viewer Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Launches a Gradio-based script to review answers and judgment results from the benchmark. Requires the 'gradio' library. ```console python qa_browser.py --share ``` -------------------------------- ### Label Dataset with Configuration Source: https://github.com/lmarena/arena-hard-auto/blob/main/BenchBuilder/README.md Labels a dataset using criteria defined in a configuration file (config.yaml). Ensure the config.yaml file is correctly set up before execution. ```python python label.py --config config.yaml ``` -------------------------------- ### Style Control with show_result.py Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Demonstrates how to apply style control features when running the `show_result.py` script. Users can control for specific features like markdown density or token length. ```console python show_result.py -f markdown length # style control ``` ```console python show_result.py -f markdown # control for markdown density only ``` ```console python show_result.py -f length # length control only ``` -------------------------------- ### Configure Judge Model for Judgments Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Demonstrates how to configure the judge model and its parameters (temperature, max_tokens) in `config/arena-hard-v2.0.yaml`, showing how to switch between GPT-4.1 and Gemini-2.5. ```yaml judge_model: gpt-4.1 temperature: 0.0 max_tokens: 16000 ``` ```yaml judge_model: gemini-2.5 temperature: 1.0 max_tokens: 32000 ``` -------------------------------- ### Run Arena Hard Evaluation Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Executes the main evaluation script to display model scores and confidence intervals based on the downloaded dataset. ```bash > python show_result.py ``` -------------------------------- ### Filter Prompts by Score and Cluster Thresholds Source: https://github.com/lmarena/arena-hard-auto/blob/main/BenchBuilder/README.md Filters prompts from a conversations file based on specified prompt and cluster quality thresholds. It requires both the conversations file and a clusters file as input. ```python python filter.py --conversations_file [your jsonl file] --clusters_file [your json file] --prompt_threshold 6 --cluster_threshold 3 ``` -------------------------------- ### Configure Bedrock Model in api_config.yaml Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Defines a new entry for a Bedrock-hosted model in the `config/api_config.yaml` file. It specifies the internal alias, Bedrock model identifier, API type, and generation parameters like parallel calls, max tokens, and temperature. ```yaml aws_nova_light_v1: model: aws_nova_light_v1 model_id: us.amazon.nova-lite-v1:0 endpoints: null api_type: aws_nova parallel: 8 max_tokens: 4096 temperature: 0.0 ``` -------------------------------- ### Topic Clustering for Dataset Source: https://github.com/lmarena/arena-hard-auto/blob/main/BenchBuilder/README.md Performs topic clustering on a given conversation file. The --min-topic-size argument specifies the minimum number of conversations required to form a topic cluster. ```python python topic_clustering.py --conv-file [your json file] --min-topic-size 8 ``` -------------------------------- ### Benchmark Results: Hard Prompt, Style Control, GPT-4.1 Judge Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md This section presents benchmark scores for language models evaluated with hard prompts and style control, using GPT-4.1 as the judge. This configuration is recommended for users who prefer the OpenAI API. Scores include percentages and confidence intervals. ```console Model Scores (%) CI (%) 0 o3-2025-04-16 87.0 (-1.0 / +1.0) 1 o4-mini-2025-04-16-high 81.7 (-1.2 / +1.2) 2 o4-mini-2025-04-16 78.0 (-1.3 / +1.4) 3 o3-mini-2025-01-31-high 64.8 (-2.1 / +1.9) 4 o1-2024-12-17-high 58.7 (-2.3 / +2.1) 5 gpt-4.1 58.3 (-2.0 / +2.3) 6 o1-2024-12-17 50.2 (-2.2 / +1.8) 7 o3-mini-2025-01-31 50.0 (-0.0 / +0.0) 8 gemini-2.5 49.1 (-2.5 / +2.4) 9 gpt-4.1-mini 48.6 (-2.7 / +1.9) 10 deepseek-r1 48.0 (-2.6 / +2.3) 11 claude-3-7-sonnet-20250219-thinking-16k 47.0 (-1.9 / +2.3) 12 Qwen3-235B-A22B 46.7 (-1.9 / +2.4) 13 gemini-2.5-flash 45.1 (-2.7 / +2.1) 14 gpt-4.5-preview 43.0 (-1.9 / +2.2) 15 QwQ-32B 36.1 (-2.0 / +2.2) 16 Qwen3-32B 35.8 (-2.1 / +2.2) 17 Qwen3-30B-A3B 28.7 (-1.4 / +2.1) 18 claude-3-5-sonnet-20241022 25.8 (-1.7 / +1.8) 19 s1.1-32B 18.3 (-2.3 / +2.2) 20 gpt-4.1-nano 15.4 (-1.1 / +1.2) 21 Athene-V2-Chat 12.6 (-1.2 / +1.3) 22 Qwen3-4B 12.6 (-1.1 / +1.5) 23 llama4-maverick-instruct-basic 12.0 (-1.0 / +1.2) 24 gemma-3-27b-it 9.7 (-0.9 / +1.1) 25 Qwen2.5-72B-Instruct 8.0 (-0.7 / +0.9) 26 Llama-3.1-Nemotron-70B-Instruct-HF 6.8 (-0.6 / +0.8) 27 OpenThinker2-32B 2.3 (-0.2 / +0.3) ``` -------------------------------- ### Benchmark Results: Hard Prompt, Style Control, Gemini-2.5 Judge Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md This section displays benchmark scores for various language models evaluated using hard prompts and style control, with Gemini-2.5 acting as the judge. The scores are presented as percentages with confidence intervals. ```console Model Scores (%) CI (%) 0 o3-2025-04-16 85.9 (-0.8 / +0.9) 1 o4-mini-2025-04-16-high 79.1 (-1.4 / +1.2) 2 gemini-2.5 79.0 (-2.1 / +1.8) 3 o4-mini-2025-04-16 74.6 (-1.8 / +1.6) 4 gemini-2.5-flash 68.6 (-1.6 / +1.6) 5 o3-mini-2025-01-31-high 66.1 (-1.5 / +2.1) 6 o1-2024-12-17-high 61.0 (-2.0 / +2.1) 7 claude-3-7-sonnet-20250219-thinking-16k 59.8 (-2.0 / +1.8) 8 Qwen3-235B-A22B 58.4 (-1.9 / +2.1) 9 deepseek-r1 58.0 (-2.2 / +2.0) 10 o1-2024-12-17 55.9 (-2.2 / +1.8) 11 gpt-4.5-preview 50.0 (-1.9 / +2.0) 12 o3-mini-2025-01-31 50.0 (-0.0 / +0.0) 13 gpt-4.1 50.0 (-1.9 / +1.7) 14 gpt-4.1-mini 46.9 (-2.4 / +2.1) 15 Qwen3-32B 44.5 (-2.2 / +2.1) 16 QwQ-32B 43.5 (-2.5 / +2.1) 17 Qwen3-30B-A3B 33.9 (-1.6 / +1.5) 18 claude-3-5-sonnet-20241022 33.0 (-2.3 / +1.8) 19 s1.1-32B 22.3 (-1.7 / +1.5) 20 llama4-maverick-instruct-basic 17.2 (-1.5 / +1.2) 21 Athene-V2-Chat 16.4 (-1.4 / +1.4) 22 gemma-3-27b-it 15.0 (-1.4 / +1.0) 23 Qwen3-4B 15.0 (-1.1 / +1.5) 24 gpt-4.1-nano 13.7 (-1.1 / +1.0) 25 Llama-3.1-Nemotron-70B-Instruct-HF 10.3 (-0.8 / +1.0) 26 Qwen2.5-72B-Instruct 10.1 (-0.9 / +1.3) 27 OpenThinker2-32B 3.2 (-0.3 / +0.3) ``` -------------------------------- ### Evaluate Benchmarks with Model Separability and Agreement Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md This section details the metrics and code used to evaluate benchmarks' ability to rank models, focusing on model separability and agreement with human preferences. It provides insights into how Arena-Hard-Auto was curated and how its performance is measured. ```Python from benchbuilder.evaluate import evaluate_benchmark # Example usage: benchmark_results = evaluate_benchmark( benchmark_path="path/to/your/benchmark", model_list=["model_a", "model_b"], human_preference_data="path/to/human_data.csv" ) print(benchmark_results.separability) print(benchmark_results.agreement) ``` -------------------------------- ### Configure API Endpoint for Models Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Sets up the API endpoint configuration in `config/api_config.yaml` to connect to various model serving engines like vLLM, SGLang, or cloud AI platforms. ```yaml ... # Add your model below for evaluation model_list: - deepseek-r1 - [YOUR-MODEL-NAME] ``` -------------------------------- ### Generate Model Answers Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Generates answers for specified models by adding their names to `config/gen_answer_config.yaml` and running the generation script. Includes caching to avoid re-generating existing answers. ```bash > python gen_answer.py ``` -------------------------------- ### Register Bedrock Model Handler in completion.py Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Registers a new model handler function in `utils/completion.py` using the `@register_api` decorator. This function defines how inputs are formatted, sent to Bedrock via boto3, and how the response is parsed. The `api_type` must match the one defined in `api_config.yaml`. ```python @register_api("aws_nova") handles Nova models # Example of a handler function: def handle_nova_model(prompt, model_id, **kwargs): # Format input for Bedrock body = create_nova_messages(prompt, **kwargs) # Invoke Bedrock model response = bedrock_client.invoke_model(modelId=model_id, body=body) # Parse response return parse_nova_response(response) ``` -------------------------------- ### Display Arena-Hard-v0.1 Leaderboard Source: https://github.com/lmarena/arena-hard-auto/blob/main/misc/past_leaderboards.md This snippet shows the output of the Arena-Hard-v0.1 leaderboard, detailing model performance metrics. It includes model names, scores, 95% confidence intervals, and average token counts for each entry. ```console Llama-3.3-Nemotron-Super-49B-v1-Feedback-Edit-ITS | score: 88.7 | 95% CI: (-2.3, 2.2) | average #tokens: 1217 claude-3-5-sonnet-20241022 | score: 86.4 | 95% CI: (-1.3, 1.3) | average #tokens: 691 claude-3-5-sonnet-20240620 | score: 82.2 | 95% CI: (-1.9, 1.6) | average #tokens: 567 o1-preview-2024-09-12 | score: 81.7 | 95% CI: (-2.2, 2.1) | average #tokens: 1193 o1-mini-2024-09-12 | score: 79.3 | 95% CI: (-2.8, 2.3) | average #tokens: 1399 gpt-4-turbo-2024-04-09 | score: 74.3 | 95% CI: (-2.4, 2.4) | average #tokens: 662 gpt-4-0125-preview | score: 73.6 | 95% CI: (-2.0, 2.0) | average #tokens: 619 athene-v2-chat | score: 72.1 | 95% CI: (-2.5, 2.5) | average #tokens: 884 gpt-4o-2024-08-06 | score: 71.1 | 95% CI: (-2.5, 2.0) | average #tokens: 594 llama-3.1-nemotron-70b-instruct | score: 71.0 | 95% CI: (-2.8, 3.1) | average #tokens: 869 gpt-4o-2024-05-13 | score: 69.9 | 95% CI: (-2.5, 2.0) | average #tokens: 696 athene-70b-0725 | score: 68.3 | 95% CI: (-2.6, 2.4) | average #tokens: 683 llama-3.1-405b-instruct-fp8 | score: 67.1 | 95% CI: (-2.2, 2.8) | average #tokens: 658 yi-lightning | score: 66.9 | 95% CI: (-3.3, 2.7) | average #tokens: 875 claude-3-opus-20240229 | score: 65.5 | 95% CI: (-2.3, 2.2) | average #tokens: 541 yi-large-preview | score: 65.1 | 95% CI: (-2.5, 2.5) | average #tokens: 720 gpt-4o-mini-2024-07-18 | score: 64.0 | 95% CI: (-3.5, 2.9) | average #tokens: 668 qwen2.5-72b-instruct | score: 63.3 | 95% CI: (-2.5, 2.3) | average #tokens: 821 Mistral-large-2407 | score: 63.1 | 95% CI: (-3.0, 2.6) | average #tokens: 623 Gemini-1.5-pro-api-0514 | score: 62.7 | 95% CI: (-3.2, 3.0) | average #tokens: 676 GLM-4-0520 | score: 61.4 | 95% CI: (-2.6, 2.4) | average #tokens: 636 Yi-large | score: 59.4 | 95% CI: (-2.8, 2.5) | average #tokens: 626 Deepseek-coder-v2 | score: 58.3 | 95% CI: (-2.8, 2.6) | average #tokens: 578 GLM-4-0116 | score: 54.2 | 95% CI: (-2.2, 2.2) | average #tokens: 622 Llama-3.1-70b-instruct | score: 51.8 | 95% CI: (-3.4, 2.1) | average #tokens: 628 GLM-4-air | score: 50.6 | 95% CI: (-2.6, 2.4) | average #tokens: 619 GPT-4-0314 | score: 50.0 | 95% CI: (0.0, 0.0) | average #tokens: 423 Claude-3-sonnet-20240229 | score: 49.9 | 95% CI: (-2.7, 2.4) | average #tokens: 552 GPT-4-0613 | score: 49.7 | 95% CI: (-2.3, 2.5) | average #tokens: 354 Qwen2-72b-instruct | score: 49.6 | 95% CI: (-2.1, 2.2) | average #tokens: 515 Gemma-2-27b-it | score: 47.5 | 95% CI: (-2.5, 2.7) | average #tokens: 577 Gemini-1.5-pro-api-0409-preview | score: 46.7 | 95% CI: (-2.6, 3.1) | average #tokens: 478 Mistral-large-2402 | score: 45.6 | 95% CI: (-2.1, 2.3) | average #tokens: 400 Claude-3-haiku-20240307 | score: 45.4 | 95% CI: (-2.5, 2.7) | average #tokens: 505 Llama-3-70b-instruct | score: 44.5 | 95% CI: (-2.4, 2.0) | average #tokens: 591 Mixtral-8x22b-instruct-v0.1 | score: 44.2 | 95% CI: (-2.7, 3.1) | average #tokens: 430 Gemini-1.5-flash-api-0514 | score: 39.9 | 95% CI: (-2.5, 2.1) | average #tokens: 642 Llama-3.1-nemotron-51b-instruct | score: 39.9 | 95% CI: (-2.9, 2.7) | average #tokens: 747 Qwen1.5-72b-chat | score: 39.9 | 95% CI: (-2.1, 2.4) | average #tokens: 474 Mistral-next | score: 39.6 | 95% CI: (-2.4, 2.7) | average #tokens: 297 Mistral-medium | score: 39.1 | 95% CI: (-2.4, 2.8) | average #tokens: 485 Phi-3-medium-4k-instruct | score: 38.8 | 95% CI: (-2.5, 2.7) | average #tokens: 517 Command-r-plus | score: 37.5 | 95% CI: (-2.4, 2.3) | average #tokens: 541 ``` -------------------------------- ### Benchmark Results: Creative Writing, Ensemble Judges Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md This configuration is optimized for creative writing tasks, utilizing an ensemble of GPT-4.1 and Gemini 2.5 as judges. The results show the performance of various models in creative writing scenarios, with scores and confidence intervals. ```console Model Scores (%) CI (%) 0 gemini-2.5 90.8 (-1.2 / +1.3) 1 o3-2025-04-16 88.8 (-1.1 / +1.0) 2 gemini-2.5-flash 83.9 (-1.3 / +1.4) 3 deepseek-r1 77.0 (-2.0 / +1.4) 4 Qwen3-235B-A22B 73.5 (-1.8 / +1.5) 5 gemma-3-27b-it 69.9 (-1.9 / +1.7) ``` -------------------------------- ### Add Style Attributes to Model Answers Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Appends style attributes, such as token length and number of headers, to model answers. This script takes input data from a specified directory and saves the enhanced data to an output directory. ```console python add_markdown_info.py --dir data/arena-hard-v0.1/model_answer --output-dir data/arena-hard-v0.1/model_answer ``` -------------------------------- ### Show Model Win Rates Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Displays model win rates for different configurations of the Arena Hard benchmark. It allows specifying judge names and control features like markdown and length. ```console python show_result.py --judge-names gpt-4.1 --control-features markdown length ``` ```console python show_result.py --judge-names gpt-4.1 gemini-2.5 --category creative_writing ``` -------------------------------- ### Generate Model Judgments Source: https://github.com/lmarena/arena-hard-auto/blob/main/README.md Generates judgments for model comparisons by adding model names to `config/arena-hard-v2.0.yaml` and running the judgment script. Supports different judge models like GPT-4.1 or Gemini-2.5 and includes caching. ```bash > python gen_judgment.py ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.