LiveIdeaBench

https://github.com/x66ccff/liveideabench
Admin
LiveIdeaBench is a benchmark framework for evaluating the scientific creativity, divergent thinking,...
Tokens:48,874
Snippets:191
Trust Score:8.8
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install Environment Dependencies

Source: https://github.com/x66ccff/liveideabench/blob/main/README.md

Install the required Python packages for the framework.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run Evaluation Examples

Source: https://github.com/x66ccff/liveideabench/blob/main/README.md

Generate and evaluate ideas using specific models, providers, and keywords.

```bash
# Generate ideas using a specified model
python run.py --idea_model "openai/gpt-4o-mini"

# Use a specific provider
python run.py --idea_model "openai/gpt-4o-mini" --provider openrouter
```

```bash
# Use a single keyword:

python run.py --idea_model "openai/gpt-4o-mini" --keyword "relativity"
# Use multiple keywords:

python run.py --idea_model "openai/gpt-4o-mini" --keyword "relativity" "periodic table"
# Do not specify a keyword (use all keywords):

python run.py --idea_model "openai/gpt-4o-mini"
```

--------------------------------

### Load Dataset with Pandas

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Initializes the pandas library and provides a commented-out example for loading the dataset from the Hugging Face hub.

```python
import pandas as pd

# df = pd.read_csv("hf://datasets/6cf/liveideabench/liveideabench_hf.csv")
```

--------------------------------

### Invoke Emission Calculation

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Example of calling the emission calculation function with required token counts and model parameters.

```python
combined_results = calculate_combined_emissions_table(
    idea_token_counts=result, 
    critic_token_counts=critic_token_results, 
    model_params_dict=model_params_dict,
    open_source_models=open_source_models
)
```

--------------------------------

### Initialize Database

Source: https://github.com/x66ccff/liveideabench/blob/main/README.md

Run the initialization script to set up the internal database.

```bash
python -c "from utils.database import init_database; init_database()"
```

--------------------------------

### Get DataFrame length

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Returns the total number of rows in the DataFrame.

```python
df.__len__()
```

--------------------------------

### Configure API Keys

Source: https://github.com/x66ccff/liveideabench/blob/main/README.md

Set up API keys via file or environment variables before running the framework.

```bash
echo "your-openrouter-api-key" > apikey
```

```bash
export OPENROUTER_API_KEY="your-openrouter-api-key"
export STEP_API_KEY="your-step-api-key"
export GEMINI_API_KEYS="key1,key2,key3"
```

--------------------------------

### Log Warning Output

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Example of a warning message generated during plot rendering.

```text
2025-04-09 22:37:03,370 - WARNING - Looks like you are using a tranform that doesn't support FancyArrowPatch, using ax.annotate instead. The arrows might strike through texts. Increasing shrinkA in arrowprops might help.
```

--------------------------------

### Initialize Environment

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Imports necessary libraries for data manipulation.

```python
import pandas as pd
import os
```

--------------------------------

### Filter Rows

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Example of removing specific rows based on model name.

```python
# # del all rows with anthropic/claude-3-haiku
# df = df[df['idea_model'] != 'anthropic/claude-3-haiku']
```

--------------------------------

### Initialize Local Tokenizer

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Loads a Qwen tokenizer from a local directory path.

```python
from transformers import AutoTokenizer
tokenizer_path = "./tokenizer/Qwen/Qwen2.5-72B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(
    tokenizer_path,
    trust_remote_code=True,
    local_files_only=True
)
```

--------------------------------

### Initialize Environment and Load Data

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Imports necessary libraries and loads the primary dataset from a parquet file.

```python
import os 
import random 
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
from matplotlib.gridspec import GridSpec
from matplotlib.lines import Line2D
from matplotlib.font_manager import fontManager
import hashlib
import pandas as pd
import random
import re
import os
import time
import json
from pprint import pprint
from config import CRITIC_MODELS, IDEA_MODELS
from utils.LLM import create_llm
from utils.utils import stable_hash, timeout_handler, run_with_timeout, append_content_new



FLUENCY_EVAL_METHOD = 'llm' # llm jaccard embedding_dense embedding_sparse tfidf

df = pd.read_parquet('./data/data.parquet')
```

--------------------------------

### Import Model Configurations

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Loads model lists from a local configuration file.

```python
from config import IDEA_MODELS, CRITIC_MODELS
```

--------------------------------

### Prepare Output Directory

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Ensures the figures directory exists.

```python
# If './figs/' does not exist, create it; if it exists, ignore it
os.makedirs('./figs/', exist_ok=True)
```

--------------------------------

### Initialize Multi-Metric Plot Layout

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Sets up a figure with three subplots for displaying different model metrics.

```python
# --- Second plot: three subplots arranged horizontally ---
fig2, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(20*scale, 9*scale))  # Arrange 3 subplots horizontally

# Update color settings
metrics = [
    {'name': 'originality', 'color': '#66ccff', 'title': 'Originality Score'},  # Use new color
    {'name': 'feasibility', 'color': '#5092e4', 'title': 'Feasibility Score'},  # Use new color
    {'name': 'clarity', 'color': '#ffa500', 'title': 'Clarity Score'}  # Use new color
]

axes = [ax1, ax2, ax3]
```

--------------------------------

### Inspect Raw Critique

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Displays a sample of the raw critique text.

```python
string = df['raw_critique'].tolist()
print(string[2])
```

--------------------------------

### Import Configuration

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Imports model constants from a local configuration file.

```python
from config import CRITIC_MODELS, IDEA_MODELS
```

--------------------------------

### Execution Logs

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Console output logs showing the progress of the bootstrap calculation for various models.

```text
开始计算 41 个模型的灵活性指标...
处理 (1/41) qwen/qwq-32b
percentile_value 6.75
```

```text
 49%|████▉     | 4912/10000 [00:00<00:00, 7175.65it/s]
```

```text
100%|██████████| 10000/10000 [00:01<00:00, 8330.22it/s]
```

```text
处理 (2/41) google/gemini-2.0-flash-exp:free
percentile_value 6.833333333333333
```

```text
100%|██████████| 10000/10000 [00:01<00:00, 9768.52it/s]
```

```text
处理 (3/41) google/gemini-2.0-flash-thinking-exp:free
percentile_value 6.833333333333333
```

```text
100%|██████████| 10000/10000 [00:01<00:00, 8782.86it/s]
```

```text
处理 (4/41) deepseek/deepseek-chat
percentile_value 6.375
```

```text
100%|██████████| 10000/10000 [00:01<00:00, 6539.99it/s]
```

```text
处理 (5/41) deepseek/deepseek-r1-distill-llama-70b
percentile_value 6.413541666666666
```

--------------------------------

### Configuring and Saving the Performance Chart

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Finalizing chart aesthetics, including labels, rotation, layout adjustment, and saving to a PDF file.

```python
# 7. Set the chart style
plt.title('Model Performance on LiveIdeaBench Across Scientific Categories')
plt.xlabel('Scientific Categories (sorted by keyword count)')
plt.ylabel('Idea Models')

# 8. Rotate the x-axis labels
plt.xticks(rotation=30, ha='right')
plt.yticks(rotation=0)

# 9. Adjust the layout
plt.tight_layout()
# Make sure the 'figs' directory exists or change the save path
plt.savefig('figs/sci_cat_2_digits.pdf', bbox_inches='tight')

# Display the chart
plt.show()
```

--------------------------------

### Prepare Data for Visualization

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Filter, scale, and rename columns for plotting, and map model names for consistency.

```python
if is_plot:
    # First, get all the numeric column names
    numeric_columns = df_merge.select_dtypes(include=['int64', 'float64']).columns

    # Add the Model_Short column to the list of columns to keep
    columns_to_keep = list(numeric_columns) + ['Model_Short']

    # Keep only the numeric columns and the Model_Short column
    df_merge = df_merge[columns_to_keep]
    
    # Use regular expressions to filter out columns containing _second or _max
    df_merge = df_merge.loc[:, ~df_merge.columns.str.contains('_second|_max')]
    df_merge = df_merge.set_index('Model_Short', drop=True)
    df_merge.iloc[:, 0:6] *= 10
    
    import seaborn as sns
    import matplotlib.pyplot as plt
    import matplotlib.gridspec as gridspec
    import numpy as np
    import pandas as pd
    import matplotlib.colors as mcolors

    df_merge.columns = ['Fluency','Feasibility','Originality','Clarity','Flexibility','Avg.','(LiveBench) Avg.','(LB) Reasoning','(LB) Coding','(LB) Math','(LB) Data Analysis','(LB) Language','(LB) IF']

    # Ensure the model order is consistent
    df_merge = df_merge.sort_values(by='(LiveBench) Avg.', ascending=False)
    
    # Create a model name mapping - assuming the index of df_view needs to be transformed by the clean_model_name function
    # If df_view already has an index, use that index to create the mapping
    if not isinstance(df_view.index, pd.RangeIndex):  # Check if df_view has an index
        model_mapping = {name: clean_model_name(name) for name in df_view.index}
    else:  # If df_view uses the default index, assume there is a Model column
        # Find the possible model column names
        possible_model_columns = [col for col in df_view.columns if 'model' in col.lower()]
        if possible_model_columns:
            model_column = possible_model_columns[0]
            model_mapping = {name: clean_model_name(name) for name in df_view[model_column]}
        else:
            raise ValueError("Cannot find a model column in df_view, please specify the model column name")
    
    # Create a reverse mapping, from the cleaned model names back to the original model names
    reverse_mapping = {}
    for orig, cleaned in model_mapping.items():
        if cleaned in df_merge.index:
            reverse_mapping[cleaned] = orig
    
    # Prepare the LiveIdeaBench confidence interval data - we will store tuples (mean_value, ci_value) instead of formatted strings
    ci_data_raw = {}
    
    # Calculate the average error for each model (for the Avg column)
    metrics = ['fluency', 'feasibility', 'originality', 'clarity', 'flexibility']
    avg_symmetric_errors = {}
    
    for view_idx in df_view.index:
```

--------------------------------

### Execution Outputs

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Sample console outputs generated during the execution of the token counting process.

```text
Output:
处理模型:  10%|▉         | 4/41 [00:16<01:44,  2.82s/it]
```

```text
Output:
idea error: nan
```

```text
Output:
处理模型: 100%|██████████| 41/41 [01:45<00:00,  2.58s/it]
```

```text
Output:
{'qwen/qwq-32b': 3272395, 'google/gemini-2.0-flash-exp:free': 289855, 'google/gemini-2.0-flash-thinking-exp:free': 268822, 'deepseek/deepseek-chat': 339689, 'deepseek/deepseek-r1-distill-llama-70b': 1054313, 'qwen/qwq-32b-preview': 3774662, 'anthropic/claude-3.7-sonnet:thinking': 4993782, 'google/gemini-2.0-pro-exp-02-05:free': 315840, 'deepseek/deepseek-r1': 1642147, 'anthropic/claude-3.5-haiku-20241022': 278554, 'deepseek/deepseek-r1-distill-qwen-32b': 1429143, 'openai/o1': 240612, 'openai/gpt-4.5-preview': 287003, 'mistralai/mistral-large-2411': 292152, 'openai/o3-mini-high': 232967, 'amazon/nova-lite-v1': 222522, 'sammcj/qwen2.5-dracarys2-72b:Q4_K_M': 261955, 'meta-llama/llama-3.1-405b-instruct': 317736, 'openai/o3-mini': 223555, 'openai/gpt-4o-2024-11-20': 322091, 'openai/gpt-4o-mini': 268069, 'google/gemma-2-27b-it': 215656, 'mistralai/mistral-small-24b-instruct-2501': 349875, 'openai/gpt-4-turbo': 300866, 'mistralai/mistral-small': 295047, 'openai/o1-mini': 292354, 'meta-llama/llama-3.1-70b-instruct': 331842, 'qwen/qwen-2.5-72b-instruct': 284860, 'x-ai/grok-2-1212': 259922, 'google/gemini-2.0-flash-lite-001': 294225, 'qwen/qwen-2.5-coder-32b-instruct': 260045, 'qwen/qwen-2.5-7b-instruct': 226013, 'qwen/qwen-max': 167514, 'anthropic/claude-3.5-sonnet': 223678, 'step-2-16k-202411': 317633, 'microsoft/phi-4': 396819, 'google/gemini-pro-1.5': 207097, 'meta-llama/llama-3.3-70b-instruct': 131481, 'anthropic/claude-3.7-sonnet': 304098, 'anthropic/claude-3-opus': 287743, 'amazon/nova-pro-v1': 198872}
```

--------------------------------

### Calculate and display model rejection statistics

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Computes total samples, rejected samples, and rejection percentages per model, then displays the results.

```python
# Calculate total samples per model
model_counts = df['idea_model'].value_counts()

# Calculate truly rejected samples per model
true_rejected_counts = df[df['true_rejection'] == True]['idea_model'].value_counts()

# Calculate true rejection rates
true_rejection_rates = (true_rejected_counts / model_counts * 100).sort_values(ascending=False)

# Create a DataFrame with model stats
rejection_stats = pd.DataFrame({
    'Total Samples': model_counts,
    'Rejected Samples': true_rejected_counts,
    'Rejection Rate (%)': true_rejection_rates.round(2)
})

# Sort by rejection rate in descending order
rejection_stats = rejection_stats.sort_values('Rejection Rate (%)', ascending=False)

# Filter out models with zero rejection rate
non_zero_rejection_stats = rejection_stats[rejection_stats['Rejection Rate (%)'] > 0]

# Display rejection rate statistics for all models
print("Model Rejection Rates Ranking (All Models):")
display(rejection_stats)

# Display only non-zero rejection models
print("\nModel Rejection Rates Ranking (Non-Zero Only):")
display(non_zero_rejection_stats)
```

--------------------------------

### LaTeX Model Performance Table

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

A LaTeX table structure for displaying comparative performance metrics of various LLM models.

```latex
\begin{table*}[htbp]
    \centering
    \renewcommand{\arraystretch}{1.2}
    \begin{tabular}{c|c|ccccc|c}
    \toprule
    \textbf{Rank} & \textbf{Model} & \textbf{Fluency} & \textbf{Feasib.} & \textbf{Origin.} & \textbf{Flexib.} & \textbf{Avg.} & \textbf{Open} \\
    \midrule
    1 & claude-3.7-sonnet:thinking & 7.48 & 5.70 & \textbf{8.06} & \textbf{7.04} & \textbf{7.22} &  \\
2 & deepseek-r1 & 6.63 & \textbf{6.52} & \underline{7.84} & 6.83 & \underline{7.18} & \checkmark \\
3 & claude-3.7-sonnet & \textbf{7.80} & 5.46 & 7.81 & \underline{6.92} & 7.12 &  \\
4 & gemini-2.0-flash-exp & 7.30 & 6.02 & 7.37 & 6.83 & 7.07 &  \\
5 & qwq-32b & 6.45 & \underline{6.35} & 7.77 & 6.75 & 7.06 & \checkmark \\
6 & gemini-2.0-flash-thinking-exp & 7.38 & 6.05 & 7.35 & 6.83 & 7.06 &  \\
7 & gemini-2.0-pro-exp-02-05 & 6.84 & 5.88 & 7.76 & 6.75 & 7.03 &  \\
8 & qwq-32b-preview & \underline{7.49} & 6.10 & 6.87 & 6.71 & 6.93 & \checkmark \\
9 & claude-3.5-sonnet & 6.90 & 5.42 & 7.83 & 6.62 & 6.92 &  \\
10 & deepseek-r1-distill-qwen-32b & 7.06 & 6.08 & 7.13 & 6.62 & 6.86 & \checkmark \\
11 & mistral-small & 7.36 & 5.97 & 6.98 & 6.62 & 6.86 & \checkmark \\
12 & gemini-pro-1.5 & 6.68 & 5.92 & 7.33 & 6.58 & 6.85 &  \\
13 & gpt-4.5-preview & 6.49 & 6.03 & 7.45 & 6.54 & 6.85 &  \\
14 & mistral-large-2411 & 6.68 & 6.06 & 7.01 & 6.50 & 6.79 & \checkmark \\
15 & deepseek-chat & 6.15 & 6.04 & 7.31 & 6.38 & 6.72 & \checkmark \\
16 & phi-4 & 6.58 & 5.80 & 7.24 & 6.42 & 6.72 & \checkmark \\
17 & deepseek-r1-distill-llama-70b & 6.66 & 6.07 & 6.98 & 6.41 & 6.71 & \checkmark \\
18 & gpt-4o-2024-11-20 & 6.12 & 5.58 & 7.64 & 6.38 & 6.69 &  \\
19 & o1-mini & 5.89 & 6.20 & 7.09 & 6.33 & 6.66 &  \\
20 & gemma-2-27b-it & 7.18 & 5.50 & 6.86 & 6.38 & 6.65 & \checkmark \\
21 & qwen2.5-dracarys2-72b & 6.22 & 6.09 & 6.80 & 6.33 & 6.63 &  \\
22 & llama-3.1-405b-instruct & 6.57 & 5.56 & 7.18 & 6.33 & 6.62 & \checkmark \\
23 & qwen-2.5-72b-instruct & 6.17 & 5.99 & 6.91 & 6.29 & 6.62 & \checkmark \\
24 & llama-3.1-70b-instruct & 6.71 & 5.49 & 7.16 & 6.38 & 6.62 & \checkmark \\
25 & o1 & 6.23 & 5.88 & 7.23 & 6.29 & 6.61 &  \\
26 & gemini-2.0-flash-lite-001 & 6.73 & 5.61 & 6.92 & 6.25 & 6.60 &  \\
27 & qwen-max & 5.64 & 5.83 & 7.51 & 6.29 & 6.59 &  \\
28 & nova-pro-v1 & 6.45 & 6.19 & 6.59 & 6.25 & 6.58 &  \\
29 & qwen-2.5-coder-32b-instruct & 6.58 & 5.76 & 6.90 & 6.25 & 6.56 & \checkmark \\
30 & gpt-4-turbo & 6.14 & 5.62 & 7.08 & 6.25 & 6.56 &  \\
31 & step-2-16k-202411 & 5.70 & 5.77 & 7.28 & 6.21 & 6.54 &  \\
32 & o3-mini & 5.57 & 5.91 & 7.45 & 6.21 & 6.51 &  \\
33 & qwen-2.5-7b-instruct & 6.66 & 6.02 & 6.34 & 6.17 & 6.47 & \checkmark \\
34 & grok-2-1212 & 5.76 & 5.82 & 6.95 & 6.17 & 6.47 &  \\
35 & o3-mini-high & 5.27 & 5.86 & 7.52 & 6.17 & 6.44 &  \\
36 & mistral-small-24b-instruct-2501 & 7.07 & 5.66 & 6.61 & 6.12 & 6.41 & \checkmark \\
37 & claude-3.5-haiku-20241022 & 5.61 & 5.05 & 7.72 & 6.08 & 6.37 &  \\
38 & claude-3-opus & 5.74 & 5.66 & 6.66 & 6.04 & 6.36 &  \\
39 & gpt-4o-mini & 5.28 & 5.86 & 6.67 & 6.00 & 6.25 &  \\
40 & llama-3.3-70b-instruct & 5.68 & 5.69 & 6.32 & 5.75 & 6.06 & \checkmark \\
41 & nova-lite-v1 & 4.51 & 6.06 & 6.60 & 5.71 & 6.05 &  \\

    \bottomrule
    \end{tabular}
    \caption{Performance Comparison of Different Models}
    \label{tab:model_performance}
    \end{table*}
```

--------------------------------

### Configure Visualization Fonts

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Sets up custom font paths and matplotlib parameters for consistent plotting.

```python
colors = ['#28abaf','#66ccff','#EE0000','#006666','#FFFF00','#0080FF']

p = './assets/fonts/Roboto_Condensed/static'
import os
all_font_flie_names = os.listdir(p)
all_ttf_file_names = [n for n in all_font_flie_names if n.endswith('.ttf')]

for ttf_file_name in all_ttf_file_names:
    fontManager.addfont(path=os.path.join(p,ttf_file_name))


# sns.set_style("white")
plt.rcParams['font.family'] = ['sans-serif'] 
plt.rcParams['font.sans-serif'] = ['Roboto Condensed'] 
plt.rcParams['font.size'] = 13
```

--------------------------------

### Ecologits Calculator Function

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Calculates energy and GHG emissions per 100 output tokens based on active and total parameter counts.

```python
def ecologits_calculator_per100_output_token(active_parameters, total_parameters):
    # data from:
    # https://huggingface.co/spaces/genai-impact/ecologits-calculator
    res = {          # (Wh  , gCO2eq)
        ('70B','70B'): (1.25, 0.762),
        ('32B','32B'): (0.714, 0.437),
        ('37B','671B'): (4.7, 2.88),
        ('175B','175B'): (5.44, 3.32),
        ('20B','20B'): (0.545, 0.335),
        ('123B','123B'): (1.99, 1.22),
        ('8B','8B'): (0.377, 0.232),
        ('72B','72B'): (1.27, 0.779),
        ('405B','405B'): (23.8, 14.5),
        ('7B','7B'): (0.363, 0.223),
        ('27B', '27B'): (0.643, 0.394),
        ('24B', '24B'): (0.601, 0.369),
        ('14B', '14B'): (0.461, 0.283),
        ('500B', '500B'): (29.1, 17.8),
        
    }
    return res[(active_parameters, total_parameters)]
```

--------------------------------

### Generate Hexbin Plots for Metrics

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Iterates through models to create hexbin visualizations comparing response length against performance metrics.

```python
# --- Plotting Section (only if is_plot is still True) ---
if is_plot and 'df_agg' in locals():

    # --- Helper Function for p-value stars ---
    def get_p_stars(p_value):
        if p_value is None or pd.isna(p_value): return ""
        if p_value < 0.001: return '***'
        elif p_value < 0.01: return '**'
        elif p_value < 0.05: return '*'
        else: return ''

    # --- Metrics and Plotting Setup ---
    metrics_to_plot = ['originality', 'feasibility', 'clarity', 'avg']
    dict_name = {
        'originality': 'Originality',
        'feasibility': 'Feasibility',
        'clarity': 'Clarity',
        # 'avg': 'Average',
        'avg': r'$\frac{1}{3}(Orig. + Feas. + Clar.)$'
    }
    target_colors = ['#66ccff', '#5092e4', '#ff9933', '#39C5BB']
    # --- REMOVED Fixed Y-axis limits ---
    # y_axis_limits = (0.5, 5.5) # Removed

    # --- Loop Through Each Model using the Aggregated Data ---
    for model_name in models_to_plot:
        print(f"\n--- Plotting for model: {model_name} ---")

        df_model = df_agg[df_agg['idea_model'] == model_name].copy()
        df_model = df_model[df_model['full_response_length'] > 0]

        if df_model.empty:
            print(f"Skipping model {model_name}.")
            continue

        df_model['log_length'] = np.log10(df_model['full_response_length'])

        print(f"Model {model_name} - Aggregated Response Length Description:")
        print(df_model['full_response_length'].describe())

        # --- Create Plot Figure for the Current Model ---
        fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(6, 6))

        axes = [ax1, ax2, ax3, ax4]

        for i, metric in enumerate(metrics_to_plot):
            print(f'  Metric: {metric}')
            ax = axes[i]

            if metric not in df_model.columns:
                print(f"  Metric '{metric}' not found.")
                ax.set_title(f"{dict_name.get(metric, metric)}\n(Data Missing)")
                ax.text(0.5, 0.5, "Data Missing", ha='center', va='center', transform=ax.transAxes)
                continue

            mask = pd.notna(df_model['log_length']) & pd.notna(df_model[metric])
            clean_df = df_model[mask]

            if clean_df.empty or len(clean_df) < 2:
                print(f"  Skipping plot for metric '{metric}' due to insufficient data.")
                ax.set_title(f"{dict_name.get(metric, metric)}\n(Insufficient Data)")
                ax.text(0.5, 0.5, "Insufficient Data", ha='center', va='center', transform=ax.transAxes)
                continue

            # --- Setup for Hexbin ---
            grid_size = 18 # Keep gridsize adjustment
            target_color = target_colors[i]
            custom_cmap = mcolors.LinearSegmentedColormap.from_list(f'cmap_{metric}', ['#FFFFFF', target_color])

            try:
                x_data = clean_df['full_response_length']
                y_data = clean_df[metric]
                log_x_data = clean_df['log_length']
```

--------------------------------

### Configure and Plot Radar Chart

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Sets up radial axes, categories, and plots data for multiple models on a radar chart.

```python
# === Right Plot (Radar Chart) ===
# Set categories (dimensions) for the radar chart, including Clarity
categories = ['Fluency', 'Feasibility', 'Clarity', 'Originality', 'Flexibility']
num_vars = len(categories)
# Calculate angle for each axis
angles = np.linspace(0, 2 * np.pi, num_vars, endpoint=False).tolist()
angles += angles[:1]  # Close the shape

# Configure the radar chart appearance
ax2.set_theta_offset(np.pi / 2) # Rotate plot to start at the top
ax2.set_theta_direction(-1) # Plot axes clockwise

# Draw grid lines and labels
ax2.set_rlabel_position(0) # Position radial labels near the center
ax2.set_rticks([5, 6, 7, 8]) # Set radial ticks (score values)
ax2.set_rlim(4.5, 8.5) # Set radial limits (score range)
ax2.grid(True, linestyle='--', alpha=0.7)

# Set angle ticks and labels (categories)
ax2.set_xticks(angles[:-1])
ax2.set_xticklabels(categories)
# Increase the distance between labels (categories) and the plot edge
ax2.xaxis.set_tick_params(pad=20)  # Adjust padding as needed

# Define color list for the radar plot lines
colors = ['#52ccc3', '#5092e4', '#ff6b6b', '#9673a6', '#ffa500'] # Example colors, adjust as needed

# Plot data for each sampled model, including Clarity
for idx, (model_idx, model) in enumerate(sampled_models.iterrows()):
    values = [model['Fluency'], model['Feasibility'],
              model['Clarity'], model['Originality'], model['Flexibility']]  # Include Clarity
    values += values[:1] # Close the shape by repeating the first value
    actual_color = colors[idx % len(colors)] # Cycle through colors if more models than colors
    ax2.plot(angles, values, 'o-', linewidth=2, label=model['Model_Short'], color=actual_color)
    ax2.fill(angles, values, alpha=0.1, color=actual_color)
```

--------------------------------

### Sample True Rejections

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Retrieves a random sample of 20 true rejections for inspection.

```python
df[df['true_rejection']][['keywords','idea_model','first_reject_response']].sample(20)
```

--------------------------------

### Visualize rejection rates with a bar chart

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Generates a bar chart of non-zero rejection rates, adds value labels, and saves the output to a PDF file.

```python
# Set figure with higher resolution and better proportions
plt.figure(figsize=(10, 6), dpi=100)

# Create the plot with better-aligned x-tick positions, using only non-zero models
x = np.arange(len(non_zero_rejection_stats.index))
bars = plt.bar(x, non_zero_rejection_stats['Rejection Rate (%)'], color='skyblue', width=0.6)

# Add value labels directly above each bar
for i, bar in enumerate(bars):
    height = bar.get_height()
    plt.text(x[i], height + 0.005, f'{height:.2f}%', 
             ha='center', va='bottom', fontsize=10)

# Set x-ticks at the centers of the bars
plt.xticks(x, non_zero_rejection_stats.index, rotation=45, ha='right', fontsize=9)

# Set chart title and axis labels
plt.title('Rejection Rates by Model (Non-Zero Only)', fontsize=14, fontweight='bold', pad=20)
plt.xlabel('Model', fontsize=12, labelpad=10)
plt.ylabel('Rejection Rate (%)', fontsize=12)
plt.yticks(fontsize=9)

# Improve grid appearance
plt.grid(axis='y', linestyle='--', alpha=0.3, zorder=0)

# Add a thin border at the bottom
plt.axhline(y=0, color='black', linewidth=0.8)

# Ensure proper layout
plt.tight_layout()
plt.subplots_adjust(bottom=0.2)  # Add more space at the bottom for x labels
plt.savefig('figs/model_rejection_rates.pdf')
# Display the chart
plt.show()
```

--------------------------------

### Calculate Model Flexibility with Bootstrap CI

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Performs bootstrap sampling to estimate percentiles and confidence intervals for model data, with optional visualization.

```python
model_data = view.groupby('keywords').agg({'feasibility':'mean','originality':'mean','flue':'mean','clarity':'mean'}).dropna().values.mean(axis=1)
    
    # Check if there are enough data points
    if len(model_data) < 5:  # Set a minimum threshold
        print(f"Warning: The model '{idea_model}' has too few valid data points ({len(model_data)})")
        return np.nan, np.nan, np.nan
    
    # Calculate the percentile of the original data
    percentile_value = np.percentile(model_data, percentile * 100)
    print('percentile_value', percentile_value)
    
    # Perform bootstrap sampling
    bootstrap_percentiles = []
    np.random.seed(42)  # Set the random seed to ensure reproducibility
    from tqdm import tqdm
    for _ in tqdm(range(n_bootstrap)):
        # Random sampling with replacement, sample size is the same as the original data
        bootstrap_sample = np.random.choice(model_data, size=len(model_data), replace=True)
        # Calculate the percentile of this sample
        bootstrap_percentile = np.percentile(bootstrap_sample, percentile * 100)
        bootstrap_percentiles.append(bootstrap_percentile)
    
    # Calculate the confidence interval
    alpha = (1 - confidence) / 2
    ci_lower = np.percentile(bootstrap_percentiles, alpha * 100)
    ci_upper = np.percentile(bootstrap_percentiles, (1 - alpha) * 100)
    
    # Visualize (if needed)
    if plot:
        plt.figure(figsize=(10, 6))
        
        # Original data distribution
        plt.subplot(2, 1, 1)
        sns.histplot(model_data, kde=True, color='purple')
        plt.axvline(x=percentile_value, color='red', linestyle='--')
        plt.title(f'{idea_model} Data Distribution and {percentile*100}% Percentile')
        plt.xlabel('Mean Value')
        
        # Bootstrap percentile distribution
        plt.subplot(2, 1, 2)
        sns.histplot(bootstrap_percentiles, kde=True, color='blue')
        plt.axvline(x=percentile_value, color='red', linestyle='--', label=f'Original {percentile*100}% Percentile')
        plt.axvline(x=ci_lower, color='green', linestyle=':', label=f'Confidence Interval Lower Bound')
        plt.axvline(x=ci_upper, color='green', linestyle=':', label=f'Confidence Interval Upper Bound')
        plt.title(f'Bootstrap Calculated {percentile*100}% Percentile Distribution (n={n_bootstrap})')
        plt.xlabel(f'{percentile*100}% Percentile Value')
        plt.legend()
        
        plt.tight_layout()
        plt.show()

    
    return percentile_value, ci_lower, ci_upper
```

--------------------------------

### Model Data Preparation for Visualization

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Python script to filter model data based on keywords and prepare the environment for plotting.

```python
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors # Needed for LinearSegmentedColormap
import numpy as np
import seaborn as sns
from scipy import stats
import pandas as pd
import os

# --- !!! Assume df is already loaded here !!! ---
# Example: df = pd.read_csv('your_data.csv')
# Ensure your df has columns: 'idea_model', 'idea', 'full_response', 'originality', 'feasibility', 'clarity', 'avg'

# --- Configuration ---
is_plot = True

if is_plot and 'df' in locals():
    # --- Define Models to Plot ---
    keywords_to_plot = ["qwq", "thinking", "reasoning", "r1", "o1", "o3"]
    if 'idea_model' not in df.columns:
        print("Error: 'idea_model' column not found in DataFrame.")
        is_plot = False
    else:
        all_model_names = df['idea_model'].dropna().unique()
        models_to_plot = [name for name in all_model_names if any(keyword in name for keyword in keywords_to_plot)]
        print(f"Found models to plot based on keywords: {models_to_plot}")
        if not models_to_plot:
            print("No models found matching the keywords. Skipping plotting.")
            is_plot = False

if is_plot and 'df' in locals():
    # --- Ensure Output Directory Exists ---
    output_dir = 'figs'
    os.makedirs(output_dir, exist_ok=True)
```

--------------------------------

### Execution Progress Output

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Console output showing the progress of the environmental impact calculation.

```text
Calculating Idea Model Environmental Impact: 100%|██████████| 41/41 [00:00<00:00, 105500.90it/s]
Calculating Critic Model Environmental Impact: 100%|██████████| 10/10 [00:00<00:00, 88674.50it/s]
```

--------------------------------

### Load Data

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Reads the primary dataset from a CSV file.

```python
df = pd.read_csv('./csvs/view.csv',index_col=0)
df
```

--------------------------------

### Define Model Parameters Dictionary

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

A dictionary mapping model identifiers to their total parameters, architecture type, and activated parameters.

```python
        'total_params': '32B',
        'architecture': 'Dense',
        'activated_params': '32B'
    },
    'qwen/qwen-2.5-7b-instruct': {
        'total_params': '7B',
        'architecture': 'Dense',
        'activated_params': '7B'
    },
    'qwen/qwen-max': { # Estimated
        'total_params': '671B',
        'architecture': 'MoE',
        'activated_params': '37B'
    },
    'anthropic/claude-3.5-sonnet': { # Estimated
        'total_params': '175B',
        'architecture': 'Dense',
        'activated_params': '175B'
    },
    'step-2-16k-202411': { # Estimated
        'total_params': '671B',
        'architecture': 'MoE',
        'activated_params': '37B'
    },
    'microsoft/phi-4': {
        'total_params': '14B',
        'architecture': 'Dense',
        'activated_params': '14B'
    },
    'google/gemini-pro-1.5': { # Estimated
        'total_params': '671B',
        'architecture': 'MoE',
        'activated_params': '37B'
    },
    'meta-llama/llama-3.3-70b-instruct': {
        'total_params': '70B',
        'architecture': 'Dense',
        'activated_params': '70B'
     },
    'anthropic/claude-3.7-sonnet': { # Estimated
        'total_params': '175B',
        'architecture': 'Dense',
        'activated_params': '175B'
    },
    'anthropic/claude-3-opus': { # Estimated
        'total_params': '500B',
        'architecture': 'Dense',
        'activated_params': '500B'
    },
    'amazon/nova-pro-v1': { # Estimated
        'total_params': '70B',
        'architecture': 'Dense',
        'activated_params': '70B'
    }
}


# print(json.dumps(model_params_dict, indent=4))
```

--------------------------------

### Pivot and Reorder Performance Data

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Creates pivot tables for sample sizes and reorders columns based on keyword counts.

```python
pivot_samples = stats_df.pivot_table(
    values='sample_size',
    index='idea_model',
    columns='category',
    aggfunc='first'
)

# 4. Rename the indices
pivot_mean.index = pivot_mean.index.map(clean_model_name)
pivot_ci.index = pivot_ci.index.map(clean_model_name)
pivot_samples.index = pivot_samples.index.map(clean_model_name)

# Reorder the columns by keyword count
# Filter out any categories that may not be present in the pivot tables
available_categories = [cat for cat in sorted_categories if cat in pivot_mean.columns]
pivot_mean = pivot_mean[available_categories]
pivot_ci = pivot_ci[available_categories]
pivot_samples = pivot_samples[available_categories]
```

--------------------------------

### Generate and Save LaTeX Table

Source: https://github.com/x66ccff/liveideabench/blob/main/stats_flexibility.ipynb

Constructs a LaTeX table string with confidence intervals and writes it to a .tex file.

```python
# Format the row for LaTeX, ensuring math mode ($...$) around scores with CIs
        latex_table_with_ci += f"{row['Model_Short']} & {open_source} & "
        latex_table_with_ci += f"${fluency_ci}$ & ${feasibility_ci}$ & ${clarity_ci}$ & "
        latex_table_with_ci += f"${originality_ci}$ & ${flexibility_ci}$ & ${average_ci}$ \\\n"

    latex_table_with_ci += "\\bottomrule\n"
    latex_table_with_ci += "\\end{tabular}\n"
    latex_table_with_ci += "}\n" # End resizebox
    latex_table_with_ci += "\\end{table*}\n" # End table*

    # Write to file
    # Ensure the 'tex' directory exists or change the path
    # import os
    # if not os.path.exists('tex'): os.makedirs('tex')
    output_tex_file = './tex/liveideabench_table_with_ci.tex'
    with open(output_tex_file, 'w') as f:
        f.write(latex_table_with_ci)
    print(f"LaTeX table with confidence intervals saved to {output_tex_file}")
```

--------------------------------

### Pretty Print Results

Source: https://github.com/x66ccff/liveideabench/blob/main/co2.ipynb

Uses the pprint module to format the dictionary of token counts for better readability.

```python
from pprint import pprint
```

```python
pprint(result)
```

--------------------------------

### Calculate Overall Rejection Rate

Source: https://github.com/x66ccff/liveideabench/blob/main/stats.ipynb

Computes and prints the percentage of rejected ideas.

```python
overall_reject_rate = df['first_was_rejected'].sum() / len(df) * 100
print('The Overall rejection rate is {:.4f}%'.format(overall_reject_rate))
```
Tokens:
Raw