### HotpotQA ReAct Task Example

Source: https://context7.com/au-clan/reasonbench/llms.txt

Demonstrates the setup and execution of a HotpotQA task using the ReAct method, which involves Search, Lookup, and Finish actions. The example shows how to initialize the state and print the final step.

```python
async def demo_react(method, env):
    from src.tasks.hotpotqa.state import StateHotpotQA
    # HotpotQA: ReAct uses Search/Lookup/Finish actions
    state = StateHotpotQA(
        puzzle="Which president was Richard Nixon named after?",
        current_state="",
        steps=[],
        randomness=None,
        lookup_paragraphs=[],
        search_paragraphs=[]
    )
    states = await method.solve(idx=0, state=state, namespace="ns_react")
    print(states[0].steps[-1])  # e.g. "Finish[Richard Nixon]"
```

--------------------------------

### HotpotQA Reasoning Example 1

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

Demonstrates a sequence of search and lookup actions to answer a question about documentaries. Includes possible and selected actions.

```text
Which documentary is about Finnish rock groups, Adam Clayton Powell or The Saimaa Gesture?

Possible Actions:
Search[Adam Clayton Powell (film)]
Search[The Saimaa Gesture (film)]
Search[Finish rock music]
Search[Finish documentaries]
Search[Juice Leskinen]
Search[Documentary film]

Selected actions:
Search[Adam Clayton Powell (film)]
Search[The Saimaa Gesture (film)]
Search[Finish documentaries]
```

--------------------------------

### Setup for Log Parsing

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/parsing.ipynb

Defines lists of models, experiments, and benchmarks to be used for log retrieval. Sets up initial variables for model and experiment.

```python
# Models we've run experiments on
models = [
    "Qwen3-235B-A22B-Thinking-2507",
    "gpt-oss-120b", 
    "DeepSeek-R1", 
    "Llama-4-Maverick-17B-128E-Instruct-FP8",     
]

# Experiments we've set up
"""
    - simple: Standard evaluation without any special prompting strategies
    - repeats: Same as simple, but the experiment is repeated multiple times
"""
experiments = [
    "simple",
    "repeats"
]

# Benchmarks we've run experiments on
benchmarks = [
        "game24",
        "hle",
        "hotpotqa",
        "humaneval",
        "matharena",
        "scibench",
        "sonnetwriting"
    ]


# Setup
model = models[0]           # Model for which we want to load logs
experiment = experiments[1] # Experiment for which we want to load the logs

calls = get_calls(
    logs_path="../logs",
    experiment=experiment,
    model=model,
    benchmarks=benchmarks,
    verbose=verbose
)
```

--------------------------------

### Install ReasonBENCH Dependencies

Source: https://github.com/au-clan/reasonbench/blob/main/README.md

Install the required Python packages for ReasonBENCH using pip. Ensure you have the requirements.txt file in your current directory.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install CacheSaver for Inference Optimization

Source: https://github.com/au-clan/reasonbench/blob/main/README.md

Install the CacheSaver library, a client-side framework for efficient, affordable, and reproducible LLM inference, which is a dependency for ReasonBENCH.

```bash
pip install cachesaver
```

--------------------------------

### Visualize Sample Category Distributions

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/traces.ipynb

Prepares for visualizing sample category distributions by printing a header and defining category names, benchmarks, and models. This is a setup for a subsequent heatmap visualization.

```python
# Visualize sample category distributions

print("\n" + "=" * 100)
print("SAMPLE CATEGORY DISTRIBUTION HEATMAP")
print("=" * 100)

# Create data for heatmap
category_names = ['Always Succeeds', 'Always Fails', 'Token-Sensitive', 'Token-Averse', 'Token-Agnostic', 'Weak Dependency']
benchmarks = sorted(df['Benchmark'].unique())
models = sorted(df['Model'].unique())

```

--------------------------------

### HotpotQA Reasoning Example 2

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

Illustrates a reasoning process involving search and lookup actions to determine who a character was named after. Shows possible and selected actions.

```text
Question: Musician and satirist Allie Goertz wrote a song about the "The Simpsons" character Milhouse, who Matt Groening named after who?
Action 1: Search[Milhouse]
Observation 1: Milhouse Mussolini Van Houten is a recurring character in the Fox animated television series The Simpsons voiced by Pamela Hayden and created by Matt Groening.

Possible Actions:
Lookup[named after]
Lookup[Allie Goertz]
Lookup[Matt Groening]
Lookup[name]
Search[Allie Goertz]
Search[The Simpsons]
Search[Allie Goertz Simspons]

Selected actions:
Lookup[named after]
Lookup[name]
Search[The Simpsons]
```

--------------------------------

### Interact with Environment for Game24

Source: https://context7.com/au-clan/reasonbench/llms.txt

Shows how to use the environment's step() method to apply an action to a state and get a new state, and how evaluate() checks for terminal conditions and returns a score. This is used to build a solution trajectory.

```python
from src import EnvironmentFactory
from src.tasks import *
from src.tasks.game24.state import StateGame24

env = EnvironmentFactory.get("game24")

# Build up a solution trajectory
state = StateGame24(puzzle="2 3 4 12", current_state="2 3 4 12", steps=[], randomness=0)

s1 = env.step(state, "3 * 4 = 12 (left: 2 12 12)")
s2 = env.step(s1,    "12 + 12 = 24 (left: 2 24)")
s3 = env.step(s2,    "24 / 2 ... wait, Answer: (3 * 4 + 12) * 2 / 2 = 24")

print(env.is_final(s3))       # True
final, score = env.evaluate(s3)
print(final, score)           # True, 1.0  (if expression evaluates to 24)

# Partial state is not terminal
print(env.is_final(s1))       # False
print(env.evaluate(s1))       # (False, 0.0)
```

--------------------------------

### HotpotQA Reasoning Example 3

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

Presents a reasoning chain to find a common profession between two individuals, using search and finish actions. Includes possible and selected actions.

```text
Question: What profession does Nicholas Ray and Elia Kazan have in common?
Action 1: Search[Nicholas Ray]
Observation 1: Nicholas Ray (born Raymond Nicholas Kienzle Jr., August 7, 1911 – June 16, 1979) was an American film director, screenwriter, and actor best known for the 1955 film Rebel Without a Cause.
Action 2: Search[Elia Kazan]
Observation 2: Elia Kazan was an American film and theatre director, producer, screenwriter and actor.

Possible Actions:
Finish[director, screenwriter, actor]
Finish[film director, screenwriter, actor]
Finish[director, screenwriter and actor]
Lookup[Nicholas Ray]
Lookup[profession]
Lookup[producer]

Selected actions:
Finish[director, screenwriter, actor]
Finish[film director, screenwriter, actor]
Finish[director, screenwriter and actor]
```

--------------------------------

### Aggregate Prompt for Selecting Best Implementations

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

This prompt guides the AI to select the top 'k' best implementations of a function from a given set, based on criteria like correctness, efficiency, readability, style, and testability. It requires returning only the indices of the chosen implementations.

```python
aggregate_prompt = """You are a programming assistant, who is helping user to write efficient and correct codes. You will be given multiple implementations of the same function. You should choose the {k} best implementation based on the following criterias:
1. Correctness: The implementation should return the correct output.
2. Efficiency: The implementation should be efficient in terms of time and space complexity.
3. Readability: The implementation should be readable and understandable.
4. Style: The implementation should follow the style guide of the language.
5. Testability: The implementation should be testable.

Remember your task is to choose the {k} best implementation based on the above criterias. Make sure to return only the indexes of the selected implementations, separated by commas. Do not include any other explanations, introduction, conclusions or thoughts. Just return the indexes of the selected implementations.
Function signature and docstring:
{prompt}
Implementations:
{implementations}
Chosen implementation:
"""
```

--------------------------------

### Import necessary libraries

Source: https://github.com/au-clan/reasonbench/blob/main/examples/cities_example.ipynb

Imports required libraries for disk caching, asynchronous OpenAI API calls, and CacheSaver pipelines. Ensure these libraries are installed.

```python
%load_ext autoreload
%autoreload 2

from diskcache import Cache

from openai import AsyncOpenAI

from cachesaver.pipelines import OnlineAPI

from src.models import OnlineLLM, API
from src.typedefs import DecodingParameters
```

--------------------------------

### BFS Prompt for Multiple Code Implementations

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

This prompt guides the AI to generate at least two different implementations for a given function signature and docstring, using distinct approaches. Each implementation must be clearly delimited by triple backticks.

```python
SIMPLE_CHAT_INSTRUCTION_BFS = """
You are an AI that only responds with {lang} code. You will be given a function signature and its docstring by the user.
Write multiple full implementations (at least two), each restating the function signature. Use a different approach for each.
Mark the start and end of each implementation using triple backticks, like this:
```
<Code implementation here>
```
Each implementation should be fully contained within its own set of backticks, without any additional markers.
"""
```

--------------------------------

### Game of 24 - Single Step Prompt

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

This prompt guides a model to find a single arithmetic step to get closer to 24, given a set of numbers. It requires the model to output only the operation, the result, and the remaining numbers.

```python
# Updated
act = '''Use numbers and basic arithmetic operations (+ - * /) to obtain 24. Each step, you are only allowed to choose two of the remaining numbers to obtain a new number. Do not explain simply list one possible next step, as well as all the remaining numbers and nothing else.

Example: 2 8 8 14
Possible next step:
14 + 2 = 16 (left: 8 8 16)

Example: 1 4 6
Possible next step:
1 * 4 = 4 (left: 4 6)

Example: 1 3
Possible next step:
1 * 3 = 3 (left: 3)

Input: {input}
Possible next step:
'''
```

--------------------------------

### Initialize FoA components

Source: https://github.com/au-clan/reasonbench/blob/main/examples/FoA.ipynb

Sets up the core components for the FoA framework, including selecting the task, loading configuration, initializing the environment, setting up a cache, configuring the LLM client and model, and creating the CacheSaver API and agent.

```python
# Choose task
task = "hotpotqa" # "hotpotqa" or "game24"

# Config
config = OmegaConf.load(f'../configs/{task}/config_foa_{task}.yaml')

# Environment
env = EnvironmentBasic.create(task=task, data_path=f"../datasets/dataset_{task}.csv.gz")

# Cache
cache = Cache(f"../caches/{task}")

# LLM Client and Model
client = AsyncTogether(api_key=os.environ.get('TOGETHER_API_KEY_PERS'))
model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo"
model = OnlineLLM(client, model=model_name)

# CacheSaver API
api = OnlineAPI(
    model=model,
    cache=cache,
    batch_size=30,
    timeout=0.5,
)

# Agent
agent = AgentLLM(api=api)
```

--------------------------------

### Example Sonnet for Aggregation

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

An example sonnet provided for the aggregation task, demonstrating the expected format and content.

```text
aggregate_examples = [
    '''Beneath the whisper of the waving grass (A)
The river sings a song of ancient lore (B)
Each golden moment holds a priceless mass (A)
Of value set beyond the richest store. (B)

The sunlight weaves a pattern soft and bright (C)
Across the meadows where the daisies sail, (D)
Yet even beauty feels a distant blight (C)
When love is locked within a silent jail. (D)

O time, who takes but seldom grants us grace, (E)
Within your tides do all our dreams entwine, (F)
Yet still the grass shall whisper of this place, (E)
A testament to moments lost in time. (F)

'''
]
```

--------------------------------

### Reasoning via Planning (MCTS) Demo

Source: https://context7.com/au-clan/reasonbench/llms.txt

Demonstrates the Reasoning via Planning (MCTS) method. It initializes a state and solves it using the MCTS algorithm, then evaluates the first resulting state.

```python
async def demo_rap(method, env):
    state = StateGame24(puzzle="4 6 8 10", current_state="4 6 8 10", steps=[], randomness=None)
    states = await method.solve(idx=3, state=state, namespace="ns_rap")
    is_final, score = env.evaluate(states[0])
    print(f"RAP result: final={is_final}, score={score}")
```

--------------------------------

### Example Sonnet for Evaluation

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

An example of a sonnet that meets the specified criteria (rhyme scheme and words), used for evaluation purposes.

```text
examples_evaluate = [
    '''Required Rhyme Scheme: ABAB CDCD EFEF GG  
Required Words: grass, value, jail

The river bends beside the morning grass (A)  
A shimmer dances on the silver tide (B)  
The hours like golden moments swiftly pass (A)  
And fortune's value flows with gentle pride (B)  

The jail of fear breaks open with the breeze (C)  
The open fields forgive the bitter rain (D)  
The grass revives beneath the waking trees (C)  
And songs of hope replace the cries of pain (D)  

Bright grasslands whisper secrets to the skies (E)  
The jail of winter thaws beneath the light (F)  
The value found in spring will never die (E)  
But bloom in endless fields beyond our sight (F)  

The grass will grow where broken dreams once fall (G)  
And value shines within the hearts of all. (G)

---END-OF-SONNET---

evaluation: 10
''',
```

```text
''',
    '''Required Rhyme Scheme: ABAB CDCD EFEF GG  
Required Words: grass, value, jail

The mountain sighs beneath the winter snow
The river carves its path with solemn might
The flowers sleep beneath the frozen glow
Awaiting touch of spring’s returning light

No jail confines the wild and restless breeze
It tumbles past the valley’s sleeping door
The grass will rise when winter grants release
Yet value fades along the rocky shore

The colors blend beneath a paling sky
The stars retreat behind a drifting veil
Though grass returns, some dreams must say goodbye
And hearts once brave grow weary and grow pale

The seasons turn, and leave us wondering still
What value clings to hope, and what to will.

---END-OF-SONNET---

evaluation: 10 
''',
```

```text
''',
    '''Required Rhyme Scheme: ABAB CDCD EFEF GG  
Required Words: grass, value, jail

Upon the cliffs the storm begins to roar
The candles flicker in the empty hall
The silver mist creeps underneath the door
A solemn hush descends upon the wall

No grassy fields are near this barren land
No value glimmers in the heavy rain
No jail can hold the fury in its hand
The waves collapse against the rocks in vain

The sailors cry into the endless dark
No flame to guide them through the cruel, wild gale
No harvest waits beyond the fading spark
Just shattered hopes imprisoned without bail

Their names are lost beneath the mourning wave
No grass, no value, only storms to brave.

---END-OF-SONNET---

evaluation: 10
''',
```

```text
''',
    '''Required Rhyme Scheme: ABAB CDCD EFEF GG  
Required Words: grass, value, jail

The twilight burns across the shattered shore (A)  
Old lanterns flicker in the dying mist (B)  
The crows descend where empty houses mourn (C)  
A bitter memory clenched within a fist (B)  

The river moans beneath a sky of ash (D)  
Its course forgotten by the sleeping stone (E)  
The mountains tremble under thunder’s crash (D)  
The plains are silent, aching and alone (F)  

No voices rise above the broken field (G)  
No banners fly beneath the heavy rain (H)  
The meadow bows beneath the turning wheel (I)  
While dreams decay and vanish in their pain (H)  

Night falls without a sound, a hollow art (J)  
And leaves the world with one forsaken heart. (J)

---END-OF-SONNET---

evaluation: 0
'''
]
```

--------------------------------

### Configure ToT environment and LLM

Source: https://github.com/au-clan/reasonbench/blob/main/examples/ToT.ipynb

Sets up the task, configuration, environment, cache, and LLM client for the ToT framework. This includes loading configuration files, initializing the environment with dataset paths, and setting up the asynchronous LLM client with API keys.

```python
# Choose task
task = "hotpotqa" # "hotpotqa" or "game24"

# Config
config = OmegaConf.load(f'../configs/{task}/config_tot_{task}.yaml')

# Environment
env = EnvironmentBasic.create(task=task, data_path=f"../datasets/dataset_{task}.csv.gz")

# Cache
cache = Cache(f"../caches/{task}")

# LLM Client and Model
client = AsyncTogether(api_key=os.environ.get('TOGETHER_API_KEY_PERS'))
model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo"
model = OnlineLLM(client, model=model_name)

# CacheSaver API
api = OnlineAPI(
    model=model,
    cache=cache,
    batch_size=30,
    timeout=0.5,
)

# Agent
agent = AgentLLM(api=api)
```

--------------------------------

### Implement Dummy Framework

Source: https://github.com/au-clan/reasonbench/blob/main/API_proposal.md

A concrete implementation of the Framework that performs a single step and evaluation. It initializes with an agent and environment, then executes a run cycle.

```python
def Dummy(Framework):
    def initialize(agent: Agent, environment: Environmet):
        self.agent = agent
        self.environment = environment

    def run(puzzle_idx: int):
        state = self.environment.reset(puzzle_idx)
        next_state = self.agent.act(state, environment)
        value = self.agent.evaluate(state)
        verification = self.environment.verify(state)
        return next_state, value, verification
```

--------------------------------

### Load Notebook Extensions and Libraries

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/results.ipynb

Loads autoreload and necessary Python libraries for data manipulation, plotting, and analysis. Ensure these libraries are installed.

```python
%load_ext autoreload
%autoreload 2

import numpy as np
import pandas as pd

from collections import defaultdict

from plots import *

import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
from matplotlib.ticker import FormatStrFormatter
import matplotlib.lines as mlines
import seaborn as sns

VERBOSE = False
```

--------------------------------

### Instantiate Reasoning Method with MethodFactory

Source: https://context7.com/au-clan/reasonbench/llms.txt

Use MethodFactory.get() to instantiate a complete reasoning method, connecting agents, models, and environments. This is the primary entry point for running any reasoning strategy. Ensure necessary components like models, caches, and configurations are set up.

```python
import asyncio
from omegaconf import OmegaConf
from src import BenchmarkFactory, EnvironmentFactory, MethodFactory
from src.tasks import *
from src.methods import *
from src.models import OnlineLLM, API
from src.typedefs import DecodingParameters
from cachesaver.pipelines import OnlineAPI
from diskcache import Cache

async def main():
    # Model + Pipeline
    model = OnlineLLM(provider="openai", api_key="OPENAI_API_KEY")
    cache = Cache("caches/example")
    pipeline = OnlineAPI(model=model, cache=cache, batch_size=10,
                         timeout=2.0, allow_batch_overflow=True, correctness=True)
    api = API(pipeline=pipeline, model="gpt-4.1-nano")

    # Parameters and config
    params = DecodingParameters(
        temperature=1.0, max_completion_tokens=10000,
        top_p=1.0, stop=None, logprobs=False
    )
    config = OmegaConf.load("scripts/configs/game24.yaml")["tot_bfs"]
    # config: num_selections=3, num_steps=4, num_evaluations=3

    env = EnvironmentFactory.get("game24")

    # Instantiate Tree-of-Thought BFS method
    method = MethodFactory.get(
        method="tot_bfs",
        benchmark="game24",
        params=params,
        model=api,
        env=env,
        config=config
    )
    return method

# Available methods: "io", "cot", "cot_sc", "react", "foa",
#                   "tot_bfs", "tot_dfs", "got", "rap"
method = asyncio.run(main())
```

--------------------------------

### Load and Print tot_bfs Configuration

Source: https://context7.com/au-clan/reasonbench/llms.txt

Loads the `game24.yaml` configuration file using OmegaConf and prints specific hyperparameters for the `tot_bfs` method. Demonstrates how to access configuration values programmatically.

```python
from omegaconf import OmegaConf

config = OmegaConf.load("scripts/configs/game24.yaml")["tot_bfs"]
print(config.num_selections)   # 3
print(config.num_steps)        # 4
print(config.num_evaluations)  # 3

```

--------------------------------

### Print Analysis Header (Python)

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/variance_analysis.ipynb

Prints a header for Analysis 3, indicating the start of the 'Token Efficiency Frontiers by Problem Type' analysis.

```python
print("\n" + "="*100)
print("ANALYSIS 3: TOKEN EFFICIENCY FRONTIERS BY PROBLEM TYPE")
print("="*100)
```

--------------------------------

### MethodFactory.get

Source: https://context7.com/au-clan/reasonbench/llms.txt

Instantiates the complete reasoning method, wiring together the appropriate agents, model, and environment. This is the primary entry point for running any reasoning strategy.

```APIDOC
## MethodFactory.get(method, benchmark, params, model, env, config)

### Description
Instantiates the complete reasoning method, wiring together the appropriate agents, model, and environment. This is the primary entry point for running any reasoning strategy.

### Parameters
#### Path Parameters
- **method** (string) - Required - The name of the reasoning method to instantiate (e.g., "tot_bfs").
- **benchmark** (string) - Required - The name of the benchmark to use.
- **params** (DecodingParameters) - Required - Decoding parameters for the model.
- **model** (API) - Required - The language model API client.
- **env** (Environment Object) - Required - The task-specific environment.
- **config** (OmegaConf object) - Required - Configuration specific to the chosen method.

### Request Example
```python
import asyncio
from omegaconf import OmegaConf
from src import BenchmarkFactory, EnvironmentFactory, MethodFactory
from src.tasks import *
from src.methods import *
from src.models import OnlineLLM, API
from src.typedefs import DecodingParameters
from cachesaver.pipelines import OnlineAPI
from diskcache import Cache

async def main():
    # Model + Pipeline
    model = OnlineLLM(provider="openai", api_key="OPENAI_API_KEY")
    cache = Cache("caches/example")
    pipeline = OnlineAPI(model=model, cache=cache, batch_size=10,
                         timeout=2.0, allow_batch_overflow=True, correctness=True)
    api = API(pipeline=pipeline, model="gpt-4.1-nano")

    # Parameters and config
    params = DecodingParameters(
        temperature=1.0, max_completion_tokens=10000,
        top_p=1.0, stop=None, logprobs=False
    )
    config = OmegaConf.load("scripts/configs/game24.yaml")["tot_bfs"]
    # config: num_selections=3, num_steps=4, num_evaluations=3

    env = EnvironmentFactory.get("game24")

    # Instantiate Tree-of-Thought BFS method
    method = MethodFactory.get(
        method="tot_bfs",
        benchmark="game24",
        params=params,
        model=api,
        env=env,
        config=config
    )
    return method

# Available methods: "io", "cot", "cot_sc", "react", "foa",
#                   "tot_bfs", "tot_dfs", "got", "rap"
method = asyncio.run(main())
```

### Response
#### Success Response (Method Object)
- Returns an instantiated reasoning method object ready to be used for solving tasks.
```

--------------------------------

### Perform basic FoA step and evaluation

Source: https://github.com/au-clan/reasonbench/blob/main/examples/FoA.ipynb

Demonstrates a single step in the FoA process by resetting the environment, taking a step using the agent, and then evaluating the resulting state. This is useful for testing individual operations.

```python
# # Basic FoA operations test

state = env.reset(0)
print(f"{state=}\n")

new_state = await agent.foa_step(
    state=state, 
    environment=env,
    namespace="0",
    request_id="0",
    cache=None,
    config=config.api.parameters
    )
print(f"{new_state=}\n")

value = await agent.evaluate(
    state=new_state,
    environment=env,
    n=3,
    namespace="0",
    request_id="1",
    cache=None,
    config=config.api.parameters
    )
print(f"{value=}\n")
```

--------------------------------

### Load and Save DataFrame

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/parsing.ipynb

Loads parsed experiment data into a pandas DataFrame and saves it for future use. Also shows an example of loading a previously saved DataFrame.

```python
# Load the data into DataFrame. Each row corresponds to a single experiment
df = pd.DataFrame(data)

# Save the DataFrame for future reference
save_dataframe(
    df=df,
    data_path="../data/strategies",
    experiment=experiment,
    model=model
)

# Loading the data back (example)
df = pd.read_parquet("../data/examples/models.parquet")
if verbose:
    display(df.head(2))
```

--------------------------------

### Run complete FoA framework and log results

Source: https://github.com/au-clan/reasonbench/blob/main/examples/FoA.ipynb

Initializes the complete FoA framework and runs it for a specified number of repeats. It processes puzzles in parallel, computes quality metrics, merges logs, and saves the trial results to a JSON file.

```python
# Complete Framework
fw = FrameworkFoA(config, agent, env)

for seed in range(config.run.repeats):
    log = {}
    step_cache = None
    value_cache = {}
    

    # Get the puzzle indexes for the current set
    set = config.run.set
    puzzle_idxs = env.data.get_set_idxs(set)[:config.run.debugging]

    # Run the puzzles in parallel
    puzzle_coroutines = [
        fw.run(
            puzzle_idx=puzzle_idx,
            namespace=str(puzzle_idx),
            seed=seed,
            value_cache=value_cache,
            step_cache=step_cache,
        )
        for puzzle_idx in puzzle_idxs
    ]
    results = await asyncio.gather(*puzzle_coroutines)
    states, verifications,logs = map(list, zip(*results))

    # Compute quality
    success = [any(v.finished and v.correct for v in vs) for vs in verifications]
    accuracy = sum(success) / len(success)

    # Merge the logs
    for l in logs:
        log.update(l)
    
    # Saving additional info in the log
    log["Info"] = {}
    
    log["Info"]["Cost"] = {
            "total": tokens2cost(agent.tokens["total"], model_name),
            "cached": tokens2cost(agent.tokens["cached"], model_name),
            "generated": tokens2cost(agent.tokens["generated"], model_name),
        }
    
    config = OmegaConf.to_container(config, resolve=True)
    log["Info"]["LLM"] = {"model": config["api"]["model"], "parameters": config["api"]["parameters"]}
    log["Info"]["Framework"] = config["framework"]
    log["Task"] = config["task"]
    log["Run"] = {"seed": seed, "set": config["run"]["set"], "debugging": config["run"]["debugging"]}
    log["Quality"] = {"accuracy": accuracy, "success": success}

    # Save the results of the trial
    log_dir = os.path.join(os.getcwd().split("cachesaver")[0] + "cachesaver", config["logs"]["log_dir"])
    os.makedirs(log_dir, exist_ok=True)
    log_path = os.path.join(log_dir, f"{config['logs']['log_name']}_{seed}.json")
    with open(log_path, "w") as f:
        json.dump(log, f, indent=4)
    
```

--------------------------------

### Perform basic ToT operations

Source: https://github.com/au-clan/reasonbench/blob/main/examples/ToT.ipynb

Demonstrates fundamental operations within the ToT framework, including resetting the environment, taking a step using the agent, and evaluating the resulting state. This is useful for testing individual components.

```python
# Basic ToA operations test

state = env.reset(0)
print(f"{state=}\n")

new_states = await agent.tot_step(
    state=state, 
    environment=env,
    namespace="0",
    request_id="0",
    cache=None,
    config=config.api.parameters
    )
print(f"{new_states=}\n")

value = await agent.evaluate(
    state=new_states[0],
    environment=env,
    n=3,
    namespace="0",
    request_id="1",
    cache=None,
    config=config.api.parameters)
print(f"{value=}\n")
```

--------------------------------

### Load and Save Trace DataFrame

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/parsing.ipynb

Loads the parsed trace data into a pandas DataFrame, with each row representing a response sample. Saves the DataFrame and provides an example of loading it back.

```python
# Load the data into DataFrame. Each row corresponds to a single response sample
df = pd.DataFrame(data)

# Save the DataFrame for future reference
save_dataframe(
    df=df,
    data_path="../data/calls",
    experiment=experiment,
    model=model
)

df = pd.read_parquet("../data/examples/calls.parquet")
if verbose:
    display(df.head(2))
```

--------------------------------

### Direct API Call with OnlineLLM and API

Source: https://context7.com/au-clan/reasonbench/llms.txt

Shows how to make direct API calls to various LLM providers using OnlineLLM and API wrappers. It includes setting up the model, API pipeline with caching, decoding parameters, and inspecting token usage. Supports multiple providers like OpenAI, Anthropic, Gemini, Together AI, and Groq.

```python
import asyncio, os
from src.models import OnlineLLM, API
from src.typedefs import DecodingParameters
from cachesaver.pipelines import OnlineAPI
from diskcache import Cache

async def direct_api_call():
    model = OnlineLLM(provider="openai", api_key="OPENAI_API_API_KEY_CLAN")
    # Supported providers: "openai", "anthropic", "gemini", "together", "groq"

    pipeline = OnlineAPI(model=model, cache=Cache("caches/dev"),
                         batch_size=5, timeout=2.0,
                         allow_batch_overflow=True, correctness=True)
    api = API(pipeline=pipeline, model="gpt-4.1-nano",
              log_path="logs/raw_calls/test.log")

    params = DecodingParameters(temperature=1.0, max_completion_tokens=512,
                                top_p=1.0, stop=None, logprobs=False)

    responses = await api.request(
        prompt="What is 3 + 4?",
        n=3,
        request_id="idx0-test",
        namespace="ns_test",
        params=params
    )
    print(responses)  # ["7", "7", "7"]

    # Inspect token usage and cost
    print(api.tokens["idx0"]["total"])
    # {"in": 12, "out": 6, "cached": 0}

    # Reset counters between runs
    api.clean()

    # Anthropic with extended thinking
    model_claude = OnlineLLM(provider="anthropic", api_key="ANTHROPIC_API_KEY",
                             reasoning_effort="5000")  # budget_tokens

asyncio.run(direct_api_call())
```

--------------------------------

### Initialize CacheSaver Client

Source: https://github.com/au-clan/reasonbench/blob/main/examples/cities_example.ipynb

Defines a function to initialize the CacheSaver client, including the cache, the online LLM model, and the online API pipeline. This setup is crucial for utilizing caching during inference.

```python
def get_cachesaver_client():
    """
    Just a random function to intialize CacheSaver client
    """
    cache = Cache(cache_path)

    # Model
    model = OnlineLLM(provider="openai")

    # Pipeline
    pipeline = OnlineAPI(
        model=model,
        cache=cache,
        batch_size=batch_size,
        timeout=timeout,
        allow_batch_overflow=allow_batch_overflow,
        correctness=bool(correctness)
    )

    # CacheSaver Client
    client_cachesaver = API(
        pipeline=pipeline,
        model="gpt-4.1"
    )
    return client_cachesaver
```

--------------------------------

### Create and Save DataFrame for Reasoning Models

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/parsing.ipynb

Converts the parsed log data into a pandas DataFrame and saves it to a specified path using parquet format. Includes an example of loading the data back.

```python
# Load the data into DataFrame. Each row corresponds to a single experiment
df = pd.DataFrame(data)

# Save the DataFrame for future reference
save_dataframe(
    df=df,
    data_path="../data/models",
    experiment=experiment,
    model=model
)

# Loading the data back (example)
df = pd.read_parquet("../data/examples/models.parquet")
if verbose:
    display(df.head(2))
```

--------------------------------

### Import necessary libraries for ToT framework

Source: https://github.com/au-clan/reasonbench/blob/main/examples/ToT.ipynb

Imports essential libraries for the Tree of Thoughts (ToT) framework, including cache management, configuration loading, API clients, and custom task/agent/model modules. Ensure these libraries are installed.

```python
%load_ext autoreload
%autoreload 2

import os
import json
import asyncio

from diskcache import Cache
from omegaconf import OmegaConf
from together import AsyncTogether

from cachesaver.pipelines import OnlineAPI

import sys
sys.path.append('..')

from src.tasks import EnvironmentBasic
from src.frameworks import FrameworkToT
from src.agents import AgentLLM
from src.models import OnlineLLM
from src.utils import tokens2cost
```

--------------------------------

### Benchmark Reasoning Method Performance

Source: https://context7.com/au-clan/reasonbench/llms.txt

Demonstrates how to use the async `benchmark` method to run a reasoning method over a dataset, measuring durations and evaluating results. It also shows how to enable an internal value cache for performance optimization.

```python
import asyncio
from src import BenchmarkFactory, EnvironmentFactory, MethodFactory
from src.tasks import *
from src.methods import *
from src.models import OnlineLLM, API
from src.typedefs import DecodingParameters
from omegaconf import OmegaConf
from cachesaver.pipelines import OnlineAPI
from diskcache import Cache

async def run_benchmark():
    model = OnlineLLM(provider="openai", api_key="OPENAI_API_KEY")
    pipeline = OnlineAPI(model=model, cache=Cache("caches/bench"),
                         batch_size=10, timeout=2.0,
                         allow_batch_overflow=True, correctness=True)
    api = API(pipeline=pipeline, model="gpt-4.1-nano")

    params = DecodingParameters(temperature=1.0, max_completion_tokens=5000,
                                top_p=1.0, stop=None, logprobs=False)
    config = OmegaConf.load("scripts/configs/game24.yaml")["cot"]
    env = EnvironmentFactory.get("game24")
    method = MethodFactory.get("cot", "game24", params, model=api, env=env, config=config)
    benchmark = BenchmarkFactory.get("game24", split="mini", max_len=5)

    # Run over the benchmark (ns_ratio=0.0 means all unique namespaces)
    durations, results = await method.benchmark(benchmark, ns_ratio=0.0)

    # Evaluate results
    for i, (duration, result_states) in enumerate(zip(durations, results)):
        best = max(env.evaluate(s)[1] for s in result_states)
        print(f"Sample {i}: duration={duration:.2f}s, best_score={best}")
    # Sample 0: duration=1.43s, best_score=1.0
    # Sample 1: duration=1.21s, best_score=0.0

    # With internal value cache (useful for ToT/GoT)
    durations, results = await method.benchmark(benchmark, ns_ratio=0.0, value_cache=True)

asyncio.run(run_benchmark())

```

--------------------------------

### React Prompt for Step-by-Step Code Generation

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

This prompt is designed for a step-by-step code generation process. It instructs the AI to think through the implementation, write code for each step, consider edge cases, and finally confirm completion. It includes placeholders for the function signature, docstring, and current state of the implementation.

```python
react = """You are a programming assistant solving a coding task. Think step by step and plan your implementation carefully. You will be given a function signature and docstring, and you need to implement the function.

For each step:
1. Think about what needs to be done
2. Write code to implement that step
3. Consider edge cases and error handling

Example:
Function signature and docstring:
def add_numbers(a: int, b: int) -> int:
    '''Add two numbers and return the result.'''

Thought: I need to implement a simple addition function. The function takes two integers and returns their sum. I should handle basic input validation.

Action: python
def add_numbers(a: int, b: int) -> int:
    '''Add two numbers and return the result.'''
    # Input validation
    if not isinstance(a, int) or not isinstance(b, int):
        raise TypeError("Both arguments must be integers")
    return a + b


Thought: The implementation looks good. It includes:
1. Type hints for parameters and return value
2. Input validation to ensure both arguments are integers
3. Simple and efficient addition operation
4. Proper docstring preservation

Action: Finish[The implementation is complete and correct]

Function signature and docstring:
{prompt}

Current implementation:
{current_state}

Remember to think step by step and write clear, efficient code."""
```

--------------------------------

### Define Models, Experiments, and Benchmarks for Reasoning Models

Source: https://github.com/au-clan/reasonbench/blob/main/notebooks/parsing.ipynb

Sets up lists of models, experiment types, and benchmarks to be used for loading and parsing logs. This configuration is used to filter and select relevant log data.

```python
# Models we've run experiments on
models = [
    "Qwen3-235B-A22B-Thinking-2507",
    "gpt-oss-120b", 
    "DeepSeek-R1", 
    "Llama-4-Maverick-17B-128E-Instruct-FP8",     
]

# Experiments we've set up
"""
    - simple: Standard evaluation without any special prompting strategies
    - repeats: Same as simple, but the experiment is repeated multiple times
"""
experiments = [
    "simple",
    "repeats"
]

# Benchmarks we've run experiments on
benchmarks = [
        "game24",
        "hle",
        "hotpotqa",
        "humaneval",
        "matharena",
        "scibench",
        "sonnetwriting"
    ]


# Setup
model = models[i]           # Model for which we want to load logs
experiment = experiments[1] # Experiment for which we want to load the logs

# Load the logs
logs = get_logs(
    logs_path="../logs",
    experiment=experiment,
    model=model,
    benchmarks=benchmarks,
    verbose=verbose
)
```

--------------------------------

### Game of 24 - Evaluation Prompt

Source: https://github.com/au-clan/reasonbench/blob/main/prompts.md

This prompt instructs the model to evaluate whether a given set of numbers can result in 24 using arithmetic operations. It requires a response of 'sure', 'likely', or 'impossible', along with brief reasoning or example calculations.

```python
# Updated
evaluate = '''Evaluate if given numbers can reach 24 by responding with the following: "sure", "likely" or "impossible". Follow the format of the following examples. Try to be brief.

Example: 10 14
10 + 14 = 24
sure

Example: 11 12
11 + 12 = 23
12 - 11 = 1
11 * 12 = 132
11 / 12 = 0.91
impossible

Example: 4 4 10
4 + 4 + 10 = 8 + 10 = 18
4 * 10 - 4 = 40 - 4 = 36
(10 - 4) * 4 = 6 * 4 = 24
sure

Example: 4 9 11
9 + 11 + 4 = 20 + 4 = 24
sure

Example: 5 7 8
5 + 7 + 8 = 12 + 8 = 20
(8 - 5) * 7 = 3 * 7 = 21
I cannot obtain 24 now, but numbers are within a reasonable range
likely

Example: 5 6 6
5 + 6 + 6 = 17
(6 - 5) * 6 = 1 * 6 = 6
I cannot obtain 24 now, but numbers are within a reasonable range
likely

Example: 10 10 11
10 + 10 + 11 = 31
(11 - 10) * 10 = 10
10 10 10 are all too big
impossible

Example: 1 3 3
1 * 3 * 3 = 9
(1 + 3) * 3 = 12
1 3 3 are all too small
impossible

Input: {input}
'''
```

--------------------------------

### Create and Inspect StateGame24 Object

Source: https://context7.com/au-clan/reasonbench/llms.txt

Demonstrates creating an initial StateGame24 object, serializing it to a dictionary, cloning it with modified attributes, and accessing attributes using dictionary-style access. States are immutable and should be cloned for modifications.

```python
from src.tasks.game24.state import StateGame24

# Create an initial state
state = StateGame24(
    puzzle="1 2 3 4",
    current_state="1 2 3 4",
    steps=[],
    randomness=42
)

# Serialize to dict for inspection
print(state.serialize())
# {'current_state': '1 2 3 4', 'steps': ''}

# Clone with new randomness (required before passing to a method)
cloned = state.clone(randomness=1234)
print(cloned.randomness)   # 1234
print(cloned.puzzle)       # "1 2 3 4" (unchanged)

# Hash for deduplication / caching
print(hash(state))         # deterministic int

# Dict-style attribute access
print(state["puzzle"])     # "1 2 3 4"
print(state.get("steps", []))  # []
```

--------------------------------

### Configuration and Parameters for LLM Inference

Source: https://github.com/au-clan/reasonbench/blob/main/examples/cities_example.ipynb

Sets up configuration variables and decoding parameters for LLM inference. This includes cache path, benchmark details, model specifics, and generation parameters like temperature and max tokens.

```python
batch_size = 1
timeout=2
allow_batch_overflow = 1
correctness = 1
ns_ratio=0
value_cache=True

cache_path="caches/developping"

benchmark="hotpotqa"
method="tot_bfs"
split="mini"

model="gpt-4.1"
temperature=1.5
max_completion_tokens=10000
top_p=1.0
stop=None
logprobs=None

# Decoding Parameters
params = DecodingParameters(
    temperature=temperature,
    max_completion_tokens=max_completion_tokens,
    top_p=top_p,
    stop=stop,
    logprobs=logprobs
)

prompt = "Give me the name of a random city from all over the world (only the name, no other text)."
```

--------------------------------

### Integrate LLM with CacheSaver Pipeline

Source: https://github.com/au-clan/reasonbench/blob/main/examples/cachesaver_api.ipynb

Sets up and uses the `OnlineAPI` pipeline with a defined `OnlineLLM` model and a `diskcache.Cache` instance. It demonstrates making both single and batch requests through the pipeline, showing how caching and batching are handled.

```python
@dataclass(frozen=True)
class Request(Request):
    temperature: float=1
    max_completion_tokens: int=None

# Pipeline
request1 = Request(prompt="What is the meaning of life?", n=2, request_id="sth1", namespace="sth", max_completion_tokens=10)
request2 = Request(prompt="Cats or dogs?", n=1, request_id="sth2", namespace="sth", max_completion_tokens=10)
batch = Batch(requests=[request1, request2])

cache = Cache("../caches/developping")

pipeline = OnlineAPI(
    model = model,
    cache=cache,
    batch_size = 2,
    timeout = 1
)

pipeline_request_response = await pipeline.request(request1)
print(f"{pipeline_request_response=}\n")

pipeline_batch_response = await pipeline.batch_request(batch)
print(f"{pipeline_batch_response=}")
```

--------------------------------

### Set LLM Provider API Keys

Source: https://github.com/au-clan/reasonbench/blob/main/README.md

Configure your API keys for LLM providers by exporting them as environment variables. Replace 'sk-...' with your actual API key.

```bash
export OPENAI_API_KEY_CLAN="sk-..."
# and/or other provider keys
```

--------------------------------

### Run a Simple Experiment via Shell Script

Source: https://github.com/au-clan/reasonbench/blob/main/README.md

Execute a basic experiment using the provided shell script. Edit the variables at the top of the script to customize the benchmark, method, model, and data split.

```bash
bash scripts/simple/simple.sh
```