### Install Project Dependencies

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Install all required Python packages for the project by running this command.

```shell
pip install -r requirements.txt
```

--------------------------------

### InfiniteBench End-to-End Evaluation Workflow

Source: https://context7.com/openbmb/infinitebench/llms.txt

This bash script outlines the steps for setting up and running the InfiniteBench evaluation. It includes installing dependencies, downloading benchmark data, running evaluations for specific models (e.g., GPT-4), and computing final scores.

```bash
# 1. Install dependencies
pip install -r requirements.txt

# 2. Download benchmark data
bash scripts/download_dataset.sh

# 3. Run evaluation (example: GPT-4 on all tasks via shell script)
bash scripts/eval_gpt4.sh

# 4. Score results for all tasks
cd src
python compute_scores.py --task all --model_name gpt4 --output_dir ../results
# Retrieve.PassKey: 1.0
# Retrieve.Number:  1.0
# Retrieve.KV:      0.89
# En.Sum:           0.1473
# En.QA:            0.2244
# ...
```

--------------------------------

### Load Task Data with eval_utils

Source: https://context7.com/openbmb/infinitebench/llms.txt

Loads examples for a specific benchmark task from a JSONL file in the specified directory. Ensure the `data_dir` points to your downloaded dataset.

```python
from eval_utils import load_data

# Load the passkey retrieval task (590 examples, avg 122.4k tokens)
examples = load_data("passkey", data_dir="../data")
print(len(examples))          # 590
print(examples[0].keys())     # dict_keys(['context', 'input', 'answer'])
print(examples[0]['input'])   # "What is the pass key?"
print(examples[0]['answer'])  # ["84729"]
```

--------------------------------

### Create Model-Specific Prompt

Source: https://context7.com/openbmb/infinitebench/llms.txt

Builds a formatted prompt for a given dataset example and model. Handles task-specific formatting and supports various LLM backends.

```python
from eval_utils import load_data, create_prompt

examples = load_data("longbook_choice_eng", data_dir="../data")
eg = examples[0]

# Build prompt for GPT-4
prompt = create_prompt(eg, data_name="longbook_choice_eng", model_name="gpt4", data_dir="../data")
print(prompt[:300])
# "Read the book and answer the question.\n\n<book context>...\n\nQuestion: Who is the main character?\n#  Only one of the following options is correct, tell me the answer using one single letter (A, B, C, or D)...\n#  A. Alice\nB. Bob\nC. Carol\nD. Dave"

# Build prompt for YaRN-Mistral / Yi-200K / ChatGLM3 (uses yarn_mistral_templates)
prompt = create_prompt(eg, data_name="longbook_choice_eng", model_name="yarn-mistral", data_dir="../data")
```

--------------------------------

### Create Chat Messages with Truncation

Source: https://context7.com/openbmb/infinitebench/llms.txt

Generates an OpenAI-style `messages` list from a dataset example, optionally truncating the input to a specified token limit using a provided tokenizer. Returns both the messages list and the raw prompt string.

```python
import tiktoken
from eval_utils import load_data, create_msgs

examples = load_data("math_calc", data_dir="../data")
eg = examples[0]
tokenizer = tiktoken.encoding_for_model("gpt-4")

msgs, prompt = create_msgs(tokenizer, eg, data_name="math_calc", data_dir="../data", model_name="gpt4")
# Before truncation: 45000
```

--------------------------------

### Generate Code Execution Dataset

Source: https://context7.com/openbmb/infinitebench/llms.txt

The `build_code_run` function generates synthetic Python function execution examples. For custom generation, use `generate_and_store_collections`, `generate_functions`, and `generate_code_run_example` to control call depth, function count, and value ranges.

```python
from construct_synthetic_dataset import (
    build_code_run,
    generate_and_store_collections,
    generate_functions,
    generate_code_run_example,
)

# Build the full code_run.jsonl dataset (400 examples)
build_code_run()

# Or build a single example manually:
generate_and_store_collections(
    n=4,              # 4 layers of function calls
    m=137,            # ~137 functions per layer
    min_val=1,
    max_val=1100,
    output_file="collections.json"
)
generate_functions("collections.json", min_add=-12, max_add=17, output_file="functions_module.py")
example = generate_code_run_example("collections.json", min_x=-10, max_x=10, functions_module="functions_module.py")
print(example['input'])   # "Please give me the exact number of the return value of func_42(7)..."
print(example['answer'])  # 23  ← computed by actually executing the generated Python code
```

--------------------------------

### Extract Ground Truth Answer with get_answer

Source: https://context7.com/openbmb/infinitebench/llms.txt

Retrieves the expected answer for a given example and task. For multiple-choice tasks, it returns both the text and the label.

```python
from eval_utils import load_data, get_answer

examples = load_data("code_debug", data_dir="../data")
eg = examples[0]
answer = get_answer(eg, "code_debug")
print(answer)   # ["func_42", "B"]  — function name + option letter

examples = load_data("passkey", data_dir="../data")
eg = examples[0]
answer = get_answer(eg, "passkey")
print(answer)   # ["84729"]
```

--------------------------------

### Load and log configurations from config.txt

Source: https://context7.com/openbmb/infinitebench/llms.txt

Use `load_and_log_configs` to read and parse the `config.txt` file into a structured dictionary. Access API keys, service models, and local API IPs from the returned config object.

```python
from eval_utils import load_and_log_configs

config = load_and_log_configs()
print(config['api_keys']['openai'])        # "sk-ாலத்தில்"
print(config['services']['anthropic'])     # "claude-3-5-sonnet-20240620"
print(config['local_api_ip']['ollama'])    # "http://127.0.0.1:11434/api/generate"
print(config['output_path'])               # "results"
```

--------------------------------

### Load InfiniteBench Dataset with Hugging Face Datasets

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Demonstrates how to load the InfiniteBench dataset using the 🤗 Datasets library, specifying custom features.

```python
from datasets import load_dataset, Value, Sequence
ft = Features({"id": Value("int64"), "context": Value("string"), "input": Value("string"), "answer": Sequence(Value("string")), "options": Sequence(Value("string"))})
dataset = load_dataset("xinrongzhang2022/InfiniteBench", features=ft)
```

--------------------------------

### Download Dataset using Script

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Use this script to download the necessary datasets. The data will be directly placed in the 'data' directory.

```shell
cd InfiniteBench
bash scripts/download_dataset.sh
```

--------------------------------

### Download InfiniteBench Dataset

Source: https://context7.com/openbmb/infinitebench/llms.txt

Downloads all 12 task JSONL files from HuggingFace to a local 'data/' directory. This script ensures all benchmark data is available for evaluation.

```bash
bash scripts/download_dataset.sh
```

--------------------------------

### Configuration Loading

Source: https://context7.com/openbmb/infinitebench/llms.txt

Details on how to load API credentials and settings from the `config.txt` file, which is automatically used as a fallback by API wrapper functions.

```APIDOC
## Configuration (`src/config.txt`)

### Description
This section details the structure of the `config.txt` file used for storing API keys, model names, and local server endpoints. These configurations are loaded automatically by the API wrapper functions if not provided directly.

### File Structure (`src/config.txt`)
```ini
[API]
anthropic_api_key = sk-ant-...
anthropic_model   = claude-3-5-sonnet-20240620
openai_api_key    = sk-...
openai_model      = gpt-4o
groq_api_key      = gsk_...
groq_model        = llama3-70b-8192
deepseek_api_key  = ...
deepseek_model    = deepseek-chat
mistral_api_key   = ...
mistral_model     = mistral-large-latest

[Local-API]
kobold_api_IP  = http://127.0.0.1:5001/api/v1/generate
llama_api_IP   = http://127.0.0.1:8080/completion
ooba_api_IP    = http://127.0.0.1:5000/v1/chat/completions
vllm_api_IP    = http://127.0.0.1:8000/v1/chat/completions
ollama_api_IP  = http://127.0.0.1:11434/api/generate
```

### Programmatic Usage
```python
from eval_utils import load_and_log_configs

config = load_and_log_configs()
print(config['api_keys']['openai'])        # Example: "sk-..."
print(config['services']['anthropic'])     # Example: "claude-3-5-sonnet-20240620"
print(config['local_api_ip']['ollama'])    # Example: "http://127.0.0.1:11434/api/generate"
print(config['output_path'])               # Example: "results"
```
```

--------------------------------

### Evaluate using a local llama.cpp server

Source: https://context7.com/openbmb/infinitebench/llms.txt

Command-line execution for evaluating tasks using a local llama.cpp server. Specify the task, API type, data directory, and output directory.

```bash
python eval_multi_api.py \
    --task passkey \
    --api llamacpp \
    --data_dir ../data \
    --output_dir ../results
```

--------------------------------

### OpenAI-compatible API wrappers for Groq, DeepSeek, Mistral

Source: https://context7.com/openbmb/infinitebench/llms.txt

These functions provide uniform wrappers for Groq, DeepSeek, and Mistral endpoints, all adhering to the same signature pattern. They allow for easy switching between these providers.

```python
from LLM_API_Calls import chat_with_groq, chat_with_deepseek, chat_with_mistral

# Groq (llama3-70b-8192 by default)
response = chat_with_groq(
    api_key="gsk_ாலத்தில்",
    input_data="<long context>",
    custom_prompt_arg="What is the pass key?",
    temp=0.0,
    system_message="You are a helpful assistant."
)

# DeepSeek (deepseek-chat by default)
response = chat_with_deepseek(
    api_key="...",
    input_data="<long context>",
    custom_prompt_arg="Find the largest number in the list.",
    temp=0.1,
)

# Mistral (mistral-large-latest by default)
response = chat_with_mistral(
    api_key="...",
    input_data="<long context>",
    custom_prompt_arg="Which function has the bug? Answer A, B, C, or D.",
    temp=0.2,
)
print(response)  # "B"
```

--------------------------------

### Multi-Backend LLM Evaluation via CLI

Source: https://context7.com/openbmb/infinitebench/llms.txt

Evaluates LLMs across multiple API backends (commercial and local) using a unified runner. Configure backends via config.txt. Use --verbose for detailed output.

```bash
cd src

# Evaluate using Anthropic Claude via API
python eval_multi_api.py \
    --task longbook_qa_eng \
    --api anthropic \
    --data_dir ../data \
    --output_dir ../results \
    --verbose
```

--------------------------------

### chat_with_groq / chat_with_deepseek / chat_with_mistral

Source: https://context7.com/openbmb/infinitebench/llms.txt

Provides uniform wrappers for Groq, DeepSeek, and Mistral APIs, all compatible with the OpenAI API signature for ease of use.

```APIDOC
## chat_with_groq / chat_with_deepseek / chat_with_mistral

### Description
Uniform wrappers for Groq, DeepSeek, and Mistral endpoints. All follow the same signature pattern, making it easy to switch between these providers.

### Method Signatures
```python
chat_with_groq(api_key: str, input_data: str, custom_prompt_arg: str = "", temp: float = 0.0, system_message: str = "") -> str
chat_with_deepseek(api_key: str, input_data: str, custom_prompt_arg: str = "", temp: float = 0.0, system_message: str = "") -> str
chat_with_mistral(api_key: str, input_data: str, custom_prompt_arg: str = "", temp: float = 0.0, system_message: str = "") -> str
```

### Parameters (Common to all three)
- **api_key** (str) - The API key for the respective service.
- **input_data** (str) - The main input prompt or data for the model.
- **custom_prompt_arg** (str, optional) - Additional instructions or prompt elements. Defaults to "".
- **temp** (float, optional) - Controls randomness. Lower values make output more deterministic. Defaults to 0.0 (or 0.1/0.2 for DeepSeek/Mistral examples).
- **system_message** (str, optional) - A system-level instruction for the assistant. Defaults to "".

### Request Examples
```python
from LLM_API_Calls import chat_with_groq, chat_with_deepseek, chat_with_mistral

# Groq (llama3-70b-8192 by default)
response = chat_with_groq(
    api_key="gsk_...",
    input_data="<long context>",
    custom_prompt_arg="What is the pass key?",
    temp=0.0,
    system_message="You are a helpful assistant."
)

# DeepSeek (deepseek-chat by default)
response = chat_with_deepseek(
    api_key="...",
    input_data="<long context>",
    custom_prompt_arg="Find the largest number in the list.",
    temp=0.1,
)

# Mistral (mistral-large-latest by default)
response = chat_with_mistral(
    api_key="...",
    input_data="<long context>",
    custom_prompt_arg="Which function has the bug? Answer A, B, C, or D.",
    temp=0.2,
)
print(response)  # Expected output: "B" (example for Mistral)
```
```

--------------------------------

### Generate Math Task Datasets

Source: https://context7.com/openbmb/infinitebench/llms.txt

Utilize `build_math_find` for generating datasets that require finding largest/smallest/median values from large integer lists, and `build_math_calc` for arithmetic expression evaluation requiring intermediate value tracking. `generate_math_qa` allows for custom parameterization.

```python
from construct_synthetic_dataset import build_math_find, build_math_calc, generate_math_qa

# Generate math_find.jsonl (7 subtasks × 50 samples)
build_math_find()
# Example record: {"prompt": "Find the largest number from the list below:"
#                  "context": "[42, 7, 99, 3, ...]",  ← 60k integers
#                  "input": "You should answer with only one number...",
#                  "answer": 99}

# Generate math_calc.jsonl (50 expression examples, ~30k operations each)
build_math_calc()
# Example record: {"context": "1 + 5 - 3 + 8 - 2 ...",
#                  "answer": [6, 3, 11, 9, ...]}  ← intermediate values

# Or call generate_math_qa directly for custom parameters
samples = generate_math_qa(
    list_length=1000,
    min_val=0,
    max_val=99,
    tasks=["largest number", "smallest number", "median"]
)
print(len(samples))   # 150 (3 tasks × 50 samples)
print(samples[0].keys())  # dict_keys(['prompt', 'context', 'input', 'answer'])
```

--------------------------------

### MultiAPILLMClient Usage

Source: https://context7.com/openbmb/infinitebench/llms.txt

Demonstrates how to use the MultiAPILLMClient to chat with different LLM backends by specifying the api_name. This client abstracts away the complexities of individual API integrations.

```APIDOC
## MultiAPILLMClient

### Description
Provides a unified interface to interact with various LLM APIs. You can switch between different LLM providers without changing your core chat logic.

### Usage
```python
from eval_multi_api import MultiAPILLMClient

client = MultiAPILLMClient("src/config.txt")

# Chat with OpenAI
response = client.chat(
    api_name="openai",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the pass key in this text? [long context] The pass key is 84729."}
    ],
    temperature=0.0,
    max_tokens=10,
)
print(response)  # Expected output: "84729"

# Switch to Anthropic
response = client.chat(
    api_name="anthropic",
    messages=[{"role": "user", "content": "Summarize the following book: [long context]"}],
    system_message="You are a helpful assistant."
)
print(response) # Expected output: A summary of the book.
```

### Supported APIs
Commercial: openai, anthropic, cohere, groq, openrouter, deepseek, mistral
Local: llamacpp, kobold, oobabooga, vllm, tabbyapi
```

--------------------------------

### Generate Passkey and Number String Datasets

Source: https://context7.com/openbmb/infinitebench/llms.txt

Use `build_passkey` to generate datasets with hidden keys at varying positions and `build_number_string` for number string retrieval tasks. These functions create JSONL files suitable for retrieval benchmarks.

```python
from construct_synthetic_dataset import build_passkey, build_number_string

# Generate passkey dataset files (passkey_8192.jsonl, passkey_16384.jsonl, etc.)
build_passkey()

# Generate number_string.jsonl (128k context, key embedded at varying positions)
build_number_string()
```

--------------------------------

### Call the OpenAI Chat Completions API

Source: https://context7.com/openbmb/infinitebench/llms.txt

Use the `chat_with_openai` function to send prompts to the OpenAI API. It supports API key fallback from config, custom prompts, temperature, and system messages.

```python
from LLM_API_Calls import chat_with_openai

response = chat_with_openai(
    api_key="sk-ாலத்தில்",
    input_data="There is a hidden key: 84729. What is the key?",
    custom_prompt_arg="Answer with only the number.",
    temp=0.0,
    system_message="You are a helpful assistant."
)
print(response)  # "84729"
```

--------------------------------

### Programmatic usage of MultiAPILLMClient

Source: https://context7.com/openbmb/infinitebench/llms.txt

Instantiate and use the MultiAPILLMClient to interact with different LLM backends. Switch between APIs like OpenAI and Anthropic without changing the core code by specifying the api_name.

```python
from eval_multi_api import MultiAPILLMClient

client = MultiAPILLMClient("src/config.txt")

# Chat with any configured backend
response = client.chat(
    api_name="openai",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the pass key in this text? [long context] The pass key is 84729."}
    ],
    temperature=0.0,
    max_tokens=10,
)
print(response)  # "84729"

# Switch to Anthropic without changing code
response = client.chat(
    api_name="anthropic",
    messages=[{"role": "user", "content": "Summarize the following book: [long context]"}],
    system_message="You are a helpful assistant.",
)
```

--------------------------------

### Configuration file template for API credentials and settings

Source: https://context7.com/openbmb/infinitebench/llms.txt

INI-formatted configuration file template for storing API keys, model names, and local endpoint IPs. This file is read automatically by API wrapper functions as a fallback.

```ini
# src/config.txt — template structure
[API]
anthropic_api_key = sk-ant-ாலத்தில்
anthropic_model   = claude-3-5-sonnet-20240620
openai_api_key    = sk-ாலத்தில்
openai_model      = gpt-4o
groq_api_key      = gsk_ாலத்தில்
groq_model        = llama3-70b-8192
deepseek_api_key  = ...
deepseek_model    = deepseek-chat
mistral_api_key   = ...
mistral_model     = mistral-large-latest

[Local-API]
kobold_api_IP  = http://127.0.0.1:5001/api/v1/generate
llama_api_IP   = http://127.0.0.1:8080/completion
ooba_api_IP    = http://127.0.0.1:5000/v1/chat/completions
vllm_api_IP    = http://127.0.0.1:8000/v1/chat/completions
ollama_api_IP  = http://127.0.0.1:11434/api/generate
```

--------------------------------

### Run Evaluation for Passkey Task

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Execute the evaluation script for the passkey task. Ensure the dataset is in the 'data' folder or specify its location using the --data_dir argument.

```python
python eval_rwkv.py --task passkey
```

--------------------------------

### Access Model-Specific Prompt Templates

Source: https://context7.com/openbmb/infinitebench/llms.txt

Access task-to-template mappings for different models. Templates include placeholders like {context}, {input}, and {question}. YaRN-Mistral adds a completion prefix.

```python
from prompt import gpt4_templates, yarn_mistral_templates, claude2_templates, kimi_templates

# GPT-4 passkey template
print(gpt4_templates["passkey"])
# "There is an important info hidden inside a lot of irrelevant text. Find it and memorize them.
#  I will quiz you about the important information there.\n\n{context}\n\n{input}"

# YaRN-Mistral adds a completion prefix to guide generation
print(yarn_mistral_templates["passkey"])
# "...{context}\n\n{input}\n\nThe pass key is"

# Claude 2 code_run template
print(claude2_templates["code_run"])
# "In the file functions_module.py, there is a function called ${func}. ...
#  Please give me the exact number of the return value of {func_call}.
#  Your response should end with the sentence 'The return value is:'."

# Available template dicts: gpt4_templates, yarn_mistral_templates, claude2_templates, kimi_templates
```

--------------------------------

### Run GPT-4 Evaluation via OpenAI API

Source: https://context7.com/openbmb/infinitebench/llms.txt

Evaluates GPT-4 on specified tasks with automatic checkpoint resuming, middle-truncation, and rate-limit delays. Use --verbose for more output.

```bash
cd src

# Evaluate GPT-4 on the KV retrieval task
python eval_gpt4.py --task kv_retrieval --data_dir ../data --output_dir ../results/gpt4

# Evaluate on English book summarization
python eval_gpt4.py --task longbook_sum_eng --data_dir ../data --output_dir ../results/gpt4 --verbose

# Run on a subset (examples 0–49)
python eval_gpt4.py --task code_debug --start_idx 0 --stop_idx 50 \
    --data_dir ../data --output_dir ../results/gpt4
# Output files: ../results/gpt4/preds_code_debug.jsonl
# Each line: {"id": 0, "prediction": "B", "ground_truth": ["func_42", "B"]}
```

--------------------------------

### chat_with_openai

Source: https://context7.com/openbmb/infinitebench/llms.txt

Directly calls the OpenAI Chat Completions API. It supports API key fallback from configuration and handles various input types, along with error logging.

```APIDOC
## chat_with_openai

### Description
Sends a prompt to the OpenAI API using the model configured in `config.txt`. Handles JSON/string/file input, API key fallback from config, and error logging.

### Method Signature
```python
chat_with_openai(api_key: str, input_data: str, custom_prompt_arg: str = "", temp: float = 0.0, system_message: str = "") -> str
```

### Parameters
- **api_key** (str) - Your OpenAI API key.
- **input_data** (str) - The main input prompt or data for the model.
- **custom_prompt_arg** (str, optional) - Additional instructions or prompt elements. Defaults to "".
- **temp** (float, optional) - Controls randomness. Lower values make output more deterministic. Defaults to 0.0.
- **system_message** (str, optional) - A system-level instruction for the assistant. Defaults to "".

### Request Example
```python
from LLM_API_Calls import chat_with_openai

response = chat_with_openai(
    api_key="sk-...",
    input_data="There is a hidden key: 84729. What is the key?",
    custom_prompt_arg="Answer with only the number.",
    temp=0.0,
    system_message="You are a helpful assistant."
)
print(response)  # Expected output: "84729"
```
```

--------------------------------

### Run YaRN-Mistral Local Model Evaluation

Source: https://context7.com/openbmb/infinitebench/llms.txt

Evaluates YaRN-Mistral-7B-128K locally using chunked KV-cache generation for long inputs. Specify model path, data directory, and output directory.

```bash
cd src

# Load model from local path and evaluate on passkey retrieval
python eval_yarn_mistral.py \
    --task passkey \
    --model_path ../../../yarn-mistral-7b-128k \
    --data_dir ../data \
    --output_dir ../results \
    --device cuda

# Evaluate on math_find with verbose output
python eval_yarn_mistral.py \
    --task math_find \
    --model_path ../../../yarn-mistral-7b-128k \
    --data_dir ../data \
    --output_dir ../results \
    --verbose
# Prints token counts before/after truncation and generation results per example
```

--------------------------------

### Run Evaluation for Longbook Sum QA Task

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Execute the evaluation script for the longbook sum QA task. Ensure the dataset is in the 'data' folder or specify its location using the --data_dir argument.

```python
python eval_gpt4.py --task longbook_sum_qa
```

--------------------------------

### Compute Scores for Predictions

Source: https://context7.com/openbmb/infinitebench/llms.txt

Loads a JSONL predictions file, computes per-example scores using get_score_one, and prints the mean accuracy for the task. Use this to programmatically score predictions.

```python
from compute_scores import compute_scores
from pathlib import Path

# Score GPT-4 predictions on the KV retrieval task
compute_scores(
    preds_path=Path("../results/gpt4/preds_kv_retrieval.jsonl"),
    data_name="kv_retrieval",
    model_name="gpt4"
)
```

--------------------------------

### Run Evaluation for KV Retrieval Task

Source: https://github.com/openbmb/infinitebench/blob/main/README.md

Execute the evaluation script for the KV retrieval task. Ensure the dataset is in the 'data' folder or specify its location using the --data_dir argument.

```python
python eval_yarn_mistral.py --task kv_retrieval
```

--------------------------------

### Chunked Generation for YaRN-Mistral

Source: https://context7.com/openbmb/infinitebench/llms.txt

Processes inputs in chunks to build a KV cache without OOM errors, then generates answers. Useful for handling very long prompts with models like YaRN-Mistral.

```python
from eval_yarn_mistral import chunk_generate, load_model

model, tok = load_model("../../../yarn-mistral-7b-128k")

responses = chunk_generate(
    model=model,
    tok=tok,
    texts=["<very long 100k token prompt>"],
    max_tokens=6,         # For passkey: only need a short answer
    sliding_window=128 * 1024,
    chunk_size=2500,
    verbose=True,
)
print(responses[0])  # "84729"
```

--------------------------------

### Batch Score Predictions via CLI

Source: https://context7.com/openbmb/infinitebench/llms.txt

Use this script to batch score predictions for single tasks or all tasks for a given model directly from the command line.

```bash
cd src

# Score a single task
python compute_scores.py --task passkey --model_name gpt4 --output_dir ../results
# Output: 1.0

# Score all tasks for a model
python compute_scores.py --task all --model_name yarn-mistral --output_dir ../results
# Iterates over all 12 tasks and prints accuracy for each
```

--------------------------------

### Stream and Write JSONL Files with iter_jsonl / dump_jsonl

Source: https://context7.com/openbmb/infinitebench/llms.txt

Lazily reads JSON objects from a .jsonl file or writes a list of dictionaries to a JSONL file. Supports streaming large files with a count limit.

```python
from eval_utils import iter_jsonl, dump_jsonl

# Read existing predictions
preds = list(iter_jsonl("../results/gpt4/preds_passkey.jsonl"))
print(preds[0])
# {"id": 0, "prediction": "84729", "ground_truth": ["84729"]}

# Write new predictions
new_preds = [
    {"id": 0, "prediction": "84729", "ground_truth": ["84729"]},
    {"id": 1, "prediction": "31052", "ground_truth": ["31052"]},
]
dump_jsonl(new_preds, "../results/my_model/preds_passkey.jsonl")

# Stream large files with a count limit
for eg in iter_jsonl("../data/kv_retrieval.jsonl", cnt=10):
    print(eg["input"])  # prints first 10 examples only
```

--------------------------------

### Call the Anthropic Messages API with retry logic

Source: https://context7.com/openbmb/infinitebench/llms.txt

Utilize `chat_with_anthropic` for interacting with the Anthropic API, featuring built-in retry mechanisms for server errors. It accommodates large contexts and custom system prompts.

```python
from LLM_API_Calls import chat_with_anthropic

response = chat_with_anthropic(
    api_key="sk-ant-ாலத்தில்",
    input_data="[100k token book context]",
    model="claude-3-5-sonnet-20240620",
    custom_prompt_arg="Summarize the book in 3 sentences.",
    max_retries=3,
    retry_delay=5,
    system_prompt="You are a helpful assistant."
)
print(response)  # "The book follows Alice as she..."
```

--------------------------------

### chat_with_anthropic

Source: https://context7.com/openbmb/infinitebench/llms.txt

Interacts with Anthropic's Messages API, offering configurable retries for server errors and support for large context windows.

```APIDOC
## chat_with_anthropic

### Description
Posts to Anthropic's `/v1/messages` endpoint with configurable retries on server errors. Supports custom system prompts and up to 4096 output tokens.

### Method Signature
```python
chat_with_anthropic(api_key: str, input_data: str, model: str, custom_prompt_arg: str = "", max_retries: int = 3, retry_delay: int = 5, system_prompt: str = "") -> str
```

### Parameters
- **api_key** (str) - Your Anthropic API key.
- **input_data** (str) - The main input prompt or data for the model.
- **model** (str) - The specific Anthropic model to use (e.g., "claude-3-5-sonnet-20240620").
- **custom_prompt_arg** (str, optional) - Additional instructions or prompt elements. Defaults to "".
- **max_retries** (int, optional) - Maximum number of retries for server errors. Defaults to 3.
- **retry_delay** (int, optional) - Delay in seconds between retries. Defaults to 5.
- **system_prompt** (str, optional) - A system-level instruction for the assistant. Defaults to "".

### Request Example
```python
from LLM_API_Calls import chat_with_anthropic

response = chat_with_anthropic(
    api_key="sk-ant-...",
    input_data="[100k token book context]",
    model="claude-3-5-sonnet-20240620",
    custom_prompt_arg="Summarize the book in 3 sentences.",
    max_retries=3,
    retry_delay=5,
    system_prompt="You are a helpful assistant."
)
print(response)  # Expected output: "The book follows Alice as she..."
```
```

--------------------------------

### Truncate Token Sequences with truncate_input

Source: https://context7.com/openbmb/infinitebench/llms.txt

Performs middle-truncation on a list of tokens to a specified maximum length, preserving the beginning and end of the sequence. This is the standard truncation strategy used in the codebase.

```python
from eval_utils import truncate_input

tokens = list(range(200_000))  # simulate a long tokenized input

# Middle truncation to 128k tokens
truncated = truncate_input(tokens, max_length=128_000, manner="middle")
print(len(truncated))      # 128000
print(truncated[:3])       # [0, 1, 2]       — start preserved
print(truncated[-3:])      # [199997, 199998, 199999]  — end preserved
```

--------------------------------

### Normalize Text with normalize_answer / normalize_zh_answer

Source: https://context7.com/openbmb/infinitebench/llms.txt

Cleans English or Chinese text for scoring by lowercasing, removing articles, punctuation, and extra whitespace. Essential for F1 and exact-match scoring.

```python
from eval_utils import normalize_answer, normalize_zh_answer

# English normalization
print(normalize_answer("The Quick Brown Fox!"))   # "quick brown fox"
print(normalize_answer("A cat, and the dog."))    # "cat dog"

# Chinese normalization (removes CN punctuation, collapses whitespace)
print(normalize_zh_answer("这是一个测试。"))        # "这是一个测试"
print(normalize_zh_answer("你好！ 世 界"))          # "你好世界"
```

--------------------------------

### Compute ROUGE-L Score with rouge_score / rouge_zh_score

Source: https://context7.com/openbmb/infinitebench/llms.txt

Calculates the ROUGE-L F1 score between a predicted and reference text. The Chinese version uses `jieba` for word segmentation.

```python
from eval_utils import rouge_score, rouge_zh_score

# English ROUGE-L (used for En.Sum)
score = rouge_score(
    prediction="The fox jumped over the lazy dog near the river.",
    ground_truth="A quick fox leaped over a lazy dog."
)
print(round(score, 4))  # ~0.4615

# Chinese ROUGE-L with jieba tokenization (used for Zh.QA)
score = rouge_zh_score(
    prediction="这只狐狸跳过了那只懒狗",
    ground_truth="一只快速的狐狸跳过了一只懒惰的狗"
)
print(round(score, 4))  # varies by segmentation
```

--------------------------------

### Compute Single Prediction Score with get_score_one

Source: https://context7.com/openbmb/infinitebench/llms.txt

Dispatches to the appropriate scoring function for a given task, returning a score between 0.0 and 1.0. Handles exact match, multiple-choice, F1, and ROUGE scores.

```python
from compute_scores import get_score_one

# Passkey retrieval: exact first-integer match
score = get_score_one("The pass key is 84729.", ["84729"], "passkey", "gpt4")
print(score)   # 1.0

# Code debug: multiple-choice letter match
score = get_score_one("The answer is B. func_42", ["func_42", "B"], "code_debug", "gpt4")
print(score)   # 1.0

# Book QA: token-level F1 score
score = get_score_one("Alice went to the market", ["Alice visited the market"], "longbook_qa_eng", "gpt4")
print(round(score, 4))  # ~0.6667

# English summarization: rougeLsum
score = get_score_one(
    "A fox leapt over a dog in the meadow.",
    "The quick brown fox jumps over the lazy dog.",
    "longbook_sum_eng", "gpt4"
)
print(round(score, 4))  # varies
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.