### Clone and Install Repository

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Initial setup steps to clone the repository and install required dependencies.

```bash
git clone <repository-url>
cd <repository-name>
```

```bash
pip install -r requirements.txt
```

--------------------------------

### Configuration File Example

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Example structure of the config.json file, including API keys, model name mappings, and task-specific maximum lengths.

```json
{
    "keys": {
        "TOGETHER_API_KEY": "your_together_api_key_here",
        "OPENAI_API_KEY": "your_openai_api_key_here",
        "GEMINI_API_KEY": "your_gemini_api_key_here",
        "DEEPSEEK_API_KEY": "your_deepseek_api_key_here"
    },

    "model2str": {
        "gpt4-1106": "gpt-4-1106-preview",
        "gpt4o": "gpt-4o-2024-08-06",
        "gpt4omini": "gpt-4o-mini-2024-07-18",
        "llama70b": "meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
        "llama3b": "meta-llama/Llama-3.2-3B-Instruct-Turbo",
        "qwen72b": "Qwen/Qwen2.5-72B-Instruct-Turbo",
        "mistral7b": "mistralai/Mistral-7B-Instruct-v0.2"
    },

    "task2maxlen": {
        "math": 128,
        "kv": 128,
        "qalb": 128,
        "sum": 2048,
        "char": 128,
        "qaib": 128
    }
}
```

--------------------------------

### Example Workflows

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Common patterns for comparing inference strategies and batch processing.

```bash
# Single model inference
python inference_sing.py --model gpt4o --task kv --len 8000 --save_dir ../outputs

# Divide and conquer inference
python inference_dc.py --model gpt4o --task kv --len 8000 --ctx 4000 --save_dir ../outputs

# Evaluate results
python compute_scores.py --input_file ../outputs/kv-gpt4o-len8000.jsonl
python compute_scores.py --input_file ../outputs/kv-gpt4o-len8000-ctx4000.jsonl
```

```bash
# Run experiments for multiple models
for model in gpt4o llama70b qwen72b; do
    python inference_sing.py --model $model --task kv --len 8000 --save_dir ../outputs
    python inference_dc.py --model $model --task kv --len 8000 --ctx 4000 --save_dir ../outputs
done
```

--------------------------------

### Configure API Keys

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Setup the configuration file and define required API keys for model access.

```bash
cd src
cp config.json.template config.json
```

```json
{
    "keys": {
        "TOGETHER_API_KEY": "your_together_api_key_here",
        "OPENAI_API_KEY": "your_openai_api_key_here",
        "GEMINI_API_KEY": "your_gemini_api_key_here"
    }
}
```

--------------------------------

### Configuration Setup

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Template for the configuration file. Copy this template and edit it with your API keys and model mappings before running experiments. Ensure you are in the 'src' directory.

```bash
cd src
cp config.json.template config.json
```

--------------------------------

### Divide and Conquer Inference Output Format

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Example of the JSONL output format for divide-and-conquer inference, including context and input details.

```json
{"id": 0, "context": "JSON data:\n{\"0198fbb7...", "input": "...", "answer": "uuid-value", "pred": "uuid-value"}
```

--------------------------------

### Single Model Inference Output Format

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Example of the JSONL output format for single model inference results, saved to the specified directory.

```json
{"id": 0, "answer": "99c9a6b5-3328-4ab0-93cd-41796ff990d2", "pred": "99c9a6b5-3328-4ab0-93cd-41796ff990d2"}
{"id": 1, "answer": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", "pred": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}
```

--------------------------------

### Define JSONL Task Data Formats

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Examples of JSONL structures for different task types including retrieval, math, QA, summarization, character identification, and multiple-choice.

```json
// Key-Value Retrieval (kv_8000.jsonl)
{"id": 0, "context": "JSON data:\n{\"uuid-key-1\": \"uuid-value-1\", ...}", "input": "\nKey: \"uuid-key-1\"\nThe value associated with the specified key is: ", "answer": "uuid-value-1"}

// Math Reasoning (math_8000.jsonl)
{"id": 0, "context": [42, 15, 78, 23, 91, 56, 34, 67], "question": "What is the largest number?", "answer": 91}

// Question Answering on Books (qaib_120000.jsonl)
{"id": 0, "context": "Chapter 1: The story begins...", "input": "Who is the main protagonist?", "answer": ["John Smith", "John"]}

// Summarization (sum_120000.jsonl)
{"id": 0, "context": "The complete book text spanning many chapters...", "answer": "A concise summary of the book content."}

// Character Identification (char_120000.jsonl)
{"id": 0, "context": "JOHN: Hello $$MASK$$, how are you?\n$$MASK$$: I am fine, thank you.", "answer": ["Mary", "MARY"]}

// Multiple Choice QA (qalb_120000.jsonl)
{"id": 0, "context": "Document text...", "question": "What is the main theme?", "choice_A": "Love", "choice_B": "War", "choice_C": "Peace", "choice_D": "Nature", "answer": "A"}
```

--------------------------------

### Single Model Scoring Output Format

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Example of the CSV output format for scoring single model inference results, detailing task, context length, model, and accuracy.

```csv
task,task_length,model,accuracy
kv,1000,gpt4o,0.95
kv,2000,gpt4o,0.92
kv,4000,gpt4o,0.88
kv,8000,gpt4o,0.82
```

--------------------------------

### Run Batch Experiments

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Automate inference and scoring across multiple models and context configurations using shell scripts.

```bash
#!/bin/bash
cd src

# Run experiments for multiple models on key-value task
for model in gpt4o llama70b qwen72b; do
    # Single model baseline
    for len in 1000 2000 4000 8000; do
        python inference_sing.py --model $model --task kv --len $len --save_dir ../outputs --n_proc 16
    done

    # Divide-and-conquer with varying context windows
    for ctx in 1000 2000 4000; do
        python inference_dc.py --model $model --task kv --len 8000 --ctx $ctx --save_dir ../outputs --n_proc 16
    done
done

# Compute all scores
for model in gpt4o llama70b qwen72b; do
    python score_sing.py --model $model --task kv
    python score_dc.py --model $model --task kv
done

# Compare results across different tasks
for task in kv math qaib sum char; do
    python inference_sing.py --model gpt4o --task $task --len 8000 --save_dir ../outputs
    python inference_dc.py --model gpt4o --task $task --len 8000 --ctx 4000 --save_dir ../outputs
done
```

--------------------------------

### Run Inference Strategies

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Execute single-model or divide-and-conquer inference tasks.

```bash
cd src
python inference_sing.py --model MODEL_NAME --task TASK --len CONTEXT_LENGTH --save_dir ../outputs --n_proc 16
```

```bash
cd src
python inference_dc.py --model MODEL_NAME --task TASK --len CONTEXT_LENGTH --ctx CONTEXT_WINDOW --save_dir ../outputs --n_proc 16
```

--------------------------------

### Generate Synthetic Datasets

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Create evaluation datasets and calculate token lengths for specific context requirements.

```python
from data import gen_math_data, gen_kv_data, tokenized_length
import json

# Generate math reasoning data with specified context length
# Creates questions asking for largest/smallest numbers
math_data = gen_math_data(len=8000, samples=100)
print(f"Generated {len(math_data)} math samples")
# Sample output:
# {"id": 0, "context": [42, 15, 78, 23, ...], "question": "What is the largest number?", "answer": 92}

# Save generated data to JSONL
with open("../data/math_8000.jsonl", "w") as f:
    for item in math_data:
        f.write(json.dumps(item) + "\n")

# Check tokenized length of text
sample_text = "This is a sample context for evaluation."
tokens = tokenized_length(sample_text)
print(f"Token count: {tokens}")  # Output: Token count: 9

# Generate key-value retrieval data (run gen_kv_data() to create files)
# Creates files: kv_1000.jsonl, kv_2000.jsonl, ..., kv_120000.jsonl
# Each file contains JSON key-value pairs with retrieval questions
```

--------------------------------

### Run Single Model Inference

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Execute inference using a single model on the entire context. Use this for baseline comparisons or when models can handle the full context length directly. Ensure you are in the 'src' directory.

```bash
cd src
python inference_sing.py --model gpt4o --task kv --len 8000 --save_dir ../outputs --n_proc 16
```

```bash
python inference_sing.py --model llama70b --task math --len 4000 --save_dir ../outputs --n_proc 8
```

```bash
python inference_sing.py --model gpt4omini --task sum --len 120000 --save_dir ../outputs --n_proc 4
```

--------------------------------

### Define Output Format

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Structure of the generated JSONL output files.

```json
{
    "id": 0,
    "answer": "ground_truth_answer",
    "pred": "model_prediction",
    "context": "truncated_context_preview..."
}
```

--------------------------------

### Run Divide and Conquer Inference

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Execute inference using the divide-and-conquer strategy, splitting contexts into chunks for worker agents. This is suitable for contexts exceeding native model limits. Ensure you are in the 'src' directory.

```bash
cd src
python inference_dc.py --model gpt4o --task kv --len 8000 --ctx 4000 --save_dir ../outputs --n_proc 16
```

```bash
python inference_dc.py --model llama70b --task qaib --len 120000 --ctx 8000 --save_dir ../outputs --n_proc 8
```

```bash
python inference_dc.py --model qwen72b --task char --len 60000 --ctx 4000 --save_dir ../outputs --n_proc 4
```

--------------------------------

### Compute Scores via CLI

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Execute evaluation scripts for different tasks and models using the command line interface.

```bash
# Compute scores for divide-and-conquer key-value results
cd src
python score_dc.py --model gpt4o --task kv

# Compute scores for QA task
python score_dc.py --model llama70b --task qaib

# Compute scores for character identification
python score_dc.py --model qwen72b --task char

# Output CSV format (../outputs/results-kv-gpt4o.csv):
# task,task_length,model,ctx,accuracy
# kv,128000,gpt4o,1000,0.75
# kv,128000,gpt4o,2000,0.82
# kv,128000,gpt4o,4000,0.88
# kv,128000,gpt4o,8000,0.91
```

--------------------------------

### Evaluate Results

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Compute performance scores for generated inference outputs.

```bash
cd src
python compute_scores.py --input_file ../outputs/RESULT_FILE.jsonl
```

--------------------------------

### Compute Metrics Programmatically

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Use the scoring module to evaluate model outputs using F1, ROUGE, and exact match metrics.

```python
from compute_scores import (
    get_score_one_kv_retrieval,
    get_score_one_longbook_sum_eng,
    get_score_one_longbook_qa_eng,
    qa_f1_score,
    normalize_answer
)

# Key-value retrieval scoring (exact match in tokens)
pred = "The value is 99c9a6b5-3328-4ab0-93cd-41796ff990d2"
label = "99c9a6b5-3328-4ab0-93cd-41796ff990d2"
score = get_score_one_kv_retrieval(pred, label, model_name="gpt4o")
print(f"KV Score: {score}")  # Output: KV Score: True

# QA F1 scoring with answer normalization
pred = "The main character is Elizabeth Bennet"
labels = ["Elizabeth Bennet", "Elizabeth", "Lizzy"]
f1 = qa_f1_score(pred, labels)
print(f"QA F1: {f1:.3f}")  # Output: QA F1: 1.000

# Summarization scoring using ROUGE
pred_summary = "The story follows a young wizard who discovers his magical powers."
ref_summary = "A young wizard discovers he has magical abilities and attends wizard school."
rouge_score = get_score_one_longbook_sum_eng(pred_summary, ref_summary, model_name="gpt4o")
print(f"ROUGE-L: {rouge_score:.3f}")  # Output: ROUGE-L: 0.421

# Answer normalization for consistent comparison
raw_answer = "The   answer is: A simple response!"
normalized = normalize_answer(raw_answer)
print(f"Normalized: '{normalized}'")  # Output: Normalized: 'answer is simple response'
```

--------------------------------

### Define Data Format

Source: https://github.com/nehzux/divideconqueragents/blob/main/README.md

Structure for input data files in JSONL format.

```json
{
    "id": 0,
    "context": "...",
    "input": "...", 
    "answer": "..."
}
```

--------------------------------

### Compute Single Model Scores

Source: https://context7.com/nehzux/divideconqueragents/llms.txt

Calculate and export accuracy scores for single model inference results. This script helps in evaluating performance across different models and tasks. Ensure you are in the 'src' directory.

```bash
cd src
python score_sing.py --model gpt4o --task kv
```

```bash
python score_sing.py --model llama70b --task math
```

```bash
python score_sing.py --model gpt4omini --task sum
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.