### Install Evaluation Dependencies

Source: https://github.com/ccm0111/adafuse/blob/main/evaluation/README.md

Command to install the necessary Python packages required to run the evaluation scripts, including libraries for number conversion and BLEU score calculation.

```bash
pip install word2number sacrebleu
```

--------------------------------

### Execute Evaluation Scripts

Source: https://github.com/ccm0111/adafuse/blob/main/evaluation/README.md

Demonstrates how to run the various evaluation scripts provided in the AdaFuse project. These scripts typically require a predictions file and sometimes a reference file to compute performance metrics.

```bash
python eval_gsm.py --pred predictions.jsonl
python eval_nq.py --pred predictions.jsonl --ref references.jsonl
python eval_trivia.py --pred predictions.jsonl --ref references.jsonl
python eval_squad.py --pred predictions.jsonl --ref references.jsonl
python eval_spbleu.py --input predictions.jsonl [--lowercase]
```

--------------------------------

### AdaFuse Key Parameter Configuration

Source: https://context7.com/ccm0111/adafuse/llms.txt

Provides a reference for key configuration parameters used across all AdaFuse scripts. These parameters control the adaptive decoding behavior, including confidence thresholds, generation step sizes, beam search dimensions, and token limits.

```bash
# Parameter reference for all AdaFuse scripts:

--theta_delta 0.7        # Confidence threshold (0.0-1.0)
                         # Higher = more conservative, fewer ensemble decisions
                         # Lower = more exploration, more ensemble decisions
                         # Recommended: 0.6-0.8

--max_words 3            # Max words per generation step (2 or 3)
                         # 3 = faster but may miss optimal boundaries
                         # 2 = more granular control

--beam_size 3            # Beam size for diversity exploration
                         # Only used when confidence < theta_delta

--max_span_rounds 5      # Maximum decoding iterations per sample
                         # Increase for longer answers (e.g., 10 for GSM8K)

--max_total_tokens 512   # Maximum total tokens for generation
--max_new_tokens 10      # Max new tokens per beam search step
--per_device_batch_size 1  # Batch size per GPU (keep at 1 for ensemble)
```

--------------------------------

### Run Four-Model Ensemble Decoding with AdaFuse

Source: https://context7.com/ccm0111/adafuse/llms.txt

Configures AdaFuse for maximum model diversity using four LLMs, ideal for complex tasks requiring multiple perspectives, such as machine translation. This script utilizes adaptive confidence-based decoding and requires specifying paths for datasets, prompts, models, and output files, along with ensemble parameters.

```bash
python AdaFuse_four_models.py \
  --test_set local_datasets/Flores/flores_eng2deu_qa.devtest.jsonl \
  --prompts local_datasets/Flores/flores_en2deu_prompt.txt \
  --model_path1 meta-llama/Llama-3.1-8B-Instruct \
  --model_path2 internlm/internlm3-8b-instruct \
  --model_path3 Qwen/Qwen2.5-7B-Instruct \
  --model_path4 mistralai/Mistral-7B-Instruct-v0.3 \
  --output_file flores_predictions.jsonl \
  --theta_delta 0.6 \
  --beam_size 3
```

--------------------------------

### Run Three-Model Ensemble Decoding with AdaFuse

Source: https://context7.com/ccm0111/adafuse/llms.txt

Implements AdaFuse ensemble decoding using three LLMs to enhance diversity and accuracy. It employs the same adaptive confidence-based strategy with cross-model perplexity scoring, suitable for tasks like arithmetic reasoning. Key parameters include model paths, dataset configurations, and adaptive decoding controls.

```bash
python AdaFuse_three_models.py \
  --test_set local_datasets/GSM/test.cleand.jsonl \
  --prompts local_datasets/GSM/gsm_prompt.txt \
  --model_path1 meta-llama/Llama-3.1-8B-Instruct \
  --model_path2 internlm/internlm3-8b-instruct \
  --model_path3 Qwen/Qwen2.5-7B-Instruct \
  --output_file gsm_predictions.jsonl \
  --theta_delta 0.7 \
  --max_words 3 \
  --max_span_rounds 10
```

--------------------------------

### Run Two-Model Ensemble Decoding with AdaFuse

Source: https://context7.com/ccm0111/adafuse/llms.txt

Executes AdaFuse with two LLMs for ensemble decoding. This script supports multi-GPU processing and automatic result merging, suitable for tasks like question answering. It takes dataset paths, model paths, and output file as input, along with parameters controlling the adaptive strategy.

```bash
python AdaFuse_two_models.py \
  --test_set local_datasets/NaturalQuestions/test/natural-questions.jsonl \
  --prompts local_datasets/NaturalQuestions/dev/nq_prompt.txt \
  --model_path1 meta-llama/Llama-3.1-8B-Instruct \
  --model_path2 internlm/internlm3-8b-instruct \
  --output_file nq_predictions.jsonl \
  --theta_delta 0.7 \
  --max_words 3 \
  --max_total_tokens 512
```

--------------------------------

### SQuAD and BLEU Evaluation

Source: https://context7.com/ccm0111/adafuse/llms.txt

CLI commands for evaluating SQuAD reading comprehension and computing spBLEU scores for machine translation.

```bash
python evaluation/eval_squad.py --pred squad_predictions.jsonl --ref local_datasets/SQUAD/squad_2500.jsonl
python evaluation/eval_spbleu.py --input flores_predictions.jsonl --lowercase
```

--------------------------------

### Standard JSONL Input Format

Source: https://github.com/ccm0111/adafuse/blob/main/evaluation/README.md

Defines the expected JSONL structure for input files across all evaluation scripts. Each line represents a single data point containing the question, the ground truth answer, and the model prediction.

```json
{
  "question": "What is the capital of France?",
  "answer": "Paris",
  "prediction": "Paris"
}
```

--------------------------------

### Natural Questions Evaluation

Source: https://context7.com/ccm0111/adafuse/llms.txt

Evaluates open-domain QA using lenient matching. Includes CLI execution and programmatic usage for answer normalization and exact match scoring.

```bash
python evaluation/eval_nq.py --pred nq_predictions.jsonl --ref local_datasets/NaturalQuestions/test/natural-questions.jsonl
```

```python
from evaluation.eval_nq import exact_match_score, normalize_answer
exact_match_score("twenty three", "23")
normalize_answer("The United States of America")
```

--------------------------------

### Evaluate GSM8K Arithmetic Reasoning with AdaFuse Predictions

Source: https://context7.com/ccm0111/adafuse/llms.txt

Evaluates the accuracy of AdaFuse predictions on the GSM8K dataset, specifically focusing on arithmetic reasoning tasks. It processes prediction files and outputs accuracy metrics. The script also includes helper functions for programmatic evaluation and answer extraction.

```bash
python evaluation/eval_gsm.py --pred gsm_predictions.jsonl
```

```python
# Programmatic usage
from evaluation.eval_gsm import compute_accuracy, extract_boxed_answer, normalize

# Extract answer from boxed format
pred = "So the total is 50+24+10=84. The answer is \\boxed{84}"
answer = extract_boxed_answer(pred)  # Returns: "84"
```

--------------------------------

### TriviaQA Evaluation

Source: https://context7.com/ccm0111/adafuse/llms.txt

Evaluates trivia QA with fuzzy matching for aliases. Provides CLI for batch evaluation and programmatic functions for loading data and checking correctness.

```bash
python evaluation/eval_trivia.py --pred trivia_predictions.jsonl --ref local_datasets/TriviaQA/wikipedia-test-6000.jsonl
```

```python
from evaluation.eval_trivia import is_correct, load_references, load_predictions
from pathlib import Path
refs = load_references(Path("references.jsonl"))
preds = load_predictions(Path("predictions.jsonl"))
is_correct(["George Washington"], "washington", is_regex=False)
```

--------------------------------

### Core Scoring and Utility Functions

Source: https://context7.com/ccm0111/adafuse/llms.txt

Internal functions for calculating confidence margins between top-2 predictions and computing span perplexity using transformer models.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
def compute_top1_margin(logits):
    probs = torch.nn.functional.softmax(logits, dim=-1)
    top_probs, _ = torch.topk(probs, k=2, dim=-1)
    return (top_probs[0, 0] - top_probs[0, 1]).item()

def compute_span_perplexity(model, tokenizer, prefix, span_tokens):
    inputs = tokenizer(prefix, return_tensors='pt').to(model.device)
    # ... (omitted implementation details for brevity)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.