XtraGPT

https://github.com/xtra-computing/xtragpt
Admin
XtraGPT is a family of open-source Large Language Models designed specifically for human-AI...
Tokens:8,419
Snippets:114
Trust Score:9
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install the package

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Install the library via pip.

```bash
pip install paper-revision-bench
```

--------------------------------

### Install and initialize the package

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Install the npm package and initialize the project structure.

```bash
npm install xtragpt-paper-revision-skill
npx xtragpt-paper-revision-skill init
```

--------------------------------

### Setup AlpacaEval for Component-wise Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Clones the AlpacaEval repository, installs dependencies, and applies custom configurations for paper revision tasks.

```bash
# Clone and install AlpacaEval
git clone https://github.com/tatsu-lab/alpaca_eval.git
cd alpaca_eval && pip install -e .

# Copy our modified configs
cp -r ../6_component_evaluation/alpaca_eval_gpt4_turbo_fn/* \
    src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/
```

```bash
cp 6_component_evaluation/glm_winrate.py $(python -c "import alpaca_eval; print(alpaca_eval.__path__[0])")/metrics/glm_winrate.py
```

--------------------------------

### Setup AI-Scientist for Full Paper Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Clones and installs the AI-Scientist repository for full paper evaluation.

```bash
git clone https://github.com/SakanaAI/AI-Scientist.git
cd AI-Scientist && pip install -e .
```

--------------------------------

### Reproduce the demo locally

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Commands to start the local server, run the demo script, and replay the session.

```bash
# from repo root
bash tests/openclaw-demo/start_vllm.sh          # serves XtraGPT-7B on :8088 and Qwen2.5-7B on :8089
python tests/openclaw-demo/run_demo.py          # runs the three backends with the skill's prompt template
asciinema play tests/openclaw-demo/demo.cast    # replay the recorded session
```

--------------------------------

### Install AlpacaEval

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Clone the AlpacaEval repository and install it in editable mode.

```bash
git clone https://github.com/tatsu-lab/alpaca_eval.git
cd alpaca_eval
pip install -e .
```

--------------------------------

### Install XtraGPT dependencies

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Clone the repository and install the required Python packages.

```bash
# Clone repository
git clone https://github.com/Xtra-Computing/XtraGPT.git
cd XtraGPT

# Install dependencies
pip install -r requirements.txt

# For training, also install LLaMA-Factory
pip install llamafactory
```

--------------------------------

### Install AI-Scientist

Source: https://github.com/xtra-computing/xtragpt/blob/main/full_paper_evaluation/README.md

Clone the repository and install the package in editable mode to set up the environment.

```bash
git clone https://github.com/SakanaAI/AI-Scientist.git
cd AI-Scientist
pip install -e .
```

--------------------------------

### Install LLaMA-Factory

Source: https://github.com/xtra-computing/xtragpt/blob/main/train/README.md

Install the required LLaMA-Factory package via pip.

```bash
pip install llamafactory
```

--------------------------------

### Quick Start evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Perform a basic evaluation of revised text against original text using a judge model.

```python
from paper_revision_bench import evaluate

results = evaluate(
    original_texts=["The dominant sequence transduction models are based on complex recurrent or convolutional neural networks."],
    revised_texts=["Sequence transduction models typically use complex RNNs or CNNs."],
    section="abstract",
    judge_model="gpt-4-1106-preview",  # matches paper
    api_key="sk-xxx",  # or set OPENAI_API_KEY env var
)

print(f"Win Rate: {results.win_rate:.1%}")
print(results.summary())
```

--------------------------------

### Evaluate Models with paper-revision-bench

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Example of running an evaluation using GPT-4-Turbo as a judge.

```python
import paper_revision_bench as prb

# Prepare your data
samples = [
    {
        "instruction": "Improve the clarity of this title",
        "input": "A Study of Neural Networks",
        "output_1": "Deep Learning for Image Classification",  # Model A output
        "output_2": "Neural Network Analysis Study",           # Model B output
    }
]

# Run evaluation with GPT-4-Turbo as judge
results = prb.evaluate(
    samples=samples,
    judge="openai/gpt-4-turbo",
    criteria="clarity"
)

print(f"Model A win rate: {results.win_rate:.1%}")
```

--------------------------------

### Evaluation Output Format

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Example of the JSON structure produced by the evaluation process.

```json
{
  "win_rate": 65.2,
  "standard_error": 1.3,
  "n_total": 1400
}
```

--------------------------------

### Scaffold into a specific directory

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Initialize the package in a custom project path.

```bash
npx xtragpt-paper-revision-skill init --dir /path/to/project
```

--------------------------------

### Run training with scripts or CLI

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Execute training using the provided shell script or the LLaMA-Factory CLI.

```bash
# Set environment variables
export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct"  # Base model
export OUTPUT_DIR="./output/xtragpt-7b"        # Output directory

# Run training
bash scripts/train.sh
```

```bash
llamafactory-cli train configs/train_config.yaml
```

--------------------------------

### Overwrite existing files

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Force initialization to overwrite existing configuration files.

```bash
npx xtragpt-paper-revision-skill init --force
```

--------------------------------

### Train using LLaMA-Factory CLI

Source: https://github.com/xtra-computing/xtragpt/blob/main/train/README.md

Execute training directly using the LLaMA-Factory CLI with a specified configuration file.

```bash
llamafactory-cli train ../configs/train_config.yaml
```

--------------------------------

### Configure Dataset

Source: https://github.com/xtra-computing/xtragpt/blob/main/train/README.md

Copy the dataset configuration file to the LLaMA-Factory data directory.

```bash
cp data/data.json /path/to/LLaMA-Factory/data/dataset_info.json
```

--------------------------------

### Train using provided script

Source: https://github.com/xtra-computing/xtragpt/blob/main/train/README.md

Set environment variables for the model path and output directory before executing the training script.

```bash
# Set your base model and output directory
export MODEL_PATH="Qwen/Qwen2.5-7B-Instruct"
export OUTPUT_DIR="./output/xtragpt-7b"

bash ../scripts/train.sh
```

--------------------------------

### Set environment variables

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Configure the base URL and API key for the XtraGPT service before running OpenClaw.

```bash
export XTRAGPT_BASE_URL=http://127.0.0.1:8088/v1
export XTRAGPT_API_KEY=dummy
```

--------------------------------

### Project Structure Overview

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Visual representation of the XtraGPT repository file structure.

```text
XtraGPT/
├── configs/
│   ├── train_config.yaml      # Training configuration
│   └── dataset_info.json      # Dataset configuration for LLaMA-Factory
├── scripts/
│   ├── train.sh               # Training script
│   └── predict.sh             # Inference script
├── 6_component_evaluation/    # Component-wise evaluation
│   ├── alpaca_eval_gpt4_turbo_fn/
│   ├── convert_predictions.py
│   └── run_eval.sh
├── full_paper_evaluation/     # Full paper evaluation
│   ├── ai_scientist_eval.py
│   ├── analyze_results.py
│   └── paper_results/
├── train/
│   ├── data/
│   └── README.md
├── examples/
│   └── inference_example.py
├── xtragpt-paper-revision-skill/
│   └── skills/skill.xtragpt-paper-revision-skill.yaml
├── requirements.txt
└── README.md
```

--------------------------------

### Recommended Usage Pattern

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

The standard workflow for combining general LLMs with XtraGPT for academic writing.

```text
General LLM → drafting / reasoning  
XtraGPT     → in-context revision / alignment
```

--------------------------------

### Scaffolded file structure

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

The directory structure created by the initialization command.

```text
openclaw/
├── openclaw.config.example.yaml
├── providers/
│   └── provider.xtragpt.yaml
├── routers/
│   └── router.auto_route_rules.yaml
└── skills/
    └── skill.xtragpt-paper-revision-skill.yaml
```

--------------------------------

### CLI Commands

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Command line interface for running evaluations and listing supported configurations.

```APIDOC
## paper-revision-bench evaluate
Runs evaluation from a JSON input file.
- **--input**: Path to JSON file.
- **--judge**: Model ID of the judge.
- **--criteria**: Evaluation criteria.
- **--output**: Path to save results.

## paper-revision-bench list-criteria
Lists all available evaluation criteria.

## paper-revision-bench list-judges
Lists all supported LLM judges.
```

--------------------------------

### Run Component-wise Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Executes the evaluation script after setting the required OpenAI API key.

```bash
export OPENAI_API_KEY="your-api-key"

bash 6_component_evaluation/run_eval.sh \
    ./formatted_predictions/xtragpt \
    ./formatted_predictions/baseline \
    ./eval_results
```

--------------------------------

### XtraGPT Workflow Visualization

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

A diagram illustrating the request routing process from the agent to the XtraGPT revision skill.

```text
User request
   ↓
Agent router detects "revision intent"
   ↓
xtragpt-paper-revision-skill skill
   ↓
XtraGPT
   ↓
Revised academic text
```

--------------------------------

### Analyze Before/After Improvements

Source: https://github.com/xtra-computing/xtragpt/blob/main/full_paper_evaluation/README.md

Compare evaluation results from two different JSON files to quantify improvements.

```bash
python analyze_results.py \
    --before before_revision_scores.json \
    --after after_revision_scores.json \
    --output analysis.json
```

--------------------------------

### User Prompt for Model Ranking

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/alpaca_eval_gpt4_turbo_fn/modified_alpaca_eval_fn.txt

Provides the instruction and model outputs to be evaluated, requesting a structured JSON output.

```json
## Prompt

{
    "instruction": """{instruction}""",
}

## Model Outputs

Here are the unordered outputs from the models. Each output is associated with a specific model, identified by a unique model identifier.

{
    {
        "model": "m",
        "output": """{output_1}"""
    },
    {
        "model": "M",
        "output": """{output_2}"""
    }
}

## Task

Evaluate and rank the models based on the quality and relevance of their outputs. The ranking should be such that the model with the highest quality output is ranked first.
```

--------------------------------

### Run Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Execute the evaluation script comparing model output against a baseline.

```bash
bash run_eval.sh \
    /path/to/model/formatted \
    /path/to/baseline/formatted \
    /path/to/output
```

--------------------------------

### Run Inference with XtraGPT

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Loads an XtraGPT model and tokenizer, formats input using the required prompt template, and generates a response.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Select the model size: "XtraGPT-1.5B", "XtraGPT-3B", "XtraGPT-7B", or "XtraGPT-14B"
model_name = "Xtra-Computing/XtraGPT-7B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Define the Prompt Template tailored for XtraGPT
prompt_template = """Act as an expert model for improving articles **PAPER_CONTENT**.
The output needs to answer the **QUESTION** on **SELECTED_CONTENT** in the input. Avoid adding unnecessary length, unrelated details, overclaims, or vague statements.
Focus on clear, concise, and evidence-based improvements that align with the overall context of the paper.
<PAPER_CONTENT>
{paper_content}
</PAPER_CONTENT>
<SELECTED_CONTENT>
{selected_content}
</SELECTED_CONTENT>
<QUESTION>
{user_question}
</QUESTION>"""

# Example Data (from the "Attention Is All You Need" paper)
paper_content = "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train."
selected_content = "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration."
user_question = "help me make it more concise."

# Format the input
formatted_prompt = prompt_template.format(
    paper_content=paper_content,
    selected_content=selected_content,
    user_question=user_question
)

messages = [
    {"role": "user", "content": formatted_prompt}
]

# Apply chat template
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# Generate
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
    temperature=0.1
)

generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```

--------------------------------

### Basic Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Perform a standard evaluation with optional instructions.

```python
from paper_revision_bench import evaluate

results = evaluate(
    original_texts=["Original text 1", "Original text 2"],
    revised_texts=["Revised text 1", "Revised text 2"],
    instructions=["Improve clarity", "Improve clarity"],  # optional
    section="abstract",  # title, abstract, introduction, background, evaluation, conclusion
    judge_model="gpt-4-1106-preview",
)
```

--------------------------------

### Copy Modified Configs

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Replace default AlpacaEval configurations with project-specific versions.

```bash
cp -r alpaca_eval_configs/* \
    /path/to/alpaca_eval/src/alpaca_eval/evaluators_configs/alpaca_eval_gpt4_turbo_fn/
```

--------------------------------

### CLI Usage for Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Commands for running evaluations and listing available criteria or judges via the CLI.

```bash
# Evaluate from JSON file
paper-revision-bench evaluate \
    --input samples.json \
    --judge openai/gpt-4-turbo \
    --criteria clarity \
    --output results.json

# List available criteria and judges
paper-revision-bench list-criteria
paper-revision-bench list-judges
```

--------------------------------

### Set OpenAI API Key

Source: https://github.com/xtra-computing/xtragpt/blob/main/full_paper_evaluation/README.md

Configure the environment variable required for LLM-based evaluation.

```bash
export OPENAI_API_KEY="your-api-key"
```

--------------------------------

### Detailed Reports

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Access evaluation metrics and export results to files.

```python
results = evaluate(...)

print(results.win_rate)    # 0.85
print(results.lose_rate)   # 0.10
print(results.tie_rate)    # 0.05

for detail in results.details:
    print(f"Sample {detail.index}: {detail.winner} - {detail.explanation}")

results.to_json("report.json")
results.to_csv("report.csv")
```

--------------------------------

### Define Evaluation Prompt Structure

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/alpaca_eval_gpt4_turbo_fn/alpaca_eval_fn.txt

The instruction field expects the prompt text used to generate the model responses.

```json
{
    "instruction": """{instruction}""",
}
```

--------------------------------

### Generate Predictions

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Run inference using the XtraGPT model to generate prediction files.

```bash
# Using XtraGPT
MODEL_PATH="Xtra-Computing/XtraGPT-7B" bash ../../scripts/predict.sh
```

--------------------------------

### Citation

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

BibTeX citation for the XtraGPT paper.

```bibtex
@misc{nuo2025xtragpt,
      title={XtraGPT: LLMs for Human-AI Collaboration on Controllable Academic Paper Revision},
      author={Nuo Chen and Andre Lin HuiKai and Jiaying Wu and Junyi Hou and Zining Zhang and Qian Wang and Xidong Wang and Bingsheng He},
      year={2025},
      eprint={2505.11336},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}
```

--------------------------------

### Run Full Paper Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Executes the AI-Scientist evaluation script on a specific paper PDF.

```bash
export OPENAI_API_KEY="your-api-key"

python full_paper_evaluation/ai_scientist_eval.py \
    --paper_path ./papers/my_paper.pdf \
    --output ./review_results.json \
    --model "gpt-4o"
```

--------------------------------

### Convert Predictions for Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Formats prediction files for the evaluation pipeline using the conversion script.

```bash
python 6_component_evaluation/convert_predictions.py \
    --input_dir ./predictions \
    --output_dir ./formatted_predictions \
    --model_name "XtraGPT-7B"
```

--------------------------------

### Convert Predictions to AlpacaEval Format

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/README.md

Transform prediction files into the format expected by the evaluation framework.

```bash
python convert_predictions.py \
    --input_dir /path/to/predictions \
    --output_dir /path/to/formatted \
    --model_name "XtraGPT-7B"
```

--------------------------------

### Configure self-hosted XtraGPT endpoint

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

Define the provider configuration for an OpenAI-compatible XtraGPT endpoint.

```yaml
model:
  provider: openai_compatible
  base_url: http://127.0.0.1:8088/v1
  model: Xtra-Computing/XtraGPT-7B
```

--------------------------------

### Retrieve Paper-Specific Prompts

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Access predefined evaluation prompts for specific paper sections.

```python
from paper_revision_bench import get_paper_eval_prompt, list_paper_sections

# Available sections: title, abstract, introduction, background, evaluation, conclusion
print(list_paper_sections())

# Get the evaluation prompt for a specific section
prompt = get_paper_eval_prompt("title")
```

--------------------------------

### Evaluate a Single Paper

Source: https://github.com/xtra-computing/xtragpt/blob/main/full_paper_evaluation/README.md

Run the evaluation script on a specific PDF file and save the output to a JSON file.

```bash
python ai_scientist_eval.py \
    --paper_path ./my_paper.pdf \
    --output ./review.json \
    --model gpt-4o
```

--------------------------------

### Async Batch Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Run evaluations concurrently using asyncio.

```python
from paper_revision_bench import evaluate_async
import asyncio

async def main():
    results = await evaluate_async(
        original_texts=large_original_list,
        revised_texts=large_revised_list,
        section="introduction",
        max_concurrent=5,
    )
    print(results.summary())

asyncio.run(main())
```

--------------------------------

### Served provider model name

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

The specific model name expected by the provider.

```text
Xtra-Computing/XtraGPT-7B
```

--------------------------------

### Paper-Specific Prompt Utilities

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Functions to retrieve evaluation prompts and list available paper sections.

```APIDOC
## list_paper_sections()
Returns a list of available paper sections for evaluation.

## get_paper_eval_prompt(section)
Returns the evaluation prompt template for a specific paper section (e.g., 'title', 'abstract').
```

--------------------------------

### evaluate_async()

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Performs an asynchronous batch evaluation of revised texts.

```APIDOC
## evaluate_async(original_texts, revised_texts, section='abstract', max_concurrent=5)

### Description
Asynchronously evaluates large batches of text revisions.

### Parameters
- **original_texts** (List[str]) - Required - Baseline texts.
- **revised_texts** (List[str]) - Required - Revised texts.
- **section** (str) - Optional - Paper section. Default: "abstract".
- **max_concurrent** (int) - Optional - Concurrent API calls. Default: 5.

### Returns
- **EvaluationResult** - An object containing evaluation metrics.
```

--------------------------------

### Define Model Outputs Structure

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/alpaca_eval_gpt4_turbo_fn/alpaca_eval_fn.txt

The model outputs are provided as a set of objects, each containing a model identifier and the corresponding output text.

```json
{
    {
        "model": "m",
        "output": """{output_1}"""
    },
    {
        "model": "M",
        "output": """{output_2}"""
    }
}
```

--------------------------------

### System Prompt for LLM Evaluation

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/alpaca_eval_gpt4_turbo_fn/modified_alpaca_eval_fn.txt

Defines the role of the evaluator to rank model responses based on human preference.

```text
<|im_start|>system
You are a highly efficient assistant, who evaluates and rank large language models (LLMs) based on the quality of their responses to given prompts. This process will create a leaderboard reflecting the most accurate and human-preferred answers.
<|im_end|>
```

--------------------------------

### evaluate()

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Performs a synchronous evaluation of revised texts against original texts using a specified judge model.

```APIDOC
## evaluate(original_texts, revised_texts, instructions=None, section='abstract', judge_model='gpt-4-1106-preview', api_key=None, temperature=0.0, max_tokens=200, max_concurrent=5)

### Description
Evaluates the quality of revised texts compared to original baseline texts using an LLM judge.

### Parameters
- **original_texts** (List[str]) - Required - Baseline texts (model "m" / output_1).
- **revised_texts** (List[str]) - Required - Model outputs to evaluate (model "M" / output_2).
- **instructions** (List[str]) - Optional - Revision instructions per sample.
- **section** (str) - Optional - Paper section (title, abstract, introduction, background, evaluation, conclusion). Default: "abstract".
- **judge_model** (str) - Optional - OpenAI model to use for judging. Default: "gpt-4-1106-preview".
- **api_key** (str) - Optional - OpenAI API key.
- **temperature** (float) - Optional - Default: 0.0.
- **max_tokens** (int) - Optional - Default: 200.
- **max_concurrent** (int) - Optional - Concurrent API calls. Default: 5.

### Returns
- **EvaluationResult** - An object containing win/loss/tie rates and detailed metrics.
```

--------------------------------

### prb.evaluate(samples, judge, criteria)

Source: https://github.com/xtra-computing/xtragpt/blob/main/README.md

Evaluates model outputs using a specified judge and criteria.

```APIDOC
## prb.evaluate

### Description
Evaluates a list of samples using an LLM judge to determine model performance based on specific criteria.

### Parameters
- **samples** (list) - Required - A list of dictionaries containing 'instruction', 'input', 'output_1', and 'output_2'.
- **judge** (string) - Required - The model ID of the judge (e.g., 'openai/gpt-4-turbo').
- **criteria** (string) - Required - The evaluation criteria (e.g., 'clarity').

### Returns
- **results** (object) - An object containing evaluation metrics such as 'win_rate'.
```

--------------------------------

### Length-Controlled Win Rate

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Calculate length-controlled win rate to account for length bias.

```bash
pip install paper-revision-bench[alpaca]
```

```python
lc = results.length_controlled_winrate(model_name="XtraGPT-7B", baseline_name="original")
print(f"LC Win Rate: {lc['length_controlled_winrate']:.1f}% ± {lc['lc_standard_error']:.1f}%")
```

--------------------------------

### compute_weighted_overall()

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Computes an overall win rate as a weighted average across different paper sections.

```APIDOC
## compute_weighted_overall(section_results)

### Description
Calculates the weighted overall win rate based on results from multiple paper sections.

### Parameters
- **section_results** (dict) - Required - A dictionary mapping section names to their respective EvaluationResult objects.
```

--------------------------------

### Skill ID

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

The identifier used for router rules or manual skill invocations.

```text
xtragpt-paper-revision-skill
```

--------------------------------

### Weighted Overall Score

Source: https://github.com/xtra-computing/xtragpt/blob/main/paper_revision_bench/README.md

Compute a weighted win rate across multiple paper sections.

```python
from paper_revision_bench import evaluate, compute_weighted_overall

section_results = {}
for section in ["title", "abstract", "introduction", "background", "evaluation", "conclusion"]:
    section_results[section] = evaluate(
        original_texts=original_list,
        revised_texts=revised_list,
        section=section,
    )

overall = compute_weighted_overall(section_results)
print(f"Overall Win Rate: {overall['weighted_win_rate']:.1%}")
```

--------------------------------

### AlpacaEval Evaluation Input Structure

Source: https://github.com/xtra-computing/xtragpt/blob/main/6_component_evaluation/alpaca_eval_gpt4_turbo_fn/alpaca_eval_fn_evaluation.txt

Defines the JSON schema for providing instructions and model outputs to the evaluation function.

```json
{
    "instruction": """{instruction}""",
}
```

```json
{
    {
        "model": "m",
        "output": """{output_1}"""
    },
    {
        "model": "M",
        "output": """{output_2}"""
    }
}
```

--------------------------------

### OpenClaw model ID

Source: https://github.com/xtra-computing/xtragpt/blob/main/xtragpt-paper-revision-skill/README.md

The identifier for the model within OpenClaw.

```text
xtragpt-7b
```
Tokens:
Raw