### Install NarraBench and Dependencies

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Installs Prolog and Git LFS, clones the NarraBench repository, and installs the package. Downloads all benchmark datasets.

```bash
# Install Prolog and enable Git LFS (required for large benchmark files)
# On Ubuntu/Debian:
sudo apt-get install swi-prolog git-lfs
git lfs install

# Clone and install NarraBench
git clone https://github.com/your-repo/narrabench.git
cd narrabench
pip install -e .

# Download all benchmark datasets
python setup.py
```

--------------------------------

### AustenAlike Benchmark Setup

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Defines characters for the AustenAlike benchmark and initializes an OpenAI client for the model being evaluated. This benchmark requires a judge model.

```python
from openai import OpenAI

# Characters span 6 novels: Emma, Mansfield Park, Northanger Abbey,
# Persuasion, Pride and Prejudice, Sense and Sensibility
CHARACTERS = [
    "Emma Woodhouse", "Elizabeth Bennet", "Fanny Price", "Anne Elliot",
    "Catherine Morland", "Elinor Dashwood", "Marianne Dashwood",
    "Fitzwilliam Darcy", "Mr. Knightley", "Colonel Brandon", ...
]

# Test client for the model being evaluated
test_client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

```

--------------------------------

### Results CSV Header and Example Data

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Illustrates the CSV format for benchmark results, including model, taxonomy metadata, and accuracy.

```csv
# Results CSV includes taxonomy metadata
# benchmark,model,feature,aspect,accuracy
# austenalike,llama3.2,Story,Agent/Attributes,0.6800
# culemo,llama3.2,Story,Agent/Emotional State,0.4523
# phantomwiki,llama3.2,Story,Social Networks/Connections,0.3200
# storysumm,llama3.2,Story,Plot/Plotline,0.7150
# tot,llama3.2,Discourse,Time/Order,0.2890
# tram,llama3.2,Discourse,Time/Order,0.5430
# traveler,llama3.2,Discourse,Time/Order,0.4100
```

--------------------------------

### Run All Benchmarks with Ollama

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Starts Ollama servers and runs all benchmarks using the unified runner script. Specify model, ports, and output file.

```bash
# Start Ollama server (default port 11434)
ollama serve

# Start judge model server on separate port for benchmarks requiring evaluation
# (e.g., AustenAlike requires a judge model)
OLLAMA_HOST=0.0.0.0:11435 ollama serve

# Run all benchmarks with a specific model
python run.py --model llama3.2 --port 11434 --judge-port 11435 --output results.csv

# Run with custom host configuration
python run.py \
    --model mistral \
    --host localhost \
    --port 11434 \
    --judge-host localhost \
    --judge-port 11435 \
    --output my_results.csv

# Example output:
# Found 7 benchmark(s): austenalike, culemo, phantomwiki, storysumm, tot, tram, traveler
# Model: llama3.2
# API: http://localhost:11434
# Judge API: http://localhost:11435
# ------------------------------------------------------------
# austenalike...
#   ✓ 0.6800
# culemo...
#   ✓ 0.4523
# ...
# Results: results.csv
```

--------------------------------

### Classify Sentiment from Text

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Uses an OpenAI client to classify the sentiment of a given text. The system prompt guides the model to respond with only 'positive', 'negative', or 'neutral'. Requires an OpenAI client setup.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

text = "I felt my heart sink as I read the letter, knowing everything had changed forever."

# Sentiment classification
sentiment_response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that analyzes sentiment. Answer with only one word: positive, negative, or neutral."},
        {"role": "user", "content": f"{text}\n\nWhat is the sentiment?"}
    ],
    temperature=0.0,
    max_tokens=10
)
predicted_sentiment = sentiment_response.choices[0].message.content.strip().lower()
# Output: "negative"
```

--------------------------------

### Classify Emotion from Text

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Uses an OpenAI client to classify the emotion expressed in a given text. The system prompt instructs the model to respond with a single emotion label. Requires an OpenAI client setup.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

text = "I felt my heart sink as I read the letter, knowing everything had changed forever."

# Emotion classification
emotion_response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that predicts emotions. Answer with only one word: the emotion label."},
        {"role": "user", "content": f"{text}\n\nWhat emotion does this express? (joy, sadness, anger, fear, disgust, surprise, guilt, shame)"}
    ],
    temperature=0.0,
    max_tokens=10
)
predicted_emotion = emotion_response.choices[0].message.content.strip().lower()
# Output: "sadness"
```

--------------------------------

### Load and Query PhantomWiki Dataset

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Loads the PhantomWiki dataset from HuggingFace and uses an OpenAI client to answer questions based on the provided corpus. It constructs context from wiki articles and evaluates the model's response against ground truth answers.

```python
from openai import OpenAI
from datasets import load_dataset

# Load PhantomWiki dataset from HuggingFace
ds_qa = load_dataset("kilian-group/phantom-wiki-v1", "question-answer", split="depth_20_size_50_seed_1")
ds_corpus = load_dataset("kilian-group/phantom-wiki-v1", "text-corpus", split="depth_20_size_50_seed_1")

# Build context from wiki articles
corpus_text = "\n\n".join([item['article'] for item in ds_corpus])

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

# Example question about character connections
example = ds_qa[0]
question = example['question']  # e.g., "Who does Character_A know?"
ground_truth_answers = example['answer']  # List of valid answers

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": f"Answer questions based on this information:\n\n{corpus_text}\n\nProvide concise answers with just the name(s)."},
        {"role": "user", "content": question}
    ],
    temperature=0.0,
    max_tokens=100
)

predicted = response.choices[0].message.content.strip().lower()
is_correct = any(ans.lower() in predicted for ans in ground_truth_answers)
```

--------------------------------

### Load and Run a Benchmark Dynamically

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Dynamically loads a benchmark wrapper script and executes its `run_benchmark` function. Returns the accuracy score.

```python
from pathlib import Path
import importlib.util

def load_and_run_benchmark(benchmark_name: str, model: str, host: str = "localhost", port: int = 11434) -> float:
    """Load a benchmark wrapper and execute it against a model."""
    wrapper_path = Path(f"tasks/{benchmark_name}/wrapper.py")

    # Dynamic module loading
    spec = importlib.util.spec_from_file_location("wrapper", wrapper_path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)

    # Execute benchmark - returns accuracy as float between 0.0 and 1.0
    accuracy = module.run_benchmark(
        model=model,
        host=host,
        port=port,
        judge_host="localhost",  # Optional: for benchmarks requiring evaluation
        judge_port=11435         # Optional: judge model port
    )

    return accuracy

# Example usage
accuracy = load_and_run_benchmark("culemo", "llama3.2")
print(f"CuLEmo accuracy: {accuracy:.4f}")
# Output: CuLEmo accuracy: 0.4523
```

--------------------------------

### Evaluate Temporal Ordering with TRAM Benchmark

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Loads TRAM ordering questions from a CSV file and uses an LLM to answer multiple-choice questions about event ordering. Requires 'openai' and 'csv' libraries. The response is expected to be a single letter.

```python
from openai import OpenAI
import csv

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

# Load TRAM ordering questions
with open("tasks/tram/tram-original/datasets/ordering_mcq.csv", 'r') as f:
    reader = csv.DictReader(f)
    data = list(reader)

row = data[0]
question = row['Question']
options = f"A. {row['Option A']}\nB. {row['Option B']}\nC. {row['Option C']}"
correct_answer = row['Answer']  # "A", "B", or "C"

prompt = f"{question}\n\n{options}\n\nAnswer with only the letter (A, B, or C):"

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "Answer multiple choice questions by providing only the letter of the correct answer."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.0,
    max_tokens=10
)

predicted = response.choices[0].message.content.strip().upper()[0]
is_correct = (predicted == correct_answer)
```

--------------------------------

### Evaluate Event Sequence Reasoning with TRaVelER Benchmark

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Loads event logs and questions for the TRaVelER benchmark. It formats events into a text log and uses an LLM to answer questions about event sequences and temporal relationships. Requires 'openai', 'datetime', and 'json' libraries.

```python
from openai import OpenAI
from datetime import datetime
import json

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

# Load events and questions
with open("tasks/traveler/traveler-original/events/100Events.json", 'r') as f:
    events = json.load(f)

with open("tasks/traveler/traveler-original/dataset/explicit/5Events.json", 'r') as f:
    questions = json.load(f)

# Format events as context
events_text = "\n".join([
    f"- {e['Subject']} {e['Action']} {e['Object']} in the {e['Location']} on {datetime.fromtimestamp(e['Timestamp']).strftime('%Y-%m-%d')}"
    for e in events
])

item = questions[0]
question = item['text']           # e.g., "How many times did John visit the library?"
ground_truth = item['gt_answers'] # e.g., "3"

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": f"Answer questions based on this event log:\n\n{events_text}\n\nProvide concise, direct answers."},
        {"role": "user", "content": question}
    ],
    temperature=0.0,
    max_tokens=50
)

predicted = response.choices[0].message.content.strip().lower()
```

--------------------------------

### NarraBench Submission Template

Source: https://github.com/srhm-ca/narrabench/blob/main/README.md

Use this template when creating a Pull Request to submit a new benchmark. Fill in all the required fields and provide a concise rationale.

```markdown
Name: <fill>
Link: <fill>
License: <fill>
Dimension: <fill>
Feature: <fill>
Aspect: <fill>
Scale: <fill>
Mode: <fill>
Variance: <fill>

Citation: 
```
<fill>
```

Rationale: <1-2 sentences describing what this benchmark provides.>
```

--------------------------------

### Evaluate Temporal Reasoning with ToT Benchmark

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Loads the ToT (Test of Time) dataset and uses an LLM to answer temporal reasoning questions. Requires 'openai', 'datasets', and 'json' libraries. The response is parsed to extract an entity ID.

```python
from openai import OpenAI
from datasets import load_from_disk
import json
import re

client = OpenAI(base_url="http://localhost:11434/v1", api_key="dummy")

# Load Test of Time dataset
ds = load_from_disk("tasks/tot/tot-original/tot_semantic/test")

example = ds[0]
prompt = example['prompt']      # Temporal facts and question
label = example['label']        # Ground truth entity (e.g., "E76")

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "Answer temporal reasoning questions. Output JSON with 'explanation' and 'answer' fields. The answer should be an entity ID (e.g., E76)."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.0,
    max_tokens=200
)

answer_text = response.choices[0].message.content.strip()

# Parse response - try JSON first, fallback to regex
try:
    parsed = json.loads(answer_text)
    predicted = parsed.get('answer', '').upper()
except:
    match = re.search(r'E\d+', answer_text)
    predicted = match.group(0).upper() if match else ''

is_correct = (predicted == label.upper())
```

--------------------------------

### Benchmark Wrapper Implementation Interface

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Defines the required `run_benchmark()` interface for implementing new benchmarks. Includes arguments for model, API host/port, and optional judge details.

```python
# tasks/mybenchmark/wrapper.py - Required implementation

def run_benchmark(model: str, host: str, port: int, judge_host: str = None, judge_port: int = None) -> float:
    """
    Run the benchmark against the specified model.

    Args:
        model: Model name to test (e.g., "llama3.2")
        host: API host address
        port: API port number
        judge_host: Optional judge model host for evaluation
        judge_port: Optional judge model port

    Returns:
        Accuracy as float between 0.0 and 1.0
    """
    # Load your dataset
    # Query the model via OpenAI-compatible API
    # Calculate and return accuracy
    pass

```

--------------------------------

### Evaluate Story Summary Consistency with LLM

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Loads the StorySumm dataset and uses an LLM to evaluate the factual consistency of a summary against a given story. Requires the 'openai' and 'json' libraries.

```python
with open("tasks/storysumm/storysumm-original/storysumm.json", 'r') as f:
    data = json.load(f)

# Example evaluation
item = list(data.values())[0]
story = item['story'].strip()
summary = ' '.join(item['summary'])
true_label = item['label']  # 1 = consistent, 0 = inconsistent

response = client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You evaluate story summaries for factual consistency."},
        {"role": "user", "content": f"Story:\n{story}\n\nSummary:\n{summary}"},
        {"role": "user", "content": "Is all of the information in the summary consistent with the story? Answer Yes or No."}
    ],
    temperature=0.0,
    max_tokens=10
)

answer = response.choices[0].message.content.strip().lower()
predicted_label = 1 if answer.startswith('yes') else 0
is_correct = (predicted_label == true_label)
```

--------------------------------

### New Benchmark Submission PR Template

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Template for submitting new benchmarks, including metadata, dimensions, and citation.

```markdown
# PR Template for New Benchmark Submission

Name: MyBenchmark
Link: https://github.com/author/mybenchmark
License: MIT
Dimension: Story|Narration|Discourse|Situatedness
Feature: Agent|Social Networks|Event|Plot|Structure|Setting|...
Aspect: name|role|attributes|emotional state|motivation|...
Scale: local|global
Mode: discrete|holistic|progressive
Variance: deterministic|consensus|perspectival

Citation:
```
@inproceedings{author2024mybenchmark,
    title = "My Benchmark Title",
    author = "Author Name",
    booktitle = "Conference",
    year = "2024",
}
```

Rationale: Brief 1-2 sentence description of what narrative understanding ability this benchmark tests.

```

--------------------------------

### run_benchmark Function Interface

Source: https://context7.com/srhm-ca/narrabench/llms.txt

The primary interface for executing individual benchmarks within the NarraBench framework. Each benchmark wrapper implements this function to evaluate a model.

```APIDOC
## run_benchmark(model, host, port, judge_host, judge_port)

### Description
Executes a specific narrative understanding benchmark against a target LLM model.

### Parameters
- **model** (string) - Required - The name of the model to evaluate.
- **host** (string) - Optional - The host address of the model server (default: localhost).
- **port** (int) - Optional - The port of the model server (default: 11434).
- **judge_host** (string) - Optional - The host address of the judge model server.
- **judge_port** (int) - Optional - The port of the judge model server.

### Response
- **accuracy** (float) - Returns an accuracy score between 0.0 and 1.0.
```

--------------------------------

### Evaluate Character Similarity Prediction

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Uses an OpenAI client to evaluate if a model's character similarity prediction is reasonable compared to an expert annotation. Requires setting up an OpenAI client with a specific base URL and API key.

```python
judge_client = OpenAI(base_url="http://localhost:11435/v1", api_key="dummy")

# Query the model for character similarity
character_list = "\n".join([f"{i+1}. {char}" for i, char in enumerate(CHARACTERS)])
question = "Based on your knowledge of Jane Austen's novels, which character from the list is Emma Woodhouse most similar to in terms of personality, social role, or narrative function?"

response = test_client.chat.completions.create(
    model="llama3.2",
    messages=[
        {"role": "system", "content": f"You are an expert on Jane Austen's novels.\n\n{character_list}"},
        {"role": "user", "content": question}
    ],
    temperature=0.0,
    max_tokens=50
)

predicted = response.choices[0].message.content.strip()
# Example output: "Elizabeth Bennet"

# Judge evaluates if prediction matches expert annotation
judge_response = judge_client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[
        {"role": "system", "content": "You are an expert evaluating character similarity predictions."},
        {"role": "user", "content": f"Expert says: Harriet Smith. Model predicted: {predicted}. Is this reasonable? YES or NO"}
    ],
    temperature=0.0,
    max_tokens=10
)
# Returns "YES" or "NO"
```

--------------------------------

### Benchmark Taxonomy Definition

Source: https://context7.com/srhm-ca/narrabench/llms.txt

Defines the classification of benchmarks by feature and aspect. Used for organizing and analyzing results.

```python
BENCHMARK_TAXONOMY = {
    'austenalike': {'feature': 'Story', 'aspect': 'Agent/Attributes'},
    'culemo':      {'feature': 'Story', 'aspect': 'Agent/Emotional State'},
    'phantomwiki': {'feature': 'Story', 'aspect': 'Social Networks/Connections'},
    'storysumm':   {'feature': 'Story', 'aspect': 'Plot/Plotline'},
    'tot':         {'feature': 'Discourse', 'aspect': 'Time/Order'},
    'tram':        {'feature': 'Discourse', 'aspect': 'Time/Order'},
    'traveler':    {'feature': 'Discourse', 'aspect': 'Time/Order'},
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.