LongMemEval

https://github.com/xiaowu0162/longmemeval
Admin
LongMemEval is a comprehensive benchmark for testing the long-term memory capabilities of chat...
Tokens:47,203
Snippets:276
Trust Score:9.2
License:MIT
Update:1 month ago
Show doc for...
Tokens:
Raw
### Start vLLM server with serve_vllm.sh

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Starts a local OpenAI API emulator for open-weight reader LLMs using vLLM. GPU is a comma-separated list of GPUs, MODEL is an alias configured in serve_vllm.sh, PORT defaults to 8001, and TP_SIZE must be <= the number of GPUs.

```bash
cd src/utils
bash serve_vllm.sh GPU MODEL PORT TP_SIZE
```

--------------------------------

### Example - split-merge over keyphrases

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/index_expansion_utils.md

Shows how to use resolve_expansion with 'turn-keyphrase' and 'split-merge' strategy. The output demonstrates that each expansion item is merged with the existing document.

```python
from src.retrieval.index_expansion_utils import resolve_expansion

corpus = ["my car is blue"]
ids = ["sx_1"]
ts = ["2023/05/01"]
corpus, ids, ts = resolve_expansion(
    'turn-keyphrase', 'split-merge', corpus, ids, ts,
    ["car color; hiking hobby"], "sx_1_1", "2023/05/01")
print(corpus)  # ['car color my car is blue', 'hiking hobby my car is blue']
print(ids)     # ['sx_1_1', 'sx_1_1']
```

--------------------------------

### Start vLLM server with max token limit

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Starts the vLLM server with a maximum token limit to control memory usage. Parameters are the same as serve_vllm.sh, with MAXLEN added.

```bash
bash serve_vllm_with_maxlen.sh GPU MODEL MAXLEN PORT TP_SIZE
```

--------------------------------

### DenseRetrievalMaster: __init__, prepare_retriever, run_flat_retrieval, and usage example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_retrieval.md

Initializes the dense retriever on a CUDA device, prepares the retriever model based on the retriever type, and runs flat retrieval. The example shows how to instantiate DenseRetrievalMaster and call run_flat_retrieval with a query, retriever name, and corpus.

```python
def __init__(self, args, gpu_id):
    self.args = args
    self.device = torch.device('cuda', gpu_id)
    self.prepare_retriever()

```

```python
def prepare_retriever(self):
    self.retriever_model = None

    if self.args.retriever == 'flat-contriever':
        model = AutoModel.from_pretrained('facebook/contriever').to(self.device)
        tokenizer = AutoTokenizer.from_pretrained('facebook/contriever')
        self.retriever_model = (tokenizer, model)

    elif self.args.retriever == 'flat-stella':
        model_dir = self.args.cache_dir + "/dunzhang_stella_en_1.5B_v5"
        vector_dim = 1024
        vector_linear_directory = f"2_Dense_{vector_dim}"
        model = AutoModel.from_pretrained(model_dir, trust_remote_code=True).to(self.device)
        model.eval()
        tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
        vector_linear = torch.nn.Linear(in_features=model.config.hidden_size, out_features=vector_dim).to(self.device)
        vector_linear_dict = {
            k.replace("linear.", ""): v for k, v in
            torch.load(os.path.join(model_dir, f"{vector_linear_directory}/pytorch_model.bin")).items()
        }
        vector_linear.load_state_dict(vector_linear_dict)
        vector_linear.to(self.device)
        self.retriever_model = (tokenizer, model, vector_linear)

    elif self.args.retriever == 'flat-gte':
        tokenizer = AutoTokenizer.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True)
        model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen2-7B-instruct', trust_remote_code=True).to(self.device)
        model.eval()
        self.retriever_model = (tokenizer, model)

```

```python
def run_flat_retrieval(self, query, retriever, corpus):

```

```python
import argparse
from src.retrieval.run_retrieval import DenseRetrievalMaster

args = argparse.Namespace(retriever='flat-contriever', cache_dir='model_cache')
master = DenseRetrievalMaster(args, gpu_id=0)
ranks = master.run_flat_retrieval(
    "What color is my car?",
    'flat-contriever',
    ["my car is blue", "I like on weekends"])
print(ranks)  # tensor of corpus indices, descending score

```

--------------------------------

### Run temp_query_search_pruning.py with session granularity

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/temp_query_search_pruning.md

Example command to run the script from the src/index_expansion directory. It passes a timestamp event file, a retrieval log, and the granularity 'session'.

```bash
cd src/index_expansion
python3 temp_query_search_pruning.py \
    /path/to/0826_..._session-userfact.ICL.json \
    ../retrieval/xxx_retrievallog_session_flat-gte \
    session
```

--------------------------------

### Example - error handling and separate join

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/index_expansion_utils.md

Demonstrates error handling for unknown expansion type and strategy, and shows the 'separate' join mode which appends expansion as a new document.

```python
from src.retrieval.index_expansion_utils import resolve_expansion

# Unknown expansion type -> NotADirectoryError-free NotImplementedError
try:
    resolve_expansion('session-xray', 'separate', ["a"], ["s1"], ["d1"],
                      ["note"], "s1", "2023/05/01")
except NotImplementedError:
    pass

# Unknown strategy -> NotImplementedError from the final else branch
try:
    resolve_expansion('session-summ', 'concatenate', ["a"], ["s1"], ["d1"],
                      ["note"], "s1", "2023/05/01")
except NotImplementedError:
    pass

# separate: appends expansion as a new document with the same id/timestamp
corpus, ids, ts = resolve_expansion('session-summ', 'separate', ["a"], ["s1"], ["d1"],
                                    ["user likes hiking"], "s1", "2023/05/01")
print(corpus, ids, ts)   # (['a', 'user likes hiking'], ['s1', 's1'], ['d1', '2023/05/01'])
```

--------------------------------

### Set up evaluation-only conda environment

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Creates and activates a minimal conda environment for evaluation-only usage. Install this when you only need to run src/evaluation/evaluate_qa.py.

```bash
conda create -n longmemeval-lite python=3.9
conda activate longmemeval-lite
pip install -r requirements-lite.txt
```

--------------------------------

### Example — missing keys yield None

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/index_expansion_utils.md

Demonstrates that missing keys or failed generations (None values) yield None, while valid string expansions are wrapped in a list.

```python
from src.retrieval.index_expansion_utils import fetch_expansion_from_cache

cache = {"sx_3": ["likes hiking", "owns a blue car"], "sx_4": None}
print(fetch_expansion_from_cache(cache, "answer_sx_3"))  # ['likes hiking', 'owns a blue car']
print(fetch_expansion_from_cache(cache, "answer_sx_4"))  # None (failed generation)
print(fetch_expansion_from_cache(cache, "answer_sx_9"))  # None (missing)
```

--------------------------------

### Example: process_item_flat_index with turn granularity

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_retrieval.md

Example usage of process_item_flat_index with a session containing one user turn with has_answer=True. The resulting ids list is ['answer_sx_3_1'] and corpus is ["my car is blue"].

```python
from src.retrieval.run_retrieval import process_item_flat_index
session = [
    {"role": "user", "content": "my car is blue", "has_answer": True},
    {"role": "assistant", "content": "nice!"},
]
corpus, ids, ts = process_item_flat_index(session, 'turn', 'answer_sx_3', '2023/05/01')
# ids == ['answer_sx_3_1'], corpus == ["my car is blue"]
```

--------------------------------

### Run print_qa_metrics.py with example output

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/qa_metrics_printers.md

Run the QA metric aggregator from the src/evaluation directory. The command takes a JSONL log file and a reference dataset JSON, then prints per-task, task-averaged, overall, and abstention accuracy values. The commented output shows the expected format with metrics rounded to 4 decimals.

```bash
cd src/evaluation
python3 print_qa_metrics.py your_hypothesis_file.log ../../data/longmemeval_oracle.json
# Evaluation results by task:
# 	single-session-user: 0.8234 (60)
# 	...
# Task-averaged Accuracy: 0.812
# Overall Accuracy: 0.8106
# Abstention Accuracy: 0.3 (30)
```

--------------------------------

### infer_time_range: infer date range from query

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/temp_query_search_pruning.md

Prompts the model constant with 10 hard-coded few-shot examples (e.g. {'date': '2023/07/01 (Sat) 23:13', 'question': '…first BBQ event in June?'} → {"start": "2023/06/01", "end": "2023/06/30"}; temporal queries return 'N/A'). Uses max_tokens=2000, temperature=0. Returns a dict {'start': 'YYYY/MM/DD', 'end': 'YYYY/MM/DD'} parsed from the model JSON (backticks stripped); {} when the model replies N/A, returns non-JSON, or when out_data['start'] in query_date (a hard-coded hack that refuses a range coinciding with the question-date string). Throws none — parse errors are swallowed; JSON-decode failures return {}.

```python
def infer_time_range(query, query_date):
    system_prompt = "You will be given a question from a human user asking about some prvious events, as well as the time the question is asked. Infer a potential time range such that the events happening in this range is likely to help to answer the question (a start date and an end date). Write a json dict two fields: \"start\" and \"end\". Write date in the form YYYY/MM/DD. If the question does not have any temporal referencea, do not attempt to guess a time range. Instead, just say N/A."

    user_prompt = "Question date: {}\nQuestion:\n{}\n\nRelevant Date Range(dict in json format; do not generate anything else):"
    ...
```

--------------------------------

### Create OpenAI client in main()

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_generation.md

Creates the OpenAI client per run inside main(), not at import time. Uses the --openai_key and --openai_base_url command-line arguments.

```python
client = OpenAI(
    api_key=args.openai_key,
    base_url=args.openai_base_url,
)
```

--------------------------------

### Prepare prompt for flat-session retrieval with prepare_prompt

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_generation.md

Assembles a prompt from a flat retrieval log entry using prepare_prompt with the 'flat-session' retriever, JSON history format, and chain-of-thought enabled. Uses an OpenAI-compatible client at localhost:8001 and tiktoken's o200k_base encoding. The max_retrieval_length is set to 128000 minus 800 and 1000 tokens, and merge_key_expansion_into_value is 'none'.

```python
import json, tiktoken
from openai import OpenAI
from src.generation.run_generation import prepare_prompt

client = OpenAI(api_key="EMPTY", base_url="http://localhost:8001/v1")
entry = json.load(open("retrieval_log/instance.jsonl")).readline()  # has retrieval_results
tok = tiktoken.get_encoding('o200k_base')
prompt = prepare_prompt(
    entry, retriever_type="flat-session", topk_context=50, useronly=False,
    history_format="json", cot=True, tokenizer=tok,
    tokenizer_backend="openai", max_retrieval_length=128000 - 800 - 1000,
    merge_key_expansion_into_value="none",
)
```

--------------------------------

### CLI usage for temp_query_search_pruning.py

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/temp_query_search_pruning.md

Shows the required positional arguments for the CLI entry point. The first argument is a JSON file of question entries, the second is a JSONL retrieval log, and the third must be 'session' or 'turn'.

```text
Usage: python3 temp_query_search_pruning.py TIMESTAMP_EVENT_FILE RETRIEVAL_LOG GRANULARITY
```

--------------------------------

### Error handling for format_date and generate_random_dates_in_range

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/sample_haystack_and_timestamp.md

Shows error handling for format_date and generate_random_dates_in_range. Passing a date not in YYYY/MM/DD format raises ValueError with message 'Date must be in the format YYYY/MM/DD'; passing an end date before the start date raises ValueError with 'End date must be after start date'.

```python
import data.custom_history.sample_haystack_and_timestamp as s

try:
    s.format_date("2023-05-20")
except ValueError as e:
    print(e)  # Date must be in the format YYYY/MM/DD

try:
    s.generate_random_dates_in_range("2023/06/01", "2023/05/01", 3)
except ValueError as e:
    print(e)  # End date must be after start date
```

--------------------------------

### generate_random_dates_in_range(start_date_str, end_date_str, n)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/sample_haystack_and_timestamp.md

Generates `n` sorted `YYYY/MM/DD` strings in the inclusive range `[start, end]`.

```APIDOC
## generate_random_dates_in_range(start_date_str, end_date_str, n)

### Description
Generates `n` sorted `YYYY/MM/DD` strings in the inclusive range `[start, end]`.

### Parameters
- **start_date_str** (str) - Required - Inclusive lower bound `YYYY/MM/DD`
- **end_date_str** (str) - Required - Inclusive upper bound `YYYY/MM/DD`
- **n** (int) - Required - Number of dates to draw

### Returns
List of `n` sorted `YYYY/MM/DD` strings in `[start, end]`.

### Throws
- `ValueError` ("End date must be after start date") when the end precedes the start.

```

--------------------------------

### DenseRetrievalMaster.__init__(self, args, gpu_id)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_retrieval.md

Initializes the DenseRetrievalMaster instance. It stores the arguments, sets the CUDA device, and calls prepare_retriever to load the retriever model.

```APIDOC
## DenseRetrievalMaster.__init__(self, args, gpu_id)

### Description
Initializes the DenseRetrievalMaster instance. It stores the arguments, sets the CUDA device, and calls prepare_retriever to load the retriever model.

### Parameters
- **args** (argparse.Namespace) - Required - Parsed args; reads `args.retriever` and `args.cache_dir`
- **gpu_id** (int) - Required - Target CUDA device id (0-based)

### Source
`src/retrieval/run_retrieval.py:69`

### Code
```python
def __init__(self, args, gpu_id):
    self.args = args
    self.device = torch.device('cuda', gpu_id)
    self.prepare_retriever()
```
```

--------------------------------

### Import fetch_expansion_from_cache and resolve_expansion

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/index_expansion_utils.md

Import path for the library helpers used at index-build time.

```python
from src.retrieval.index_expansion_utils import fetch_expansion_from_cache, resolve_expansion
```

--------------------------------

### Run print_qa_metrics.py with in_file and ref_file

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/configuration.md

Run the QA metrics printer with an input JSONL evaluation log and a reference dataset JSON. Both arguments are required.

```bash
python print_qa_metrics.py in_file ref_file
```

--------------------------------

### generate_random_dates_in_range: random dates within a range

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/sample_haystack_and_timestamp.md

Generates n sorted YYYY/MM/DD strings uniformly in the inclusive range [start_date, end_date]. Raises ValueError if the end date precedes the start date.

```python
def generate_random_dates_in_range(start_date_str, end_date_str, n):
    start_date = datetime.strptime(start_date_str, "%Y/%m/%d")
    end_date = datetime.strptime(end_date_str, "%Y/%m/%d")
    delta_days = (end_date - start_date).days
    if delta_days < 0:
        raise ValueError("End date must be after start date")
    random_dates = [
        start_date + timedelta(days=random.randint(0, delta_days))
        for _ in range(n)
    ]
    random_dates.sort()
    return [date.strftime("%Y/%m/%d") for date in random_dates]
```

--------------------------------

### Example usage of evaluate_retrieval_turn2session

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/retrieval_eval_utils.md

Shows usage of evaluate_retrieval_turn2session with turn-level ids and rankings. The comment indicates the expected output (1.0, 1.0, 1.0) when session s1 is covered by rank 0 alone.

```python
from src.retrieval.eval_utils import evaluate_retrieval_turn2session

turn_ids = ["answer_s1_1", "filler_a_1", "filler_b_1", "answer_s1_2"]
rankings = [0, 3, 1, 2]                      # both answer turns at ranks 0 and 1
correct_turns = ["answer_s1_1", "answer_s1_2"]
print(evaluate_retrieval_turn2session(rankings, correct_turns, turn_ids, k=1))
# (1.0, 1.0, 1.0)  — session s1 is covered by rank 0 alone, widened k reaches 1 session
```

--------------------------------

### Example: multi-type evaluation with abstention detection

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/evaluate_qa.md

Shows how to call get_anscheck_prompt for a multi-type evaluation with abstention detection. The abstention flag is set to True when the question_id contains the '_abs' suffix, which marks abstention questions. The resulting prompt is printed.

```python
from src.evaluation.evaluate_qa import get_anscheck_prompt

entry = {"question_id": "q123", "question": "…", "answer": "…"}
hyp = "I don't have that information"
prompt = get_anscheck_prompt(
    "single-session-user", entry["question"], entry["answer"], hyp,
    abstention="_abs" in entry["question_id"],   # '_abs' suffix marks abstention
)
print(prompt)
```

--------------------------------

### prepare_prompt(...) - build prompt from entry

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_generation.md

Builds the final prompt string from a dataset entry. Selection logic depends on retriever_type: no-retrieval, orig-session/orig-turn, oracle-session/oracle-turn, flat-session/flat-turn. Asserts retriever_type == "no-retrieval" or history_string != "".

```python
def prepare_prompt(entry, retriever_type, topk_context: int, useronly: bool, history_format: str, cot: bool, tokenizer, tokenizer_backend, max_retrieval_length, merge_key_expansion_into_value, con=False, con_client=None, con_model=None):
```

--------------------------------

### Run generation with long context (bash)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/run_generation.md

Runs the generation pipeline from the shell with a long-context session. The arguments set the data file, model, history format, top-k, output format, user-only flag, and conversation flag. This is the documented long-context example.

```bash
cd src/generation
bash run_generation.sh DATA_FILE MODEL full-history-session 1000 json false con
```

--------------------------------

### Example usage of evaluate_retrieval

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/retrieval_eval_utils.md

Shows how to import and call evaluate_retrieval with a sample corpus and rankings. The first call returns recall_any=1.0, recall_all=1.0, and ndcg_score=0.8155. The second call demonstrates a failure mode where correct documents are absent from the top-k, returning 0.0 for all metrics.

```python
from src.retrieval.eval_utils import evaluate_retrieval

corpus_ids = ["answer_s1_1", "filler_a_1", "filler_b_2", "answer_s1_2"]
rankings = [1, 0, 3, 2]                      # retrieved_by_index
correct = ["answer_s1_1", "answer_s1_2"]
recall_any, recall_all, ndcg_score = evaluate_retrieval(rankings, correct, corpus_ids, k=5)
print(recall_any, recall_all, ndcg_score)    # 1.0 1.0 0.8155  (DCG 1 + 1/log2(3) over ideal 1 + 1/log2(2))

# Failure mode: correct documents absent from top-k
r2, r3, n2 = evaluate_retrieval([2, 3], correct, corpus_ids, k=1)
print(r2, r3, n2)                            # 0.0 0.0 0.0
```

--------------------------------

### Set up full conda environment with PyTorch and CUDA

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Creates and activates a full conda environment with PyTorch 2.3.1 and CUDA 12.1 support. Use this when running the memory systems introduced in the paper; tested on Linux with CUDA 12.1.

```bash
conda create -n longmemeval python=3.9
conda activate longmemeval
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements-full.txt
```

--------------------------------

### Run retrieval-augmented generation with run_generation.sh

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Runs retrieval-augmented generation. RETRIEVAL_LOG_FILE must be the output from the retrieval step, relying on the retrieval_results field. EXP must be in the form [RETRIEVER]-[GRANULARITY] such as flat-stella-session. Other parameters are as introduced above.

```bash
cd src/generation
bash run_generation.sh RETRIEVAL_LOG_FILE MODEL EXP TOPK [HISTORY_FORMAT] [USERONLY] [READING_METHOD]
```

--------------------------------

### Import eval_utils functions

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/retrieval_eval_utils.md

Import the ranking-metrics helpers from the library module. Requires the repository root on PYTHONPATH.

```python
from src.retrieval.eval_utils import dcg, ndcg, evaluate_retrieval, evaluate_retrieval_turn2session
```

--------------------------------

### extract_session_userfact: timestamped event extraction from session

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/index_expansion_batch_scripts.md

Extracts timestamped user events from a conversation session. Requires a `sess_date` conversation date string and a `sess_entry` list of turns; optionally accepts few-shot `examples`. Uses `chat_completions_with_backoff` with `temperature=0` and `max_tokens=2000`. Returns a list of dicts with `date` and `event` fields, or `None` if JSON parsing fails. The CLI stores results under `timestamped_facts` and writes the whole question list to `.session-userfact.{mode}.json`, which is consumed by `temp_query_search_pruning.py`.

```python
def extract_session_userfact(sess_date, sess_entry, model_name, examples=None):
    system_prompt = "You will be given a list of messages from a human user to an AI assistant, as well as the time the conversation took place. Extract all events related to the user as long as its date is specified or could be inferred. If the time some event took place cannot be inferred, do not extract that event. Return the events in a json list where each item contains two fields: \"date\" and \"event\". Write date in the form YYYY/MM/DD. If there is no specific event, just write an empty list."

    user_prompt = "Conversation date: {}\nHuman user messages:\n{}\n\nPersonal facts about the user with dates (a list of dicts in json format; do not generate anything else):"
    dialogue_string = ""
    for turn_entry in sess_entry:
        if turn_entry['role'] == 'user':
            dialogue_string += f"\n{turn_entry['role']}: {turn_entry['content']}"
    summarization_prompt = user_prompt.format(sess_date, dialogue_string)
    if examples is None:
        messages = [{"role": "system", "content": system_prompt},
                    {"role": "user", "content": summarization_prompt}]
    else:
        messages = [{"role": "system", "content": system_prompt}]
        for example_input_dialogue_string, example_output in examples:
            messages += [{"role": "user", "content": example_input_dialogue_string},
                         {"role": "assistant", "content": example_output}]
        messages += [{"role": "user", "content": summarization_prompt}]
    kwargs = {'model': model_name, 'messages': messages, 'n': 1, 'temperature': 0, 'max_tokens': 2000}
    completion = chat_completions_with_backoff(client,**kwargs)
    try:
        out_string = completion.choices[0].message.content.strip()
        out_string = out_string.replace('```json', '').replace('```', '').strip()
        print([json.dumps(json.loads(out_string))])
        return json.loads(out_string.strip())
    except:
        return None
```

--------------------------------

### Run evaluate_qa.py with positional CLI arguments

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/configuration.md

Run the QA evaluation script with three required positional arguments: metric_model (a key from the model_zoo, e.g., gpt-4o, gpt-4o-mini, or llama-3.1-70b-instruct), hyp_file (JSONL with {question_id, hypothesis}, with JSON fallback), and ref_file (reference dataset JSON, with JSONL fallback). The judge call uses n=1, temperature=0, max_tokens=10, and verbose=True; output is written to hyp_file + '.eval-results-' + metric_model.

```bash
python evaluate_qa.py metric_model hyp_file ref_file
```

--------------------------------

### Run evaluate_qa.py CLI with gpt-4o

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/api-reference/evaluate_qa.md

Run the CLI with a supported metric model alias (e.g., gpt-4o). Set OPENAI_API_KEY first; OPENAI_ORGANIZATION is optional for single-org keys. The command expects the hypothesis file and reference dataset path as positional arguments.

```bash
export OPENAI_API_KEY=YOUR_API_KEY
export OPENAI_ORGANIZATION=YOUR_ORGANIZATION     # optional; omit for single-org keys
cd src/evaluation
python3 evaluate_qa.py gpt-4o your_hypothesis_file ../../data/longmemeval_oracle.json
```
Tokens:
Raw