### prepare_prompt Function Usage Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

An example demonstrating how to call the `prepare_prompt` function with a sample entry and various parameters to generate a complete prompt.

```python
entry = {
    'question': 'What is my favorite hobby?',
    'question_date': '2024/01/15',
    'haystack_dates': ['2024/01/10', '2024/01/12'],
    'haystack_sessions': [
        [{'role': 'user', 'content': 'I love photography'}, ...],
        [{'role': 'user', 'content': 'I also enjoy hiking'}, ...]
    ],
    'retrieval_results': {...}
}

prompt = prepare_prompt(
    entry, 'flat-session', topk_context=5, useronly=True,
    history_format='json', cot=True,
    tokenizer=tokenizer, tokenizer_backend='openai',
    max_retrieval_length=90000,
    merge_key_expansion_into_value='none'
)
# Returns complete prompt with history formatted as JSON
```

--------------------------------

### Example: Multi-Session Synthesis Questions

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Creates multi-session synthesis questions with a fixed number of filler sessions. This example demonstrates setting the minimum and maximum filler sessions to the same value.

```bash
# Create 10 multi-session questions with 500-500 filler sessions (all 500)
python sample_haystack_and_timestamp.py multi_session_synthesis 10 500 500
```

--------------------------------

### Start vLLM Server

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Use this command to start the vLLM server for testing open-weight reader LLMs locally. Ensure the PORT matches the one specified in run_generation.sh.

```bash
cd src/utils
bash serve_vllm.sh GPU MODEL PORT TP_SIZE
```

--------------------------------

### BM25 Baseline Retrieval Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

This example demonstrates how to run retrieval using the BM25 baseline. Ensure the input file and output directory are correctly specified.

```bash
python run_retrieval.py \
  --in_file ../../data/longmemeval_s.json \
  --out_dir retrieval_logs \
  --retriever flat-bm25 \
  --granularity session
```

--------------------------------

### Setup Conda Environment for Evaluation Only

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Create and activate a conda environment named 'longmemeval-lite' with Python 3.9 and install the minimal requirements for evaluation.

```bash
conda create -n longmemeval-lite python=3.9
conda activate longmemeval-lite
pip install -r requirements-lite.txt
```

--------------------------------

### Setup Conda Environment for Full Support

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Create and activate a conda environment named 'longmemeval' with Python 3.9, install specific PyTorch versions, and then install the full requirements.

```bash
conda create -n longmemeval python=3.9
conda activate longmemeval
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements-full.txt
```

--------------------------------

### Start vLLM Server with Max Length

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Use this command to start the vLLM server with a specified maximum sequence length, useful for managing memory requirements. Ensure the PORT matches the one specified in run_generation.sh.

```bash
bash serve_vllm_with_maxlen.sh GPU MODEL MAXLEN PORT TP_SIZE
```

--------------------------------

### Output Filename Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Illustrates the naming convention for output JSON files, which includes parameters like task, session count, source ratios, and token length.

```text
202410_custom_haystack1_single_hop_10-80haysess_user0.5sharegpt0.25ultrachat0.25_enflen115000.json
```

--------------------------------

### Input Log Format Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_retrieval_metrics.md

This is an example of the JSONL log file format generated by run_retrieval.py, which contains retrieval results including session and turn-level metrics.

```json
{
  "question_id": "single_hop_user_1",
  "question_type": "single-session-user",
  "question": "What is my favorite color?",
  "answer": "Blue",
  "question_date": "2024/01/15 14:30",
  "haystack_dates": ["2024/01/10", "2024/01/12"],
  "haystack_sessions": [...],
  "haystack_session_ids": ["answer_sess_1", "sess_2"],
  "answer_session_ids": ["answer_sess_1"],
  "retrieval_results": {
    "query": "What is my favorite color?",
    "ranked_items": [...],
    "metrics": {
      "session": {
        "recall_any@5": 0.8,
        "ndcg_any@5": 0.75,
        "recall_all@5": 0.6,
        "recall_any@10": 0.9,
        "ndcg_any@10": 0.82,
        "recall_all@10": 0.8,
        "recall_any@30": 1.0,
        "ndcg_any@30": 0.88,
        "recall_all@30": 1.0,
        "recall_any@50": 1.0,
        "ndcg_any@50": 0.88,
        "recall_all@50": 1.0
      },
      "turn": {
        "recall_any@5": 0.7,
        ...
      }
    }
  }
}
```

--------------------------------

### Example Output File Names

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Illustrates the expected output file naming convention based on various parameters like input filename, context size, history format, user-only flag, fact expansion, and timestamp.

```text
longmemeval_oracle.json_testlog_top5context_jsonformat_useronly{true}_20240115-1430
```

```text
longmemeval_oracle.json_testlog_top5context_jsonformat_useronly{false}_factexpansionmerge_20240115-1430
```

--------------------------------

### Oracle Retrieval Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

Use this command to perform retrieval using the Oracle method. Specify the input file, output directory, retriever type, and granularity.

```bash
python run_retrieval.py \
  --in_file ../../data/longmemeval_oracle.json \
  --out_dir retrieval_logs \
  --retriever oracle \
  --granularity session
```

--------------------------------

### Example: Resolving Expansions with Different Strategies

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/index_expansion_utils.md

Demonstrates the application of 'separate', 'split-merge', and 'replace' strategies using the resolve_expansion function. Shows the resulting corpus, IDs, and timestamps for each strategy.

```python
# Before: corpus=['What is my hobby?'], ids=['sess_1'], ts=['2024/01/15']
# Expansion: keyphrase='photography; hiking; cooking'

result_separate = resolve_expansion(
    'session-keyphrase', 'separate',
    ['What is my hobby?'], ['sess_1'], ['2024/01/15'],
    ['photography; hiking; cooking'], 'sess_1', '2024/01/15'
)
# Result: corpus=['What is my hobby?', 'photography', 'hiking', 'cooking'],
#         ids=['sess_1', 'sess_1', 'sess_1', 'sess_1']

result_merge_split = resolve_expansion(
    'session-keyphrase', 'split-merge',
    ['What is my hobby?'], ['sess_1'], ['2024/01/15'],
    ['photography; hiking; cooking'], 'sess_1', '2024/01/15'
)
# Result: corpus=['photography What is my hobby?', 'hiking What is my hobby?', 'cooking What is my hobby?'],
#         ids=['sess_1', 'sess_1', 'sess_1']

result_replace = resolve_expansion(
    'session-keyphrase', 'replace',
    ['What is my hobby?'], ['sess_1'], ['2024/01/15'],
    ['photography; hiking; cooking'], 'sess_1', '2024/01/15'
)
# Result: corpus=['photography; hiking; cooking'],
#         ids=['sess_1'],
#         timestamps=['2024/01/15']
```

--------------------------------

### Example Script Output

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_qa_metrics.md

Displays the typical output format of the print_qa_metrics.py script, showing per-task accuracy, task-averaged accuracy, overall accuracy, and abstention accuracy.

```text
Evaluation results by task:
	single-session-user: 0.8452 (84)
	single-session-preference: 0.8571 (84)
	single-session-assistant: 0.8205 (83)
	multi-session: 0.8193 (83)
	temporal-reasoning: 0.7952 (83)
	knowledge-update: 0.8072 (83)

Task-averaged Accuracy: 0.8241
Overall Accuracy: 0.8243
Abstention Accuracy: 0.8667 (30)
```

--------------------------------

### Statistics Output Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Shows the typical output printed by the script, including counts of different haystack sources and statistics about the question database token counts.

```text
Haystack source: 245 user, 123 sharegpt, 132 ultrachat
Current question db: 2_questions/0822_all_500_questions_final_v3.json

Max token count 125432; Min token count 45123; Mean token count 98456.23
```

--------------------------------

### Example: Single Session User Questions

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Generates single-session-user questions with a specified number of filler sessions and an enforced token limit. This is useful for creating specific test cases with controlled history lengths.

```bash
# Create 25 single-session-user questions with 10-80 filler sessions, enforce 115k tokens
python sample_haystack_and_timestamp.py single_hop 25 10 80 115000
```

--------------------------------

### Example: Temporal Reasoning Questions

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Generates temporal reasoning questions without enforcing a specific token limit. This is suitable for scenarios where the history length is less critical than the temporal aspects.

```bash
# Create temporal reasoning without token limit
python sample_haystack_and_timestamp.py temp_reasoning_explicit 50 40 40
```

--------------------------------

### JSON History Formatting Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Demonstrates the JSON format for representing chat history, including session dates and content with role-based messages.

```json
### Session 1:
Session Date: 2024/01/15 10:30
Session Content:
[{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]

### Session 2:
...
```

--------------------------------

### Main Retrieval Execution

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

The main function to execute the retrieval process. Call this with parsed arguments to start retrieval.

```python
main(args: argparse.Namespace) -> None
```

--------------------------------

### Natural Language History Formatting Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Illustrates the natural language format for chat history, presenting sessions with dates and alternating user/assistant turns.

```string
### Session 1:
Session Date: 2024/01/15 10:30
Session Content:

user: ...
assistant: ...

user: ...
assistant: ...

### Session 2:
...
```

--------------------------------

### Typical Usage in run_retrieval.py

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/index_expansion_utils.md

Demonstrates the three main steps for integrating expansions: building the base corpus, fetching and applying expansions, and running retrieval on the augmented corpus. Ensure 'expansion_cache.json' is loaded.

```python
# Step 1: Build base corpus from sessions
corpus, corpus_ids, corpus_timestamps = [], [], []
for sess_id, session_entry, ts in zip(session_ids, sessions, timestamps):
    items, ids, tss = process_item_flat_index(session_entry, granularity, sess_id, ts)
    corpus += items
    corpus_ids += ids
    corpus_timestamps += tss

# Step 2: Fetch and apply expansions
index_expansion_cache = json.load(open('expansion_cache.json'))
for sess_id, session_entry, ts in zip(session_ids, sessions, timestamps):
    expansions = fetch_expansion_from_cache(cache, sess_id)
    corpus, corpus_ids, corpus_timestamps = resolve_expansion(
        expansion_type='session-keyphrase',
        resolution_strategy='merge',
        existing_corpus=corpus,
        existing_corpus_ids=corpus_ids,
        existing_corpus_timestamps=corpus_timestamps,
        cur_item_expansions=expansions,
        cur_sess_id=sess_id,
        ts=ts
    )

# Step 3: Run retrieval on augmented corpus
rankings = retriever.run_retrieval(query, corpus)
```

--------------------------------

### Load and Apply Session Summary Expansions

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/index_expansion_utils.md

Demonstrates loading a cache, fetching a session summary, and applying it to the corpus as separate items.

```python
import json
from src.retrieval.index_expansion_utils import (
    fetch_expansion_from_cache,
    resolve_expansion
)

# Load cache
cache = json.load(open('session_summaries.json'))

# Initial corpus
corpus = ['User discussed their childhood memories.']
corpus_ids = ['answer_session_5']
timestamps = ['2024/01/15 10:00']

# Fetch summary
summary = fetch_expansion_from_cache(cache, 'session_5')
# summary = ['Important memory about school years.']

# Apply as separate items
corpus, corpus_ids, timestamps = resolve_expansion(
    'session-summ', 'separate',
    corpus, corpus_ids, timestamps,
    summary, 'answer_session_5', '2024/01/15 10:00'
)

# Result: corpus has 2 items, both with ID 'answer_session_5'
```

--------------------------------

### Launch Local vLLM Server

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/README.md

Launch a local vLLM server for using open-source models. Specify the GPU IDs, model name, and port. Use the provided base URL and an empty key for scripts.

```bash
# Launch vLLM server first
bash src/utils/serve_vllm.sh 0,1 llama-3.1-70b-instruct 8001 2

# Then use in scripts
--openai_base_url http://localhost:8001/v1
--openai_key EMPTY
```

--------------------------------

### generate_random_dates_in_range

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Generates a specified number of random dates that fall within a given start and end date range.

```APIDOC
## generate_random_dates_in_range(start_date_str: str, end_date_str: str, n: int) -> list

### Description
Generates n random dates within [start_date, end_date] range.

### Parameters
#### Path Parameters
- **start_date_str** (string) - Required - The start date of the range in `YYYY/MM/DD` format.
- **end_date_str** (string) - Required - The end date of the range in `YYYY/MM/DD` format.
- **n** (integer) - Required - Number of dates to generate

### Return Type
`list[str]` - Sorted ascending
```

--------------------------------

### Run Generation with GPT-4o (Context of Notes)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Utilizes GPT-4o with flat session retrieval, a topk context of 10, and enables chain-of-thought along with context notes.

```bash
python run_generation.py \
  --in_file retrieval_results.jsonl \
  --out_dir generation_logs \
  --model_name gpt-4o \
  --model_alias gpt4o \
  --openai_key sk-... \
  --retriever_type flat-session \
  --topk_context 10 \
  --history_format json \
  --useronly false \
  --cot true \
  --con true
```

--------------------------------

### Argument Parsing and Checking

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Utility functions for parsing and checking command-line arguments. Use `parse_args` to get arguments and `check_args` to validate them.

```python
parse_args() -> argparse.Namespace
```

```python
check_args(args: argparse.Namespace) -> None
```

--------------------------------

### Configure Local vLLM Server

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Configure connection to a local vLLM server using base URL and an empty API key. Ensure the server is running at the specified address.

```bash
--openai_base_url http://localhost:8001/v1
--openai_key EMPTY
```

--------------------------------

### Per-Task-Type Accuracy Example

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_qa_metrics.md

Accuracy metrics are reported per task type, showing the calculated mean accuracy and the count of questions for that type.

```text
single-session-user: 0.8500 (84)
```

--------------------------------

### Get Random Same-Day Timestamps

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Generates a list of random timestamps occurring on the same day. Use this to create data points with identical dates but varying times.

```python
get_random_same_day_timestamps(n: int, base_date: str = None) -> list
```

--------------------------------

### Display QA Evaluation Output

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/evaluate_qa.md

Example output from the QA evaluation script, showing overall accuracy and accuracy broken down by question type, along with the save location of the results.

```text
Accuracy: 0.8234
	single-session-user: 0.85 (84)
	single-session-assistant: 0.82 (83)
	temporal-reasoning: 0.79 (83)
	knowledge-update: 0.81 (83)
	multi-session: 0.80 (83)
	single-session-preference: 0.84 (84)
Saved to model_predictions.jsonl.eval-results-gpt-4o
```

--------------------------------

### Run Generation with GPT-4o (Retrieved Context)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Runs the generation script with GPT-4o, using flat session retrieval with a topk context of 5, and enabling user-only mode and chain-of-thought.

```bash
python run_generation.py \
  --in_file retrieval_results.jsonl \
  --out_dir generation_logs \
  --model_name gpt-4o-2024-08-06 \
  --model_alias gpt4o \
  --openai_key sk-... \
  --openai_organization org-... \
  --retriever_type flat-session \
  --topk_context 5 \
  --history_format json \
  --useronly true \
  --cot true
```

--------------------------------

### Run Retrieval Preparation

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Prepare retrieval data by running the retrieval script. Specify input file, output directory, retriever type, and granularity.

```python
python src/retrieval/run_retrieval.py \
  --in_file data/longmemeval_oracle.json \
  --out_dir logs \
  --retriever flat-stella \
  --granularity session
```

--------------------------------

### Generate Random Dates in Range

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Generates a list of random dates within a specified start and end date range. Use this for creating data points within a defined period.

```python
generate_random_dates_in_range(start_date_str: str, end_date_str: str, n: int) -> list
```

--------------------------------

### prepare_prompt

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Prepares a prompt for the generation module, incorporating retrieval context if specified.

```APIDOC
## prepare_prompt

### Description
Prepares a prompt for the generation module, potentially including retrieved context.

### Function Signature
```python
prepare_prompt(entry: dict, retriever_type: str, topk_context: int, useronly: bool, history_format: str, cot: bool, tokenizer, tokenizer_backend: str, max_retrieval_length: int, merge_key_expansion_into_value: str, con: bool = False, con_client = None, con_model: str = None) -> str
```

### Parameters
- **entry** (dict) - The current entry or turn data.
- **retriever_type** (str) - The type of retriever to use (e.g., 'flat-session', 'no-retrieval').
- **topk_context** (int) - The number of top contexts to retrieve.
- **useronly** (bool) - Whether to only include user turns in the history.
- **history_format** (str) - The desired format for the conversation history.
- **cot** (bool) - Whether to enable chain-of-thought prompting.
- **tokenizer** - The tokenizer object.
- **tokenizer_backend** (str) - The backend used by the tokenizer.
- **max_retrieval_length** (int) - The maximum length for retrieved context.
- **merge_key_expansion_into_value** (str) - Strategy for merging key expansions.
- **con** (bool, optional) - Whether to use conversational context. Defaults to False.
- **con_client** - The conversational client (if `con` is True).
- **con_model** (str, optional) - The conversational model name (if `con` is True).

### Returns
- str: The prepared prompt string.
```

--------------------------------

### Run Generation

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Execute the generation script with input file, output directory, model details, API key, retriever type, context settings, history format, and generation flags.

```bash
python src/generation/run_generation.py \
  --in_file data.json \
  --out_dir results \
  --model_name gpt-4o \
  --model_alias gpt4o \
  --openai_key sk-... \
  --retriever_type flat-session \
  --topk_context 5 \
  --history_format json \
  --useronly true \
  --cot true
```

--------------------------------

### Generate Random Dates Within a Date Range

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Generates a specified number of random dates that fall within a given start and end date range. The returned list of dates is sorted chronologically.

```python
def generate_random_dates_in_range(
    start_date_str: str,
    end_date_str: str,
    n: int
) -> list:
    # Return Type: list[str] - Sorted ascending
```

--------------------------------

### prepare_retriever()

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

Loads and initializes the specified dense retrieval model. Supports Contriever, Stella V5, and GTE-Qwen2.

```APIDOC
## prepare_retriever()

### Description
Loads and initializes the specified dense retrieval model. Supports Contriever, Stella V5, and GTE-Qwen2.

### Signature
```python
def prepare_retriever(self) -> None
```

### Supported Models

| Model | Identifier |
|-------|-----------|
| Contriever | `flat-contriever` |
| Stella V5 1.5B | `flat-stella` |
| GTE-Qwen2-7B | `flat-gte` |
```

--------------------------------

### Run Time-Aware Query Expansion

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Implement time-aware query expansion to prune the search space. This involves downloading timestamped event data and running a Python script from the 'src/index_expansion' directory, providing the event file, retrieval log, and granularity.

```python
cd src/index_expansion
python3 temp_query_search_pruning.py TIMESTAMP_EVENT_FILE RETRIEVAL_LOG GRANULARITY
```

--------------------------------

### Run Generation with Llama 3.1 70B (Full History Baseline)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Executes the generation script using the Llama 3.1 70B Instruct model with original session retrieval and chain-of-thought enabled.

```bash
python run_generation.py \
  --in_file ../../data/longmemeval_s.json \
  --out_dir generation_logs \
  --model_name meta-llama/Meta-Llama-3.1-70B-Instruct \
  --model_alias llama-70b \
  --openai_base_url http://localhost:8001/v1 \
  --openai_key EMPTY \
  --retriever_type orig-session \
  --topk_context 1000 \
  --history_format json \
  --useronly false \
  --cot true
```

--------------------------------

### Run Generation with GPT-4o (User Facts Merged)

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Configures the generation script for GPT-4o with flat session retrieval, merging key expansion into values, and enabling chain-of-thought.

```bash
python run_generation.py \
  --in_file retrieval_with_facts.jsonl \
  --out_dir generation_logs \
  --model_name gpt-4o \
  --model_alias gpt4o \
  --openai_key sk-... \
  --retriever_type flat-session \
  --topk_context 10 \
  --history_format json \
  --useronly false \
  --cot true \
  --merge_key_expansion_into_value merge
```

--------------------------------

### Create Custom Subset Bash Command

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/sample_haystack_and_timestamp.md

Demonstrates how to create a custom subset of the data, specifying the number of questions, session range, and a token limit.

```bash
python sample_haystack_and_timestamp.py single_hop 50 200 300 200000
```

--------------------------------

### Print QA Metrics

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Display the QA evaluation metrics from a log file against a reference JSON file.

```bash
python src/evaluation/print_qa_metrics.py eval_results.log reference.json
```

--------------------------------

### Evaluate Pre-Generated Predictions with GPT-4o

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/README.md

Run the evaluation script for QA with GPT-4o and then print the aggregated metrics. Ensure the OPENAI_API_KEY is set.

```bash
# 1. Run evaluation with GPT-4o
export OPENAI_API_KEY="sk-..."
cd src/evaluation
python evaluate_qa.py gpt-4o predictions.jsonl ../../data/longmemeval_oracle.json

# 2. Print aggregated metrics
python print_qa_metrics.py predictions.jsonl.eval-results-gpt-4o ../../data/longmemeval_oracle.json
```

--------------------------------

### DenseRetrievalMaster Constructor

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

Initializes the DenseRetrievalMaster with command-line arguments and a specified GPU ID.

```APIDOC
## DenseRetrievalMaster Constructor

### Description
Initializes the DenseRetrievalMaster with command-line arguments and a specified GPU ID.

### Signature
```python
class DenseRetrievalMaster:
    def __init__(self, args: argparse.Namespace, gpu_id: int)
```

### Parameters
#### Path Parameters
- **args** (argparse.Namespace) - Required - Command-line arguments from parse_args()
- **gpu_id** (int) - Required - GPU device ID to use for this instance
```

--------------------------------

### Retrieval with Index Expansion

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

This command shows how to perform retrieval with index expansion enabled. It includes parameters for the expansion method, result cache, and join mode, along with a cache directory for the model.

```bash
python run_retrieval.py \
  --in_file ../../data/longmemeval_m.json \
  --out_dir retrieval_logs \
  --retriever flat-stella \
  --granularity session \
  --index_expansion_method session-keyphrase \
  --index_expansion_result_cache expansion_cache.json \
  --index_expansion_result_join_mode merge \
  --cache_dir /path/to/cache
```

--------------------------------

### Execute print_qa_metrics Script

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_qa_metrics.md

Run the script from the command line, providing paths to the evaluation log and reference data files. The script outputs formatted accuracy metrics to standard output.

```bash
python print_qa_metrics.py in_file ref_file
```

--------------------------------

### Execute print_retrieval_metrics.py Script

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_retrieval_metrics.md

Run the script from the command line, providing the path to the retrieval log file as an argument. The script outputs formatted retrieval metrics to stdout.

```bash
python print_retrieval_metrics.py in_file
```

--------------------------------

### Run Retrieval Evaluation

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Initiate retrieval evaluation with specified input file, output directory, retriever type, granularity, and cache directory.

```bash
python src/retrieval/run_retrieval.py \
  --in_file data.json \
  --out_dir results \
  --retriever flat-stella \
  --granularity session \
  --cache_dir /path/to/models
```

--------------------------------

### Prepare Generation Prompt

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Prepares a prompt for the generation module, incorporating retrieval results and conversation history. Use this to format input for language models.

```python
prepare_prompt(entry: dict, retriever_type: str, topk_context: int, useronly: bool, history_format: str, cot: bool, tokenizer, tokenizer_backend: str, max_retrieval_length: int, merge_key_expansion_into_value: str, con: bool = False, con_client = None, con_model: str = None) -> str
```

--------------------------------

### Run Basic QA Evaluation with GPT-4o

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/evaluate_qa.md

Execute the QA evaluation script using the GPT-4o model. Ensure OpenAI API keys and organization are set as environment variables. The script requires the model name, a JSONL file of model predictions, and a reference JSON file.

```bash
export OPENAI_API_KEY='sk-...'
export OPENAI_ORGANIZATION='org-...'
cd src/evaluation
python evaluate_qa.py gpt-4o model_predictions.jsonl ../../data/longmemeval_oracle.json
```

--------------------------------

### Run Print QA Metrics Script

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_qa_metrics.md

Executes the print_qa_metrics.py script with the results log and oracle data file as arguments. This command generates a detailed evaluation report.

```bash
python print_qa_metrics.py gpt-4o_results.log ../../data/longmemeval_oracle.json
```

--------------------------------

### Prepare Retriever Model

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_retrieval.md

Loads and initializes the specified dense retrieval model. Supported models include Contriever, Stella V5, and GTE-Qwen2.

```python
def prepare_retriever(self) -> None
```

--------------------------------

### Run Long-Context Generation Baseline

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Execute this command to run the long-context generation baseline, providing the model with the full conversation history. Adjust parameters like DATA_FILE, MODEL, and TOPK as needed.

```bash
cd src/generation
bash run_generation.sh DATA_FILE MODEL full-history-session TOPK [HISTORY_FORMAT] [USERONLY] [READING_METHOD]
```

--------------------------------

### Merge Keyphrases into Document Text

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/index_expansion_utils.md

Shows how to fetch keyphrases for each turn and merge them into the document text, prepending them to the original content.

```python
# For each turn, fetch keyphrases and merge
for turn_id in turn_ids:
    keyphrases = fetch_expansion_from_cache(cache, turn_id)
    
    corpus, corpus_ids, timestamps = resolve_expansion(
        'turn-keyphrase', 'split-merge',
        corpus, corpus_ids, timestamps,
        keyphrases, turn_id, timestamp
    )

# Now each document contains keyphrases prepended
# "hobby music travel the user mentioned they enjoy music and travel"
```

--------------------------------

### Main Execution Function

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

Orchestrates prompt generation, model inference, and result logging. Initializes clients, loads data, prepares prompts, calls the model, extracts responses, and logs results.

```python
import argparse

def main(args: argparse.Namespace) -> None:
    pass
```

--------------------------------

### Project Data Directory Structure

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/README.md

Overview of the directory structure for the project's data files.

```tree
data/
├── longmemeval_oracle.json
├── longmemeval_s_cleaned.json
├── longmemeval_m_cleaned.json
└── custom_history/
    ├── 1_attr_bg/
    ├── 2_questions/
    ├── 5_filler_sess/
    ├── 6_session_cache/
    └── sample_haystack_and_timestamp.py
```

--------------------------------

### Script Execution for evaluate_qa

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/evaluate_qa.md

Execute the QA evaluation script with specified metric model and file paths. Ensure required environment variables like OPENAI_API_KEY are set.

```bash
python evaluate_qa.py metric_model hyp_file ref_file
```

--------------------------------

### Project Source Module Structure

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/README.md

Overview of the directory structure for the project's source code modules.

```tree
src/
├── evaluation/
│   ├── evaluate_qa.py
│   ├── print_qa_metrics.py
│   └── print_retrieval_metrics.py
├── retrieval/
│   ├── run_retrieval.py
│   ├── eval_utils.py
│   └── index_expansion_utils.py
├── generation/
│   ├── run_generation.py
│   └── run_generation.sh
├── index_expansion/
│   ├── batch_expansion_session_keyphrases.py
│   ├── batch_expansion_session_summ.py
│   ├── batch_expansion_session_userfact.py
│   ├── batch_expansion_turn_keyphrases.py
│   ├── batch_expansion_turn_userfact.py
│   ├── batch_expansion_session_temp_event.py
│   └── temp_query_search_pruning.py
└── utils/
    ├── serve_vllm.sh
    └── serve_vllm_with_maxlen.sh
```

--------------------------------

### Download LongMemEval Dataset

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Download the LongMemEval dataset files to the data directory. Ensure the data directory exists before downloading.

```bash
mkdir -p data/
cd data/
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_m_cleaned.json
cd ..
```

--------------------------------

### Prompt Templates for Different Retrieval Strategies

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/run_generation.md

These templates define the structure of the prompt based on the retrieval strategy used. They include placeholders for history, question, and optional chain-of-thought instructions.

```string
```
{question}
[+ "Answer step by step." if cot else ""]
```
```

```string
```
I will give you several history chats between you and a user. Please answer the question based on the relevant chat history. [+ "Answer the question step by step: first extract all the relevant information, and then reason over the information to get the answer." if cot else ""]

History Chats:

{history_string}

Current Date: {question_date}
Question: {question}
Answer[+(step by step) if cot else ""]
```
```

```string
```
I will give you several history chats between you and a user, as well as the relevant user facts extracted from the chat history. Please answer the question based on the relevant chat history and the user facts. [+ CoT instruction if cot]

History Chats:

{history_string}

Current Date: {question_date}
Question: {question}
Answer...
```
```

```string
```
I will give you several facts extracted from history chats between you and a user. Please answer the question based on the relevant facts. [+ CoT instruction if cot]

History Chats:

{history_string}

Current Date: {question_date}
Question: {question}
Answer...
```
```

--------------------------------

### Report Turn Metrics from Dense Retrieval

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/print_retrieval_metrics.md

Use this command to report both session-level and turn-level metrics for dense retrieval. This provides a more granular view of performance.

```bash
python print_retrieval_metrics.py retrieval_logs/longmemeval_m.json_retrievallog_turn_flat-stella
```

--------------------------------

### Sample Custom History Data

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Run a Python script to sample Haystack and timestamp data for custom history generation.

```bash
cd data/custom_history
python sample_haystack_and_timestamp.py single_hop 500 80 80 115000
```

--------------------------------

### Run Retrieval-Augmented Generation

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Execute the question answering process using retrieved memory. This command is run from the 'src/generation' directory and requires the retrieval log file, model experiment type, and top-k value. Optional parameters for history format, user-only mode, and reading method can also be provided.

```bash
cd src/generation
bash run_generation.sh RETRIEVAL_LOG_FILE MODEL EXP TOPK [HISTORY_FORMAT] [USERONLY] [READING_METHOD]
```

--------------------------------

### OpenAI Chat Completions with Backoff

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/evaluate_qa.md

A wrapper for the OpenAI chat completions API that includes exponential backoff retry logic for handling rate limits and API errors. Requires an initialized OpenAI client.

```python
from openai import OpenAI
import backoff

@backoff.on_exception(backoff.expo, (openai.RateLimitError, openai.APIError))
def chat_comcompletions_with_backoff(client, **kwargs) -> Any:
    # ... function implementation ...
    pass

client = OpenAI(api_key='your-key')
result = chat_completions_with_backoff(
    client,
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Is this answer correct?'}],
    temperature=0,
    max_tokens=10
)
answer = result.choices[0].message.content
```

--------------------------------

### Fetch Expansion from Cache

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/index_expansion_utils.md

Retrieves pre-computed expansion data for a given session or turn ID from a cache. Handles different data formats (string, list, None) and cleans IDs before lookup. Returns a list of expansion strings or None if not found or failed.

```python
cache = {
    'sess_1': 'Session summary text',
    'sess_2': ['user fact 1', 'user fact 2'],
    'sess_3': None  # Failed expansion
}

expansion = fetch_expansion_from_cache(cache, 'answer_sess_1')
# Returns: ['Session summary text'] (string converted to list)

expansion = fetch_expansion_from_cache(cache, 'sess_2')
# Returns: ['user fact 1', 'user fact 2'] (already list)

expansion = fetch_expansion_from_cache(cache, 'sess_3')
# Returns: None (failed/missing entry)
```

--------------------------------

### Run Baseline Memory Retrieval

Source: https://github.com/xiaowu0162/longmemeval/blob/main/README.md

Execute the baseline memory retrieval process. Ensure you are in the 'src/retrieval' directory and provide the input file, retriever type, and granularity. The script outputs logs and evaluation metrics.

```bash
cd src/retrieval
bash run_retrieval.sh IN_FILE RETRIEVER GRANULARITY
```

--------------------------------

### Run QA Evaluation with Local Llama Model

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/evaluate_qa.md

Execute the QA evaluation script with a local Llama model served via vLLM. This command assumes a vLLM server is running on the default port 8001. Specify the model name, predictions file, and reference data file.

```bash
cd src/evaluation
python evaluate_qa.py llama-3.1-70b-instruct predictions.jsonl data.json
```

--------------------------------

### Run QA Evaluation

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Execute the QA evaluation script with predicted and reference JSONL files.

```bash
python src/evaluation/evaluate_qa.py gpt-4o predictions.jsonl reference.json
```

--------------------------------

### Fetch Expansion from Cache

Source: https://github.com/xiaowu0162/longmemeval/blob/main/_autodocs/INDEX.md

Retrieves index expansion results from a cache. Use this to avoid redundant computations during index expansion.

```python
fetch_expansion_from_cache(index_expansion_result_cache: dict, cur_sess_id: str) -> list
```