LongMemEval-V2

https://github.com/xiaowu0162/longmemeval-v2
Admin
LongMemEval-V2 is a benchmark that evaluates whether long-term agent memory systems can help agents...
Tokens:4,644
Snippets:75
Trust Score:9.2
License:Apache-2.0
Update:2 months ago
Show doc for...
Tokens:
Raw
### Initialize Environment and Install Package

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Commands to create the conda environment and install the package in editable mode.

```bash
PYTHONNOUSERSITE=1 conda env create -f environment.yml
conda activate lme-v2-release
pip install -e .
```

--------------------------------

### Install PyTorch for CUDA 12.4

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Command to install the specific PyTorch build required for CUDA 12.4 environments.

```bash
pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 \
  --extra-index-url https://download.pytorch.org/whl/cu124
```

--------------------------------

### Download and Prepare Dataset

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Downloads, prepares, and validates the dataset using the provided Python scripts.

```bash
python data/download_data.py --data-root data/longmemeval-v2
export DATA_ROOT="$(pwd)/data/longmemeval-v2"
python data/prepare_data.py --data-root "$DATA_ROOT" --mode symlink
python data/validate_data.py --data-root "$DATA_ROOT" --tier small
```

--------------------------------

### Build a single operating point

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/leaderboard/README.md

Run this command for each latency operating point to process web and enterprise run data.

```bash
python leaderboard/build_submission_step_1_single_operating_point.py \
  runs/agentrunbook_c_web_small \
  runs/agentrunbook_c_enterprise_small \
  submission_1 \
  fast \
  small
```

```bash
python leaderboard/build_submission_step_1_single_operating_point.py \
  runs/my_method_fast_web_small \
  runs/my_method_fast_enterprise_small \
  submission_1 \
  fast \
  small \
  --method my_method
```

--------------------------------

### Build the final submission package

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/leaderboard/README.md

Combine multiple operating points into a final package after completing step 1 for all points.

```bash
python leaderboard/build_submission_step_2_build_package.py \
  submission_1 \
  SYSTEM_DESCRIPTION.md \
  path/to/code_file.py \
  leaderboard/submissions/submission_1/operating_points/fast \
  leaderboard/submissions/submission_1/operating_points/balanced \
  leaderboard/submissions/submission_1/operating_points/accurate
```

--------------------------------

### Step 2 output structure

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/leaderboard/README.md

The directory structure and final tarball generated after running the second step.

```text
leaderboard/submissions/submission_1/
  SYSTEM_DESCRIPTION.md
  code_file.py
  submission_overview.json
  operating_points/
    fast/
    balanced/
    accurate/

leaderboard/submissions/submission_1.tar.gz
```

--------------------------------

### Verify submission package

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/leaderboard/README.md

Inspect the contents of the final tarball and the submission overview file before submitting.

```bash
tar -tzf leaderboard/submissions/submission_1.tar.gz | head
python -m json.tool leaderboard/submissions/submission_1/submission_overview.json
```

--------------------------------

### Step 1 output structure

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/leaderboard/README.md

The directory structure generated after running the first step of the submission process.

```text
leaderboard/submissions/submission_1/
  operating_points/
    fast/
      metric_overview.json
      operating_point_metadata.json
      web/
        aggregated_metrics.json
        per_question.jsonl
        run_args.json
        runtime_inputs/
      enterprise/
        aggregated_metrics.json
        per_question.jsonl
        run_args.json
        runtime_inputs/
```

--------------------------------

### Memory Configuration JSON

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Define the memory type and parameters in a JSON file to run a new backend.

```json
{
  "memory_type": "your_memory_type",
  "memory_params": {}
}
```

--------------------------------

### Repository Directory Structure

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Overview of the project's primary directories and their purposes.

```text
data/                 download, preparation, and validation scripts
evaluation/           evaluation runner, scoring code, configs, and shell wrappers
leaderboard/          metric merging, LAFS scoring, and submission packaging
memory_modules/       memory backend implementations
```

--------------------------------

### Configure Model Endpoints

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Sets environment variables for reader, controller, and embedding models.

```bash
# for all experiments
export READER_BASE_URL=http://localhost:8023/v1
export READER_MODEL=Qwen/Qwen3.5-9B

# additionally for RAG and AgentRunbook-R
export LME_CONTROLLER_BASE_URL=http://localhost:8023/v1
export LME_CONTROLLER_MODEL=Qwen/Qwen3.5-9B
export LME_EMBEDDING_BASE_URL=http://localhost:8114/v1
export LME_EMBEDDING_MODEL=Qwen/Qwen3-Embedding-8B
```

--------------------------------

### Configure LLM Judge API Key

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Sets the OpenAI API key for the LLM judge.

```bash
export OPENAI_API_KEY=...
```

--------------------------------

### Run Baseline Evaluations

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Executes various baseline evaluation scripts with specified environment variables.

```bash
export DATA_ROOT=/path/to/longmemeval-v2
export OUTPUT_ROOT=runs
export TIER=small

evaluation/scripts/run_no_retrieval.sh
evaluation/scripts/run_rag_query_to_slice.sh
evaluation/scripts/run_rag_query_to_slice_notes.sh
evaluation/scripts/run_agentrunbook_r.sh
evaluation/scripts/run_codex.sh
evaluation/scripts/run_agentrunbook_c.sh
```

--------------------------------

### Configure Codex Environment

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Sets environment variables required for Codex and AgentRunbook-C execution.

```bash
export CODEX_BINARY=/path/to/codex-binary
export CODEX_MODEL=gpt-5.4-mini
export CODEX_REASONING_EFFORT=xhigh
```

--------------------------------

### Screenshot Directory Structure

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

The expected directory structure for extracted trajectory screenshots.

```text
screenshots/<trajectory_id>/<step>.png
```

--------------------------------

### Combine Aggregated Metrics

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Combines enterprise and web evaluation metrics into a single JSON file.

```bash
python leaderboard/combine_aggregated_metrics.py \
  runs/agentrunbook_r_enterprise_small/aggregated_metrics.json \
  runs/agentrunbook_r_web_small/aggregated_metrics.json \
  -o runs/agentrunbook_r_small_combined_metrics.json
```

--------------------------------

### Memory Query Return Format

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

The query method must return a list of dictionaries containing either text or image types.

```python
[
    {"type": "text", "value": "retrieved notes or evidence"},
    {"type": "image", "value": "/absolute/or/relative/path/to/image.png"},
]
```

--------------------------------

### Memory Module Output Schema

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/memory_modules/assets/agentrunbook_c/INSTRUCTION.md

The required JSON structure for the final output file memory_module_output.json.

```json
{
  "memory_markdown": "## Support Analysis\n...\n\n## Relevant Procedure and Hint Notes\n...",
  "trajectory_spans": [
    {
      "trajectory_id": "<trajectory id>",
      "start_state_index": 0,
      "end_state_index": 0
    }
  ]
}
```

--------------------------------

### Inspect Trajectory Data

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/memory_modules/assets/agentrunbook_c/INSTRUCTION.md

Use these commands to inspect trajectory summaries, specific states, or spans of states without manually searching raw JSON files.

```bash
python scripts/inspect_trajectory.py <trajectory_id>
```

```bash
python scripts/inspect_trajectory.py <trajectory_id> --state 7
```

```bash
python scripts/inspect_trajectory.py <trajectory_id> --span 6:8
```

```bash
python scripts/inspect_trajectory.py <trajectory_id> --match "Delete Review|Previous"
```

--------------------------------

### Cite LongMemEval-V2 in BibTeX

Source: https://github.com/xiaowu0162/longmemeval-v2/blob/main/README.md

Use this BibTeX entry to cite the LongMemEval-V2 paper in academic publications.

```bibtex
@article{wu2026longmemevalv2,
      title={LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues}, 
      author={Di Wu and Zixiang Ji and Asmi Kawatkar and Bryan Kwan and Jia-Chen Gu and Nanyun Peng and Kai-Wei Chang},
      year={2026},
      eprint={2605.12493},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2605.12493}, 
}
```
Tokens:
Raw