### Deploy MemAgent Locally with vLLM

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Starts a vLLM server for local MemAgent deployment and runs quickstart inference.

```bash
vllm serve BytedTsinghua-SIA/RL-MemoryAgent-14B --tensor_parallel_size 2
```

```bash
python MemAgent/quickstart.py --model BytedTsinghua-SIA/RL-MemoryAgent-14B
```

--------------------------------

### Run MemAgent Quickstart with vLLM

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Execute the quickstart script to use MemAgent with a locally deployed vLLM server. Ensure the model name matches the one served.

```bash
python quickstart.py --model BytedTsinghua-SIA/RL-MemoryAgent-14B
```

--------------------------------

### Execute Split Placement Example Script

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/split_placement/README.md

Run the shell script to execute the split placement example. This command initiates the PPO algorithm with the configured split placement strategy.

```bash
bash run_deepseek7b_llm.sh
```

--------------------------------

### Install Dependencies

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Installs necessary Python packages and Ray for distributed computing. Ensure you have the specified httpx version.

```bash
pip install httpx==0.23.1 aiohttp -U ray[serve,default] vllm
```

--------------------------------

### Dependency Installation Output

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

This output shows the process of installing the verl package and its dependencies. It indicates successful installation and lists all satisfied requirements.

```text
Obtaining file:///teamspace/studios/this_studio/verl_repo
  Installing build dependencies ... [?25ldone
[?25h  Checking if build backend supports build_editable ... [?25ldone
[?25h  Getting requirements to build editable ... [?25ldone
[?25h  Preparing editable metadata (pyproject.toml) ... [?25ldone
[?25hRequirement already satisfied: accelerate in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (1.1.1)
Requirement already satisfied: codetiming in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (1.4.0)
Requirement already satisfied: datasets in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (3.1.0)
Requirement already satisfied: dill in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (0.3.8)
Requirement already satisfied: hydra-core in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (1.3.2)
Requirement already satisfied: numpy in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (1.26.4)
Requirement already satisfied: pybind11 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (2.13.6)
Requirement already satisfied: ray in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (2.10.0)
Requirement already satisfied: tensordict in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (0.5.0)
Requirement already satisfied: transformers in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (4.46.3)
Requirement already satisfied: vllm<=0.6.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from verl==0.1) (0.5.4)
Requirement already satisfied: cmake>=3.21 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (3.31.1)
Requirement already satisfied: ninja in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (1.11.1.2)
Requirement already satisfied: psutil in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (6.1.0)
Requirement already satisfied: sentencepiece in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.2.0)
Requirement already satisfied: requests in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (2.32.3)
Requirement already satisfied: tqdm in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (4.67.1)
Requirement already satisfied: py-cpuinfo in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (9.0.0)
Requirement already satisfied: tokenizers>=0.19.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.20.3)
Requirement already satisfied: fastapi in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.115.4)
Requirement already satisfied: aiohttp in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (3.10.10)
Requirement already satisfied: openai in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (1.55.3)
Requirement already satisfied: uvicorn[standard] in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.32.0)
Requirement already satisfied: pydantic>=2.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (2.9.2)
Requirement already satisfied: pillow in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (10.4.0)
Requirement already satisfied: prometheus-client>=0.18.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.21.0)
Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (7.0.0)
Requirement already satisfied: tiktoken>=0.6.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.7.0)
Requirement already satisfied: lm-format-enforcer==0.10.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.10.3)
Requirement already satisfied: outlines<0.1,>=0.0.43 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from vllm<=0.6.3->verl==0.1) (0.0.46)

```

--------------------------------

### Start vLLM Server for MemAgent

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Use this command to start a vLLM server with the specified MemAgent model. Adjust tensor_parallel_size based on your hardware.

```bash
vllm serve BytedTsinghua-SIA/RL-MemoryAgent-14B --tensor_parallel_size 2
```

--------------------------------

### Setup AsyncLLMGenerationManager

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_async_generation_output.ipynb

Initializes the Dummy class which inherits from AsyncLLMGenerationManager. The chat template is set using the provided tokenizer.

```python
from recurrent.interface import AsyncOutput
from recurrent.chat_template.utils import set_chat_template

from recurrent.async_generation_manager import AsyncLLMGenerationManager
class Dummy(AsyncLLMGenerationManager):
    def __init__(self, tokenizer):
        self.tokenizer = tokenizer
        set_chat_template(tokenizer)

manager = Dummy(
    tokenizer,
)
```

--------------------------------

### Configure API Keys

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Copy the example environment file and edit it with your API credentials for LLM access. Supports OpenAI and Azure OpenAI.

```bash
cp .env.example .env

# Edit .env with your credentials
# For OpenAI:
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-5-chat
OPENAI_MODEL_EMBED=text-embedding-3-large

# For Azure OpenAI:
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_KEY=your_azure_openai_key_here
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-5-chat
AZURE_OPENAI_API_VERSION=your_api_version
```

--------------------------------

### Configure API Keys

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Copies the example environment file and prompts the user to fill in API credentials for OpenAI or Microsoft Azure. Ensure you replace placeholders with your actual keys and model names.

```bash
cp .env.example .env
```

```bash
OPENAI_API_KEY=your_openai_api_key_here
OPENAI_MODEL=gpt-5-chat
OPENAI_MODEL_EMBED=text-embedding-3-large
```

```bash
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_KEY=your_azure_openai_key_here
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-5-chat
AZURE_OPENAI_API_VERSION=your_api_version
AZURE_OPENAI_DEPLOYMENT_NAME_EMBED=text-embedding-3-large
AZURE_OPENAI_API_VERSION_EMBED=your_embed_api_version
```

--------------------------------

### Start Local Ray Cluster

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/tests/ray/detached_worker/README.md

Use this command to initiate a local Ray cluster. Ensure the port is available.

```bash
ray start --head --port=6379
```

--------------------------------

### Configure and Run MemAgent with Online LLM Service

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Sets environment variables for an online LLM service and runs MemAgent quickstart inference.

```bash
export URL="https://your-endpoint/v1"
export API_KEY="your-api-key"
```

```bash
python MemAgent/quickstart.py --model gpt-4o-2024-11-20
```

--------------------------------

### Install verl and Dependencies

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Navigate to the cloned repository directory and install the verl package and its dependencies using pip. This command ensures the project is installed in editable mode.

```bash
!cd $HOME/verl_repo && pip3 install -e . -U
```

--------------------------------

### Install verl Package

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Use this command to install the verl package if you haven't cloned the repository yet. Ensure you have the necessary environment set up.

```python
# In case you run this notebook and have not cloned verl yet:

```

--------------------------------

### Run Server

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/tests/ray/detached_worker/README.md

Execute the server script after the Ray cluster has started.

```bash
python3 server.py
```

--------------------------------

### Install and Upgrade Python Dependencies

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Use these commands to upgrade pip, setuptools, and wheel, and to install specific versions of PyTorch and torchvision. Ensure you have the correct CUDA version for PyTorch.

```python
!pip3 install --upgrade pip setuptools wheel
```

```python
!pip3 install torch==2.4.0 torchvision==0.19.0
```

```python
!pip3 list | grep torch
```

```python
!pip3 install flash-attn --no-build-isolation
```

--------------------------------

### Initialize Local Ray Cluster

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Starts a local Ray cluster. This is the first step for distributed execution on a single machine.

```python
ray.init()
```

--------------------------------

### Download Hugging Face Model

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

This command-line instruction downloads a specified Hugging Face model to a local directory. Ensure `huggingface-cli` is installed and accessible in your PATH.

```bash
!huggingface-cli download Qwen/Qwen2.5-0.5B-Instruct --local-dir $HOME/models/Qwen2.5-0.5B-Instruct
```

--------------------------------

### Import Ray and Warnings

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Imports necessary libraries for Ray and warning management. Ensure Ray is installed and configured for your environment.

```python
import os
```

```python
import warnings

import ray
import torch

warnings.filterwarnings("ignore")
```

--------------------------------

### Install Data Processing Dependencies

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Installs Python packages required for data processing, including NLTK, PyYAML, BeautifulSoup, and others.

```bash
cd taskutils/memory_data
pip install nltk pyyaml beautifulsoup4 html2text wonderwords tenacity fire
```

--------------------------------

### Call Function with Python Requests

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_tool_tokenize.ipynb

Use the requests library to send a POST request to the chat completions endpoint. This example demonstrates calling a function with specific arguments and calculating a sum.

```python
import requests
chat_completion = requests.post(
    url = "http://localhost:8000/v1/chat/completions",
    json=dict(
        model="Qwen2.5-Coder-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "wrtie xml code: <request></request> to call function, e.g. <tool_call>{'name': 'add', 'args': {'a': 1, 'b': 2}}, calculate 3+3.",
        },
    ],
    temperature=1,
    )
)
data = chat_completion.json()
data
```

--------------------------------

### Restart Python Kernel

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Use this snippet to restart the Python kernel after installing new packages to ensure they are recognized by the environment.

```python
import IPython

# Restart the kernel to pickup the latest python packages
IPython.get_ipython().kernel.do_shutdown(restart=True)
```

--------------------------------

### Custom AsyncRAgent Implementation

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Implement a custom asynchronous agent that processes data in chunks, maintains memory, and generates final responses. Requires setup with an AsyncRAgent, ChatCompletionProxy, tokenizer, and configuration objects.

```python
from recurrent.interface import AsyncRAgent, AsyncOutput, RConfig
from recurrent.async_utils import ChatCompletionProxy
from verl.protocol import DataProtoItem
import torch

class CustomAsyncAgent(AsyncRAgent):
    def __init__(self, proxy: ChatCompletionProxy, tokenizer, config: RConfig, rollout_config):
        super().__init__(proxy, tokenizer, config, rollout_config)

    async def rollout(self, gen_item: DataProtoItem) -> AsyncOutput:
        """Rollout a single sample asynchronously."""
        timing_raw = {}
        conversations = []
        memory = None

        # Process chunks iteratively
        for step in range(self.config.max_chunks):
            chunk = self.get_chunk(gen_item, step)

            conversation = [{"role": "user", "content": f"Memory: {memory}\nChunk: {chunk}"}]

            # Async LLM call
            completions, err = await self.proxy.get_chat_completions(
                messages=conversation,
                max_completion_tokens=self.config.max_memorization_length
            )

            memory = completions.choices[0].message.content
            conversations.append(conversation + [{"role": "assistant", "content": memory}])

        # Final response generation
        final_conv = [{"role": "user", "content": f"Based on memory: {memory}, answer: {gen_item.non_tensor_batch['prompt']}"}]
        completions, err = await self.proxy.get_chat_completions(messages=final_conv)
        conversations.append(final_conv + [{"role": "assistant", "content": completions.choices[0].message.content}])

        sample_index = torch.full((len(conversations),), gen_item.batch['sample_index'].item())
        final_mask = torch.zeros(len(conversations), dtype=torch.bool)
        final_mask[-1] = True

        return AsyncOutput(conversations, sample_index, final_mask, timing_raw)
```

--------------------------------

### Run PPO Algorithm with Qwen 2.5-0.5B

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Execute the PPO training pipeline using the main_ppo.py script. This command configures various parameters for data loading, model paths, batch sizes, and GPU settings. Adjust micro batch sizes to mitigate out-of-memory issues.

```bash
!PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
 data.train_files=$HOME/data/gsm8k/train.parquet \
 data.val_files=$HOME/data/gsm8k/test.parquet \
 data.train_batch_size=256 \
 data.max_prompt_length=512 \
 data.max_response_length=256 \
 actor_rollout_ref.model.path=$HOME/models/Qwen2.5-0.5B-Instruct \
 actor_rollout_ref.actor.optim.lr=1e-6 \
 actor_rollout_ref.actor.ppo_mini_batch_size=64 \
 actor_rollout_ref.actor.ppo_micro_batch_size=1 \
 actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1 \
 actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
 actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
 actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
 critic.optim.lr=1e-5 \
 critic.model.path=$HOME/models/Qwen2.5-0.5B-Instruct \
 critic.ppo_micro_batch_size=1 \
 algorithm.kl_ctrl.kl_coef=0.001 \
 trainer.val_before_train=False \
 trainer.default_hdfs_dir=null \
 trainer.n_gpus_per_node=1 \
 trainer.nnodes=1 \
 trainer.save_freq=10 \
 trainer.test_freq=10 \
 trainer.total_epochs=15 \
 trainer.logger=\[console\]
```

--------------------------------

### Cold-start Base Model with SFT

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Optionally cold-starts the base model using Supervised Fine-Tuning (SFT) before GRPO training. This script is specific to the Qwen3 4B model.

```bash
bash verl_custom/scripts/run_qwen3_4b_sft.sh
```

--------------------------------

### Build and Run Docker Environment

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Build the Docker image for PersonaMem and run a container with GPU support. Mount the local project directory to the container for development.

```bash
docker build -t persona_mem .

docker run -it --gpus all -v /path/to/PersonaMem-v2:/workspace persona_mem /bin/bash
```

--------------------------------

### PPO Configuration Output

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

This output shows the initial configuration parameters loaded for the PPO algorithm, including settings for the actor, critic, and rollout components. Note that 'ppo_micro_batch_size' in the output may differ from the command-line argument if overridden by default values or other configurations.

```yaml
2025-01-10 21:40:29,298	INFO worker.py:1752 -- Started a local Ray instance.
(main_task pid=28294) {'actor_rollout_ref': {'actor': {'clip_ratio': 0.2,
(main_task pid=28294)                                  'entropy_coeff': 0.001,
(main_task pid=28294)                                  'fsdp_config': {'grad_offload': False,
(main_task pid=28294)                                                  'optimizer_offload': False,
(main_task pid=28294)                                                  'param_offload': False,
(main_task pid=28294)                                                  'wrap_policy': {'min_num_params': 0}},
(main_task pid=28294)                                  'grad_clip': 1.0,
(main_task pid=28294)                                  'optim': {'lr': 1e-06,
(main_task pid=28294)                                            'lr_warmup_steps_ratio': 0.0,
(main_task pid=28294)                                            'min_lr_ratio': None,
(main_task pid=28294)                                            'total_training_steps': -1,
(main_task pid=28294)                                            'warmup_style': 'constant'},
(main_task pid=28294)                                  'ppo_epochs': 1,
(main_task pid=28294)                                  'ppo_micro_batch_size': 4,
(main_task pid=28294)                                  'ppo_mini_batch_size': 64,
(main_task pid=28294)                                  'shuffle': True,
(main_task pid=28294)                                  'strategy': 'fsdp'},

```

--------------------------------

### Build and Run Docker Container

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Builds a Docker image for the project and runs it, mounting the local PersonaMem-v2 directory into the container. Use this for setting up the development environment.

```bash
# Build
docker build -t persona_mem .

# Run with all GPUs
docker run -it --gpus all -v /path/to/PersonaMem-v2:/workspace persona_mem /bin/bash
```

--------------------------------

### Download Qwen2.5-Instruct Models

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Downloads Qwen2.5-Instruct models using 'hfd.sh'. Note that Qwen2.5-Instruct models require manual configuration of 'config.json' to activate YaRN.

```bash
bash hfd.sh Qwen/Qwen2.5-7B-Instruct --tool aria2c -x 10
bash hfd.sh Qwen/Qwen2.5-14B-Instruct --tool aria2c -x 10
bash hfd.sh Qwen/Qwen2.5-32B-Instruct --tool aria2c -x 10
```

--------------------------------

### Organize Qwen2.5 Models

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Moves downloaded Qwen2.5 models to a specified root directory and appends '-128K' to their names, likely for context length configuration.

```bash
export MODELROOT=/your/path/to/models
mv Qwen2.5-7B-Instruct $MODELROOT/Qwen2.5-7B-Instruct-128K
mv Qwen2.5-14B-Instruct $MODELROOT/Qwen2.5-14B-Instruct-128K
mv Qwen2.5-32B-Instruct $MODELROOT/Qwen2.5-32B-Instruct-128K
```

--------------------------------

### Run Digit Completion Experiment

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/tests/e2e/arithmetic_sequence/rl/README.md

Execute the PPO trainer for digit completion. Specify the configuration path and name. The model configuration path can be overridden if needed.

```bash
# cd examples/arithmetic_sequence/rl

# Specify the config path and config name (current working dir)
python3 -m verl.trainer.ppo.ray_megatron_train_synchronous --config-path=$(pwd)/config --config-name='ray_megatron'
```

```bash
# The default relative path of model config is 'config/model_config', if you want to change it, you can rewrite it in ray_megatron.yaml or using:
python3 -m verl.trainer.ppo.ray_megatron_train_synchronous --config-path=$(pwd)/config --config-name='ray_megatron' ++model.base_path=config/model_config
```

--------------------------------

### Get GSM8k Reward Function Source

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Retrieves the source code for the GSM8k reward computation function. This function is used to score model outputs against ground truth.

```python
import inspect

from verl.utils.reward_score.gsm8k import compute_score as gsm8k_reward

print(inspect.getsource(gsm8k_reward))
```

--------------------------------

### Download QA Dataset

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Navigates to the QA data directory and downloads the dataset using a bash script. This is a prerequisite for testing.

```bash
cd taskutils/memory_data
bash download_qa_dataset.sh
```

--------------------------------

### Stop Ray Processes

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

Use this command to stop all running Ray processes and clean up resources. It's useful for ensuring a clean state before starting new Ray jobs or when encountering issues.

```bash
!ray stop

```

--------------------------------

### Find Substrings in a List

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_token_template.ipynb

Defines a function to find all occurrences of substrings within a list, delimited by specified start and end lists. This function is useful for parsing structured data within lists.

```python
def find_substrings(lst, startlist, endlist):
    start_len = len(startlist)
    end_len = len(endlist)
    substrings = []
    
    i = 0
    while i <= len(lst) - start_len - end_len:
        # Find start part
        if lst[i:i+start_len] == startlist:
            j = i + start_len
            # Find end part
            while j <= len(lst) - end_len and lst[j:j+end_len] != endlist:
                j += 1
            if j <= len(lst) - end_len and lst[j:j+end_len] == endlist:
                substrings.append(lst[i+start_len:j])
        i += 1
    
    return substrings

# Example usage
lst = [1, 2, 3, 4, 5]
startlist = [1, 2] # <summary>
endlist = [4, 5] # </summary>
print(find_substrings(lst, startlist, endlist))
```

--------------------------------

### Download Base Model Checkpoint

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Downloads the initial model checkpoint required for training. This script fetches the necessary pre-trained weights.

```bash
bash verl_custom/scripts/download_model.sh
```

--------------------------------

### Initialize Ray Resource Pool

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Sets up a resource pool for Ray, specifying the number of GPUs to use and the maximum number of collocated tasks.

```python
resource_pool = RayResourcePool([4], use_gpu=True, max_colocate_count=1)
```

--------------------------------

### Run Client

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/tests/ray/detached_worker/README.md

Connect to the running server and Ray cluster by executing the client script in a separate terminal.

```bash
python3 client.py
```

--------------------------------

### Initialize Qwen2.5 Tokenizer

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_tool_tokenize.ipynb

Initializes the AutoTokenizer for the Qwen/Qwen2.5-0.5B-Instruct model. This is the first step before applying chat templates or tokenizing text.

```python
from transformers import AutoTokenizer

# Initialize tokenizer
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Configure Training Scripts

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Sets the PROJ_ROOT and DATASET_ROOT environment variables in the provided shell scripts before launching training.

```bash
# First specify `PROJ_ROOT` (for checkpoints) and `DATASET_ROOT` (for training data, should be the same as used in testing) in `run_memory_7B.sh` and `run_memory_14B.sh`.
```

--------------------------------

### Data Generation Main Script Usage

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Illustrates the usage of the main data generation script with key command-line arguments for controlling pipeline steps, models, and generation parameters.

```python
# data_generation/main.py usage
import argparse
from data_generation.main import main

# Key command-line arguments:
# --model: LLM model name (default: gpt-5-chat)
# --step: Pipeline step (generate_conv, generate_qa, build_chat_history, fill_category, add_pref_others)
# --num_persona: Number of personas to generate (default: 1000)
# --data_types: Conversation types (email, chat_message, creative_writing, etc.)
# --context_length: Total context length in tokens (default: 32000)
# --version: Context version ('32k' or '128k')
# --parallel: Enable parallel processing
# --rate_limit_per_min: API rate limit
# --validate_qa: Enable QA validation filtering
```

--------------------------------

### Data Generation Main Script Help

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Displays help information for the main data generation script, showing all available options for customization. Use this to understand parameters like persona count and rate limits.

```python
PYTHONPATH=. python data_generation/main.py --help
```

--------------------------------

### Prepare DAPO Data

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recipe/dapo/README.md

This script downloads the necessary datasets for DAPO training. It defaults to saving them in the ${HOME}/verl/data directory.

```bash
bash prepare_dapo_data.sh # This downloads the datasets to ${HOME}/verl/data by default
```

--------------------------------

### Initialize and Use Tokenizer with TokenTemplate

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_token_template.ipynb

Initializes a tokenizer and uses TokenTemplate to format a string with provided keyword arguments. This is useful for preparing input for language models.

```python
import re
from transformers import AutoTokenizer

# Initialize tokenizer
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')

from utils import TokenTemplate


# Example usage
TEMPLATE = """Here is a problem, a section of a article that may contain the answer to the problem and a previous memory. Please read carefully and update the memory based on given section to help answer the problem. 
You should keep the relevant information in the memory while adding new information.

<problem> 
{problem}
</problem>

<memory>
{memory}
</memory>

<section>
{section}
</section>

Updated memory (should be enclosed in <memory> and </memory>)
<memory>
"""

processor = TokenTemplate(TEMPLATE)
processor.init(tokenizer)

# Assume input token ids (should be obtained from the model or elsewhere in actual use)
kwarg_text = dict(
    problem="What is the capital of France?",
    section="Here is a introduction to France. France is a country in Western Europe. Its capital is Paris.",
    memory="No previous memory",
)
kwargs_token_ids = {
    k: tokenizer.encode(v, add_special_tokens=False) for k, v in kwarg_text.items()
}
# Format template
formatted_template = processor.format(**kwargs_token_ids)
print(tokenizer.decode(formatted_template))
print(TEMPLATE.format(**kwarg_text) == tokenizer.decode(formatted_template))
```

--------------------------------

### MemAgent Data Preprocessing

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Preprocess data for MemAgent agentic memory training using MCQ format. Requires navigating to the MemAgent directory and specifying paths to benchmark data and configuration files.

```bash
# Preprocess for MemAgent training
cd MemAgent
python data/data_preprocess.py \
    --text_benchmark_csv ../data/benchmark/text/benchmark.csv \
    --local_dir data/implicit_persona \
    --config_file verl/trainer/config/ppo_trainer.yaml \
    --script_file run_qwen3_4b_grpo.sh \
    --model_path ../verl_custom/hub/models--Qwen--Qwen3-4B-Instruct-2507
```

--------------------------------

### Run Full Data Generation Pipeline

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Executes the complete pipeline for regenerating the benchmark dataset from scratch. This includes multiple sequential steps for data generation.

```bash
bash scripts/data_gen_scripts/run_generate_all.sh
```

--------------------------------

### Run Evaluation Script

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Executes the main evaluation script. This process can take several days. It utilizes all available GPUs.

```bash
cd taskutils/memory_eval
python run.py
```

--------------------------------

### Convert Benchmark to VERL Format

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Preprocess PersonaMem benchmark data into VERL-compatible parquet format for GRPO training. Includes options for prompt length and enabling thinking steps.

```bash
# Preprocess data for GRPO training
python verl_custom/data_preprocess_rft.py \
    --text_train_csv benchmark/text/train.csv \
    --text_val_csv benchmark/text/benchmark.csv \
    --local_dir verl_custom/data/implicit_persona \
    --model_path verl_custom/hub/models--Qwen--Qwen3-4B-Instruct-2507 \
    --max_prompt_length 38000 \
    --enable_thinking

# Check existing parquet files
python verl_custom/data_preprocess_rft.py --check-only
```

--------------------------------

### Configure Online LLM Service for MemAgent

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Set environment variables for URL and API_KEY to use MemAgent with online LLM services. The URL format depends on whether you are using normal services or Azure OpenAI.

```bash
export URL=
export API_KEY=
python quickstart.py --model gpt-4o-2024-11-20
```

--------------------------------

### Instantiate and Use Remote Actor

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Instantiates a remote 'Accumulator' actor and demonstrates retrieving its initial value and then adding to it. Note that remote calls return immediately.

```python
# Instantiate an accumulator. Accumulator can be viewed as a process, acting as an RPC service.
accumulator = Accumulator.remote()
```

```python
value_ref = accumulator.get_value.remote()  # Check the current value. Note that this function returns immediately and does not actually wait for the remote execution to complete.
# Get the value
value = ray.get(value_ref)
print(value)
```

```python
# Accumulate, then check the result.
accumulator.add.remote(10)  # Similarly, the 'add' here will return immediately.
new_value = ray.get(accumulator.get_value.remote())
print(new_value)
```

--------------------------------

### Async LLM Query with MemAgent Memory Pattern

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Demonstrates the core asynchronous function for querying LLMs using the MemAgent memory chunking pattern. Requires initialization of a tokenizer and preparation of input data.

```python
import asyncio
from transformers import AutoTokenizer
from MemAgent.quickstart import async_query_llm, RECURRENT_CHUNK_SIZE, RECURRENT_MAX_NEW

# Initialize tokenizer
tokenizer = AutoTokenizer.from_pretrained("BytedTsinghua-SIA/RL-MemoryAgent-14B")

# Prepare input with long context
item = {
    "context": "This is a very long conversation history... " * 1000,
    "input": "What is the user's preferred communication style?",
    "_id": 0
}

# Run async query with memory chunking
result = asyncio.run(async_query_llm(
    item=item,
    model="BytedTsinghua-SIA/RL-MemoryAgent-14B",
    tokenizer=tokenizer,
    temperature=0.7,
    top_p=0.95
))

print(f"Response: {result}")
```

--------------------------------

### Train Model with GRPO (Agentic Memory)

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Runs training with GRPO for a model using the Agentic Memory framework. This script is specific to the Qwen3 4B model.

```bash
bash MemAgent/run_qwen3_4b_grpo.sh
```

--------------------------------

### Prepare GSM8k Dataset

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

This command executes a Python script to download and preprocess the GSM8k dataset, saving it to a specified local directory in parquet format. This is part of the dataset preparation for training.

```bash
!mkdir -p $HOME/data/gsm8k
!python3 $HOME/verl_repo/examples/data_preprocess/gsm8k.py --local_dir $HOME/data/gsm8k
```

--------------------------------

### Initialize Tokenizer

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_async_generation_output.ipynb

Loads the tokenizer from a pre-trained model. Ensure the model name is correct.

```python
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Apply Chat Template with System Message and Tool

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_tool_tokenize.ipynb

Applies a chat template including a system message and a user message, along with a defined tool. This prepares the input for the model, incorporating system instructions and tool capabilities. The output is a formatted string due to `tokenize=False`.

```python
async def tool(a: int, b: int) -> tuple[int, int]:
    """
    Returns the sum of two numbers.

    Args:
        a: aaa
        b: bbb
        

    Returns:
        c: ccc
        d: ddd
    """
    return a + b, a-b
a = tokenizer.apply_chat_template(
    [
        {"role": "system", "content": "666"},
        {"role": "user", "content": "hello"},
    ],
    tools = [tool],
    tokenize=False
)
print(a)
```

--------------------------------

### Initialize Local Ray Cluster

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Initializes a local Ray cluster on the current machine. This is useful for development and testing purposes.

```python
# Build a local ray cluster. The head node and worker node are on this machine
ray.init()
```

--------------------------------

### Convert Huggingface Model to mcore GPTModel

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/verl/models/mcore/readme.md

This process involves converting Huggingface configurations to mcore TransformerConfig, initializing the mcore GPTModel, and loading Huggingface weights. It is crucial for leveraging mcore's computational efficiency.

```python
from megatron.core.transformer import TransformerConfig
from megatron.core.models.gpt import GPTModel

# Assuming hf_config is a Huggingface model configuration object
# and hf_model is a loaded Huggingface model

# 1. Convert Huggingface config to mcore TransformerConfig
# This step requires a custom converter function based on the specific model architecture.
# Example placeholder:
# mcore_config = convert_hf_config_to_mcore_config(hf_config)

# For demonstration, let's assume a basic config
mcore_config = TransformerConfig(
    num_layers=12,
    hidden_size=768,
    num_attention_heads=12,
    # ... other mcore config parameters
)

# 2. Initialize the mcore GPTModel with the converted config
model = GPTModel(config=mcore_config)

# 3. Load Huggingface model weights to the GPTModel
# This step involves mapping weights from the Huggingface model to the mcore model structure.
# This is typically done using a utility function.
# Example placeholder:
# load_hf_weights_to_mcore_model(model, hf_model)

print("Model initialized and weights loading process outlined.")
```

--------------------------------

### Create Evaluation Datasets with Varying Documents

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Generates evaluation datasets with a different number of documents included. Requires the 'convert_to_eval.py' script to be run first.

```bash
python different_docs_eval.py.py
```

--------------------------------

### Download HotpotQA Dataset

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Downloads the HotpotQA dataset using the 'hfd.sh' script and sets the DATAROOT environment variable. The '-x 10' flag specifies 10 parallel downloads.

```bash
cd ../..
bash hfd.sh BytedTsinghua-SIA/hotpotqa --dataset --tool aria2c -x 10
export DATAROOT=$(pwd)/hotpotqa
```

--------------------------------

### Inference with Trained Checkpoint (verl framework)

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Runs inference using a model checkpoint trained with the verl framework. This script is specific to the Qwen3 4B model.

```bash
bash verl_custom/scripts/run_qwen3_4b_inference.sh
```

--------------------------------

### Inference with Trained Checkpoint (Agentic Memory)

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Runs inference using a model checkpoint trained with the Agentic Memory framework. This script is specific to the Qwen3 4B model.

```bash
bash MemAgent/run_qwen3_4b_inference.sh
```

--------------------------------

### Train Model with GRPO (verl framework)

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Initiates training for a model using GRPO (Proximal Policy Optimization) over a long context with the verl framework. This script is for the Qwen3 4B model.

```bash
bash verl_custom/scripts/run_qwen3_4b_grpo.sh
```

--------------------------------

### Generate PersonaMem Benchmark Data

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Execute the bash script to generate the complete PersonaMem-v2 benchmark from scratch. Individual steps can also be run separately.

```bash
# Run the complete data generation pipeline
bash scripts/data_gen_scripts/run_generate_all.sh

# Or run individual steps:
# Step 1: Prepare image embeddings
PYTHONPATH=. python data_generation/image_matcher.py \
    --model gpt-5-chat \
    --recreate \
    --parallel \
    --rate_limit_per_min 5

# Step 2: Generate conversations
PYTHONPATH=. python data_generation/main.py \
    --model gpt-5-chat \
    --step generate_conv \
    --conv_output_dir data/raw_data/ \
    --num_persona 1000 \
    --data_types personal_email professional_email creative_writing chat_message \
    --rate_limit_per_min 5 \
    --parallel

# Step 3: Generate Q&A pairs
PYTHONPATH=. python data_generation/main.py \
    --model gpt-5-chat \
    --step generate_qa \
    --conv_output_dir data/raw_data/ \
    --validate_qa \
    --parallel

# Step 4: Build chat histories (32k and 128k versions)
PYTHONPATH=. python data_generation/main.py \
    --step build_chat_history \
    --conv_output_dir data/raw_data/ \
    --version 32k

# Step 5: Prepare benchmark CSV
PYTHONPATH=. python data_generation/prepare_benchmark.py \
    --split \
    --benchmark-size 5000 \
    --train-val-split 0.9
```

--------------------------------

### Prepare OOD Task Data

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Prepares data for Out-of-Distribution (OOD) tasks, including downloading essays, QA datasets, and preparing ruler data. Ensure DATAROOT is set.

```bash
export DATAROOT="your_dir_to_hotpotqa_dev.parquet"
python download_paulgraham_essay.py
bash download_qa_dataset.sh
bash ruler_data_prepare.sh
```

--------------------------------

### MemAgent Evaluation Configuration

Source: https://context7.com/bowen-upenn/personamem-v2/llms.txt

Configure and run comprehensive evaluations for MemAgent models on long-context QA tasks. Sets up environment variables and model configurations for evaluation.

```python
from MemAgent.taskutils.memory_eval.run import Config, ENV, RULER_HQA_TESTS

# Configure evaluation
env = ENV(
    MAX_INPUT_LEN=120000,
    MAX_OUTPUT_LEN=10000,
    RECURRENT_MAX_CONTEXT_LEN=100000000000,
    RECURRENT_CHUNK_SIZE=5000,
    RECURRENT_MAX_NEW=1024
)

# Create model config
model_config = Config(
    name="MemoryAgent-14B",
    ckpt="BytedTsinghua-SIA/RL-MemoryAgent-14B",
    tp=2,  # Tensor parallelism
    method="recurrent",  # Use memory agent method
    env=env,
    concur=256  # Concurrent requests
)

# Run HotpotQA evaluation
model_config.run(
    tests=RULER_HQA_TESTS,  # [50, 100, 200, 400, 800, 1600, 3200, 6400]
    serve=True,
    force=False
)
```

--------------------------------

### Import VeRL Ray Components

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Imports core components from the VeRL Ray library for building distributed workers and managing resource pools.

```python
from verl.single_controller.base import Worker
from verl.single_controller.ray.base import RayClassWithInitArgs, RayResourcePool, RayWorkerGroup, merge_resource_pool
```

--------------------------------

### Convert to Evaluation Format

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/README.md

Converts the processed 'hotpotqa_dev' dataset into 'eval_200.json' format. Ensure DATAROOT is set correctly.

```bash
export DATAROOT="your_dir_to_hotpotqa_dev.parquet"
python convert_to_eval.py
```

--------------------------------

### Implement Policy Gradient Loss Aggregation Logic

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recipe/dapo/README.md

Python code demonstrating different policy gradient loss aggregation modes: 'token-mean', 'seq-mean-token-sum', and 'seq-mean-token-mean'. Handles invalid modes with a ValueError.

```python
if loss_agg_mode == "token-mean":
    pg_loss = verl_F.masked_mean(pg_losses, eos_mask)
elif loss_agg_mode == "seq-mean-token-sum":
    pg_loss = torch.sum(pg_losses * eos_mask, dim=-1) / torch.sum(eos_mask, dim=-1)
    pg_loss = torch.mean(pg_loss)
elif loss_agg_mode == "seq-mean-token-mean":
    pg_loss = torch.sum(pg_losses * eos_mask, dim=-1) / torch.sum(eos_mask, dim=-1)
    pg_loss = torch.mean(pg_loss)
else:
    raise ValueError(f"Invalid loss_agg_mode: {loss_agg_mode}")
```

--------------------------------

### Load Tokenizer

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_async_generation_output.ipynb

Imports the AutoTokenizer from the transformers library. This is typically the first step before tokenizing text for NLP tasks.

```python
from transformers import AutoTokenizer
```

--------------------------------

### Alternative Model Loading with Transformers

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ppo_trainer/verl_getting_started.ipynb

This Python code provides an alternative method to load a Hugging Face model using the `transformers` library's `pipeline` function. It's useful if `huggingface-cli` is unstable.

```python
import transformers
transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Define Tool and Apply Chat Template

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/recurrent/test/test_tool_tokenize.ipynb

Defines an asynchronous tool function that returns the sum and difference of two integers. It then applies a chat template to format a user message and the tool definition, preparing it for the model. The `tokenize=False` argument ensures the output is a string.

```python
async def tool(a: int, b: int) -> tuple[int, int]:
    """
    Returns the sum of two numbers.

    Args:
        a: aaa
        b: bbb
        

    Returns:
        c: ccc
        d: ddd
    """
    return a + b, a-b
a = tokenizer.apply_chat_template(
    [
        {"role": "user", "content": "hello"},
    ],
    tools = [tool],
    tokenize=False
)
print(a)
```

--------------------------------

### Initialize RayWorkerGroup with GPUAccumulatorDecorator

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/examples/ray/tutorial.ipynb

Initializes a `RayWorkerGroup` named `gpu_accumulator_decorator` using the `GPUAccumulatorDecorator` class and a specified resource pool.

```python
class_with_args = RayClassWithInitArgs(cls=GPUAccumulatorDecorator)
gpu_accumulator_decorator = RayWorkerGroup(resource_pool_merge, class_with_args)
```

--------------------------------

### Data Preprocessing for Training

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/README.md

Preprocesses data for training models using the verl framework. Two scripts are provided for different data formats (RFT and SFT).

```python
python verl_custom/data_preprocess_rft.py
```

```python
python verl_custom/data_preprocess_sft.py
```

--------------------------------

### Offline Weight Conversion Script

Source: https://github.com/bowen-upenn/personamem-v2/blob/main/MemAgent/verl/models/mcore/readme.md

Utilize this script to convert Huggingface model weights to mcore weights and save them in the mcore dist_checkpointing format. This method offers faster loading and lower memory consumption compared to runtime loading.

```python
# Example usage of the offline conversion script
# python verl/scripts/converter_hf_to_mcore.py \
#   --hf_model_path /path/to/huggingface/model \
#   --mcore_output_path /path/to/mcore/checkpoint \
#   --dtype bfloat16

print("Offline conversion script command example.")
```