### Environment Setup Script

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/multinode.md

Commands to set up the environment on each node, including cloning the Verl repository and installing dependencies.

```bash
setup: |
  rm -rf verl
  git clone https://github.com/volcengine/verl.git
  cd verl
  pip3 install -v -e .[vllm]
```

--------------------------------

### Install and Initialize Pre-commit

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/index.md

Install pre-commit and set it up to run automatically before each commit.

```bash
pip install pre-commit
pre-commit install
```

--------------------------------

### Install Apex

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/amd_tutorial/amd_build_dockerfile_page.md

Clones and installs the Apex library from a GitHub repository.

```bash
RUN pip uninstall -y apex && \
    git clone git@github.com:ROCm/apex.git && \
    cd apex && \
    python setup.py install && \
    cd /workspace/
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/sppo/README.md

Clone the verl repository, install it with sglang support, and set up environment variables for Weights & Biases. This is the initial setup for reproducing experiments.

```bash
git clone git@github.com:volcengine/verl.git
cd verl
python3 -m uv pip install -e "[sglang]"

export WANDB_API_KEY=<YOUR_WANDB_API_KEY>
```

--------------------------------

### Install FaithLens Package (Method 1)

Source: https://github.com/s1s-z/faithlens/blob/master/README.md

Clone the repository and install the FaithLens package locally using pip.

```bash
git clone https://github.com/S1s-Z/FaithLens.git
cd Faithlens
pip install .
```

--------------------------------

### Setup Environment with Docker

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/spin.md

This snippet shows how to set up a Docker container for the SPIN experiment, including GPU access, shared memory, and environment variables. It also covers installing Python 3.10, venv, and activating a virtual environment within the container.

```bash
# Start a container with GPU access and shared memory
docker run -it --name spin_test --gpus all \
    --shm-size=32g \
    --ipc=host \
    -v /path/to/host/.cache:/root/.cache \
    -e HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN> \
    lmsysorg/sglang:latest \
    /bin/bash

# Inside the container or on your host machine:
# Ensure /tmp is writable
mkdir -p /tmp
chmod 1777 /tmp

# Install Python 3.10 (if not present) and venv
sudo apt update
sudo apt install -y python3.10 python3.10-venv tmux
python3 -m ensurepip --upgrade

# Create and activate a virtual environment
python3 -m venv ~/.python/spin_env
source ~/.python/spin_env/bin/activate

# Install uv (fast package installer)
python3 -m pip install uv
```

--------------------------------

### Docker Setup for SPIN Experiment

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/spin/README.md

This command starts a Docker container with GPU access and shared memory, essential for running the SPIN experiment. It mounts a local cache directory and sets the Hugging Face token.

```bash
# Start a container with GPU access and shared memory
docker run -it --name spin_test --gpus all \
    --shm-size=32g \
    --ipc=host \
    -v /path/to/host/.cache:/root/.cache \
    -e HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN> \
    lmsysorg/sglang:latest \
    /bin/bash
```

--------------------------------

### Setup SkyPilot for GCP

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/multinode.md

Install and configure SkyPilot for use with Google Cloud Platform, including environment setup and authentication.

```bash
conda create -y -n sky python=3.10
conda activate sky
pip install "skypilot[gcp]"

conda install -c conda-forge google-cloud-sdk
gcloud init

# Run this if you don't have a credential file.
# This will generate ~/.config/gcloud/application_default_credentials.json.
gcloud auth application-default login

# Check if the GCP credential is correctly setup.
sky check gcp
```

--------------------------------

### Example Script for Running with Rollout IS

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/rollout_is_migration.md

This shell script provides a basic example of how to run with Rollout Importance Sampling, specifically using the token-level truncate mode. It serves as a starting point for users to implement and test the new IS functionality.

```bash
# examples/rollout_importance_sampling/run_with_rollout_is.sh
# Basic example with token-level truncate
```

--------------------------------

### Install Fast Package Installer (uv)

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/spin/README.md

Installs 'uv', a fast Python package installer, within the activated virtual environment.

```bash
# Install uv (fast package installer)
python3 -m pip install uv
```

--------------------------------

### FSDP2 Configuration Example

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/one_step_off.md

Example command to launch the PPO trainer with FSDP2 strategy. This configuration places actor and rollout separately and specifies resource allocation per node.

```shell
python3 -m recipe.one_step_off_policy.async_main_ppo \
    --config-path=config \
    --config-name='one_step_off_ppo_trainer.yaml' \
    actor_rollout_ref.actor.strategy=fsdp2 \
    # actor and rollout are placed separately
    actor_rollout_ref.hybrid_engine=False \
    # actor and rollout resource
    trainer.nnodes=1 \
    trainer.n_gpus_per_node=6 \
    rollout.nnodes=1 \
    rollout.n_gpus_per_node=2
```

--------------------------------

### Install Ray

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/multinode.md

Install the Ray library locally before submitting jobs to a Ray cluster.

```shell
pip install ray
```

--------------------------------

### Install MLflow for Tracing

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/agentic_rl.md

Install the MLflow library to enable viewing tool call and LLM traces during training.

```bash
pip install mlflow
```

--------------------------------

### Install FaithLens Package (Method 2)

Source: https://github.com/s1s-z/faithlens/blob/master/README.md

Install the FaithLens package directly from the GitHub repository using pip, ensuring all dependencies are included.

```bash
pip install "faithlens @ git+https://github.com/S1s-Z/FaithLens.git@master"
```

--------------------------------

### Manual Installation of Flash Attention

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/sppo.md

Provides commands to manually install the flash-attn library if the automatic installation fails.

```bash
python3 -m uv pip install wheel
python3 -m uv pip install packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps
```

--------------------------------

### Install verl from Source

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/install.md

Clone the repository, navigate to the directory, and install the package in editable mode. This method allows for code customization.

```bash
git clone https://github.com/volcengine/verl.git
cd verl
pip install --no-deps -e .
```

--------------------------------

### Megatron Configuration Example

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/one_step_off.md

Example command to launch the PPO trainer with Megatron strategy. This configuration is similar to FSDP2 but uses Megatron for model parallelism.

```shell
python3 -m recipe.one_step_off_policy.async_main_ppo \
    --config-path=config \
    --config-name='one_step_off_ppo_megatron_trainer.yaml' \
    actor_rollout_ref.actor.strategy=megatron \
    # actor and rollout are placed separately
    actor_rollout_ref.hybrid_engine=False \
    # actor and rollout resource
    trainer.nnodes=1 \
    trainer.n_gpus_per_node=6 \
    rollout.nnodes=1 \
    rollout.n_gpus_per_node=2
```

--------------------------------

### Install SkyPilot for Azure

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/examples/skypilot_examples.md

Installs SkyPilot with Azure support. Use this command if your target platform is Azure.

```bash
pip install "skypilot[azure]"
```

--------------------------------

### Basic Rollout IS Script Example

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/rollout_is_migration.md

This shell script demonstrates a basic example of running with token-level truncate mode for Rollout Importance Sampling. It shows how to update parameters from the old `tis_imp_ratio_cap` to the new Rollout IS parameters.

```bash
# recipe/dapo/run_dapo_qwen2.5_32b_tis.sh
# Updated from tis_imp_ratio_cap to rollout IS parameters
# Added comprehensive comments
```

--------------------------------

### Install SkyPilot for Google Cloud Platform

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/examples/skypilot_examples.md

Installs SkyPilot with GCP support. Use this command if your target platform is Google Cloud Platform.

```bash
pip install "skypilot[gcp]"
```

--------------------------------

### Install and Run Pre-commit Hooks

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/CONTRIBUTING.md

Install pre-commit and run its hooks to ensure code consistency. This includes running hooks on staged changes or all files in the repository.

```bash
pip install pre-commit
pre-commit install
# for staged changes
pre-commit run
# for all files in the repo
pre-commit run --all-files
# run a specific hook with pre-commit
# pre-commit run --all-files --show-diff-on-failure --color=always <hood-id>
pre-commit run --all-files --show-diff-on-failure --color=always ruff
pre-commit run --all-files --show-diff-on-failure --color=always autogen-trainer-cfg
```

--------------------------------

### Install TensorDict from Source

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/faq/faq.md

These commands provide a workaround for the 'NotImplementedError' by uninstalling the existing TensorDict, cloning the repository, checking out a specific version, and installing it in develop mode.

```bash
pip uninstall tensordict
git clone https://github.com/pytorch/tensordict.git
cd tensordict/
git checkout v0.6.2
python setup.py develop
pip install -v -e .
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/sppo.md

Clone the verl repository, install necessary dependencies including sglang, and set up environment variables for Weights & Biases and CUDA.

```bash
git clone git@github.com:volcengine/verl.git
cd verl
python3 -m uv pip install -e "".[sglang]"

export WANDB_API_KEY=<YOUR_WANDB_API_KEY>

python3 examples/data_preprocess/math_dataset.py --local_dir ~/data/math
huggingface-cli download Qwen/Qwen2.5-7B-Instruct --local-dir $HOME/models/Qwen2.5-7B-Instruct

export CUDA_VISIBLE_DEVICES=0,1,2,3
bash recipe/sppo/run_qwen2.5-7b_rm.sh
```

--------------------------------

### Set Up Local Retrieval Engine with Conda

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/sglang_multiturn/search_tool_example.md

Installs Miniconda, creates and activates a Conda environment named 'retriever', installs PyTorch with GPU support, and necessary Python packages including faiss-gpu and FastAPI.

```bash
# Download the Miniconda installer script
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh

# Install to $HOME/miniconda3 in batch mode
bash ~/miniconda.sh -b -p $HOME/miniconda3

# Activate conda (only in the current shell)
eval "$($HOME/miniconda3/bin/conda shell.bash hook)"

# (Optional) Add conda to your default shell startup
conda init

# Reload shell config
source ~/.bashrc

# Create and activate the retriever environment with Python 3.10
conda create -n retriever python=3.10 -y
conda activate retriever

# Install PyTorch (with GPU support) and related libraries
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.1 -c pytorch -c nvidia -y

# Install other Python packages
pip install transformers datasets pyserini huggingface_hub

# Install the GPU version of faiss
conda install faiss-gpu=1.8.0 -c pytorch -c nvidia -y

# Install the API service framework
pip install uvicorn fastapi
```

--------------------------------

### Build and Preview Documentation Locally

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/CONTRIBUTING.md

Steps to build the project's HTML documentation and preview it locally. Ensure the project is installed and documentation dependencies are met.

```bash
# Ensure verl is on your PYTHONPATH, e.g.:
pip install -e .[test]

# Install documentation dependencies
pip install -r requirements-docs.txt

# Generate HTML docs
make clean
make html

# Preview locally
python -m http.server -d _build/html/
```

--------------------------------

### Install Verl Dependencies with install.sh

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/install.md

Executes the provided install.sh script to install Verl and its dependencies within an activated conda environment. Use USE_MEGATRON=0 for FSDP-only setups.

```bash
# Make sure you have activated verl conda env
# If you need to run with megatron
bash scripts/install_vllm_sglang_mcore.sh
# Or if you simply need to run with FSDP
USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh
```

--------------------------------

### Serve Verl Documentation Locally

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/README.md

Starts a local HTTP server to view the generated HTML documentation in a browser.

```bash
python -m http.server -d _build/html/
```

--------------------------------

### Testing Interaction Workflow

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/sglang_multiturn/interaction_system.md

Provides a pytest example for testing the complete interaction workflow, including starting an interaction, generating a response, and finalizing it. Asserts the expected types and values for the response.

```python
import pytest
from unittest.mock import patch

@pytest.mark.asyncio
async def test_interaction_workflow():
    interaction = YourInteraction({})

    # Test complete workflow
    instance_id = await interaction.start_interaction(ground_truth="expected_answer")


    messages = [{"role": "user", "content": "user_content"}, {"role": "assistant", "content": "assistant_content"}]
    should_terminate, response, reward, metadata = await interaction.generate_response(instance_id, messages)

    assert should_terminate in [True, False]
    assert isinstance(reward, float)

    await interaction.finalize_interaction(instance_id)
```

--------------------------------

### Qwen2.5 Training Command with PPO

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/ppo.md

Example command for training a Qwen2.5 model using PPO. This command specifies GPU usage, model paths, batch sizes, and PPO-specific parameters.

```bash
bash run_gemma.sh \
  trainer.n_gpus_per_node=1 \
  actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
  trainer.logger=console \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  data.train_batch_size=256 \
  actor_rollout_ref.actor.ppo_mini_batch_size=64 \
  actor_rollout_ref.actor.ppo_micro_batch_size=2 \
  critic.ppo_micro_batch_size=2
```

--------------------------------

### Start RL Training

Source: https://github.com/s1s-z/faithlens/blob/master/README.md

Run this command to initiate the Rule-Based Reinforcement Learning stage for further optimization of the FaithLens model.

```bash
bash training/verl/rl_training.sh
```

--------------------------------

### Install verl and Dependencies

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/spin.md

Installs the 'verl' library and its dependencies, including flash-attn for performance. It clones the repository, installs flash-attn with specific flags, and then installs 'verl' with 'sglang' extras.

```bash
# Clone the verl repository and checkout the spin branch
cd ~
git clone git@github.com:volcengine/verl.git && cd verl

# Install flash-attn (handle potential build issues)
python3 -m uv pip install wheel packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps

# Install verl with sglang extras
python3 -m uv pip install -e ".[sglang]"
```

--------------------------------

### Start Training with Tool Calls and MLflow Tracing

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/agentic_rl.md

Begin training with tool calls enabled and MLflow tracing activated for debugging rollout details.

```bash
bash examples/sglang_multiturn/run_qwen2.5-3b_gsm8k_tool_agent_mlflow.sh
```

--------------------------------

### Start SFT Training

Source: https://github.com/s1s-z/faithlens/blob/master/README.md

Execute this script to begin the Cold-Start Supervised Fine-Tuning stage for the FaithLens model.

```bash
bash training/sft/train_llama8b_instruct.sh
```

--------------------------------

### Install LiveCodeBench Dependencies

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/verl/utils/reward_score/prime_code/README.md

Installs the necessary Python packages for LiveCodeBench. Ensure CUDA version is greater than 12.0 before installation.

```bash
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```

--------------------------------

### Build Verl Documentation

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/README.md

Installs dependencies, cleans previous builds, and generates HTML documentation using Make.

```bash
# If you want to view auto-generated API docstring, please make sure verl is available in python path. For instance, install verl via:
# pip install .. -e[test]

# Install dependencies needed for building docs.
pip install -r requirements-docs.txt

# Build the docs.
make clean
make html
```

--------------------------------

### Install verl Upstream and Dependencies

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/sglang_multiturn/search_tool_example.md

Clones the verl repository, installs verl and its SGLang-specific requirements using uv, and manually installs flash-attn.

```bash
cd ~
git clone https://github.com/volcengine/verl.git
cd verl

# Install verl
python3 -m uv pip install .
python3 -m uv pip install -r ./requirements_sglang.txt

# Manually install flash-attn
python3 -m uv pip install wheel
python3 -m uv pip install packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps
```

--------------------------------

### Create Dataset with Custom Parameters

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/langgraph_agent/example/README.md

Example command to run the dataset creation script with specific training and testing sizes, and a custom output directory.

```bash
python recipe/langgraph_agent/example/create_dataset.py \
  --train_size 10000 \
  --test_size 1000 \
  --output_dir data/math_expression_tool
```

--------------------------------

### Initialize and Run PPO Trainer

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/examples/ppo_code_architecture.md

Initializes the RayPPOTrainer with configuration, tokenizer, and worker mappings, then fits the trainer to start the PPO training process.

```python
trainer = RayPPOTrainer(config=config,
                        tokenizer=tokenizer,
                        role_worker_mapping=role_worker_mapping,
                        resource_pool_manager=resource_pool_manager,
                        ray_worker_group_cls=ray_worker_group_cls,
                        reward_fn=reward_fn,
                        val_reward_fn=val_reward_fn)
trainer.init_workers()
trainer.fit()
```

--------------------------------

### Install verl and vLLM 0.7.3

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/README_vllm0.7.md

Steps to create a conda environment, clone the verl repository, install verl, and install vLLM version 0.7.3 along with flash-attn.

```bash
conda create -n verl python==3.10
conda activate verl

git clone https://github.com/volcengine/verl.git
cd verl
pip3 install -e .

pip3 install vllm==0.7.3 

pip3 install flash-attn --no-build-isolation

```

--------------------------------

### Start MLflow UI for Trace Viewing

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/agentic_rl.md

Launch an MLflow server to visualize traces after training is complete. The backend store is set to a local SQLite database.

```bash
mlflow ui -h 0.0.0.0 -p 5000 --backend-store-uri sqlite:////tmp/mlruns.db
```

--------------------------------

### PPO Training Configuration

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/amd_tutorial/amd_build_dockerfile_page.md

Command to launch PPO training with extensive configuration parameters for model path, data, actor, and rollout settings.

```bash
python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=data/gsm8k/train.parquet \
    data.val_files=data/gsm8k/test.parquet \
    data.train_batch_size=1024 \
    data.val_batch_size=1312 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    actor_rollout_ref.model.path=$MODEL_PATH \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=256 \
    actor_rollout_ref.actor.use_dynamic_bsz=True \
    actor_rollout_ref.actor.ppo_max_token_len_per_gpu=24000 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.model.enable_gradient_checkpointing=Flase \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=$ENGINE \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
    actor_rollout_ref.rollout.n=5 \
    actor_rollout_ref.ref.fsdp_config.param_offload=False \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.critic_warmup=0 \
    trainer.logger=console \
    trainer.project_name=$YOUR_PROJECT_NAME \
    trainer.experiment_name=$YOUR_RUN_NAME \
    trainer.n_gpus_per_node=$GPUS_PER_NODE \
    trainer.val_before_train=False \
    trainer.nnodes=1 \
    trainer.save_freq=-1 \
    trainer.test_freq=10 \
    trainer.total_epochs=15
```

--------------------------------

### Start Ray Worker Nodes

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/multinode.md

Starts Ray worker nodes in a loop, connecting them to the specified head node IP and port. Includes a small delay between starting each worker.

```bash
# number of nodes other than the head node
worker_num=$((SLURM_JOB_NUM_NODES - 1))

for ((i = 1; i <= worker_num; i++)); do
    node_i=${nodes_array[$i]}
    echo "Debug: Starting worker on node_i = ${node_i}"
    if [ -z "$node_i" ]; then
        echo "Error: Empty node name for worker $i"
        continue
    fi
    echo "Starting WORKER $i at $node_i"
    srun --nodes=1 --ntasks=1 -w "$node_i" \
        docker exec "${CONTAINER_NAME}" \
            ray start --address "$ip_head" --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
```

--------------------------------

### Run PPO Experiment with SGLang Backend

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/workers/sglang_worker.md

Execute this command to start a PPO experiment on a single machine using SGLang as the inference backend. This command configures various training parameters, including dataset paths, batch sizes, model paths, learning rates, and distributed training settings. The output is logged to 'verl_demo.log'.

```bash
export SGL_DISABLE_TP_MEMORY_INBALANCE_CHECK=True
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=4096 \
    data.max_prompt_length=4096 \
    data.max_response_length=4096 \
    actor_rollout_ref.rollout.name=sglang \
    actor_rollout_ref.model.path=Qwen/Qwen2-7B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
    critic.optim.lr=1e-5 \
    critic.model.path=Qwen/Qwen2-7B-Instruct \
    critic.ppo_micro_batch_size_per_gpu=4 \
    critic.model.fsdp_config.param_offload=True \
    critic.model.fsdp_config.optimizer_offload=True \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.logger=console \
    trainer.val_before_train=False \
    trainer.n_gpus_per_node=4 \
    trainer.nnodes=1 \
    trainer.save_freq=-1 \
    trainer.test_freq=10 \
    trainer.total_epochs=15 2>&1 | tee verl_demo.log
```

--------------------------------

### Install Megatron-LM AMD Version

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/amd_tutorial/amd_build_dockerfile_page.md

Installs the AMD version of Megatron-LM from a specific GitHub repository.

```bash
RUN pip uninstall -y megatron-core && \
    git clone https://github.com/yushengsu-thu/Megatron-LM-amd_version.git && \
    cd Megatron-LM-amd_version && \
    pip install -vvv -e . && \
    cd /workspace/
```

--------------------------------

### Download Base Model

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/spin/README.md

Downloads the Qwen2.5-3B-Instruct base model from Hugging Face to a local directory.

```bash
# Download the base model (Example: Qwen2.5-3B-Instruct)
huggingface-cli download Qwen/Qwen2.5-3B-Instruct --local-dir $HOME/models/Qwen2.5-3B-Instruct
```

--------------------------------

### DAPO Configuration Example

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/algo/dapo.md

An example configuration snippet for DAPO, specifying clip ratios for the actor.

```yaml
actor_rollout_ref:
  actor:
    clip_ratio_low: 0.2
    clip_ratio_high: 0.28
```

--------------------------------

### Install SkyPilot for AWS

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/examples/skypilot_examples.md

Installs SkyPilot with AWS support. Use this command if your target platform is AWS.

```bash
pip install "skypilot[aws]"
```

--------------------------------

### Initializing a dstack Project

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/multinode.md

Initialize a new project directory and set it up as a dstack repository. This is a prerequisite for using dstack for distributed training.

```bash
mkdir myproject && cd myproject
dstack init
```

--------------------------------

### Install vLLM V1 Engine

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/README_vllm0.7.md

Steps to install the vLLM V1 engine, which offers improved stability and performance over the V0 engine. This involves cloning the vLLM repository, checking out a specific commit, applying a patch, and performing an editable installation.

```bash
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout 2275784
sed -i "903a\    data_parallel_size = world_size // pipeline_model_parallel_size // tensor_model_parallel_size" ./vllm/distributed/parallel_state.py
VLLM_USE_PRECOMPILED=1 pip install --editable .

```

--------------------------------

### New Rollout IS Configuration

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/rollout_is_migration.md

Example of the updated configuration for the new rollout IS system, including main control and application settings.

```yaml
algorithm:
  rollout_is_threshold: 2.0  # Main control
  rollout_is: true           # Apply to loss (default: false)
  rollout_is_level: token
  rollout_is_mode: truncate

actor_rollout_ref:
  rollout:
    calculate_log_probs: true  # Still required!
```

--------------------------------

### Install SkyPilot for Kubernetes

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/examples/skypilot_examples.md

Installs SkyPilot with Kubernetes support. Use this command if your target platform is exclusively Kubernetes.

```bash
pip install "skypilot[kubernetes]"
```

--------------------------------

### Start Local Ray Cluster

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/tests/single_controller/detached_worker/README.md

Starts a local Ray cluster with the head node and specifies the port for communication.

```bash
ray start --head --port=6379
```

--------------------------------

### Download Qwen2.5-0.5B-Instruct Model

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/quickstart.md

Downloads the Qwen2.5-0.5B-Instruct model for post-training. This step ensures the base model is accessible for the training process. For Supervised Fine-Tuning (SFT) before RL, refer to the complete GSM8K example and SFT Trainer documentation.

```python
import transformers
transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Example RoPE Scaling Configuration

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/advance/rope.md

This JSON structure shows an example of RoPE Scaling configuration that can be applied to a model.

```json
{
    ...,
    "rope_scaling": {
        "factor": 4.0,
        "original_max_position_embeddings": 32768,
        "type": "yarn"
    }
}
```

--------------------------------

### Install Common Python Packages

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/amd_tutorial/amd_build_dockerfile_page.md

Installs essential Python packages including IPython, orjson, python-multipart, torchao, and pybind11.

```bash
RUN pip install IPython orjson python-multipart torchao pybind11
```

--------------------------------

### DAPO Early Training Script Example

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/recipe/dapo/README.md

This script is for running DAPO reproduction experiments with early stopping for token-level loss and dynamic sampling, using the Qwen2.5 32B model. It specifies the image, commit, and environment variables for the run.

```bash
#!/bin/bash

# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script is for running DAPO reproduction experiments with early stopping for token-level loss and dynamic sampling.
# It assumes the environment is set up according to runtime_env.yaml.

# Example usage:
# bash recipe/dapo/run_dapo_early_qwen2.5_32b.sh

# Set the base directory for the script
BASE_DIR=$(dirname "$0")

# Source the common functions script
source "$BASE_DIR/../../common/scripts/common_functions.sh"

# Define the experiment name
EXPERIMENT_NAME="dapo_early_qwen2.5_32b"

# Define the model name
MODEL_NAME="Qwen2.5-32B"

# Define the configuration file
CONFIG_FILE="$BASE_DIR/dapo_early_config.yaml"

# Define the output directory
OUTPUT_DIR="$BASE_DIR/../../output/${EXPERIMENT_NAME}"

# Create the output directory if it doesn't exist
create_dir "$OUTPUT_DIR"

# Run the training script
run_train "$EXPERIMENT_NAME" "$MODEL_NAME" "$CONFIG_FILE" "$OUTPUT_DIR"

```

--------------------------------

### Install Verl with Framework Support

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/start/install.md

Installs the nightly version of Verl with support for specific frameworks like vLLM or SGLang.

```bash
# install the nightly version (recommended)
git clone https://github.com/volcengine/verl && cd verl
pip3 install -e .[vllm]
pip3 install -e .[sglang]
```

--------------------------------

### PPO Training with SGLang Backend

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/workers/sglang_worker.md

Full command to initiate PPO training using SGLang as the inference backend across multiple machines with TP=16. This script configures various training parameters, including data paths, model settings, and distributed training configurations.

```bash
DATA_DIR=$HOME/data/gsm8k

python3 -m verl.trainer.main_ppo \
    actor_rollout_ref.rollout.name=sglang \
    data.train_files=$DATA_DIR/train.parquet \
    data.val_files=$DATA_DIR/test.parquet \
    data.train_batch_size=4096 \
    data.max_prompt_length=4096 \
    data.max_response_length=4096 \
    actor_rollout_ref.model.path=meta-llama/Llama-3.1-8B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=16 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=True \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=16 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=16 \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
    actor_rollout_ref.rollout.free_cache_engine=True \
    actor_rollout_ref.ref.log_prob_micro_batch_size=16 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    critic.optim.lr=1e-5 \
    critic.model.use_remove_padding=True \
    critic.model.path=meta-llama/Llama-3.1-8B-Instruct \
    critic.model.enable_gradient_checkpointing=True \
    critic.ppo_micro_batch_size=16 \
    critic.model.fsdp_config.param_offload=True \
    critic.model.fsdp_config.optimizer_offload=True \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.critic_warmup=0 \
    trainer.logger=console \
    trainer.val_before_train=True \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=2 \
    trainer.save_freq=-1 \
    trainer.test_freq=10 \
    trainer.total_epochs=15 2>&1 | tee verl_demo.log
```

--------------------------------

### Install Verl Nightly Version

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docker/README.md

Installs the nightly version of Verl without its dependencies. This is recommended for the latest features and bug fixes.

```sh
git clone https://github.com/volcengine/verl && cd verl
pip3 install --no-deps -e .
```

--------------------------------

### Install verl and vLLM 0.8.3

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/README_vllm0.8.md

Installs verl and the specified version of vLLM. Ensure you have a Python 3.10 conda environment activated.

```bash
conda create -n verl python==3.10
conda activate verl

git clone https://github.com/volcengine/verl.git
cd verl
pip3 install -e .

pip3 install vllm==0.8.3

pip3 install flash-attn --no-build-isolation

```

--------------------------------

### Start Model Training with PPO

Source: https://github.com/s1s-z/faithlens/blob/master/training/verl/docs/amd_tutorial/amd_build_dockerfile_page.md

Initiates model training using the PPO algorithm via a SLURM job. This command configures numerous training parameters, including data paths, batch sizes, learning rates, and model configurations.

```bash
PYTHONUNBUFFERED=1 srun --overlap --nodes=${SLURM_NNODES} --ntasks=1 -w "$head_node" \
    docker exec "${CONTAINER_NAME}" \
    python3 -m verl.trainer.main_ppo \
    data.train_files=$train_files \
    data.val_files=$val_files \
    data.train_batch_size=1024 \
    data.max_prompt_length=1024 \
    data.max_response_length=1024 \
    actor_rollout_ref.model.path=$MODEL_PATH \
    actor_rollout_ref.model.enable_gradient_checkpointing=False \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=256 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=8 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=16 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=16 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    critic.optim.lr=1e-5 \
    critic.model.use_remove_padding=True \
    critic.model.path=$MODEL_PATH \
    critic.model.enable_gradient_checkpointing=False \
    critic.ppo_micro_batch_size_per_gpu=8 \
    critic.model.fsdp_config.param_offload=False \
    critic.model.fsdp_config.optimizer_offload=False \
    algorithm.kl_ctrl.kl_coef=0.0001 \
    trainer.critic_warmup=0 \
    trainer.logger='["console","wandb"]' \
    trainer.project_name='verl_example' \
    trainer.experiment_name='Qwen2.5-32B-Instruct_function_rm' \
    trainer.n_gpus_per_node=${SLURM_GPUS_PER_NODE} \
    trainer.val_before_train=False \
    trainer.nnodes=${SLURM_NNODES} \
    trainer.save_freq=-1 \
    trainer.test_freq=10 \
    trainer.total_epochs=15
```