Dynamic Fine-Tuning (DFT)

https://github.com/yongliang-wu/dft
Admin
Dynamic Fine-Tuning (DFT) is a method to improve Supervised Fine-Tuning (SFT) for LLMs by...
Tokens:149,678
Snippets:536
Trust Score:8.2
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install verl from Source

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Clones the verl repository from GitHub and installs it in editable mode without dependencies. This allows for customization of the code.

```bash
git clone https://github.com/volcengine/verl.git
cd verl
pip install --no-deps -e .
```

--------------------------------

### Docker Image Installation

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Instructions for launching a Docker container and executing commands inside it to install Verl. This includes cloning the repository and installing dependencies for either vLLM or SGLang inference engines.

```bash
docker create --runtime=nvidia --gpus all --net=host --shm-size="10g" --cap-add=SYS_ADMIN -v .:/workspace/verl --name verl <image:tag>
docker start verl
docker exec -it verl bash
```

```bash
# install the nightly version (recommended)
git clone https://github.com/volcengine/verl && cd verl
# pick your choice of inference engine: vllm or sglang
# pip3 install -e .[vllm]
# pip3 install -e .[sglang]
# or install from pypi instead of git via:
# pip3 install verl[vllm]
# pip3 install verl[sglang]
```

--------------------------------

### PPO Training Example for AMD GPUs

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

Example command to run PPO training using the built ROCm Docker image. It includes setting environment variables for Ray, specifying data files, batch sizes, model paths, learning rates, and engine configurations.

```bash
YOUR_PROJECT_NAME=r1-verl-ppo-upstream
YOUR_RUN_NAME=r1-training_ppo-upstream 
# export HYDRA_FULL_ERROR=1

# [ray] < 2.45.0
#export RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES=1

# [ray] >= 2.45.0
export RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES=1 # Patch with https://github.com/ray-project/ray/pull/52794

GPUS_PER_NODE=8
MODEL_PATH=Qwen/Qwen2.5-0.5B-Instruct
python3 examples/data_preprocess/gsm8k.py --local_dir data/gsm8k
python3 -c "import transformers; transformers.pipeline('text-generation', model='$MODEL_PATH')"
ENGINE=vllm #sglang

PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
 data.train_files=data/gsm8k/train.parquet \
 data.val_files=data/gsm8k/test.parquet \
 data.train_batch_size=256 \
 data.val_batch_size=1312 \
 data.max_prompt_length=512 \
 data.max_response_length=256 \
 actor_rollout_ref.model.path=$MODEL_PATH \
 actor_rollout_ref.actor.optim.lr=1e-6 \
 actor_rollout_ref.actor.ppo_mini_batch_size=64 \
 actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
 actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
 actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
 actor_rollout_ref.rollout.name=$ENGINE \
 actor_rollout_ref.rollout.gpu_memory_utilization=0.8 \
 actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
 critic.optim.lr=1e-5 \
 critic.model.path=$MODEL_PATH \

```

--------------------------------

### Download Qwen2.5-0.5B-Instruct Model

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/quickstart.rst

This Python command downloads the Qwen2.5-0.5B-Instruct model using the Hugging Face `transformers` library. This model serves as the base for post-training. Ensure `transformers` is installed.

```python
import transformers
transformers.pipeline('text-generation', model='Qwen/Qwen2.5-0.5B-Instruct')
```

--------------------------------

### Training Example with verl and Qwen2.5-0.5B GRPO

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/ascend_tutorial/ascend_quick_start.rst

Executes a training job using the verl library with the PPO algorithm for the Qwen2.5-0.5B model on GRPO. It includes environment variable setup and detailed configuration parameters for the training process.

```bash
set -x

export VLLM_ATTENTION_BACKEND=XFORMERS

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=128 \
    data.max_prompt_length=512 \
    data.max_response_length=128 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
    actor_rollout_ref.actor.optim.lr=5e-7 \
    actor_rollout_ref.model.use_remove_padding=False \
    actor_rollout_ref.actor.entropy_coeff=0.001 \
    actor_rollout_ref.actor.ppo_mini_batch_size=64 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=20 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.rollout.enable_chunked_prefill=False \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=5 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.kl_ctrl.kl_coef=0.001 \
    trainer.critic_warmup=0 \
    trainer.logger=['console'] \
    trainer.project_name='verl_grpo_example_gsm8k' \
    trainer.experiment_name='qwen2_7b_function_rm' \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.save_freq=-1 \
    trainer.test_freq=5 \
    trainer.total_epochs=1 \
    trainer.device=npu $@

```

--------------------------------

### Run Split Placement Example

Source: https://github.com/yongliang-wu/dft/blob/master/verl/examples/split_placement/README.md

Provides the command to execute the split placement example using a bash script.

```bash
bash run_deepseek7b_llm.sh
```

--------------------------------

### Install verl and Dependencies

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/spin.md

This snippet details the installation process for the verl library and its dependencies, including cloning the repository, installing flash-attn with specific flags to handle build issues, and installing verl with sglang extras.

```bash
# Clone the verl repository and checkout the spin branch
cd ~
git clone git@github.com:volcengine/verl.git && cd verl

# Install flash-attn (handle potential build issues)
python3 -m uv pip install wheel packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps

# Install verl with sglang extras
python3 -m uv pip install -e ".[sglang]"
```

--------------------------------

### Setup Environment with Docker

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/spin.md

This snippet demonstrates how to set up a Docker environment for the SPIN recipe, including GPU access, shared memory, volume mounting for cache, and setting Hugging Face token. It also covers installing Python 3.10, venv, and uv within the container.

```bash
# Start a container with GPU access and shared memory
docker run -it --name spin_test --gpus all \
    --shm-size=32g \
    --ipc=host \
    -v /path/to/host/.cache:/root/.cache \
    -e HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN> \
    lmsysorg/sglang:latest \
    /bin/bash

# Inside the container or on your host machine:
# Ensure /tmp is writable
mkdir -p /tmp
chmod 1777 /tmp

# Install Python 3.10 (if not present) and venv
sudo apt update
sudo apt install -y python3.10 python3.10-venv tmux
python3 -m ensurepip --upgrade

# Create and activate a virtual environment
python3 -m venv ~/.python/spin_env
source ~/.python/spin_env/bin/activate

# Install uv (fast package installer)
python3 -m pip install uv
```

--------------------------------

### Preprocess GSM8K Dataset

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/quickstart.rst

This command preprocesses the GSM8K dataset into a parquet format for efficient RL reward computation. It requires the `verl` library and its dependencies to be installed. The `--local_dir` argument specifies the directory to save the processed data.

```bash
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k
```

--------------------------------

### Setup Environment with Docker

Source: https://github.com/yongliang-wu/dft/blob/master/verl/recipe/spin/README.md

This snippet demonstrates how to set up a Docker environment for the SPIN recipe, including GPU access, shared memory, volume mounting for cache, and setting Hugging Face token. It also covers installing Python 3.10, venv, and uv within the container.

```bash
# Start a container with GPU access and shared memory
docker run -it --name spin_test --gpus all \
    --shm-size=32g \
    --ipc=host \
    -v /path/to/host/.cache:/root/.cache \
    -e HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN> \
    lmsysorg/sglang:latest \
    /bin/bash

# Inside the container or on your host machine:
# Ensure /tmp is writable
mkdir -p /tmp
chmod 1777 /tmp

# Install Python 3.10 (if not present) and venv
sudo apt update
sudo apt install -y python3.10 python3.10-venv tmux
python3 -m ensurepip --upgrade

# Create and activate a virtual environment
python3 -m venv ~/.python/spin_env
source ~/.python/spin_env/bin/activate

# Install uv (fast package installer)
python3 -m pip install uv
```

--------------------------------

### Install verl and Dependencies

Source: https://github.com/yongliang-wu/dft/blob/master/verl/recipe/spin/README.md

This snippet details the installation process for the verl library and its dependencies, including cloning the repository, installing flash-attn with specific flags to handle build issues, and installing verl with sglang extras.

```bash
# Clone the verl repository and checkout the spin branch
cd ~
git clone git@github.com:volcengine/verl.git && cd verl

# Install flash-attn (handle potential build issues)
python3 -m uv pip install wheel packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps

# Install verl with sglang extras
python3 -m uv pip install -e ".[sglang]"
```

--------------------------------

### Install verl Upstream

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/sglang_multiturn/search_tool_example.rst

Clones the 'verl' repository, navigates into the directory, and installs the project and its dependencies using 'uv'. It also includes a manual installation step for 'flash-attn' to ensure compatibility.

```bash
cd ~
git clone https://github.com/volcengine/verl.git
cd verl

# Install verl
python3 -m uv pip install .
python3 -m uv pip install -r ./requirements_sglang.txt

# Manually install flash-attn
python3 -m uv pip install wheel
python3 -m uv pip install packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps
```

--------------------------------

### Manual Flash-Attention Installation

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/sppo.md

Instructions for manually installing flash-attn if the automated installation fails. This involves installing wheel, packaging, and then flash-attn without build isolation or dependencies.

```bash
python3 -m uv pip install wheel
python3 -m uv pip install packaging
python3 -m uv pip install flash-attn --no-build-isolation --no-deps
```

--------------------------------

### Example Training Script Reference

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/examples/sandbox_fusion_example.rst

A reference to an example shell script used for training with Sandbox Fusion integration. This script likely contains the full implementation details for setting up and running the training job.

```Shell
examples/ppo_trainer/run_deepseek7b_llm_sandbox_fusion.sh
```

--------------------------------

### Start Legacy Ray Debugger

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/multinode.rst

This snippet demonstrates how to start a Ray cluster with the legacy debugger enabled. It includes commands for starting both the head node and worker nodes with the necessary flags.

```bash
# start head node
RAY_DEBUG=legacy ray start --head --dashboard-host=0.0.0.0 --ray-debugger-external
# start worker node
RAY_DEBUG=legacy ray start --address='10.124.46.192:6379' --ray-debugger-external
```

--------------------------------

### Launch Script Example (Bash)

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/spin.md

Provides an example bash script to execute the training process, specifying the configuration file and potentially other command-line arguments.

```bash
#!/bin/bash

# Example: Launch training with a specific config file
python main_spin.py --config-name spin_trainer

# Example: Launch training with overrides
# python main_spin.py data.train_batch_size=2 trainer.nodes=2

```

--------------------------------

### Dockerfile for ROCm

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

This Dockerfile is designed for AMD GPUs with ROCm support. It sets up the environment, installs necessary libraries like vLLM and PyTorch, and copies the project files. It's configured for specific ROCm architectures and includes installations for various Python packages and a custom PyTorch memory saver.

```bash
# Build the docker in the repo dir:
# docker build -f docker/Dockerfile.rocm -t verl-rocm .
# docker images # you can find your built docker


# Support - Traing: fsdp; Inference: vllm
# FROM rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4
# Support - Traing: fsdp; Inference: vllm, sglang
FROM lmsysorg/sglang:v0.4.6.post5-rocm630

# Set working directory
# WORKDIR $PWD/app

# Set environment variables
ENV PYTORCH_ROCM_ARCH="gfx90a;gfx942"

ENV HIPCC_COMPILE_FLAGS_APPEND="--amdgpu-target=gfx90a;gfx942 -D__HIP_PLATFORM_AMD__"
ENV CFLAGS="-D__HIP_PLATFORM_AMD__"
ENV CXXFLAGS="-D__HIP_PLATFORM_AMD__"

# Install vllm
RUN pip uninstall -y vllm && \
    rm -rf vllm && \
    git clone -b v0.6.3 https://github.com/vllm-project/vllm.git && \
    cd vllm && \
    MAX_JOBS=$(nproc) python3 setup.py install && \
    cd .. && \
    rm -rf vllm

# Copy the entire project directory
COPY . .

# Install dependencies
RUN pip install "tensordict<0.6" --no-deps && \
    pip install accelerate \
    codetiming \
    datasets \
    dill \
    hydra-core \
    liger-kernel \
    numpy \
    pandas \
    peft \
    "pyarrow>=15.0.0" \
    pylatexenc \
    "ray[data,train,tune,serve]">=2.45.0 \
    torchdata \
    transformers \
    wandb \
    orjson \
    pybind11 && \
    pip install -e . --no-deps

# Install torch_memory_saver
RUN pip install git+https://github.com/ExtremeViscent/torch_memory_saver.git --no-deps

```

--------------------------------

### Install cuDNN

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Installs a specific version of cuDNN for Ubuntu 22.04. This involves downloading a .deb package, installing it, copying a keyring, updating apt, and installing the cuDNN package for CUDA 12.

```bash
# change directory to anywher you like, in verl source code directory is not recommended
wget https://developer.download.nvidia.com/compute/cudnn/9.8.0/local_installers/cudnn-local-repo-ubuntu2204-9.8.0_1.0-1_amd64.deb
dpkg -i cudnn-local-repo-ubuntu2204-9.8.0_1.0-1_amd64.deb
cp /var/cudnn-local-repo-ubuntu2204-9.8.0/cudnn-*-keyring.gpg /usr/share/keyrings/
apt-get update
apt-get -y install cudnn-cuda-12
```

--------------------------------

### Test Ray Initialization in Docker Container

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

This snippet tests the initialization of Ray within a Docker container. It checks the cluster status, including the number of nodes and their liveness, and then shuts down Ray. This is useful for verifying distributed setup before proceeding with other tasks.

```bash
echo "Testing Ray initialization in the slurm nodes..."
docker exec "${CONTAINER_NAME}" python3 -c '
import ray
try:
    ray.init(address="auto")
    print("\n=== Ray Cluster Status ===")
    print(f"Number of nodes: {len(ray.nodes())}")
    for node in ray.nodes():
        print("Node: {}, Status: {}".format(node["NodeManagerHostname"], node["Alive"]))
        # print(f"Node: {node}")
    ray.shutdown()
    print("Ray initialization successful!")
except Exception as e:
    print(f"Ray initialization failed: {str(e)}")
'
```

--------------------------------

### Run Training Experiment

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/sglang_multiturn/search_tool_example.rst

Launches the training process for the Qwen2.5-3B model with search R1-like multi-turn capabilities. It configures GPU usage, experiment naming, and logs output.

```bash
# Ensure the now() function is defined
# Create a logs directory
mkdir -p logs

# Set GPUs and run with a suitable log path
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

nohup bash examples/sglang_multiturn/search_r1_like/run_qwen2.5-3b_instruct_search_multiturn.sh \
  trainer.experiment_name=qwen2.5-3b-it_rm-searchR1-like-sgl-multiturn-$(now) \
  > logs/searchR1-like$(now).log 2>&1 &

# Define a timestamp function
function now() {
    date '+%Y-%m-%d-%H-%M'
}
```

--------------------------------

### Main Entry Point (Python)

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/spin.md

Illustrates the main Python script for launching the training process. It utilizes Hydra for configuration management and initializes the SpinTrainer class.

```python
import hydra
from omegaconf import DictConfig
from spin_trainer import SpinTrainer

@hydra.main(config_path="config", config_name="spin_trainer", version_base=None)
def main(cfg: DictConfig):
    trainer = SpinTrainer(cfg)
    trainer.train()

if __name__ == "__main__":
    main()

```

--------------------------------

### Custom Environment CUDA Installation

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Steps to install the CUDA toolkit from a local deb package for custom environments. This involves downloading the package, configuring APT, and installing the specific CUDA toolkit version.

```bash
# change directory to anywher you like, in verl source code directory is not recommended
wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda-repo-ubuntu2204-12-4-local_12.4.1-550.54.15-1_amd64.deb
dpkg -i cuda-repo-ubuntu2204-12-4-local_12.4.1-550.54.15-1_amd64.deb
cp /var/cuda-repo-ubuntu2204-12-4-local/cuda-*-keyring.gpg /usr/share/keyrings/
apt-get update
apt-get -y install cuda-toolkit-12-4
update-alternatives --set cuda /usr/local/cuda-12.4
```

--------------------------------

### Compile and Install vllm-ascend

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/ascend_tutorial/ascend_quick_start.rst

Steps to clone the vllm-ascend repository, set the environment variable for custom kernel compilation, and install the package.

```bash
git clone -b v0.7.3.post1 --depth 1 https://github.com/vllm-project/vllm-ascend.git
cd vllm-ascend
export COMPILE_CUSTOM_KERNELS=1
python setup.py install
```

--------------------------------

### Install verl Dependencies with Megatron/FSDP

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Executes a script to install vLLM and SGLang dependencies, with options to include Megatron backend support. Ensure the 'verl' Conda environment is activated before running.

```bash
# Make sure you have activated verl conda env
# If you need to run with megatron
bash scripts/install_vllm_sglang_mcore.sh
# Or if you simply need to run with FSDP
USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh
```

--------------------------------

### Install NVIDIA Apex

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Installs the NVIDIA Apex library, which is required for Megatron-LM and FSDP training. It's recommended to set the MAX_JOBS environment variable to speed up the process, but caution is advised to avoid memory overload.

```bash
# change directory to anywher you like, in verl source code directory is not recommended
git clone https://github.com/NVIDIA/apex.git && \
cd apex && \
MAX_JOB=32 pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" ./
```

--------------------------------

### Python Development Setup

Source: https://github.com/yongliang-wu/dft/blob/master/verl/CONTRIBUTING.md

Installs the verl package in editable mode with testing and specific backend dependencies (vLLM or SGLang).

```python
pip install -e .[test,vllm]
# or
pip install -e .[test,sglang]
```

--------------------------------

### ROCM Dockerfile for AMD GPUs

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

A Dockerfile snippet for building a Docker image with ROCM kernel support for AMD GPUs (MI300). It specifies the base image, sets environment variables for PyTorch ROCm architecture, and installs a specific version of vLLM and other dependencies.

```dockerfile
FROM rocm/vllm:rocm6.2_mi300_ubuntu20.04_py3.9_vllm_0.6.4

# Set working directory
# WORKDIR $PWD/app

# Set environment variables
ENV PYTORCH_ROCM_ARCH="gfx90a;gfx942"

# Install vllm
RUN pip uninstall -y vllm && \
    rm -rf vllm && \
    git clone -b v0.6.3 https://github.com/vllm-project/vllm.git && \
    cd vllm && \
    MAX_JOBS=$(nproc) python3 setup.py install && \
    cd .. && \
    rm -rf vllm

# Copy the entire project directory
COPY . .

# Install dependencies
RUN pip install "tensordict<0.6" --no-deps && \
    pip install accelerate \
    codetiming \
    datasets \
    dill \
    hydra-core \
    liger-kernel \
    numpy \
    pandas \
    datasets \
    peft \

```

--------------------------------

### RayPPOTrainer Initialization and Execution

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/examples/ppo_code_architecture.rst

Initializes and runs the RayPPOTrainer with the provided configuration, tokenizer, worker mappings, resource pool manager, and reward functions. It includes steps for worker initialization and the main training loop.

```python
trainer = RayPPOTrainer(config=config,
                        tokenizer=tokenizer,
                        role_worker_mapping=role_worker_mapping,
                        resource_pool_manager=resource_pool_manager,
                        ray_worker_group_cls=ray_worker_group_cls,
                        reward_fn=reward_fn,
                        val_reward_fn=val_reward_fn)
trainer.init_workers()
trainer.fit()
```

--------------------------------

### Login & Download Data/Model

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/spin.md

This snippet shows how to log in to Weights & Biases (optional), download the GSM8K dataset to a specified local directory, and download the Qwen2.5-3B-Instruct base model using huggingface-cli.

```bash
# Login to Weights & Biases (optional, for logging)
export WANDB_API_KEY=<YOUR_WANDB_API_KEY>
# wandb login

# Download the GSM8K dataset
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k # Adjusted path

# Download the base model (Example: Qwen2.5-3B-Instruct)
huggingface-cli download Qwen/Qwen2.5-3B-Instruct --local-dir $HOME/models/Qwen2.5-3B-Instruct
```

--------------------------------

### Setup and Run Pre-commit Hooks

Source: https://github.com/yongliang-wu/dft/blob/master/verl/CONTRIBUTING.md

Installs pre-commit, sets up the git hooks, and runs pre-commit checks on staged or all files to maintain code consistency.

```bash
pip install pre-commit
pre-commit install
# for staged changes
pre-commit run
# for all files in the repo
# pre-commit run --all-files
```

--------------------------------

### Build Docker Image

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Builds the Docker image for the DFT project with the tag 'verl-rocm'. This command assumes a Dockerfile is present in the current directory.

```bash
docker build -t verl-rocm .
```

--------------------------------

### Qwen2.5 Training Command with Dual-clip PPO Parameters

Source: https://github.com/yongliang-wu/dft/blob/master/verl/examples/ppo_trainer/README.md

This bash script demonstrates how to initiate the training of a Qwen2.5 model using specific configurations for Dual-clip PPO. It highlights parameters like the number of GPUs, logger settings, model paths, batch sizes, and micro-batch sizes for both actor and critic components.

```bash
bash run_gemma.sh \
  trainer.n_gpus_per_node=1 \
  actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
  trainer.logger=['console'] \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  data.train_batch_size=256 \
  actor_rollout_ref.actor.ppo_mini_batch_size=64 \
  actor_rollout_ref.actor.ppo_micro_batch_size=2 \
  critic.ppo_micro_batch_size=2
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Creates a new Conda environment named 'verl' with Python 3.10 and activates it. This is recommended for managing dependencies.

```bash
conda create -n verl python==3.10
conda activate verl
```

--------------------------------

### Login & Download Data/Model

Source: https://github.com/yongliang-wu/dft/blob/master/verl/recipe/spin/README.md

This snippet shows how to log in to Weights & Biases (optional), download the GSM8K dataset to a specified local directory, and download the Qwen2.5-3B-Instruct base model using huggingface-cli.

```bash
# Login to Weights & Biases (optional, for logging)
export WANDB_API_KEY=<YOUR_WANDB_API_KEY>
# wandb login

# Download the GSM8K dataset
python3 examples/data_preprocess/gsm8k.py --local_dir ~/data/gsm8k # Adjusted path

# Download the base model (Example: Qwen2.5-3B-Instruct)
huggingface-cli download Qwen/Qwen2.5-3B-Instruct --local-dir $HOME/models/Qwen2.5-3B-Instruct
```

--------------------------------

### dft Launch Script Example (Bash)

Source: https://github.com/yongliang-wu/dft/blob/master/verl/recipe/spin/README.md

Provides an example bash script for launching a training run using the dft project. It demonstrates how to set up the environment and execute the main Python script with the specified configuration file.

```bash
# run_spin.sh

# Set environment variables if needed
# export CUDA_VISIBLE_DEVICES=0,1,2,3

# Launch the training script with Hydra configuration
python main_spin.py --config-name spin_trainer

```

--------------------------------

### Build Docker Image

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

Command to build the Docker image for ROCm environment. This command should be executed in the repository directory.

```bash
docker build -t verl-rocm .
```

--------------------------------

### Submit Slurm Script

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

This command submits the slurm script for execution. It is the final step to launch the configured training job on the Slurm cluster.

```bash
sbatch slurm_script.sh
```

--------------------------------

### Run PPO Training with Verl

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/quickstart.rst

This command initiates PPO training using the `verl` library. It configures various training parameters such as dataset paths, batch sizes, learning rates, and model paths for both actor and critic. The output is logged to `verl_demo.log`. Ensure `VERL_USE_MODELSCOPE` is set if using ModelScope.

```bash
PYTHONUNBUFFERED=1 python3 -m verl.trainer.main_ppo \
 data.train_files=$HOME/data/gsm8k/train.parquet \
 data.val_files=$HOME/data/gsm8k/test.parquet \
 data.train_batch_size=256 \
 data.max_prompt_length=512 \
 data.max_response_length=256 \
 actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
 actor_rollout_ref.actor.optim.lr=1e-6 \
 actor_rollout_ref.actor.ppo_mini_batch_size=64 \
 actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \
 actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=8 \
 actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
 actor_rollout_ref.rollout.gpu_memory_utilization=0.4 \
 actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=4 \
 critic.optim.lr=1e-5 \
 critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
 critic.ppo_micro_batch_size_per_gpu=4 \
 algorithm.kl_ctrl.kl_coef=0.001 \
 trainer.logger=['console'] \
 trainer.val_before_train=False \
 trainer.default_hdfs_dir=null \
 trainer.n_gpus_per_node=1 \
 trainer.nnodes=1 \
 trainer.save_freq=10 \
 trainer.test_freq=10 \
 trainer.total_epochs=15 2>&1 | tee verl_demo.log
```

--------------------------------

### Verl Model Merger Module

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/quickstart.rst

Demonstrates how to use the `verl.model_merger` module in Python to merge model checkpoints. This example shows the basic structure for merging.

```python
import verl.model_merger

# Example usage (conceptual, actual parameters depend on the module's API)
# verl.model_merger.merge(
#     backend='fsdp',
#     local_dir='checkpoints/your_project/your_experiment/global_step_1/actor',
#     target_dir='checkpoints/your_project/your_experiment/global_step_1/actor/huggingface'
# )
```

--------------------------------

### Ray Cluster Initialization (Head and Workers)

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

This section details the process of initializing a Ray cluster across multiple nodes. It first identifies the head node and its IP address, then starts the Ray head node within the Docker container. Subsequently, it iterates through the remaining nodes to start Ray worker nodes, connecting them to the head node. Environment variables like `ip_head`, `SLURM_CPUS_PER_TASK`, and `SLURM_GPUS_PER_NODE` are used to configure the Ray cluster.

```bash
# Getting the node names
nodes_array=($(scontrol show hostnames "$SLURM_JOB_NODELIST" | tr '\n' ' '))

head_node=${nodes_array[0]}
head_node_ip=$(srun --nodes=1 --ntasks=1 -w "$head_node" hostname --ip-address)

# if we detect a space character in the head node IP, we'll
# convert it to an ipv4 address. This step is optional.
if [[ "$head_node_ip" == *" "* ]]; then
    IFS=' ' read -ra ADDR <<<"$head_node_ip"
if [[ ${#ADDR[0]} -gt 16 ]]; then
    head_node_ip=${ADDR[1]}
else
    head_node_ip=${ADDR[0]}
fi
    echo "IPV6 address detected. We split the IPV4 address as $head_node_ip"
fi

port=6379
ip_head=$head_node_ip:$port
export ip_head
echo "IP Head: $ip_head"

# make sure we set environment variables before Ray initialization

# Print out all env variables
printenv

echo "Starting HEAD at $head_node"
srun --nodes=1 --ntasks=1 -w "$head_node" \
    docker exec "${CONTAINER_NAME}" \
        ray start --head --node-ip-address="$head_node_ip" --port=$port \
        --dashboard-port=8266 \
        --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
# optional, though may be useful in certain versions of Ray < 1.0.
sleep 10

# number of nodes other than the head node
worker_num=$(($SLURM_JOB_NUM_NODES - 1))

for ((i = 1; i <= worker_num; i++)); do
    node_i=${nodes_array[$i]}
    echo "Debug: Starting worker on node_i = ${node_i}"
    if [ -z "$node_i" ]; then
        echo "Error: Empty node name for worker $i"
        continue
    fi
    echo "Starting WORKER $i at $node_i"
    srun --nodes=1 --ntasks=1 -w "$node_i" \
        docker exec "${CONTAINER_NAME}" \
            ray start --address "$ip_head" --num-cpus "${SLURM_CPUS_PER_TASK}" --num-gpus "${SLURM_GPUS_PER_NODE}" --block &
    sleep 5
done

```

--------------------------------

### Install verl with NPU Requirements

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/ascend_tutorial/ascend_quick_start.rst

Cloning the verl repository and installing its dependencies, specifically those required for NPU support.

```bash
git clone https://github.com/volcengine/verl.git
cd verl
pip install -r requirements-npu.txt
pip install -e .
```

--------------------------------

### Launch Docker Container (Non-Root User)

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Launches the Docker container with user-specific UID and GID mapping to avoid running as root. This requires passing environment variables HOST_UID and HOST_GID.

```bash
docker run --rm -it \
  -e HOST_UID=$(id -u) \
  -e HOST_GID=$(id -g) \
  --device /dev/dri \
  --device /dev/kfd \
  -p 8265:8265 \
  --group-add video \
  --cap-add SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --privileged \
  -v $HOME/.ssh:/root/.ssh \
  -v $HOME:$HOME \
  --shm-size 128G \
  -w $PWD \
  verl-rocm \
  /bin/bash
```

--------------------------------

### Verl Worker Remote Call Example

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/hybrid_flow.rst

Illustrates how a controller process invokes methods on remote workers managed by a WorkerGroup, specifically demonstrating the pattern for initializing models across multiple workers.

```python
# Assuming actor_rollout_ref_wg is an instance of WorkerGroup

# Initialize models on all workers
results = []
for worker in actor_rollout_ref_wg.workers:
    results.append(worker.init_model.remote())

# Wait for all initialization tasks to complete
ray.get(results)

# Generate sequences using workers
data = "..."
dp_size = len(actor_rollout_ref_wg.workers)
data_dp_lst = data.split(dp_size)
output_dp_lst = []
for i, worker in enumerate(actor_rollout_ref_wg.workers):
    output_dp_lst.append(worker.generate_sequences.remote(data_dp_lst[i]))

# Process collected outputs...
```

--------------------------------

### Qwen2.5 Training Command with Dual-clip PPO Parameters

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/algo/ppo.md

This bash script demonstrates how to initiate the training of a Qwen2.5 model using specific configurations for Dual-clip PPO. It highlights parameters like the number of GPUs, logger settings, model paths, batch sizes, and micro-batch sizes for both actor and critic components.

```bash
bash run_gemma.sh \
  trainer.n_gpus_per_node=1 \
  actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
  trainer.logger=['console'] \
  critic.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \
  data.train_batch_size=256 \
  actor_rollout_ref.actor.ppo_mini_batch_size=64 \
  actor_rollout_ref.actor.ppo_micro_batch_size=2 \
  critic.ppo_micro_batch_size=2
```

--------------------------------

### Initialize dstack Repository

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/multinode.rst

Commands to create a new project directory and initialize it as a dstack repository.

```bash
mkdir myproject && cd myproject
dstack init
```

--------------------------------

### Launch Docker Container

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/install.rst

Launches a Docker container named 'verl-rocm' with specific device mappings, port forwarding, volume mounts, and working directory. It includes configurations for AMD GPU support and SSH access.

```bash
docker run --rm -it \
  --device /dev/dri \
  --device /dev/kfd \
  -p 8265:8265 \
  --group-add video \
  --cap-add SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --privileged \
  -v $HOME/.ssh:/root/.ssh \
  -v $HOME:$HOME \
  --shm-size 128G \
  -w $PWD \
  verl-rocm \
  /bin/bash
```

--------------------------------

### Run ROCm Docker Container

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

Command to run the Docker container with necessary device mappings and volume mounts for AMD GPU access. Includes options for user permissions and shared memory.

```bash
docker run --rm -it \
  --device /dev/dri \
  --device /dev/kfd \
  -p 8265:8265 \
  --group-add video \
  --cap-add SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --privileged \
  -v $HOME/.ssh:/root/.ssh \
  -v $HOME:$HOME \
  --shm-size 128G \
  -w $PWD \
  verl-rocm \
  /bin/bash
```

--------------------------------

### Launch Training with DFT

Source: https://github.com/yongliang-wu/dft/blob/master/README.md

This snippet demonstrates how to launch the training process for the DFT project using torchrun. It configures various parameters related to data, model, and training, including distributed training settings.

```bash
nproc_per_node=8
project_name=numina-cot

experiment_name=numina-cot-dft-qwen-2.5-math-1.5b
save_path=checkpoints/$experiment_name

torchrun --standalone --nnodes=1 --nproc_per_node=$nproc_per_node \
        -m verl.trainer.fsdp_dft_trainer \
    data.train_files=data/numina_cot/train.parquet \
    data.val_files=data/math500/test.parquet \
    data.prompt_key=extra_info \
    data.response_key=extra_info \
    data.train_batch_size=256 \
    data.max_length=2048 \
    optim.lr=5e-5 \
    data.prompt_dict_keys=['question'] \
    data.response_dict_keys=['answer'] \
    data.micro_batch_size_per_gpu=4 \
    model.partial_pretrain=Qwen/Qwen2.5-Math-1.5B \
    model.use_liger=True \
    model.fsdp_config.model_dtype=bf16 \
    trainer.default_local_dir=$save_path \
    trainer.project_name=$project_name \
    trainer.experiment_name="$experiment_name-$(date +%Y%m%d-%H%M%S)" \
    trainer.logger=['console','tensorboard'] \
    trainer.default_hdfs_dir=null \
    trainer.test_freq=10 \
    trainer.save_freq=50 \
    trainer.total_epochs=1 \
    ulysses_sequence_parallel_size=1 \
    use_remove_padding=true
```

--------------------------------

### Qwen3 236b Training Script

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/perf/dpsk.md

This shell script provides an example of how to run training for the Qwen3-236b model using the verl framework and Megatron. It indicates the required hardware setup, specifying 128 H20(96GB) GPUs for execution.

```shell
#!/bin/bash

# Example usage for Qwen3-236b training
# Assumes 128 H20(96GB) GPUs are available

# This is a placeholder for the actual script path.
# The actual script is located at: examples/grpo_trainer/run_qwen3-236b_megatron.sh
# echo "Running Qwen3 236b training..."
# ./examples/grpo_trainer/run_qwen3-236b_megatron.sh --num-gpus 128
```

--------------------------------

### Run Data Preprocessing for GSM8K Dataset

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/amd_tutorial/amd_build_dockerfile_page.rst

This command executes a Python script to preprocess the GSM8K dataset. It specifies the local directory where the dataset files will be stored. This step is crucial for preparing the data for model training.

```bash
echo "Starting data preprocessing..."
docker exec "${CONTAINER_NAME}" \
    python3 "examples/data_preprocess/gsm8k.py" "--local_dir" "../data/gsm8k"
```

--------------------------------

### Enable Wandb for Experiment Tracking

Source: https://github.com/yongliang-wu/dft/blob/master/verl/docs/start/quickstart.rst

Configure the trainer to use 'wandb' for logging experiments. This involves setting the logger to include 'wandb' and specifying the project and experiment names.

```bash
trainer.logger=['console','wandb'] \
    trainer.project_name=$YOUR_PROJECT_NAME \
    trainer.experiment_name=$YOUR_RUN_NAME \
```
Tokens:
Raw