### Install Trinity-RFT and Flash-Attention using uv

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Installs specific versions of Trinity-RFT and FlashAttention using the `uv` package manager. This offers a faster alternative to `pip` for installing packages from PyPI.

```bash
uv pip install trinity-rft==0.3.1
uv pip install flash-attn==2.8.1

```

--------------------------------

### Install Trinity-RFT via uv (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Installs Trinity-RFT and flash-attn using the `uv` package manager from PyPI. This offers an alternative to pip for installing pre-compiled packages.

```bash
uv pip install trinity-rft==0.3.1
uv pip install flash-attn==2.8.1
```

--------------------------------

### Build and Run Trinity-RFT Docker Image

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Builds a Docker image for Trinity-RFT and runs it, mounting the current directory and a specified data path. This allows for a quick and isolated environment setup, including GPU access.

```bash
git clone https://github.com/modelscope/Trinity-RFT
cd Trinity-RFT

# 构建 Docker 镜像
## 提示：可根据需要修改 Dockerfile 添加镜像源或设置 API 密钥
docker build -f scripts/docker/Dockerfile -t trinity-rft:latest .

# 运行容器，请将 <path_to_your_data_and_checkpoints> 替换为实际需要挂载的路径
docker run -it \
  --gpus all \
  --shm-size="64g" \
  --rm \
  -v $PWD:/workspace \
  -v <path_to_your_data_and_checkpoints>:/data \
  trinity-rft:latest

```

--------------------------------

### Install Trinity-RFT and Flash-Attention using uv (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Installs Trinity-RFT and flash-attn using the `uv` package manager. This is an alternative to pip for faster dependency management.

```bash
uv sync --extra dev --extra flash_attn
```

--------------------------------

### Clone Trinity-RFT Repository (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Clones the Trinity-RFT repository from GitHub and navigates into the cloned directory. This is the first step for source installation.

```bash
git clone https://github.com/modelscope/Trinity-RFT
cd Trinity-RFT
```

--------------------------------

### Install Trinity-RFT and Flash-Attention via PyPI

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Installs specific versions of Trinity-RFT and FlashAttention directly from PyPI using pip. This is suitable for users who do not need to modify the source code.

```bash
pip install trinity-rft==0.3.1
pip install flash-attn==2.8.1

```

--------------------------------

### Install Trinity-RFT via PyPI (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Installs Trinity-RFT and flash-attn directly from PyPI using pip. This method is suitable for users who do not need to modify the source code.

```bash
pip install trinity-rft==0.3.1
pip install flash-attn==2.8.1
```

--------------------------------

### Clone Trinity-RFT Repository using Git

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

This command clones the Trinity-RFT repository from GitHub. It's the first step in source installation and requires Git to be installed.

```bash
git clone https://github.com/modelscope/Trinity-RFT
cd Trinity-RFT

```

--------------------------------

### Create venv Virtual Environment and Install Dependencies (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Creates and activates a Python virtual environment using venv, then installs Trinity-RFT with development and flash-attn dependencies. Includes a fallback for flash-attn installation issues.

```bash
python3.10 -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"
pip install -e ".[flash_attn]"
# 如果安装 flash-attn 时遇到问题，可尝试：
# pip install flash-attn==2.8.1 --no-build-isolation
```

--------------------------------

### Install Trinity-RFT Dependencies using uv

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Uses the `uv` package manager to install development and FlashAttention dependencies for Trinity-RFT. `uv` is a fast, modern alternative to pip.

```bash
uv sync --extra dev --extra flash_attn

```

--------------------------------

### Build and Run Trinity-RFT Docker Image (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Builds the Trinity-RFT Docker image and runs a container with GPU support, shared memory, and volume mounts for code and data. Assumes the repository has been cloned.

```bash
git clone https://github.com/modelscope/Trinity-RFT
cd Trinity-RFT

# 构建 Docker 镜像
## 提示：可根据需要修改 Dockerfile 添加镜像源或设置 API 密钥
docker build -f scripts/docker/Dockerfile -t trinity-rft:latest .

# 运行容器，请将 <path_to_your_data_and_checkpoints> 替换为实际需要挂载的路径
docker run -it \
  --gpus all \
  --shm-size="64g" \
  --rm \
  -v $PWD:/workspace \
  -v <path_to_your_data_and_checkpoints>:/data \
  trinity-rft:latest
```

--------------------------------

### Webshop Workflow Agent Prompt with Example - Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/common/workflows/envs/webshop/webshop_workflow

Defines the system prompt for a webshop agent, including an example of interaction. It guides the agent on how to search and click within a virtual shopping environment to fulfill user instructions and buy items. The prompt specifies the action format and provides a detailed example of a multi-turn interaction.

```python
# -*- coding: utf-8 -*-
from typing import List, Optional

from trinity.common.experience import Experience
from trinity.common.models.model import ModelWrapper
from trinity.common.workflows.workflow import WORKFLOWS, MultiTurnWorkflow, Task

SPARSE_REWARD = False

EXAMPLE_PROMPT = """
Observation:
Webshop
Instruction:
i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
[button] Search [button_]

Response:
<think>OK, let's search for 3 ounce bright citrus deodorant sensitive skin</think><action>search[3 ounce bright citrus deodorant sensitive skin]</action>

Observation:
Instruction:
i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
[button] Back to Search [button_]
Page 1 (Total results: 50)
[button] Next > [button_]
[button] B078GWRC1J [button_]
Bright Citrus Deodorant by Earth Mama | Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[button] B078GTKVXY [button_]
Ginger Fresh Deodorant by Earth Mama | Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
$10.99
[button] B08KBVJ4XN [button_]
Barrel and Oak - Aluminum-Free Deodorant, Deodorant for Men, Essential Oil-Based Scent, 24-Hour Odor Protection, Cedar & Patchouli Blend, Gentle on Sensitive Skin (Mountain Sage, 2.7 oz, 2-Pack)
$15.95

Response:
<think>button B078GWRC1J and B078GTKVXY are bright citrus deodorant less then 50 dollars. I can check B078GWRC1J first</think><action>click[B078GWRC1J]</action>

Observation:
Instruction:
i would like a 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
[button] Back to Search [button_]
[button] < Prev [button_]
scent [button] assorted scents [button_][button] bright citrus [button_][button] calming lavender [button_][button] ginger fresh [button_][button] simply non-scents [button_]
size [button] travel set (4-pack) [button_][button] 3 ounce (pack of 1) [button_][button] 3-ounce (2-pack) [button_]
Bright Citrus Deodorant by Earth Mama | Natural and Safe for Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic Calendula 3-Ounce
Price: $10.99
Rating: N.A.
[button] Description [button_]
[button] Features [button_]
[button] Reviews [button_]
[button] Buy Now [button_]

Response:
<think>For 3 ounce bottle of bright citrus deodorant for sensitive skin, the item has options 'bright citrus' and '3 ounce (pack of 1)' and seems good to buy. </think><action>click[bright citrus]</action>

Observation:
You have clicked bright citrus.
...

Response:
<think>Now I should select the 3 ounce (pack of 1) option</think><action>click[3 ounce (pack of 1)]</action>

Observation:
You have clicked 3 ounce (pack of 1).
...

Response:
<think>I can buy the item</think><action>click[Buy Now]</action>
""


WebShop_SYSTEM_PROMPT_WITH_EXAMPLE = f"""
You are an agent interacting with a virtual text-based web shopping environment to test out your ability. Your job is to find follow the Instruction provided and mimic the steps to buy the item that are closest to the Instruct provided.

## Action Format:
You should give both the action_name and action_arg like the format `action_name[action_arg]`. You can execute two types of actions, search and click.
- When the button `[button] Search [button_]` is available in the current observation, you can execute the action <action>search[xxx]</action> (you should type the query you want to search in the square brackets here).
- You can click buttons `[button] xxx [button_]` that is available in the current observation, by execute the action <action>click[xxx]</action>.

Below are some examples of action formats.
- <action>search[white shoes]</action>
- <action>click[Buy Now]</action>

## Example:
Here is an example:
```
{EXAMPLE_PROMPT}
```

## Notes:
At each step, you should first think then perform action to fulfill the instruction. You should ALWAYS wrap your thinking with the <think> </think> tag and wrap your action with the <action> </action> tag.
You should ALWAYS take one action each step.
You should finish the task and buy the item within 15 steps.
DONOT try to interact with the user at anytime. Finish the task and buy the item by yourself.
"""

WebShop_SYSTEM_PROMPT = """
You are an agent interacting with a virtual text-baed web shopping environments to testout your ability. Your job is to find follow the Instruction provided and mimic the steps to buy the item that are closest to the Instruct provided.

"""

```

--------------------------------

### Create Conda Virtual Environment and Install Dependencies (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/trinity_installation

Creates and activates a Conda virtual environment with Python 3.10, then installs Trinity-RFT with development and flash-attn dependencies. Includes a fallback for flash-attn installation issues.

```bash
conda create -n trinity python=3.10
conda activate trinity

pip install -e ".[dev]"
pip install -e ".[flash_attn]"
# 如果安装 flash-attn 时遇到问题，可尝试：
# pip install flash-attn==2.8.1 --no-build-isolation
```

--------------------------------

### Install AgentScope Library (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_react

This command installs the AgentScope library, a dependency for Trinity-RFT, ensuring version 1.0.4 or higher is used. This is a prerequisite for running the training examples.

```bash
pip install agentscope>=1.0.4
```

--------------------------------

### Prepare Experience Pipeline and Models

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/explorer/explorer

This asynchronous method prepares the environment before running. It initializes the experience pipeline, ensures all rollout models (including auxiliary models) are ready by starting their API servers, and sets up the weight synchronization group if NCCL is not used. It also initializes and starts a scheduler if the mode is not 'serve' and performs an evaluation on startup if configured.

```python
async def prepare(self) -> None:
        """Preparation before running."""
        try:
            # prepare experience pipeline
            await self.experience_pipeline.prepare.remote()
            self.logger.info("Experience pipeline is ready.")
            # make sure all rollout models are ready
            run_api_ref = [model.run_api_server.remote() for model in self.models]
            run_api_ref.extend(
                model.run_api_server.remote()
                for models in self.auxiliary_models
                for model in models
            )
            await asyncio.gather(*run_api_ref)
            self.logger.info("All models are ready.")

            if not self.use_nccl_sync:
                master_address, master_port = await self.models[0].get_available_address.remote()
                await self.setup_weight_sync_group(master_address, master_port)
            if self.config.mode != "serve":
                self.scheduler = Scheduler(self.config, self.models, self.auxiliary_models)
                await self.scheduler.start()
            if self.config.explorer.eval_on_startup and self.explore_step_num == 0:
                await self.eval()

            await self.synchronizer.set_explorer_status.remote(RunningStatus.REQUIRE_SYNC)
        except Exception as e:
            self.logger.error(f"Error during explorer preparation: {traceback.format_exc()}")
            await self.shutdown()
            raise e
```

--------------------------------

### Run Script for Async GRPO Example (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_async_mode

Bash script to execute the asynchronous GRPO example. This command initiates the trainer and explorer processes as defined in their respective configuration files.

```bash
bash examples/async_gsm8k/run.sh
```

--------------------------------

### Start Ray Cluster for Trinity-RFT

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_data_functionalities

These commands demonstrate how to start a Ray cluster, essential for running the Trinity-RFT pipeline. It includes commands for starting the head node and worker nodes.

```bash
# Start ray cluster
# On the head node:
ray start --head
# On the worker nodes:
ray start --address=<master_address>
```

--------------------------------

### Async GRPO Trainer Configuration (YAML)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_async_mode

Configuration file for the trainer process in asynchronous GRPO. It defines training mode, model path, cluster setup, buffer settings for training data, and optimizer parameters.

```yaml
# trainer.yaml
project: <project_name>
name: <experiment_name>
mode: train
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: grpo
  repeat_times: 8
  optimizer:
    lr: 1e-6
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:
  node_num: 1
  gpu_per_node: 4
buffer:
  total_epochs: 1
  train_batch_size: 512
  explorer_input:
    taskset:
      name: gsm8k
      storage_type: file
      path: 'openai/gsm8k'
      subset_name: 'main'
      format:
        prompt_key: 'question'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
      default_workflow_type: 'math_workflow'
  trainer_input:
    experience_buffer:
      name: gsm8k_buffer
      storage_type: queue
      path: 'sqlite:///gsm8k.db'
synchronizer:
  sync_method: 'checkpoint'
  sync_interval: 10
trainer:
  grad_clip: 1.0
  use_dynamic_bsz: true
  max_token_len_per_gpu: 16384
  ulysses_sequence_parallel_size: 1

```

--------------------------------

### Dynamic Scaling Explorer Configuration (YAML)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_async_mode

Configuration for adding a new explorer dynamically to an existing asynchronous GRPO setup. It adjusts cluster, explorer, and buffer settings to integrate the new node efficiently.

```yaml
# explorer_new.yaml
project: <project_name>
name: <experiment_name>
mode: explore
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: grpo
  repeat_times: 8
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:  # important
  node_num: 1
  gpu_per_node: 8
explorer:
  name: 'explorer_new'  # important
  runner_per_model: 8
  rollout_model:
    engine_num: 8
buffer:
  total_epochs: 1
  batch_size: 64
  explorer_input:
    taskset:  # important
      name: gsm8k
      storage_type: file
      path: 'openai/gsm8k'
      subset_name: 'main'
      format:
        prompt_key: 'question'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
      default_workflow_type: 'math_workflow'
  trainer_input:
    experience_buffer:
      name: gsm8k_buffer
      storage_type: queue
      path: 'sqlite:///gsm8k.db'
synchronizer:
  sync_method: 'checkpoint'
  sync_interval: 10
# other configs are the same as explorer.yaml

```

--------------------------------

### Install AgentScope and Trinity-RFT Dependencies

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_react

This command installs the AgentScope library version 1.0.4 or higher, which is a prerequisite for running Trinity-RFT. Ensure you have Python and pip installed.

```bash
pip install agentscope>=1.0.4

```

--------------------------------

### Run Asynchronous GSM8K Example (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_async_mode

This bash script executes the asynchronous GRPO example using the GSM8K dataset. It is the command used to launch the entire training and exploration process.

```bash
examples/async_gsm8k/run.sh

```

--------------------------------

### Install NVIDIA Apex for Mixed Precision Training

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_megatron

Installs NVIDIA's Apex library from source, which is required for mixed-precision training. This command ensures that custom C++ and CUDA extensions are built, enabling performance optimizations for training.

```bash
pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation \
    --config-settings "--build-option=--cpp_ext" \
    --config-settings "--build-option=--cuda_ext" \
    --resume-retries 10 git+https://github.com/NVIDIA/apex.git
```

--------------------------------

### Run Trinity-RFT Pipeline

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_data_functionalities

This command initiates the Trinity-RFT pipeline using a specified configuration file. Ensure the path to the Trinity-RFT configuration is correctly provided.

```bash
trinity run --config <Trinity-RFT_config_path>
```

--------------------------------

### Configuration Initialization for RL Components

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/trainer/verl_trainer

This snippet shows how various reinforcement learning components like advantage functions, KL divergence functions, and sampling strategies are initialized based on configuration settings. It demonstrates dynamic loading of classes using `.get()` with associated arguments.

```python
            self.advantage_fn = ADVANTAGE_FN.get(self.algorithm_config.advantage_fn)(
                **self.algorithm_config.advantage_fn_args
            )
            self.kl_fn = KL_FN.get(self.algorithm_config.kl_penalty_fn)(
                **self.algorithm_config.kl_penalty_fn_args
            )
        self.sample_strategy = SAMPLE_STRATEGY.get(global_config.algorithm.sample_strategy)(
            buffer_config=global_config.buffer,
            trainer_type=global_config.trainer.trainer_type,
            **global_config.algorithm.sample_strategy_args,
        )
        super().__init__(
            config,
            tokenizer,
            role_worker_mapping,
            resource_pool_manager,
            ray_worker_group_cls,
            processor=processor,
        )
        self.init_workers()
        self.last_full_save_step = None

```

--------------------------------

### 下载 Qwen2.5-1.5B-Instruct 模型

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_reasoning_basic

使用 Modelscope 或 Huggingface CLI 下载 Qwen2.5-1.5B-Instruct 模型到指定的本地目录。此步骤是模型准备的关键，确保模型文件可供 Trinity-RFT 使用。

```shell
# 使用 Modelscope
modelscope download Qwen/Qwen2.5-1.5B-Instruct --local_dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

# 使用 Huggingface
huggingface-cli download Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Create and Activate venv Virtual Environment for Trinity-RFT

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Installs Trinity-RFT and its development dependencies using Python's built-in `venv` module. This is an alternative to Conda for creating isolated Python environments.

```bash
python3.10 -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"
pip install -e ".[flash_attn]"
# 如果安装 flash-attn 时遇到问题，可尝试：
# pip install flash-attn==2.8.1 --no-build-isolation

```

--------------------------------

### Create and Activate Conda Virtual Environment for Trinity-RFT

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/trinity_installation

Installs Trinity-RFT and its development dependencies within a Conda virtual environment. This method is recommended for managing Python versions and project dependencies, especially when CUDA is involved.

```bash
conda create -n trinity python=3.10
conda activate trinity

pip install -e ".[dev]"
pip install -e ".[flash_attn]"
# 如果安装 flash-attn 时遇到问题，可尝试：
# pip install flash-attn==2.8.1 --no-build-isolation

```

--------------------------------

### DPO Configuration Example

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_dpo

This YAML configuration outlines key settings for a DPO experiment. It specifies the project name, experiment name, mode ('train'), algorithm type ('dpo'), KL loss function arguments, checkpoint directory, model path, cluster settings, buffer configurations including dataset path and format, and trainer settings.

```yaml
project: <project_name>
name: <experiment_name>
mode: train
algorithm:
  algorithm_type: dpo
  kl_loss_fn: k1
  kl_loss_fn_args:
    kl_coef: 0.1  # DPO 中 beta 的值
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 2
  train_batch_size: 64
  trainer_input:
    experience_buffer:
      name: human_like_dpo
      storage_type: file
      path: $DATASET_PATH/human_like_dpo_dataset
      format:
        prompt_type: plaintext
        prompt_key: prompt
        chosen_key: chosen
        rejected_key: rejected
trainer:
  save_interval: 30
  trainer_config:
    ... # 省略其他配置
```

--------------------------------

### Get OpenAI API Server URL in Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/common/models/vllm_model

Retrieves the URL for the OpenAI API server. It first attempts to start the server using `run_api_server()` and then constructs the URL using the determined host and port. Returns None if the API server cannot be started.

```python
async def get_api_server_url(self) -> Optional[str]:
        """Get the URL of the OpenAI API server.

        Returns:
            api_url (str): The URL of the OpenAI API server.
        """
        if not await self.run_api_server():
            return None
        return f"http://{self.api_server_host}:{self.api_server_port}"
```

--------------------------------

### 导入 Workflow - Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_multi_turn

在 Trinity-RFT 的 `__init__.py` 文件中导入新注册的 workflow 类，使其可用于全局访问。

```python
# -*- coding: utf-8 -*-
"""Workflow module"""
from .workflow import WORKFLOWS, MathWorkflow, SimpleWorkflow
from .envs.alfworld.alfworld_workflow import AlfworldWorkflow

__all__ = [
    "WORKFLOWS",
    "SimpleWorkflow",
    "MathWorkflow",
    "AlfworldWorkflow",
]
```

--------------------------------

### 下载模型: Qwen2.5-1.5B-Instruct

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_reasoning_basic

使用 Modelscope 或 Huggingface CLI 将 Qwen2.5-1.5B-Instruct 模型下载到本地目录。确保已安装相应的 CLI 工具。此步骤是模型准备的关键。

```bash
# 使用 Modelscope
modelscope download Qwen/Qwen2.5-1.5B-Instruct --local_dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

# 使用 Huggingface
huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Get Actor in Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/build_api/trinity.trainer

A class method to retrieve an actor, optionally configuring it with save strategy and directories. Used in distributed setups.

```python
_classmethod _get_actor(_namespace : str_, _save_strategy : SaveStrategy | None = None_, _default_local_dir : str | None = None_, _default_hdfs_dir : str | None = None_)
```

--------------------------------

### 运行 Trinity-RFT 实验

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_reasoning_basic

使用 Trinity-RFT CLI 命令来启动配置好的实验。此命令会加载指定的 YAML 配置文件并执行 RFT 流程。

```bash
trinity run --config examples/grpo_gsm8k/gsm8k.yaml
```

--------------------------------

### Example Configuration for Mix Algorithm (YAML)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_mix_algo

Provides a sample YAML configuration for running experiments with the 'mix' algorithm. It specifies parameters such as repeat times, data ratio, policy loss function arguments (mu, clip range, loss aggregation modes, batch sizes), and device configuration.

```yaml
algorithm:
  algorithm_type: mix
  repeat_times: 8
  sample_strategy_args:
    expert_data_ratio: 0.25
  policy_loss_fn_args:
    mu: 0.1
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: True
    repeat_times: 8
    ppo_mini_batch_size: 256
    ppo_micro_batch_size_per_gpu: 4
    ngpus_trainer: 4
    train_batch_size_expert: 64
    train_batch_size_usual: 192

```

--------------------------------

### Run SFT Experiment with Trinity CLI

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_dpo

This command executes the Supervised Fine-Tuning (SFT) experiment using the Trinity command-line interface. It requires the path to the SFT configuration file.

```shell
trinity run --config examples/sft_mot/sft.yaml

```

--------------------------------

### Get DLC Environment Variables (Python)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/utils/dlc_utils

Retrieves and validates essential environment variables required for a distributed Ray cluster setup in a DLC environment. It checks for RANK, WORLD_SIZE, MASTER_ADDR, and MASTER_PORT, raising a ValueError if any are not set.

```Python
import os
import sys

from trinity.utils.log import get_logger

logger = get_logger(__name__)

def get_dlc_env_vars() -> dict:
    envs = {
        "RANK": int(os.environ.get("RANK", -1)),  # type: ignore
        "WORLD_SIZE": int(os.environ.get("WORLD_SIZE", -1)),  # type: ignore
        "MASTER_ADDR": os.environ.get("MASTER_ADDR", None),
        "MASTER_PORT": os.environ.get("MASTER_PORT", None),
    }
    for key, value in envs.items():
        if value is None or value == -1:
            logger.error(f"DLC env var `{key}` is not set.")
            raise ValueError(f"DLC env var `{key}` is not set.")
    return envs
```

--------------------------------

### AgentScopeWorkflowAdapter Initialization and Setup (Python)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/common/workflows/agentscope_workflow

Initializes the AgentScopeWorkflowAdapter, requiring a Task and Model. It ensures AgentScope is installed and sets up the chat model with specified generation parameters. The workflow function must be provided in the task's workflow_args.

```python
from typing import Awaitable, Callable, Dict, List, Optional

import openai

from trinity.common.experience import Experience
from trinity.common.models.model import ModelWrapper
from trinity.common.workflows.workflow import WORKFLOWS, Task, Workflow



@WORKFLOWS.register_module("agentscope_workflow_adapter")
class AgentScopeWorkflowAdapter(Workflow):
    """Adapter to wrap a agentscope trainable workflow function into a Trinity Workflow."""

    is_async: bool = True



    def __init__(
        self,
        *,
        task: Task,
        model: ModelWrapper,
        auxiliary_models: Optional[List[openai.OpenAI]] = None,
    ):
        """Initialize the adapter with the task and model."""
        try:
            from agentscope.model import TrinityChatModel
        except ImportError:
            raise ImportError(
                "This workflow requires agentscope >= 0.1.6, please install "
                "it via `pip install agentscope>=0.1.6`",
            )

        super().__init__(
            task=task,
            model=model,
            auxiliary_models=auxiliary_models,
        )
        self.workflow_func: Callable[
            [Dict, TrinityChatModel], Awaitable[float]
        ] = task.workflow_args.get("workflow_func", None)

        if self.workflow_func is None:
            raise ValueError(
                "The 'workflow_func' is not provided.",
            )

        self.chat_model: TrinityChatModel = TrinityChatModel(
            model.get_openai_async_client(),
            generate_kwargs={
                "temperature": self.task.rollout_args.temperature,
                "max_tokens": self.task.rollout_args.max_tokens or 4096,
                "top_logprobs": self.task.rollout_args.logprobs,
            },
        )

```

--------------------------------

### Setup Weight Synchronization Group (Python)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/trainer/verl/fsdp_workers

Configures the distributed training environment for weight synchronization using NCCL and FSDP strategies. It initializes process groups, gathers model parameter metadata, and sets up communication channels via Ray. Dependencies include PyTorch and Ray.

```python
import torch
import torch.distributed
import ray
from accelerate.utils import get_device_id
from deepspeed.accelerator.meta import get_device_id
from deepspeed.ops.comm import init_process_group
from fms.distributed.fsdp import FSDP, FlatParameter
from fms.utils.dist_utils import Dispatch
from fms.utils.types import AlgorithmConfig, DataProto

# Assuming SyncMethod, Synchronizer, ROLLOUT_WEIGHT_SYNC_GROUP_NAME, FSDP_PREFIX are defined elsewhere

@register(dispatch_mode=Dispatch.ONE_TO_ALL)
def setup_weight_sync_group(self):
    if self.config.synchronizer.sync_method == SyncMethod.NCCL:
        model = self.actor_module_fsdp
        self.named_modules = []
        self.state_dict_meta = []
        with self._fsdp_offload_context():
            if self.config.actor.strategy == "fsdp":
                for name, module in model.named_modules():
                    if isinstance(module, FSDP):
                        self.named_modules.append((name, module))
                for name_prefix, module in self.named_modules:
                    with FSDP.summon_full_params(module, recurse=False):
                        for name, param in module.named_parameters():
                            if isinstance(param, FlatParameter):
                                continue
                            realname = (
                                name_prefix[len(FSDP_PREFIX) :] + "." + name
                            ) if name_prefix else name
                            self.state_dict_meta.append(
                                (realname, str(param.dtype), tuple(param.shape))
                            )
                        param = None
                    torch.cuda.empty_cache()
            else:  # fsdp2
                for name, param in model.named_parameters():
                    self.state_dict_meta.append((name, str(param.dtype), tuple(param.shape)))

        if torch.distributed.get_rank() == 0:
            import ray

            master_address, master_port = self.get_availale_master_addr_port()
            world_size = self.config.synchronizer.explorer_world_size + 1
            print(f"Trainer init_process_group {master_address}:{master_port} ({world_size}).")
            synchronizer = Synchronizer.get_actor(
                namespace=self.config.synchronizer.ray_namespace
            )
            setup_ref = synchronizer.setup_weight_sync_group.remote(
                master_address, master_port, self.state_dict_meta
            )
            timeout = self.config.synchronizer.sync_timeout

            self._model_update_group = init_process_group(
                host=master_address,
                port=master_port,
                group_name=ROLLOUT_WEIGHT_SYNC_GROUP_NAME,
                backend="nccl",
                timeout=timeout,
                world_size=world_size,
                device_id=torch.device(f"cuda:{get_device_id()}"),
                rank=0,
            )
            torch.distributed.barrier(group=self._model_update_group)
            ray.get(setup_ref)
```

--------------------------------

### Initialize MLflow Monitor

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/utils/monitor

Initializes the MlflowMonitor, setting up MLflow tracking. It checks for MLflow installation, configures environment variables for authentication if provided, sets the tracking URI and experiment name, and starts a new MLflow run with specified parameters and tags.

```python
def __init__(
        self,
        project: str,
        group: str,
        name: str,
        role: str,
        config: Config = None,
    ) -> None:
        assert (
            mlflow is not None
        ), "mlflow is not installed. Please install it to use MlflowMonitor."
        monitor_args = config.monitor.monitor_args or {}
        if username := monitor_args.get("username"):
            os.environ["MLFLOW_TRACKING_USERNAME"] = username
        if password := monitor_args.get("password"):
            os.environ["MLFLOW_TRACKING_PASSWORD"] = password
        mlflow.set_tracking_uri(config.monitor.monitor_args.get("uri", "http://localhost:5000"))
        mlflow.set_experiment(project)
        mlflow.start_run(
            run_name=f"{name}_{role}",
            tags={
                "group": group,
                "role": role,
            },
        )
        mlflow.log_params(config.flatten())
        self.console_logger = get_logger(__name__, in_ray_actor=True)
```

--------------------------------

### Run Explorer in Serving Mode - Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/explorer/explorer

Starts the explorer in serving mode, which involves launching an OpenAI-compatible server. This allows separate agent applications to interact with the explorer via an API. An example demonstrates how to use the OpenAI client to send a chat completion request.

```python
@Experimental
    async def serve(self) -> None:
        """Run the explorer in serving mode.

        In serving mode, the explorer starts an OpenAI compatible server to handle requests.
        Agent applications can be deployed separately and interact with the explorer via the API.


        .. code-block:: python

            import openai


            client = openai.OpenAI(
                base_url=f"{explorer_server_url}/v1",
                api_key="EMPTY",
            )
            response = client.chat.completions.create(
                model=config.model.model_path,
                messages=[{"role": "user", "content": "Hello!"}]
            )
        """
        from trinity.explorer.api.service import ExplorerService

        self.service = ExplorerService(
            self,
            listen_address=self.config.explorer.listen_address,
            port=self.config.explorer.api_port,
        )
        await self.service.serve()
```

--------------------------------

### Run Off-Policy RFT with OPMD

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_reasoning_advanced

This command executes the Off-Policy Reinforcement Learning from Human Feedback (RFT) using the OPMD algorithm. It requires a configuration file specifying the experiment parameters, such as the `sync_interval` which controls model synchronization between the explorer and trainer. This setup creates a challenging off-policy scenario.

```bash
trinity run --config examples/opmd_gsm8k/opmd_gsm8k.yaml

```

--------------------------------

### Download Qwen2.5 Model using ModelScope and Huggingface

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_dpo

This snippet shows how to download the Qwen2.5-1.5B-Instruct model locally using either the ModelScope CLI or Huggingface CLI. Ensure you have the respective CLIs installed and configured.

```shell
# Using Modelscope
modelscope download Qwen/Qwen2.5-1.5B-Instruct --local_dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

# Using Huggingface
huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Start Inference Model for Debug Mode

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/develop_workflow

This command initiates the inference model in Trinity-RFT's debug mode. It loads the `explorer.rollout_model` and `explorer.auxiliary_models` from the specified configuration file. The inference model will then run continuously, awaiting debugging instructions from another terminal. This setup avoids repeated model loading and initialization, significantly speeding up the development and testing of Workflows.

```bash
trinity debug --config <config_file_path> --module inference_model

```

--------------------------------

### Setup Ray Cluster in DLC Environment (Python)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/utils/dlc_utils

Initializes or reuses a Ray cluster within a DLC environment. It checks for existing Ray instances and starts a new one if necessary, configuring it as either a head or worker node based on environment variables. It also waits for cluster readiness and worker nodes to join before returning the cluster address.

```Python
import os
import subprocess
import sys
import time

import ray

from trinity.utils.log import get_logger

logger = get_logger(__name__)

CLUSTER_ACTOR_NAME = "cluster_status"


class ClusterStatus:
    def __init__(self):
        self.finished = False

    def finish(self) -> None:
        self.finished = True

    def running(self) -> bool:
        return not self.finished

def get_dlc_env_vars() -> dict:
    envs = {
        "RANK": int(os.environ.get("RANK", -1)),  # type: ignore
        "WORLD_SIZE": int(os.environ.get("WORLD_SIZE", -1)),  # type: ignore
        "MASTER_ADDR": os.environ.get("MASTER_ADDR", None),
        "MASTER_PORT": os.environ.get("MASTER_PORT", None),
    }
    for key, value in envs.items():
        if value is None or value == -1:
            logger.error(f"DLC env var `{key}` is not set.")
            raise ValueError(f"DLC env var `{key}` is not set.")
    return envs

def is_running() -> bool:
    """Check if ray cluster is running."""
    ret = subprocess.run("ray status", shell=True, capture_output=True)
    return ret.returncode == 0

def wait_for_ray_setup() -> None:
    while True:
        if is_running():
            break
        else:
            logger.info("Waiting for ray cluster to be ready...")
            time.sleep(1)

def wait_for_ray_worker_nodes(world_size: int) -> None:
    while True:
        alive_nodes = [node for node in ray.nodes() if node["Alive"]]
        if len(alive_nodes) >= world_size:
            break
        else:
            logger.info(
                f"{len(alive_nodes)} nodes have joined so far, waiting for {world_size - len(alive_nodes)} nodes..."
            )
            time.sleep(1)

def setup_ray_cluster(namespace: str) -> str:
    """Setup a ray cluster in DLC environment.

    This function will start a ray cluster if it is not running, otherwise it will reuse the existing ray cluster.

    Returns:
        str: The address of the ray cluster.
    """
    env_vars = get_dlc_env_vars()
    is_master = env_vars["RANK"] == 0

    if is_running():
        # reuse existing ray cluster
        return "auto"
    else:
        if is_master:
            cmd = f"ray start --head --port={env_vars['MASTER_PORT']} --node-ip-address={env_vars['MASTER_ADDR']}"
        else:
            cmd = f"ray start --address={env_vars['MASTER_ADDR']}:{env_vars['MASTER_PORT']}"
        ret = subprocess.run(cmd, shell=True, capture_output=True)
        logger.info(f"Starting ray cluster: {cmd}")
        if ret.returncode != 0:
            logger.error(f"Failed to start ray cluster: {cmd}")
            logger.error(f"ret.stdout: {ret.stdout!r}")
            logger.error(f"ret.stderr: {ret.stderr!r}")
            sys.exit(1)

        wait_for_ray_setup()
        time.sleep(5)
        ray.init(
            address=f"{env_vars['MASTER_ADDR']}:{env_vars['MASTER_PORT']}",
            namespace=namespace,
            ignore_reinit_error=True,
        )
        if is_master:
            # master wait for worker nodes to join
            wait_for_ray_worker_nodes(env_vars["WORLD_SIZE"])
            ray.shutdown()
            return f"{env_vars['MASTER_ADDR']}:{env_vars['MASTER_PORT']}"
        else:
            # worker wait on the cluster status actor
            cluster_status = (
                ray.remote(ClusterStatus)
                .options(
                    name=CLUSTER_ACTOR_NAME,
                    namespace=namespace,
                    get_if_exists=True,
                )
                .remote()
            )
            while True:
                if ray.get(cluster_status.running.remote()):
                    ret = subprocess.run("ray status", shell=True, capture_output=True)
                    print(ret.stdout.decode())
                    time.sleep(5)
                else:
                    logger.info("Ray cluster is not running, exiting.")
                    break
            sys.exit(0)
```

--------------------------------

### Download Model with Modelscope or Huggingface CLI

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_dpo

This snippet demonstrates how to download the Qwen2.5-1.5B-Instruct model locally using either the Modelscope CLI or Huggingface CLI. Ensure you have the respective CLIs installed and configured. The downloaded model will be stored in the specified local directory.

```shell
# 使用 Modelscope
modelscope download Qwen/Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct

# 使用 Huggingface
huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir $MODEL_PATH/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Download Models and Datasets (Bash)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_react

These commands download necessary models and datasets from Hugging Face. 'Qwen/Qwen3-8B' is downloaded as a model, and 'openai/gsm8k' is downloaded as a dataset.

```bash
huggingface-cli download Qwen/Qwen3-8B
huggingface-cli download openai/gsm8k --repo-type dataset
```

--------------------------------

### Prepare Training Environment and Load Checkpoint

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/trainer/verl_trainer

Sets up the weight synchronization group and algorithm configuration for the actor rollout. It initializes the global step counter and loads a model checkpoint before any training operations begin.

```python
def prepare(self):
        self.actor_rollout_wg.setup_weight_sync_group()
        self.actor_rollout_wg.set_algorithm(self.algorithm_config)

        # The global step counter, initialized to 0
        # It represents the total number of training steps completed so far
        # We increment this counter at the beginning of each training step
        self.global_steps = 0

        # load checkpoint before doing anything
        self._load_checkpoint()
```

--------------------------------

### Initialize Reference Model with FSDP

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/trainer/verl/fsdp_workers

Initializes the reference model using FSDP, loading from a specified path and applying FSDP configurations. This setup is for the reference policy and does not involve an optimizer or LR scheduler at this stage.

```python
local_path = copy_to_local(self.config.model.path, use_shm=use_shm)
self.ref_module_fsdp = self._build_model_optimizer(
    model_path=local_path,
    fsdp_config=self.config.ref.fsdp_config,
    optim_config=None,
    override_model_config=override_model_config,
    use_remove_padding=use_remove_padding,
    use_fused_kernels=use_fused_kernels,
    trust_remote_code=self.config.model.get("trust_remote_code", False),
    use_liger=self.config.model.get("use_liger", False),
    role="ref",
)[0]
OmegaConf.set_struct(self.config.ref, True)
with open_dict(self.config.ref):
    self.config.ref.use_remove_padding = use_remove_padding
    self.config.ref.use_fused_kernels = use_fused_kernels

```

--------------------------------

### Configure SFT Process with sft.yaml

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_dpo

This configuration file defines the parameters for the Supervised Fine-Tuning (SFT) process. It specifies the project name, experiment name, algorithm type, model path, cluster settings, buffer size, and trainer input, including dataset path and format. The `prompt_type` is set to 'messages' and `messages_key` to 'messages' for datasets in message format.

```yaml
project: <project_name>
name: <experiment_name>
mode: train
algorithm:
  algorithm_type: sft
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,Qwen/Qwen2.5-1.5B-Instruct}
cluster:
  node_num: 1
  gpu_per_node: 2
buffer:
  total_epochs: 5
  train_batch_size: 64
  trainer_input:
    experience_buffer:
      name: <sft_dataset_name>
      storage_type: file
      path: $DATASET_PATH/Mixture-of-Thoughts
      split: train
      format:
        prompt_type: messages
        messages_key: messages
trainer:
  save_interval: 50
  trainer_config:
    ... # 省略其他配置

```

--------------------------------

### Notify Started in Python

Source: https://modelscope.github.io/Trinity-RFT/zh/main/build_api/trinity.trainer

Asynchronously notifies that a process has started, including node and job IDs. Used for distributed task coordination.

```python
_async _notify_started(_node_id : str_, _job_id : str_)
```

--------------------------------

### Install Data-Juicer Dependencies

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_sources/tutorial/example_data_functionalities

Installs the necessary dependencies for the 'data' branch of Trinity-RFT, which includes Data-Juicer functionalities. This is crucial for enabling the data processor service.

```shell
pip install -e ".[data]"
```

--------------------------------

### Run SFT Experiment (Command Line)

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_dpo

This command executes a Supervised Fine-Tuning (SFT) experiment using the Trinity framework. It requires a path to the SFT configuration file, which specifies all the necessary parameters for the training process.

```bash
trinity run --config examples/sft_mot/sft.yaml
```

--------------------------------

### Start the Runner Management Service

Source: https://modelscope.github.io/Trinity-RFT/zh/main/_modules/trinity/explorer/scheduler

Asynchronously starts the runner management service by setting the running flag to True. If the service is already running, this operation does nothing.

```python
    async def start(self) -> None:
        if self.running:
            return
        self.running = True
```

--------------------------------

### Run Trinity-RFT Training Task

Source: https://modelscope.github.io/Trinity-RFT/zh/main/tutorial/example_react

This command initiates a Trinity-RFT training task using a specified configuration file. First, navigate to the Trinity-RFT root directory, then execute the `trinity run` command with the path to the configuration YAML file.

```bash
# Navigate to the Trinity-RFT root directory
cd /path/to/Trinity-RFT

# Run the training for GSM8k dataset:
trinity run --config examples/agentscope_react/gsm8k.yaml

```