AReaL

https://inclusionai.github.io/AReaL
Admin
AReaL is a framework for Agentic Reinforcement Learning, offering tools for training, evaluation,...
Tokens:94,421
Snippets:712
Trust Score:8.3
Update:9 months ago
Show doc for...
Tokens:
Raw
### Start Ray Head Node for Distributed Training

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation_npu

This command initiates the Ray head node process. It is typically run on the first node in a distributed training setup and is essential for coordinating worker nodes.

```bash
ray start --head

```

--------------------------------

### Launch Ray Worker Node

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Starts a Ray worker node and connects it to the Ray head node. Replace the placeholder with the actual IP address of the head node.

```bash
# Replace with the actual IP address of the first node
RAY_HEAD_IP=xxx.xxx.xxx.xxx
ray start --address $RAY_HEAD_IP
```

--------------------------------

### Run AReaL-lite Experiment (Distributed with Ray)

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/quickstart

Launch a distributed AReaL-lite experiment using the Ray launcher. This command is for multi-node setups and requires a pre-configured Ray cluster. GPUs are allocated at the node granularity.

```bash
# Launch with Ray launcher. 4 nodes (4 GPUs each), 3 nodes for generation, 1 node for training.
python3 -m areal.launcher.ray examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    experiment_name=<your experiment name> \
    trial_name=<your trial name> \
    allocation_mode=sglang:d12p1t1+d4p1t1 \
    cluster.n_nodes=4 \
    cluster.n_gpus_per_node=4 \

```

--------------------------------

### Run AReaL-lite Experiment (Distributed with Slurm)

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/quickstart

Launch a distributed AReaL-lite experiment using the Slurm launcher. This command is for multi-node setups and requires a pre-configured Slurm cluster. GPUs are allocated at the node granularity.

```bash
# Launch with Slurm launcher. 16 nodes (8 GPUs each), 12 nodes for generation, 4 nodes for training
python3 -m areal.launcher.slurm examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    experiment_name=<your experiment name> \
    trial_name=<your trial name> \
    allocation_mode=sglang:d96p1t1+d32p1t1 \
    cluster.n_nodes=16 \
    cluster.n_gpus_per_node=8 \

```

--------------------------------

### Launch Distributed Experiments with SkyPilot

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/quickstart

Launches distributed experiments on different infrastructures using SkyPilot. Requires SkyPilot installation and a SkyPilot YAML configuration file. The command specifies the cluster name, the configuration file, and the target infrastructure (GCP, AWS, or K8s).

```bash
# Launch on GCP
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra gcp
# Launch on AWS
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra aws
# Launch on your K8s Cluster
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra k8s
```

--------------------------------

### Launch AReaL Experiment with SkyPilot (GCP, AWS, K8s)

Source: https://inclusionai.github.io/AReaL/tutorial/quickstart

Launches an AReaL-lite distributed experiment using SkyPilot on different infrastructures. This command utilizes a SkyPilot YAML configuration file to define the cluster setup and deployment targets, supporting Google Cloud Platform (GCP), Amazon Web Services (AWS), and Kubernetes (K8s). Ensure SkyPilot is installed and configured.

```bash
# Launch on GCP
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra gcp
# Launch on AWS
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra aws
# Launch on your K8s Cluster
sky launch -c areal-test examples/skypilot/ray_cluster.sky.yaml --infra k8s

```

--------------------------------

### GCP Setup for SkyPilot

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Configures the Google Cloud SDK for SkyPilot integration. This involves installing the SDK, initializing gcloud, setting the project ID, and creating Application Default Credentials.

```bash
# Install Google Cloud SDK
conda install -y -c conda-forge google-cloud-sdk

# Initialize gcloud and select your account/project
gcloud init

# (Optional) choose a project explicitly
gcloud config set project <PROJECT_ID>

# Create Application Default Credentials
gcloud auth application-default login
```

--------------------------------

### Launch Ray Head Node

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Starts the Ray head node, which is the first step in setting up a Ray cluster for distributed training with AReaL. This command should be run on the primary node.

```bash
ray start --head
```

--------------------------------

### Docker Installation for AReaL Runtime

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Installs the AReaL runtime environment using Docker. This method is recommended for its ease of use and includes pre-built dependencies. It pulls the official AReaL image, runs it as a container, and sets up the AReaL package within the container.

```bash
docker pull ghcr.io/inclusionai/areal-runtime:v0.5.1
docker run -it --name areal-node1 \
   --privileged --gpus all --network host \
   --shm-size 700g -v /path/to/mount:/path/to/mount \
   ghcr.io/inclusionai/areal-runtime:v0.5.1 \
   /bin/bash
git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install -e . --no-deps
```

--------------------------------

### Custom AReaL Installation with uv and pip

Source: https://inclusionai.github.io/AReaL/tutorial/installation

This snippet details the process of installing AReaL and its dependencies within a custom environment using uv and pip. It includes cloning the repository and installing all optional dependencies.

```bash
git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install uv
uv pip install -e "".[all]""
```

--------------------------------

### Customize AReaL-lite Configuration (Single Node)

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/quickstart

Launch a customized AReaL-lite experiment on a single node by modifying the YAML configuration file and adding command-line options. This allows fine-tuning of models, resources, and algorithm parameters.

```bash
python3 -m areal.launcher.local examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    experiment_name=<your experiment name> \
    trial_name=<your trial name> \
    allocation_mode=sglang:d2p1t1+d2p1t1 \
    cluster.n_nodes=1 \
    cluster.n_gpus_per_node=4 \
    gconfig.max_new_tokens=2048 \
    train_dataset.batch_size=1024 \
    +sglang.attention_backend=triton
```

--------------------------------

### Example Command for GRPO Training with MoE Model

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/megatron

This bash command provides an example of initiating GRPO training on a Qwen3 30B-A3B MoE model using the GSM8K dataset on a 32-GPU Ray cluster. It assumes the necessary configurations, including the Megatron backend, are already in place.

```bash
# Example command to run GRPO on Qwen3 30B-A3B MoE model
# Assumes a 32-GPU ray cluster and necessary configurations are set up.
```

--------------------------------

### Conda Environment Setup for AReaL

Source: https://inclusionai.github.io/AReaL/tutorial/installation

This snippet shows how to create and activate a Conda virtual environment for AReaL with a specific Python version. This is the first step in setting up a custom installation environment.

```bash
conda create -n areal python=3.12
conda activate areal
```

--------------------------------

### Run AReaL-lite Experiment (Single Node)

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/quickstart

Execute an AReaL-lite experiment on a single node using the LocalLauncher. This command trains an LLM on the GSM8K dataset with GRPO and function-based rewards, automatically downloading the dataset and model.

```bash
python3 -m areal.launcher.local examples/math/gsm8k_rl.py --config examples/math/gsm8k_grpo.yaml experiment_name=<your experiment name> trial_name=<your trial name>
```

--------------------------------

### Install SkyPilot with GCP and Kubernetes Support

Source: https://inclusionai.github.io/AReaL/tutorial/installation

This snippet installs the SkyPilot library using pip, including extra dependencies for Google Cloud Platform (GCP) and Kubernetes integration. It specifies a version compatibility note for Python.

```bash
# In your conda environment
# NOTE: SkyPilot requires 3.7 <= python <= 3.13
pip install -U "skypilot[gcp,kubernetes]"
```

--------------------------------

### Install AReaL with Ascend NPU Dependencies

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation_npu

This command sequence clones the AReaL repository, checks out the 'ascend' branch, and then installs the package with all necessary dependencies for NPU acceleration. It uses 'uv pip' for installation.

```bash
git clone https://github.com/inclusionAI/AReaL
cd AReaL

# Checkout to ascend branch
git checkout ascend

# Install AReaL
uv pip install -e .[all_npu] --system

```

--------------------------------

### Installing SkyPilot using Pip

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Installs SkyPilot, a tool that simplifies running AReaL on various cloud platforms and Kubernetes, using pip. This is an optional step for users who want to leverage SkyPilot's capabilities for easier deployment and management.

```bash
# In your conda environment

```

--------------------------------

### Install CAMEL-AI and OpenAI Agents SDK

Source: https://inclusionai.github.io/AReaL/tutorial/agentic_rl

Installs necessary agent frameworks for integration with AReaL. Ensure Python and pip are set up before running these commands.

```bash
# For CAMEL-AI
pip install camel-ai

# For OpenAI Agents SDK
pip install openai-agents

```

--------------------------------

### Example GRPO Command for Qwen3 30B-A3B MoE

Source: https://inclusionai.github.io/AReaL/tutorial/megatron

An example command to run GRPO on a specific MoE model (Qwen3 30B-A3B) using a 32-GPU Ray cluster and the GSM8K dataset. This command likely incorporates the previously discussed Megatron and SGLang configurations.

```bash
# Example command (actual command not provided in text, illustrating typical usage)
# grpo --model Qwen3_30B-A3B --dataset GSM8K --cluster ray --gpus 32 --config allocation_mode=sglang:d4+megatron:t4 ...
```

--------------------------------

### Run AReaL CAMEL Training Example

Source: https://inclusionai.github.io/AReaL/tutorial/agentic_rl

Executes the complete CAMEL training example using AReaL on a single node. This command requires specifying the training script, configuration file, experiment name, and trial name.

```bash
python3 -m areal.launcher.local examples/camel/train.py \
    --config examples/camel/config.yaml \
    experiment_name=<your experiment name> \
    trial_name=<your trial name>
```

--------------------------------

### Install OpenAI Agents SDK

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/agentic_rl

Installs the OpenAI Agents SDK, a framework for building agentic applications with OpenAI models, enabling features like tool use and conversation management.

```bash
pip install openai-agents
```

--------------------------------

### Launch GSM8K RL Example with AREAL and Megatron

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/megatron

This command launches the GSM8K Reinforcement Learning example using the AREAL library's ray launcher. It configures the experiment name, trial name, allocation mode, cluster resources, actor model path, and enables deterministic algorithms for Megatron.

```bash
python3 -m areal.launcher.ray examples/math/gsm8k_rl.py --config <megatron_config.yaml> \
    experiment_name=megatron-moe-gsm8k-grpo trial_name=trial-0 allocation_mode=sglang:d4t4+megatron:(attn:d1p4t2c2|ffn:d1p4t1e4) \
    cluster.n_nodes=4 cluster.n_gpus_per_node=8 actor.path=Qwen/Qwen3-30B-A3B \
    actor.megatron.use_deterministic_algorithms=True
```

--------------------------------

### Custom Environment Setup for AReaL with Conda and Pip

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Sets up a custom AReaL environment using Conda for package management and Pip (with uv) for installing Python dependencies. This method allows for more control over the environment but requires manual installation of Miniconda/Anaconda.

```bash
conda create -n areal python=3.12
conda activate areal
git clone https://github.com/inclusionAI/AReaL
cd AReaL
pip install uv
uv pip install -e "'.[all]'"
```

--------------------------------

### Project Configuration Parameters

Source: https://inclusionai.github.io/AReaL/_sources/cli_reference

This section details the configuration parameters for the project, including discount, GAE lambda, advantage normalization, KL divergence control, and SAPO loss parameters.

```APIDOC
## Project Configuration Parameters

### Description
This section details the configuration parameters for the project, including discount, GAE lambda, advantage normalization, KL divergence control, and SAPO loss parameters.

### Parameters

#### Request Body Parameters (Implicitly for configuration settings)

- **discount** (float) - Optional - Discount factor for future rewards. Default: `1.0`
- **gae_lambda** (float) - Optional - Lambda parameter for GAE. Default: `1.0`
- **adv_norm** (NormConfig | None) - Optional - Normalization configuration for advantages. Default: `None`
- **kl_ctl** (float) - Optional - KL divergence coefficient. Default: `0.1`
- **kl_estimator** (string) - Optional - KL divergence estimator. Choices: `k1`, `k2`, `k3`. Default: `"k1"`
- **use_sapo_loss** (boolean) - Optional - Use SAPO loss (mutually exclusive with PPO clipping). Default: `False`
- **sapo_tau_pos** (float) - Optional - SAPO temperature for positive advantages. Default: `1.0`
- **sapo_tau_neg** (float) - Optional - SAPO temperature for negative advantages. Default: `1.05`

### Request Example
```json
{
  "discount": 0.99,
  "gae_lambda": 0.95,
  "adv_norm": {
    "type": "rms",
    "epsilon": 1e-5
  },
  "kl_ctl": 0.2,
  "kl_estimator": "k2",
  "use_sapo_loss": true,
  "sapo_tau_pos": 1.1,
  "sapo_tau_neg": 1.15
}
```

### Response

#### Success Response (200)

This endpoint typically returns a status indicating success or details about the applied configuration. Since this describes parameters rather than an endpoint, a direct success response example is not applicable. However, a successful configuration load might return:

- **status** (string) - The status of the operation (e.g., "Configuration loaded successfully").

#### Response Example
```json
{
  "status": "Configuration loaded successfully"
}
```
```

--------------------------------

### Start Ray Cluster Worker Node

Source: https://inclusionai.github.io/AReaL/tutorial/installation_npu

This command starts a Ray worker node and connects it to an existing Ray cluster. Replace 'xxx.xxx.xxx.xxx' with the actual IP address of the Ray head node. This command should be executed on all nodes intended to be workers.

```bash
# Replace with the actual IP address of the first node
RAY_HEAD_IP=xxx.xxx.xxx.xxx
ray start --address $RAY_HEAD_IP

```

--------------------------------

### Initializing Training Engines (FSDPPPOActor) in Python

Source: https://inclusionai.github.io/AReaL/_sources/lite/gsm8k_grpo

Shows the initialization of training engines, specifically `FSDPPPOActor` for the policy model and optionally a reference model for KL divergence penalties. This process involves creating process groups and initializing the engines with provided configurations and model specifications. Model weights are loaded from paths specified in the configuration.

```python
# Initialize actor (policy) engine
actor = FSDPPPOActor(config=config.actor)
actor.create_process_group(parallel_strategy=parallel_strategy)
actor.initialize(None, ft_spec)
actor.connect_engine(rollout, weight_update_meta)

# Initialize reference model (frozen) for KL divergence penalty
ref = None
if config.actor.kl_ctl > 0 and config.ref is not None:
    ref = FSDPPPOActor(config=config.ref)
    ref.create_process_group(parallel_strategy=parallel_strategy)
    ref.initialize(None, ft_spec)
```

--------------------------------

### Initializing PyTorch FSDP2 Training Engines for PPO

Source: https://inclusionai.github.io/AReaL/lite/gsm8k_grpo

Initializes training engines for policy and reference models using PyTorch FSDP2. This setup involves creating process groups and connecting the engines to the rollout and weight update mechanisms. It supports KL divergence penalties by optionally initializing a reference model.

```python
# Initialize actor (policy) engine
actor = FSDPPPOActor(config=config.actor)
actor.create_process_group(parallel_strategy=parallel_strategy)
actor.initialize(None, ft_spec)
actor.connect_engine(rollout, weight_update_meta)

# Initialize reference model (frozen) for KL divergence penalty
ref = None
if config.actor.kl_ctl > 0 and config.ref is not None:
    ref = FSDPPPOActor(config=config.ref)
    ref.create_process_group(parallel_strategy=parallel_strategy)
    ref.initialize(None, ft_spec)

```

--------------------------------

### Install SkyPilot for AReaL

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Installs SkyPilot with necessary providers (GCP, Kubernetes) for AReaL. Requires Python versions 3.7 to 3.13.

```bash
pip install -U "skypilot[gcp,kubernetes]"
```

--------------------------------

### Verify SkyPilot Installation

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Checks if SkyPilot is correctly installed and configured for GCP or Kubernetes. Successful verification indicates readiness to use SkyPilot with AReaL.

```bash
sky check
```

--------------------------------

### Example Usage of M2PO in AReal

Source: https://inclusionai.github.io/AReaL/algorithms/m2po

Demonstrates how to launch M2PO training using different backends (local, Ray, SLURM) by specifying configuration files and other optional arguments. This is crucial for setting up and running experiments with M2PO.

```bash
python3 -m areal.launcher.local examples/math/gsm8k_rl.py --config examples/math/gsm8k_m2po.yaml --<other_args_to_overwrite>
```

```bash
python3 -m areal.launcher.ray examples/math/gsm8k_rl.py --config examples/math/gsm8k_m2po.yaml --<other_args_to_overwrite>
```

```bash
python3 -m areal.launcher.slurm examples/math/gsm8k_rl.py --config examples/math/gsm8k_m2po.yaml --<other_args_to_overwrite>
```

--------------------------------

### Validating AReaL Installation

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation

Runs a validation script to ensure that the AReaL installation is successful and all dependencies are correctly set up. This script should be executed after the runtime environment has been configured.

```bash
python3 areal/tools/validate_installation.py
```

--------------------------------

### Run Training with PPOTrainer in Python

Source: https://inclusionai.github.io/AReaL/tutorial/agentic_rl

This Python code snippet demonstrates how to use AReaL's PPOTrainer for training. It sets up datasets, initializes the trainer, creates workflow objects for training and evaluation, and starts the training process. Dependencies include the areal library and its experimental trainer module, along with configurations for datasets and agent workflows.

```python
from areal.experimental.trainer import PPOTrainer

def main(args):
    config, _ = load_expr_config(args, AgentRLConfig)
    tokenizer = load_hf_tokenizer(config.tokenizer_path)

    train_dataset = get_custom_dataset(
        split="train",
        dataset_config=config.train_dataset,
        tokenizer=tokenizer,
    )

    valid_dataset = get_custom_dataset(
        split="test",
        dataset_config=config.valid_dataset,
        tokenizer=tokenizer,
    )

    with PPOTrainer(
        config,
        train_dataset=train_dataset,
        valid_dataset=valid_dataset,
    ) as trainer:
        # Create workflow with config parameters
        workflow = OpenAIAgentWorkflow(
            agent_builder_path=config.agent_builder_path,
            agent_builder_kwargs=config.agent_builder_kwargs,
            reward_fn_path=config.reward_fn_path,
            gconfig=config.gconfig,
            tokenizer=tokenizer,
        )

        eval_workflow = OpenAIAgentWorkflow(
            agent_builder_path=config.agent_builder_path,
            agent_builder_kwargs=config.agent_builder_kwargs,
            reward_fn_path=config.reward_fn_path,
            gconfig=config.gconfig,
            tokenizer=tokenizer,
        )

        # Start training
        trainer.train(workflow, eval_workflow)

```

--------------------------------

### Run Training with PPOTrainer in Python

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/agentic_rl

This Python code demonstrates how to use AReaL's PPOTrainer for training. It loads configurations, tokenizers, and datasets, then initializes the trainer with custom workflows for training and evaluation. The trainer then starts the training process.

```python
from areal.experimental.trainer import PPOTrainer

def main(args):
    config, _ = load_expr_config(args, AgentRLConfig)
    tokenizer = load_hf_tokenizer(config.tokenizer_path)

    train_dataset = get_custom_dataset(
        split="train",
        dataset_config=config.train_dataset,
        tokenizer=tokenizer,
    )

    valid_dataset = get_custom_dataset(
        split="test",
        dataset_config=config.valid_dataset,
        tokenizer=tokenizer,
    )

    with PPOTrainer(
        config,
        train_dataset=train_dataset,
        valid_dataset=valid_dataset,
    ) as trainer:
        # Create workflow with config parameters
        workflow = OpenAIAgentWorkflow(
            agent_builder_path=config.agent_builder_path,
            agent_builder_kwargs=config.agent_builder_kwargs,
            reward_fn_path=config.reward_fn_path,
            gconfig=config.gconfig,
            tokenizer=tokenizer,
        )

        eval_workflow = OpenAIAgentWorkflow(
            agent_builder_path=config.agent_builder_path,
            agent_builder_kwargs=config.agent_builder_kwargs,
            reward_fn_path=config.reward_fn_path,
            gconfig=config.gconfig,
            tokenizer=tokenizer,
        )

        # Start training
        trainer.train(workflow, eval_workflow)
```

--------------------------------

### Project Configuration Parameters

Source: https://inclusionai.github.io/AReaL/_sources/cli_reference

This section details the configuration parameters available for a project, covering training, data types, optimizers, and distributed training backends.

```APIDOC
## Project Configuration Parameters

### Description
Configuration options for training, data types, optimizers, and distributed training engines.

### Method
N/A (Configuration Schema)

### Endpoint
N/A (Configuration Schema)

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **gradient_checkpointing** (boolean) - Optional - Enable gradient checkpointing. Defaults to `False`.
- **dtype** (string) - Optional - Parameter data type. Defaults to `"bfloat16"`.
- **grad_reduce_dtype** (string) - Optional - Gradient reduction data type. Defaults to `"float32"`.
- **optimizer** (OptimizerConfig | None) - Optional - Optimizer configuration. `None` means no training. Refers to [`OptimizerConfig`](section-optimizer).
- **weight_update_mode** (string) - Optional - Weight update backend type. Choices: `disk`, `xccl`. Defaults to `"xccl"`.
- **fsdp** (FSDPEngineConfig) - Required - FSDP engine configuration. Refers to [`FSDPEngineConfig`](section-fsdp-engine).
- **archon** (ArchonEngineConfig) - Required - Archon engine configuration. Refers to [`ArchonEngineConfig`](section-archon-engine).
- **megatron** (MegatronEngineConfig) - Required - Megatron engine configuration. Refers to [`MegatronEngineConfig`](section-megatron-engine).
- **use_lora** (boolean) - Optional - Whether to use LoRA. Only supports FSDP. Note that should be enabled together with vLLM/SGLang. Defaults to `False`.
- **lora_rank** (integer) - Optional - LoRA rank. Defaults to `32`.
- **lora_alpha** (integer) - Optional - LoRA alpha. Defaults to `16`.
- **target_modules** (list of string) - Required - LoRA target modules.
- **peft_type** (string) - Optional - PEFT method type. Only LoRA is supported for now. Defaults to `"lora"`.

### Request Example
```json
{
  "gradient_checkpointing": false,
  "dtype": "bfloat16",
  "grad_reduce_dtype": "float32",
  "optimizer": null,
  "weight_update_mode": "xccl",
  "fsdp": {
    "some_fsdp_config": "value"
  },
  "archon": {
    "some_archon_config": "value"
  },
  "megatron": {
    "some_megatron_config": "value"
  },
  "use_lora": false,
  "lora_rank": 32,
  "lora_alpha": 16,
  "target_modules": [
    "query_key_value",
    "dense"
  ],
  "peft_type": "lora"
}
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the success of the operation.

#### Response Example
```json
{
  "status": "Configuration updated successfully"
}
```

### Error Handling
- **400 Bad Request**: Invalid configuration values or missing required fields.
- **500 Internal Server Error**: An unexpected error occurred on the server.
```

--------------------------------

### Launch Experiments with AReaL Launchers (Bash)

Source: https://inclusionai.github.io/AReaL/_sources/lite/gsm8k_grpo

Demonstrates how to launch training experiments using AReaL's built-in launchers for local, Ray, and Slurm cluster environments. These commands specify the training script, configuration file, and any additional command-line arguments.

```bash
# Local machine (using subprocesses)
python -m areal.launcher.local <training script> --config <config file> <cli args>

# Ray cluster
python -m areal.launcher.ray <training script> --config <config file> <cli args>

# Slurm cluster
python -m areal.launcher.slurm <training script> --config <config file> <cli args>
```

--------------------------------

### Launching RL Training with Ray on AReaL

Source: https://inclusionai.github.io/AReaL/_sources/algorithms/grpo_series

This command initiates reinforcement learning training using the AReaL framework, distributed with Ray. It points to the specific training script and a configuration file that defines the algorithm and its parameters. Replace `<algo>` with the target algorithm like 'grpo' or 'gspo'.

```bash
python3 -m areal.launcher.ray examples/math/gsm8k_rl.py --config examples/math/gsm8k_<algo>.yaml
```

--------------------------------

### Create Docker Container for AReaL on Ascend NPU

Source: https://inclusionai.github.io/AReaL/_sources/tutorial/installation_npu

This command pulls the specified AReaL Docker image and runs a container with necessary Ascend NPU devices, volumes, and network settings. It's designed for using A2 or A3 hardware types and mounts the host workspace to the container.

```bash
WORK_DIR=<your_workspace>
CONTAINER_WORK_DIR=<your_container_workspace>

# Use A2/A3 image depending on your hardware type
# IMAGE=swr.cn-north-9.myhuaweicloud.com/areal/areal_npu:v0.5.0-a2
IMAGE=swr.cn-north-9.myhuaweicloud.com/areal/areal_npu:v0.5.0-a3
CONTAINER_NAME=areal_npu

cd ${WORK_DIR}

docker pull ${IMAGE}

docker run -itd --cap-add=SYS_PTRACE --net=host \
--device=/dev/davinci0 \
--device=/dev/davinci1 \
--device=/dev/davinci2 \
--device=/dev/davinci3 \
--device=/dev/davinci4 \
--device=/dev/davinci5 \
--device=/dev/davinci6 \
--device=/dev/davinci7 \
--device=/dev/davinci8 \
--device=/dev/davinci9 \
--device=/dev/davinci10 \
--device=/dev/davinci11 \
--device=/dev/davinci12 \
--device=/dev/davinci13 \
--device=/dev/davinci14 \
--device=/dev/davinci15 \
--device=/dev/davinci_manager \
--device=/dev/devmm_svm \
--device=/dev/hisi_hdc \
--shm-size=1200g \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /var/log/npu/:/usr/slog \
-v ${WORK_DIR}:${CONTAINER_WORK_DIR} \
--privileged=true \
--name ${CONTAINER_NAME} \
${IMAGE}  \
/bin/bash

```

--------------------------------

### Launch Standalone SGLang Server for AReaL Debugging

Source: https://inclusionai.github.io/AReaL/_sources/best_practices/debugging

Starts a standalone, inference-only SGLang server for debugging AReaL applications. This server allows for repeated testing without requiring restarts, making debugging sessions faster and more IDE-friendly. It requires Python and the areal library, and outputs the server address to a log file.

```bash
nohup python -m areal.launcher.local examples/math/gsm8k_rl.py \
    --config examples/math/gsm8k_grpo.yaml \
    allocation_mode=sglang:d4p1t1 > llm_server.log 2>&1 &

```

--------------------------------

### Running Decoupled PPO with Proximal Approximation

Source: https://inclusionai.github.io/AReaL/_sources/algorithms/prox_approx

This command demonstrates how to run a decoupled PPO training script with a configuration that utilizes proximal log-probability approximation for enhanced speed. It uses the `areal.launcher.local` module.

```bash
python -m areal.launcher.local examples/math/gsm8k_rl.py \
  --config examples/experimental/prox_approx/gsm8k_grpo_prox_approx.yaml
```
Tokens:
Raw