FlagScale

https://docs.flagos.io/projects/FlagScale/en/latest/
Admin
FlagScale is an open-source AI toolkit that provides a unified interface and CLI for model training,...
Tokens:5,890
Snippets:62
Trust Score:5.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install FlagScale from source

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs FlagScale from source by cloning the repository and running pip install. This is Option 2 for installing FlagScale.

```bash
git clone https://github.com/flagos-ai/FlagScale.git
cd FlagScale
pip install .

```

--------------------------------

### Install FlagGems from source

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs FlagGems from source: first installs build dependencies, clones the FlagGems repository, and then installs the package without build isolation. See more details in FlagGems.

```bash
pip install -U scikit-build-core==0.11 pybind11 ninja cmake
git clone https://github.com/flagos-ai/FlagGems
cd FlagGems
pip install --no-build-isolation .

```

--------------------------------

### Download demo dataset and Qwen tokenizer

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Download the demo dataset (bin and idx) and the Qwen tokenizer files into ./data and ./qwentokenizer directories. Required before training with the Qwen3 example.

```bash
mkdir -p ./data && cd ./data
wget https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/datasets/enron_emails_demo_text_document_qwen/enron_emails_demo_text_document_qwen.idx
wget https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/datasets/enron_emails_demo_text_document_qwen/enron_emails_demo_text_document_qwen.bin

```

```bash
mkdir -p ./qwentokenizer && cd ./qwentokenizer
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/tokenizers/qwentokenizer/tokenizer_config.json" -O tokenizer_config.json
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/tokenizers/qwentokenizer/qwen.tiktoken" -O qwen.tiktoken
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/tokenizers/qwentokenizer/qwen_generation_utils.py" -O qwen_generation_utils.py
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/tokenizers/qwentokenizer/tokenization_qwen.py" -O tokenization_qwen.py

```

--------------------------------

### Start and stop serving with flagscale serve

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Start the vLLM serving server with the flagscale serve command, using either --config or -c to specify the experiment-level YAML. Stop the server with the --stop flag.

```bash
flagscale serve qwen3 --config ./examples/qwen3/conf/serve.yaml
# or
flagscale serve qwen3 -c ./examples/qwen3/conf/serve.yaml

```

```bash
flagscale serve qwen3 --stop

```

--------------------------------

### Experiment-level YAML configuration for serving (examples/qwen3/conf/serve.yaml)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Defines the global context for a flagscale run, including output directory, backend engine, and the task-level file to load. This example uses the vllm backend and the serve/8b task configuration.

```yaml
# Example: examples/qwen3/conf/serve.yaml
defaults:
- _self_
- serve: 8b

experiment:
  exp_name: qwen3_8b
  exp_dir: outputs/${experiment.exp_name}
  task:
    type: serve
    backend: vllm
  runner:
    hostfile: null
    deploy:
      use_fs_serve: false
  envs:
    CUDA_VISIBLE_DEVICES: 0
    CUDA_DEVICE_MAX_CONNECTIONS: 1

action: run

hydra:
  run:
    dir: ${experiment.exp_dir}/hydra

```

--------------------------------

### Install TransformerEngine-FL

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs TransformerEngine-FL version 0.1.0+te2.9.0 from the FlagOS PyPI repository. See more details in TransformerEngine-FL.

```bash
pip install transformer_engine==0.1.0+te2.9.0 --extra-index-url https://resource.flagos.net/repository/flagos-pypi-hosted/simple

```

--------------------------------

### Start reinforcement learning with flagscale rl

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Starts reinforcement learning using the FlagScale CLI. The --config (or -c) flag points to the experiment-level YAML file. Output can be checked in the experiment directory.

```shell
flagscale rl qwen3 --config ./examples/qwen3/conf/rl.yaml
# or
flagscale rl qwen3 -c ./examples/qwen3/conf/rl.yaml

```

--------------------------------

### Install vLLM-plugin-FL 0.1.0

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs the vLLM-plugin-FL package version 0.1.0+vllm0.13.0 from the FlagOS PyPI repository. See more details in vllm-plugin-FL.

```bash
pip install vllm-plugin-fl==0.1.0+vllm0.13.0 --extra-index-url https://resource.flagos.net/repository/flagos-pypi-hosted/simple

```

--------------------------------

### Install vLLM 0.13.0

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs vLLM version 0.13.0 inside the flagscale-inference container. Required for the inference backend.

```bash
pip install vllm==0.13.0

```

--------------------------------

### Install FlagScale via pip

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs FlagScale via pip from the FlagOS PyPI repository. This is Option 1 for installing FlagScale.

```bash
pip install flagscale --extra-index-url https://resource.flagos.net/repository/flagos-pypi-hosted/simple

```

--------------------------------

### Install verl-FL

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs verl-FL version 0.1.0+verl0.7.0 from the FlagOS PyPI repository. See more details in veRL-FL for full installation instructions.

```bash
pip install verl==0.1.0+verl0.7.0 --extra-index-url https://resource.flagos.net/repository/flagos-pypi-hosted/simple

```

--------------------------------

### Set up inference backend with flagscale-inference Docker image

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Pulls and runs the recommended flagscale-inference Docker image, then activates the flagscale-inference conda environment inside the container. Use this to set up the inference/serving backend before installing vLLM, vLLM-plugin-FL, and FlagGems.

```bash
docker pull harbor.baai.ac.cn/flagscale/flagscale-inference:dev-cu128-py3.12-20260302102033
docker run -itd --privileged --gpus all --net=host --ipc=host --device=/dev/infiniband --shm-size 512g --ulimit memlock=-1 --name <name>  harbor.baai.ac.cn/flagscale/flagscale-inference:dev-cu128-py3.12-20260302102033
docker exec -it <name> /bin/bash
conda activate flagscale-inference

```

--------------------------------

### Install Megatron-LM-FL (megatron_core)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Installs Megatron-LM-FL (megatron_core) version 0.1.0+megatron0.15.0rc7 from the FlagOS PyPI repository. See more details in Megatron-LM-FL.

```bash
pip install megatron_core==0.1.0+megatron0.15.0rc7 --extra-index-url https://resource.flagos.net/repository/flagos-pypi-hosted/simple

```

--------------------------------

### Set up RL backend with flagscale-train Docker image

Source: https://docs.flagos.io/projects/FlagScale/en/latest/getting_started/install.html

Pulls and runs the recommended flagscale-train Docker image, then activates the flagscale-train conda environment inside the container. Use this to set up the RL training backend before installing Megatron-LM-FL, TransformerEngine-FL, and verl-FL.

```bash
docker pull harbor.baai.ac.cn/flagscale/flagscale-train:dev-cu128-py3.12-20260319182856
docker run -itd --gpus all --shm-size=500g --name <name>  harbor.baai.ac.cn/flagscale/flagscale-train:dev-cu128-py3.12-20260319182856 /bin/bash
docker exec -it <name> /bin/bash
conda activate flagscale-train

```

--------------------------------

### Start inference with flagscale inference

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Start inference with the flagscale inference command, using either --config or -c to specify the experiment-level YAML.

```bash
flagscale inference qwen3 --config ./examples/qwen3/conf/inference_fl.yaml
# or
flagscale inference qwen3 -c ./examples/qwen3/conf/inference_fl.yaml

```

--------------------------------

### Start and stop distributed training with flagscale train

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Start the distributed training job with the flagscale train command, using either --config or -c to specify the experiment-level YAML. Stop the job with the --stop flag.

```bash
flagscale train qwen3 --config ./examples/qwen3/conf/train.yaml
# or
flagscale train qwen3 -c ./examples/qwen3/conf/train.yaml

```

```bash
flagscale train qwen3 --stop

```

--------------------------------

### Task-level YAML configuration for serving (examples/qwen3/conf/serve/8b.yaml)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Specifies the model, host, port, and engine arguments for the serving task. Parameters map directly to backend engine arguments with hyphens replaced by underscores. Uses environment variables QWEN3_PATH and QWEN3_PORT.

```yaml
# Example: examples/qwen3/conf/serve/8b.yaml
- serve_id: vllm_model
  engine_args:
    model: ${oc.env:QWEN3_PATH}
    host: 0.0.0.0
    uvicorn_log_level: warning
    port: ${oc.env:QWEN3_PORT}
    gpu_memory_utilization: 0.9
    trust_remote_code: true
    no_enable_prefix_caching: true
    compilation_config: '{"full_cuda_graph": true}'

```

--------------------------------

### Download Qwen3-4B inference model

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Download the Qwen3-4B inference model using modelscope into ./Qwen3-4B. Required before running inference.

```bash
modelscope download --model Qwen/Qwen3-4B --local_dir ./Qwen3-4B

```

--------------------------------

### Download Qwen3-0.6B model with modelscope

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Downloads the Qwen3-0.6B model using the ModelScope CLI. Requires the verl-FL environment. The model is saved to the local directory ./Qwen3-0.6B.

```shell
modelscope download --model Qwen/Qwen3-0.6B --local_dir ./Qwen3-0.6B

```

--------------------------------

### Download GSM8K datasets with wget

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Creates a gsm8k directory and downloads the training and test datasets for the GSM8K task. The datasets are downloaded from the BAAI FlagScale storage.

```shell
mkdir gsm8k && cd gsm8k
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/rl/datasets/gsm8k/train.parquet"
wget "https://baai-flagscale.ks3-cn-beijing.ksyuncs.com/rl/datasets/gsm8k/test.parquet"

```

--------------------------------

### Stop reinforcement learning with flagscale rl --stop

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Stops the reinforcement learning job using the FlagScale CLI. This is the recommended way to stop the training.

```shell
flagscale rl qwen3 --stop

```

--------------------------------

### Configure experiment-level YAML (rl.yaml)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Experiment-level YAML configuration. Modify exp_dir to set your experiment directory and runtime_env to point to the runtime_env.yaml file in the verl-FL environment. The exp_name is set to 0_6b.

```yaml
experiment:
   exp_name: 0_6b
   exp_dir: /workspace/qwen3-rl/ # modify: Set your experiment directory
   runner:
      runtime_env: /path/to/verl-FL/verl/trainer/runtime_env.yaml # modify: Set your runtime_env.yaml

```

--------------------------------

### Modify serve model name in experiment-level YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Experiment-level YAML snippet from ./examples/qwen3/conf/serve.yaml. Modify the serve value to match the corresponding config file name and adjust exp_name as needed.

```yaml
defaults:
- _self_
- serve: 0_6b         # modify: Serve value must match its corresponding config file name
experiment:
exp_name: qwen3-0.6b  # modify as needed for test clarity
exp_dir: outputs/${experiment.exp_name}
task:
   type: serve
   backend: vllm
runner:
   hostfile: null
   deploy:
   use_fs_serve: false
envs:
   CUDA_VISIBLE_DEVICES: 0
   CUDA_DEVICE_MAX_CONNECTIONS: 1

```

--------------------------------

### Configure data paths in task-level YAML (0_6b.yaml)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Task-level YAML configuration for the RL training data. Modify the train_files and val_files paths to point to your downloaded datasets. The other parameters (batch size, prompt/response lengths, filtering, truncation) are set to the provided defaults.

```yaml
data:
   train_files: /workspace/data/gsm8k/train.parquet # modify: Set your train dataset
   val_files: /workspace/data/gsm8k/test.parquet # modify: Set your test dataset
   train_batch_size: 1024
   max_prompt_length: 512
   max_response_length: 1024
   filter_overlong_prompts: true
   truncation: "error"

```

--------------------------------

### Force stop Ray cluster with ray stop

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Force stops the Ray cluster. Use this alternative if the normal stop command does not work.

```shell
ray stop

```

--------------------------------

### Configure model path in task-level YAML (0_6b.yaml)

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Task-level YAML configuration for the model checkpoint path. Modify the path to point to your downloaded model checkpoint. The other model settings (use_remove_padding, enable_gradient_checkpointing, trust_remote_code) are set to true.

```yaml
actor_rollout_ref:
   model:
      path: /workspace/data/ckpt/Qwen3-0.6B # modify: Set your model checkpoint directory
      use_remove_padding: true
      enable_gradient_checkpointing: true
      trust_remote_code: true

```

--------------------------------

### Modify model path in serve task-level YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Task-level YAML snippet from ./examples/qwen3/conf/serve/0_6b.yaml. Modify the model path and ensure the port is available in your environment.

```yaml
- serve_id: vllm_model
engine_args:
   model: ./Qwen3-0.6B          # modify: Set model directory
   host: 0.0.0.0
   max_model_len: 4096
   max_num_seqs: 4
   uvicorn_log_level: warning
   port: 30000                  # A port available in your env, for example: 30000

```

--------------------------------

### Modify data_path and tokenizer_path in train YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Task-level YAML snippet from ./examples/qwen3/conf/train/0_6b.yaml. Modify the data_path and tokenizer_path values to point to the downloaded dataset and tokenizer.

```yaml
data:
 data_path: ./data/enron_emails_demo_text_document_qwen    # modify data_path here
 split: 1
 no_mmap_bin_files: true
 tokenizer:
     legacy_tokenizer: true
     tokenizer_type: QwenTokenizerFS
     tokenizer_path: ./qwentokenizer   # modify tokenizer_path here
     vocab_size: 151936
     make_vocab_size_divisible_by: 64

```

--------------------------------

### Modify inference model name in experiment-level YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Experiment-level YAML snippet from ./examples/qwen3/conf/inference_fl.yaml. The inference value must match the corresponding config file name (e.g., 4b.yaml).

```yaml
defaults:
- _self_
- inference: 4b    # modify: Inference value must match its corresponding config file name

```

--------------------------------

### Modify model path in inference task-level YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Task-level YAML snippet from ./examples/qwen3/conf/inference/4b.yaml. Modify the model path to point to the downloaded model directory.

```yaml
llm:
 model: ./Qwen3-4B         # modify: Set model directory
 trust_remote_code: true
 tensor_parallel_size: 1
 pipeline_parallel_size: 1
 gpu_memory_utilization: 0.9
 seed: 1234

```

--------------------------------

### Modify train model name in experiment-level YAML

Source: https://docs.flagos.io/projects/FlagScale/en/latest/user_guide/user-guide.html

Experiment-level YAML snippet from ./examples/qwen3/conf/train.yaml. The train value must match the corresponding config file name (e.g., 0_6b.yaml).

```yaml
defaults:
- _self_
- train: 0_6b  # modify: train value must match its corresponding config file name

```
Tokens:
Raw