Awesome ML SYS Tutorial

https://github.com/zhaochenyang20/awesome-ml-sys-tutorial
Admin
Awesome ML SYS Tutorial provides a comprehensive collection of learning notes, guides, and tutorials...
Tokens:1,099,711
Snippets:4,894
Trust Score:9.8
License:Apache-2.0
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### FSDP One-Click Start

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/slime/fsdp/readme_en.md

Commands to download model weights and datasets, install the required dependencies, and execute the FSDP training script.

```bash
# If you need to use WANDB, you need to set the environment variable WANDB_API_KEY in advance
# Download model weights (Qwen3-4B)
hf download Qwen/Qwen3-4B --local-dir /root/Qwen3-4B

# Download training dataset (dapo-math-17k)
hf download --repo-type dataset zhuzilin/dapo-math-17k \
  --local-dir /root/dapo-math-17k

# Download evaluation dataset (aime-2024)
hf download --repo-type dataset zhuzilin/aime-2024 \
  --local-dir /root/aime-2024
  
# Clone code and install dependencies
git clone https://github.com/radixark/miles.git
cd miles
pip install -e .


# FSDP does not require weight conversion, natively supports huggingface format
# Enable reference model, train Qwen3-4B in colocate mode
bash /root/miles/scripts/run-qwen3-4B-fsdp.sh
```

--------------------------------

### Install and serve Higgs TTS with sgl-omni serve

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/sglang/sglang-omni/boson-ch.md

Sets up the SGLang-Omni environment and starts the Higgs TTS server. The first block pulls the dev Docker image, runs a container with GPU and shared memory flags, clones the repository, and installs the package with uv. The second block downloads the Higgs model and launches the server on port 8000.

```bash
docker pull lmsys/sglang-omni:dev
docker run -it --gpus all --shm-size 32g --ipc host --network host --privileged \
  lmsys/sglang-omni:dev /bin/zsh

git clone git@github.com:sgl-project/sglang-omni.git && cd sglang-omni
uv venv .venv -p 3.12 && source .venv/bin/activate
uv pip install -v -e .
```

```bash
hf download bosonai/higgs-audio-v3-tts-4b

sgl-omni serve \
  --model-path bosonai/higgs-audio-v3-tts-4b \
  --port 8000
```

--------------------------------

### Setup and run GRPO experiment with SGLang

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/GRPO/README.md

Commands to clone the TRL repo, install with SGLang extras, set WANDB API key, start the SGLang server, and run the training script. Note that CUDA config may need modification in run.sh and grpo_config.yaml or via CUDA_VISIBLE_DEVICES.

```bash
git clone git@github.com:huggingface/trl.git
cd trl
python3 -m uv pip install -e ".[sglang]"

export WANDB_API_KEY=<YOUR_WANDB_API_KEY>

# start sglang-server
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct

# modify cuda config in run.sh and grpo_config.yaml
# or run "export CUDA_VISIBLE_DEVICES"
# run script
sh ./trl/scripts/grpo_test/run.sh
```

--------------------------------

### 固定窗口抓取 trace 的 start_profile 请求

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/sglang/interactive-profiling/Interactive_Profiling_cn.md

在 server 运行期间，通过设置 num_steps 为 10 来抓取固定 10 个 step 的 trace。此请求只包含 num_steps 参数，其他参数使用默认值。

```bash
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{ 
    "num_steps": 10 
  }'
```

--------------------------------

### FSDP LOCAL_STATE_DICT state_dict example

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/sys-design/readme-1-EN.md

Example of LOCAL_STATE_DICT mode in FSDP state_dict. Each rank only gets the shard it is responsible for, without slicing information.

```python
# Each rank only gets the shard it is responsible for
{
    'layer1.weight': tensor([256, 1024]),  # Shard for the current rank (1/4)
    'layer1.bias': tensor([256]),          # Shard for the current rank (1/4)
    # ... only the parameter shards for the current rank
}
```

--------------------------------

### VerlEngine.__init__ 中启动 launch_server_from_verl_engine

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/verl/server-based/veRL-server-based-rollout.md

该代码展示了在 VerlEngine 的初始化中，在 tp_rank == 0 的进程内启动一个新线程来运行 launch_server_from_verl_engine，并设置端口为 30000 + tp_rank 以避免冲突。此方案因 TokenizerManager 并发访问问题而失败。

```python
class VerlEngine:
    def __init__(
        self,
        device_mesh_cpu: DeviceMesh,
        nnodes: int = 1,
        **kwargs,
    ):
        self._device_mesh_cpu = device_mesh_cpu
        self._tp_rank = device_mesh_cpu.get_local_rank()
        self._tp_size = device_mesh_cpu.size()
        tp_size_per_node = self._tp_size // nnodes
        node_rank = self._tp_rank // tp_size_per_node
        first_rank_in_node = self._tp_rank % tp_size_per_node == 0

        if first_rank_in_node:
            os.environ["SGLANG_BLOCK_NONZERO_RANK_CHILDREN"] = "0"
            self._engine = Engine(
                **kwargs, tp_size=self._tp_size, node_rank=node_rank, nnodes=nnodes
            )
        else:
            self._engine = None

        if self._tp_rank == 0:
            import copy

            new_server_args = copy.deepcopy(self._engine.server_args)
            new_server_args.port = 30000 + self._tp_rank
            print(f"launch_server_from_verl_engine {new_server_args.port}")

            def server_thread_wrapper(tokenizer_manager, scheduler_info, server_args):
                print(f"Server thread begin")
                launch_server_from_verl_engine(
                    tokenizer_manager=tokenizer_manager,
                    scheduler_info=scheduler_info,
                    server_args=server_args,
                )

            server_thread = threading.Thread(
                target=server_thread_wrapper,
                args=(
                    self._engine.tokenizer_manager,
                    self._engine.scheduler_info,
                    new_server_args,
                ),
                daemon=True,
            )
            server_thread.start()

        dist.barrier(group=self._device_mesh_cpu.get_group())

```

--------------------------------

### FSDP SHARDED_STATE_DICT state_dict example

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/sys-design/readme-1-EN.md

Example of SHARDED_STATE_DICT mode in FSDP state_dict. Each rank gets a ShardedTensor object containing metadata about the shard and slicing, building upon LOCAL_STATE_DICT.

```python
# Each rank gets a shard object containing metadata
{
    'layer1.weight': ShardedTensor {
        metadata: {
            "world_size": 4,           # Total number of shards
            "rank": 1,                 # Current shard index (0-3)
            "shape": [1024, 1024],     # Shape of the full tensor
            "dtype": torch.float16,    # Data type
        },
        local_shard: tensor([256, 1024]),  # Shard data for the current rank
    },
    'layer1.bias': ShardedTensor {
        metadata: { "world_size": 4, "rank": 1, "shape": [1024], "dtype": torch.float16 },
        local_shard: tensor([256]),    # Shard data for the current rank
    }
}
```

--------------------------------

### Run PPO training with SGLang backend

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/OpenRLHF/openrlhf-sglang.md

Starts a Ray head node with 3 GPUs and submits a PPO training job using the SGLang backend. Requires setting RAY_TEMP_DIR, RLHF_CKPT_DIR, and WANDB_API_KEY; find RAY_HEAD_IP in the Ray output before submitting.

```bash
# 启动 ray，自行设置 RAY_TEMP_DIR，RLHF_CKPT_DIR WANDB_API_KEY 环境变量
ray start --head --node-ip-address 127.0.0.1 --num-gpus 3 --port 1234 --temp-dir=$RAY_TEMP_DIR

# 在 ray 的输出中找到 RAY_HEAD_IP，然后提交任务
ray job submit --address="<RAY_HEAD_IP>:1234" \
--runtime-env-json="{
  \"working_dir\": \"${RLHF_CKPT_DIR}\",
}" \
   -- python3 -m openrlhf.cli.train_ppo_ray \
   --backend sglang \
   --ref_num_nodes 1 \
   --ref_num_gpus_per_node 1 \
   --reward_num_nodes 1 \
   --reward_num_gpus_per_node 1 \
   --critic_num_nodes 1 \
   --critic_num_gpus_per_node 1 \
   --actor_num_nodes 1 \
   --actor_num_gpus_per_node 1 \
   --vllm_num_engines 1 \
   --vllm_tensor_parallel_size 1 \
   --colocate_critic_reward \
   --colocate_actor_ref \
   --pretrain OpenRLHF/Llama-3-8b-sft-mixture \
   --reward_pretrain OpenRLHF/Llama-3-8b-rm-mixture \
   --save_path ${RLHF_CKPT_DIR}/examples/llama3-8b-rlhf \
   --save_steps 5 \
   --micro_train_batch_size 8 \
   --train_batch_size 32 \
   --micro_rollout_batch_size 32 \
   --rollout_batch_size 1024 \
   --max_samples 100000 \
   --max_epochs 2 \
   --prompt_max_len 1024 \
   --generate_max_len 1024 \
   --zero_stage 3 \
   --bf16 \
   --actor_learning_rate 5e-7 \
   --critic_learning_rate 9e-6 \
   --init_kl_coef 0.01 \
   --prompt_data OpenRLHF/prompt-collection-v0.1 \
   --input_key context_messages \
   --apply_chat_template \
   --packing_samples \
   --normalize_reward \
   --adam_offload \
   --flash_attn \
   --gradient_checkpointing \
   --use_wandb $WANDB_API_KEY \
   --wandb_run_name sglang \
   --wandb_project openrlhf
```

--------------------------------

### Start local flat e5 retrieval server

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/verl/multi-turn/tool_examples/verl-multiturn-searchR1-like_ZH.md

Starts the retrieval server using the downloaded index and corpus. First run downloads the model and loads the index; normal startup takes 1-2 minutes. Each GPU uses about 5-7GB memory.

```bash
conda activate retriever

index_file=$save_path/e5_Flat.index
corpus_file=$save_path/wiki-18.jsonl
retriever_name=e5
retriever_path=intfloat/e5-base-v2

python examples/sglang_multiturn/search_r1_like/local_dense_retriever/retrieval_server.py --index_path $index_file --corpus_path $corpus_file --topk 3 --retriever_name $retriever_name --retriever_model $retriever_path --faiss_gpu
```

--------------------------------

### Dockerfile example for building a CUDA-based image

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/engineer/how-to-use-docker/readme.md

A complete Dockerfile example that builds an image with CUDA 12.1.1 and cuDNN 8.9.0, sets a working directory, configures an environment variable, installs development tools, copies an app.py file, and defines the default startup command.

```dockerfile
# 导入 Nvidia 官方镜像
# 该镜像把 CUDA 12.1.1 和 cuDNN 8.9.0 都装好了，不需要再处理版本兼容问题
FROM nvidia/cuda:12.1.1-cudnn8-devel-ubuntu22.04

# 设置工作目录为 /sgl-workspace
WORKDIR /sgl-workspace

# 设置环境变量，如 huggingface token 用于下载模型
ENV HF_TOKEN=hf_xxxyyyzzz

# 安装开发工具
RUN apt-get update && apt-get install -y \
    vim \
    tmux \
    wget \
    git \
    && rm -rf /var/lib/apt/lists/* \
    && apt-get clean

# 将当前目录下的 app.py 文件复制到容器内的工作目录
COPY app.py /sgl-workspace/app.py

# 默认执行命令，启动 Python 脚本
CMD ["python3", "app.py"]
```

--------------------------------

### Install and Serve MOSS-TTS-Local with Docker and sgl-omni

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/sglang/sglang-omni/moss-tts-local-v15-blog.md

Sets up the SGLang-Omni environment inside a Docker container, clones the repository, creates a Python 3.12 virtual environment, installs dependencies, downloads the MOSS-TTS-Local model, and starts the server on port 8000. Requires a GPU and the --privileged flag for full access.

```bash
docker pull lmsysorg/sglang-omni:dev
docker run -it --gpus all --shm-size 32g --ipc host --network host --privileged \
  lmsysorg/sglang-omni:dev /bin/zsh

git clone git@github.com:sgl-project/sglang-omni.git
cd sglang-omni
uv venv .venv -p 3.12
source .venv/bin/activate
uv pip install -v -e .

hf download OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5

sgl-omni serve \
  --model-path OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 \
  --port 8000
```

--------------------------------

### Install Dynamo from PyPI (with outlines)

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/sglang/nvidia-dynamo/dynamo.md

Install Dynamo from PyPI with all extras and pin outlines to 0.0.46 for compatibility.

```bash
pip install ai-dynamo[all] && pip install "outlines==0.0.46"
```

--------------------------------

### Reproduction script for multi-stage wake up error

Source: https://github.com/zhaochenyang20/awesome-ml-sys-tutorial/blob/main/rlhf/verl/multi-turn/tool_examples/debug-multi-stage.md

This bash script sets up a Docker container with sglang, installs Python and dependencies, clones the verl repository on the 'multi-stage-wake-up' branch, installs verl with sglang support, and runs a training script to reproduce the 'illegal memory access' error. It includes environment setup and launching the training with nohup.

```bash
docker run \
    -it \
    --shm-size 32g \
    --gpus all \
    -v {Huggingface-Cache-Path}:/root/.cache \
    --ipc=host \
    --network=host \
    --privileged \
    --name sglang_{your-name} \
    lmsysorg/sglang:dev \
    /bin/zsh

apt update
apt install -y python3.10 python3.10-venv

python3 -m venv ~/.python/veRL-multiturn-rollout
source ~/.python/veRL-multiturn-rollout/bin/activate

python3 -m pip install uv

cd ~
git clone -b multi-stage-wake-up https://github.com/volcengine/verl.git
cd verl

python3 -m uv pip install -e ".[sglang]"
python3 -m uv pip install wheel
python3 -m uv pip install packaging
python3 -m uv pip install flash-attn==2.8.0.post2 --no-build-isolation --no-deps

# reinstall latest sglang for verl multi-stage wake up
python3 -m uv pip install "sglang[all]==0.4.8.post1"


function now() {
    date '+%Y-%m-%d-%H-%M'
}

python3 ./examples/data_preprocess/gsm8k_multiturn_w_tool.py


mkdir -p logs


export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

nohup bash examples/sglang_multiturn/run_qwen2.5-3b_gsm8k_multiturn.sh trainer.experiment_name=qwen2.5-3b_rm-gsm8k-sgl-multiturn-$(now) > logs/gsm8k-$(now).log 2>&1 &
```
Tokens:
Raw