SpecForge

https://sgl-project.github.io/SpecForge/
Admin
SpecForge is an SGLang-native framework for training, evaluating, and exporting speculative decoding...
Tokens:58,639
Snippets:453
Trust Score:9.7
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Install from PyPI with uv

Source: https://sgl-project.github.io/SpecForge/get_started/installation.html

Installs SpecForge from PyPI using uv, allowing prerelease versions.

```bash
uv pip install specforge --prerelease=allow
```

--------------------------------

### Install from PyPI with pip

Source: https://sgl-project.github.io/SpecForge/get_started/installation.html

Installs SpecForge from PyPI using pip.

```bash
pip install specforge
```

--------------------------------

### Install from source (recommended)

Source: https://sgl-project.github.io/SpecForge/get_started/installation.html

Clones the SpecForge repository, creates a Python 3.11 virtual environment with uv, and installs the package in editable mode with prerelease allowed.

```bash
# git clone the source code
git clone https://github.com/sgl-project/SpecForge.git
cd SpecForge

# create a new virtual environment
uv venv -p 3.11 --seed
source .venv/bin/activate

# install specforge
uv pip install -v -e . --prerelease=allow
```

--------------------------------

### Install on Intel GPU (XPU)

Source: https://sgl-project.github.io/SpecForge/get_started/installation.html

Inside an XPU SGLang container, clones SpecForge, swaps in the XPU-specific pyproject.toml, and installs the package.

```bash
# Inside the XPU SGLang container
git clone https://github.com/sgl-project/SpecForge.git /sgl-workspace/SpecForge
cd /sgl-workspace/SpecForge
cp pyproject.toml pyproject.toml.backup
cp pyproject_xpu.toml pyproject.toml
python -m pip install -e .
```

--------------------------------

### Install SpecForge from source with uv

Source: https://sgl-project.github.io/SpecForge

Clones the SpecForge repository, creates a Python 3.11 virtual environment with uv, and installs the package in editable mode. Requires a CUDA build of PyTorch matching the host driver; installing from source is recommended for latest recipes and patches.

```bash
git clone https://github.com/sgl-project/SpecForge.git
cd SpecForge
uv venv -p 3.11 --seed
source .venv/bin/activate
uv pip install -e .
```

--------------------------------

### Start SGLang server for Qwen3-8B

Source: https://sgl-project.github.io/SpecForge/basic_usage/data_preparation.html

Starts an SGLang server for Qwen3-8B non-reasoning regeneration. Requires the server to be running before the recipe; the recipe does not launch or stop servers.

```bash
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp-size 1 \
    --dtype bfloat16 \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 \
    --port 30000
```

--------------------------------

### Start SGLang server for Qwen3.6-27B

Source: https://sgl-project.github.io/SpecForge/basic_usage/data_preparation.html

Starts an SGLang server for Qwen3.6-27B structured-reasoning regeneration. Uses the Qwen3 reasoning parser so the OpenAI-compatible response includes reasoning_content.

```bash
python -m sglang.launch_server \
    --model-path Qwen/Qwen3.6-27B \
    --tp-size 1 \
    --dtype bfloat16 \
    --mem-fraction-static 0.8 \
    --reasoning-parser qwen3 \
    --host 0.0.0.0 \
    --port 30000
```

--------------------------------

### Start Mooncake master server

Source: https://sgl-project.github.io/SpecForge/recipes/deepseek-v4-flash-dspark-disaggregated.html

Starts the Mooncake metadata server required for the disaggregated capture setup. The `--default_kv_lease_ttl=5m` flag is required for multi-hundred-MB feature objects; the Kimi K3 runbook explains the failure modes.

```bash
mooncake_master \
  --enable_http_metadata_server=true \
  --http_metadata_server_host=0.0.0.0 \
  --rpc_port=35551 \
  --http_metadata_server_port=35880 \
  --metrics_port=35903 \
  --default_kv_lease_ttl=5m
```

--------------------------------

### Launch Qwen3-8B two-node disaggregated example with rcli

Source: https://sgl-project.github.io/SpecForge/basic_usage/disaggregated_training.html

Launch a separate-inference-node Qwen3-8B example using rcli. Sets the store ID and run root environment variables, then executes the wrapper script per node. The wrapper coordinates node-local services and lifecycle markers; it does not contain another trainer or construct torchrun.

```bash
export DISAGG_STORE_ID=qwen3-8b-two-node-attempt-001
export DISAGG_RUN_ROOT=/shared/specforge/$DISAGG_STORE_ID
rcli exec --per-node <job> \
  'bash examples/disagg/run_qwen3_8b_dflash_disagg_2node.sh'
```

--------------------------------

### Start trainer node with specforge train

Source: https://sgl-project.github.io/SpecForge/recipes/kimi-k3-dspark-disaggregated.html

Starts the trainer node with one CPU producer and a four-rank FSDP consumer on the same host. Resolve capture-node through DNS or override the three endpoint fields with the capture node's IP. WANDB_API_KEY is read from a protected node-local file; do not put credentials in YAML.

```bash
export MOONCAKE_LOCAL_HOSTNAME="$TRAINER_IP"
export MC_TCP_BIND_ADDRESS="$TRAINER_IP"
export MC_TRANSFER_TIMEOUT=300
export WANDB_API_KEY="$(< /protected/path/wandb-api-key)"
export WANDB_ENTITY=your-entity
unset RANK LOCAL_RANK WORLD_SIZE MASTER_ADDR MASTER_PORT NODE_RANK
CUDA_VISIBLE_DEVICES=0,1,2,3 specforge train \
  -c examples/configs/online/disaggregated/external/kimi-k3-dspark-disaggregated.yaml \
  --role both \
  "deployment.disaggregated.server_urls=[\"http://$CAPTURE_IP:30000\"]" \
  "deployment.disaggregated.mooncake_metadata_server=http://$CAPTURE_IP:35880/metadata" \
  "deployment.disaggregated.mooncake_master_server_addr=$CAPTURE_IP:35551"
```

--------------------------------

### Train with specforge train for managed-local DFlash2

Source: https://sgl-project.github.io/SpecForge/concepts/DFlash2.html

Run the managed-local Qwen3.6-27B DFlash2 example. Update the model/data paths and GPU allocation in the referenced YAML before running.

```bash
specforge train \
  -c examples/configs/online/disaggregated/managed-local/qwen3.6-27b-dflash2-disaggregated.yaml
```

--------------------------------

### Run single-node 4-server DP4 disaggregated training with specforge train

Source: https://sgl-project.github.io/SpecForge/recipes/qwen3.8-27b-dflash2-disaggregated.html

Run the single-node disaggregated training recipe with four capture servers and DP4. Add `--plan` first to validate the schema and print the owned Mooncake, capture-server, producer and consumer commands without starting anything. The launcher starts Mooncake, four capture servers on GPUs 0-3, then the producer and the four trainer ranks on GPUs 4-7, and tears the stack down when the consumer finishes. Logs live under the recipe's `control_dir/logs/`.

```bash
specforge train -c \
  examples/configs/online/disaggregated/managed-local/qwen3.8-27b-dflash2-4server-dp4-disaggregated.yaml
```

--------------------------------

### Install SpecForge with --no-deps

Source: https://sgl-project.github.io/SpecForge/basic_usage/Ascend/ascend_npu.html

Clones the SpecForge repository and installs it without dependencies to avoid pulling CUDA wheels over the existing NPU torch/sglang stack. Use --no-deps to keep the working environment intact.

```bash
git clone https://github.com/sgl-project/SpecForge.git
cd SpecForge
python -m pip install -e . --no-deps
```

--------------------------------

### Start Mooncake master with --default_kv_lease_ttl=500

Source: https://sgl-project.github.io/SpecForge/basic_usage/AMD/amd_rocm.html

Start the Mooncake master for online disaggregated training. Set --default_kv_lease_ttl=500 to match the consumer's teardown drain of about 19.5s; the shorter managed TTL avoids waiting several seconds for an expired read lease during normal shutdown.

```bash
mooncake_master --enable_http_metadata_server=true \
  --rpc_port=35551 --http_metadata_server_port=35880 \
  --metrics_port=35903 --enable_metric_reporting=false \
  --default_kv_lease_ttl=500 &
```

--------------------------------

### Start patched capture server on NPU 0

Source: https://sgl-project.github.io/SpecForge/basic_usage/Ascend/ascend_npu.html

Starts the patched capture server on NPU 0. The --spec-capture-aux-layer-ids must match the draft's target_layer_ids (for DFlash: 1 8 15 22 29). A mismatch produces zero features with no error.

```bash
ASCEND_RT_VISIBLE_DEVICES=0 \
MOONCAKE_LOCAL_HOSTNAME=127.0.0.1 \
MOONCAKE_METADATA_SERVER=http://127.0.0.1:35880/metadata \
MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:35551 \
MOONCAKE_PROTOCOL=tcp \
MOONCAKE_GLOBAL_SEGMENT_SIZE=$((32<<30)) \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-4B \
  --trust-remote-code \
  --skip-tokenizer-init \
  --tp-size 1 \
  --mem-fraction-static 0.8 \
  --attention-backend ascend \
  --enable-spec-capture --spec-capture-method dflash \
  --spec-capture-aux-layer-ids 1 8 15 22 29 \
  --host 127.0.0.1 --port 30000 &
```

--------------------------------

### Online deployment configuration

Source: https://sgl-project.github.io/SpecForge/basic_usage/disaggregated_training.html

Example YAML configuration for an online disaggregated deployment, specifying trainer resources, Mooncake backend, and server URLs.

```yaml
deployment:
  mode: disaggregated
  trainer:
    nnodes: 1
    nproc_per_node: 4
  disaggregated:
    control_dir: outputs/qwen3-8b-dflash-disaggregated/control
    consumer_state_dir: outputs/qwen3-8b-dflash-disaggregated/consumer-state
    backend: mooncake
    server_urls:
      - http://127.0.0.1:30000
    mooncake_metadata_server: http://127.0.0.1:35880/metadata
    mooncake_master_server_addr: 127.0.0.1:35551
    mooncake_protocol: tcp
```

--------------------------------

### Example run config YAML

Source: https://sgl-project.github.io/SpecForge/basic_usage/training.html

A run config has seven typed sections (`model`, `data`, `training`, `tracking`, `profiling`, `runtime`, and `deployment`) plus `run_id` and `output_dir`. Paths are resolved from the current working directory.

```yaml
model:
  target_model_path: Qwen/Qwen3-8B
  draft_model_config: configs/qwen3-8b-eagle3.json
  target_backend: sglang
  vocab_mapping_path: cache/vocab_mapping/qwen3-8b.pt
  torch_dtype: bfloat16

data:
  train_data_path: ./cache/dataset/sharegpt_train.jsonl
  max_length: 4096
  chat_template: qwen
  cache_dir: ./cache

training:
  strategy: eagle3
  num_epochs: 10
  max_steps: 10000
  batch_size: 1
  learning_rate: 1.0e-4
  save_interval: 1000

run_id: qwen3-8b-eagle3-disaggregated
output_dir: outputs/qwen3-8b-eagle3-disaggregated

deployment:
  mode: disaggregated
  trainer:
    nnodes: 1
    nproc_per_node: 1
  disaggregated:
    control_dir: outputs/qwen3-8b-eagle3-disaggregated/control
    consumer_state_dir: outputs/qwen3-8b-eagle3-disaggregated/consumer-state
    backend: mooncake
    server_urls:
      - http://127.0.0.1:30000
    mooncake_metadata_server: http://127.0.0.1:35880/metadata
    mooncake_master_server_addr: 127.0.0.1:35551
```

--------------------------------

### Launch a run with distributed training (nproc_per_node > 1)

Source: https://sgl-project.github.io/SpecForge/basic_usage/training.html

When `deployment.trainer.nproc_per_node` is greater than one, the CLI starts torch distributed itself. Use `-c` as a shorthand for `--config`.

```bash
specforge train -c examples/configs/online/disaggregated/external/qwen3-30b-a3b-eagle3-online.yaml
```

--------------------------------

### Start Mooncake master

Source: https://sgl-project.github.io/SpecForge/basic_usage/Ascend/ascend_npu.html

Starts the Mooncake master with HTTP metadata server enabled. Required before starting the capture server for online training.

```bash
mooncake_master --enable_http_metadata_server=true \
  --rpc_port=35551 --http_metadata_server_port=35880 \
  --metrics_port=35903 --enable_metric_reporting=false &
```

--------------------------------

### Install data regeneration dependencies

Source: https://sgl-project.github.io/SpecForge/basic_usage/data_preparation.html

Install the data regeneration utility in the source-checkout environment before running the repository script. This installs the package with the 'data' extra.

```bash
pip install -e '.[data]'
```

--------------------------------

### Launch Ascend NPU training with environment variables

Source: https://sgl-project.github.io/SpecForge/basic_usage/training.html

Launches a four-device Ascend NPU training run using the unified launcher. Requires vendor-matched PyTorch and torch_npu packages, and uses an external NPU-compatible SGLang capture server with SDPA consumers. The runtime selects the active device dynamically and uses HCCL when torch_npu is active.

```bash
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export HCCL_CONNECT_TIMEOUT=7200
export HCCL_EXEC_TIMEOUT=7200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

specforge train -c examples/configs/online/disaggregated/external/qwen3.5-4b-dflash-online-npu.yaml

```

--------------------------------

### Install ROCm environment for SpecForge

Source: https://sgl-project.github.io/SpecForge/basic_usage/training.html

Installs the checked-in ROCm environment before installing SpecForge. Use a model/backend combination supported by that PyTorch ROCm environment; HF + SDPA recipes are the portable baseline.

```bash
python -m pip install -r requirements-rocm.txt
python -m pip install -e . --no-deps

```

--------------------------------

### Install wandb and TensorBoard in ROCm container

Source: https://sgl-project.github.io/SpecForge/basic_usage/AMD/amd_rocm.html

Installs wandb and TensorBoard because the SGLang ROCm images ship neither. TensorBoard and its dependencies are installed with --no-deps to avoid pulling CUDA wheels.

```bash
python -m pip install wandb
python -m pip install --no-deps tensorboard tensorboard-data-server absl-py markdown werkzeug
```

--------------------------------

### Start SGLang server for target model

Source: https://sgl-project.github.io/SpecForge/basic_usage/data_preparation.html

Start the SGLang server for the target model (meta-llama/Llama-3.1-8B-Instruct) on port 30000. This is the first step to regenerate the dataset.

```shell
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --cuda-graph-max-bs 128 \
    --dtype bfloat16 \
    --mem-fraction-static 0.8 \
    --port 30000
```

--------------------------------

### Run managed-local Qwen3.6 DFlash one-server DP2 recipe

Source: https://sgl-project.github.io/SpecForge/basic_usage/disaggregated_training.html

Runs the Qwen3.6 DFlash one-server recipe, which owns a TP=1 server on GPU 0 and a DP=2 trainer on GPUs 1–2.

```bash
specforge train -c \
  examples/configs/online/disaggregated/managed-local/qwen3.6-27b-dflash-1server-dp2-disaggregated.yaml
```

--------------------------------

### Install Mooncake transfer engine for CUDA

Source: https://sgl-project.github.io/SpecForge/basic_usage/disaggregated_training.html

Install the Mooncake transfer engine. Use mooncake-transfer-engine for CUDA versions earlier than 13, and mooncake-transfer-engine-cuda13 for CUDA 13 or later.

```bash
pip install mooncake-transfer-engine
```

```bash
pip install mooncake-transfer-engine-cuda13
```

--------------------------------

### Launch two-node 8-server DP8 disaggregated training with rcli exec

Source: https://sgl-project.github.io/SpecForge/recipes/qwen3.8-27b-dflash2-disaggregated.html

Launch the two-node disaggregated training recipe with eight capture servers and DP8. Rank 0 applies the capture patch, starts `mooncake_master` with a 10 s read lease, starts eight TP1 capture servers on GPUs 0-7 (ports 30000-30007) with a 64 GiB Store segment each, waits for every `/health`, then runs the CPU producer. Rank 1 waits for readiness and runs the eight-rank consumer. Both nodes must see the fresh `DISAGG_RUN_ROOT`; feature tensors travel only through Mooncake. `DRY_RUN=1` prints every command instead of running it. Override `SERVER_COUNT`, `SERVER_GPUS`, `TRAINER_GPUS` and `TRAINER_NPROC` for another allocation, `MOONCAKE_PROTOCOL=rdma` plus `MOONCAKE_RDMA_DEVICES=<hca>` for an InfiniBand fabric, and `TARGET_MODEL_PATH=Qwen/Qwen3.8-27B-FP8` for Hopper.

```bash
export DISAGG_STORE_ID=qwen3.8-27b-dflash2-attempt-001
export DISAGG_RUN_ROOT=/shared/specforge/$DISAGG_STORE_ID

# The launcher supplies RCLI_NODE_RANK, RCLI_NUM_NODES, and RCLI_HEAD_IP.
rcli exec --per-node <job> \
  'bash examples/disagg/run_qwen3.8_27b_dflash2_disagg_2node.sh'
```

--------------------------------

### Weights-only warm start with model.draft_checkpoint_path

Source: https://sgl-project.github.io/SpecForge/basic_usage/training.html

Configures a weights-only warm start for a new run. The draft checkpoint path can point to a Hugging Face model directory/repository, a SpecForge checkpoint directory, training_state.pt, or a run root. Warm start never restores optimizer state, counters, data position, or RNG, and is mutually exclusive with training.resume_from.

```yaml
model:
  target_model_path: Qwen/Qwen3-8B
  draft_checkpoint_path: ./outputs/base/base-step1000
```

--------------------------------

### Launch training on GPU 1 with specforge train

Source: https://sgl-project.github.io/SpecForge/basic_usage/AMD/amd_rocm.html

Launch training on GPU 1, supervising both producer and consumer. The trainer needs no AITER env; it runs flex_attention on ROCm, only the capture server drives the Mamba target. Before rerunning, clear stale control state with rm -rf outputs/qwen3.5-4b-dflash-online. Success criteria: producer log shows drive_producer returning produced=<N> prompts_failed=0, consumer log shows step N: {...loss..., acc...} lines with no could not drain error, and checkpoint outputs/qwen3.5-4b-dflash-online/qwen3.5-4b-dflash-online-step20/ contains training_state.pt and training_state_rank0.pt.

```bash
CUDA_VISIBLE_DEVICES=1 HIP_VISIBLE_DEVICES=1 \
MOONCAKE_LOCAL_HOSTNAME=127.0.0.1 \
MOONCAKE_METADATA_SERVER=http://127.0.0.1:35880/metadata \
MOONCAKE_MASTER_SERVER_ADDR=127.0.0.1:35551 \
MOONCAKE_PROTOCOL=tcp \
MOONCAKE_GLOBAL_SEGMENT_SIZE=$((32<<30)) \
specforge train -c examples/configs/online/disaggregated/external/qwen3.5-4b-dflash-online-amd.yaml \
  training.max_steps=20 training.num_epochs=1 \
  training.save_interval=20 training.log_interval=5
```

--------------------------------

### Install on AMD ROCm without dependencies

Source: https://sgl-project.github.io/SpecForge/get_started/installation.html

Inside a ROCm SGLang container, clones SpecForge and installs it without dependencies to avoid pulling CUDA wheels over the working ROCm stack.

```bash
# Inside the ROCm SGLang container
git clone https://github.com/sgl-project/SpecForge.git /workspace/SpecForge
cd /workspace/SpecForge
python -m pip install -e . --no-deps
```

--------------------------------

### Launch consumer trainers with --node-rank for multi-node setup

Source: https://sgl-project.github.io/SpecForge/basic_usage/AMD/amd_rocm.html

Launch consumer trainers on separate hosts for multi-node training. Each host passes its node-local identity via --node-rank, while the shared topology (nnodes, nproc_per_node, master_addr, master_port) is recorded once in run.yaml.

```bash
specforge train -c run.yaml --role consumer --node-rank 0   # trainer-0
```

```bash
specforge train -c run.yaml --role consumer --node-rank 1   # trainer-1
```

--------------------------------

### Run managed-local DFlash and Domino one-server DP7 recipes

Source: https://sgl-project.github.io/SpecForge/basic_usage/disaggregated_training.html

Runs the historical self-contained DFlash and Domino topologies, each recording one TP=1 capture server on GPU 0 and a DP=7 trainer on GPUs 1–7.

```bash
specforge train -c \
  examples/configs/online/disaggregated/managed-local/qwen3-8b-dflash-1server-dp7-disaggregated.yaml

specforge train -c \
  examples/configs/online/disaggregated/managed-local/qwen3-8b-domino-1server-dp7-disaggregated.yaml
```

--------------------------------

### Prepare data with canonical presets

Source: https://sgl-project.github.io/SpecForge/basic_usage/data_preparation.html

Run the data preparation script from the repository root for the ultrachat and sharegpt presets. Each command writes <preset>_train.jsonl under cache/dataset by default; use --output-path to change the output directory and --sample-size to cap the number of source rows.

```bash
# ultrachat
python scripts/prepare_data.py --dataset ultrachat

# sharegpt
python scripts/prepare_data.py --dataset sharegpt
```

--------------------------------

### Run training with a config file

Source: https://sgl-project.github.io/SpecForge/advanced_features/customization.html

Runs a training run with a typed YAML config file. Pick the closest checked-in file under examples/configs, change model and data paths, and keep the same training entry point.

```bash
specforge train --config ./my-run.yaml
```
Tokens:
Raw