### Run HF Quickstart Example

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/hf-quickstart/README.md

Execute the main script for the HF Quickstart example. Ensure you have the specified prerequisites and configuration.

```bash
./examples/hf-quickstart/run.sh
```

--------------------------------

### Quick Setup with vLLM

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Install TorchSpec with vLLM support using the build script and activate the torchspec environment.

```bash
# Install with vLLM
./tools/build_conda.sh 1 vllm
micromamba activate torchspec
```

--------------------------------

### Quick Setup with SGLang

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Install TorchSpec with SGLang support using the build script and activate the torchspec environment.

```bash
# Or install with SGLang
./tools/build_conda.sh
micromamba activate torchspec
```

--------------------------------

### Start Ray Head Node

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Execute this script on the head node to initiate the Ray cluster setup.

```bash
NODE_ROLE=head bash examples/kimi-k25-2node-h200/setup_ray_cluster.sh
```

--------------------------------

### Run Qwen3-8B Single Node Example

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/README.md

Run the qwen3-8b-single-node example for production workloads with async inference. This example uses the SGLang backend by default.

```bash
./examples/qwen3-8b-single-node/run.sh
```

--------------------------------

### Switch to vLLM Backend for Qwen3-8B Example

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/README.md

Configure the qwen3-8b-single-node example to use the vLLM backend instead of the default SGLang. This requires specifying a vLLM configuration file.

```bash
./examples/qwen3-8b-single-node/run.sh \
    --config configs/vllm_qwen3_8b.yaml \

```

--------------------------------

### Adjust Training Steps in HF Quickstart

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/hf-quickstart/README.md

Modify the number of training steps for the HF Quickstart example by passing the training.num_train_steps argument.

```bash
./examples/hf-quickstart/run.sh training.num_train_steps=50
```

--------------------------------

### TorchSpec Configuration Example

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/code_architecture.md

A sample hierarchical YAML configuration file defining various aspects of the training and inference setup. This includes dataset paths, model details, training parameters, and inference engine settings.

```yaml
dataset:
  chat_template: llama3
  train_data_path: /path/to/data
  max_seq_length: 8192

model:
  target_model_path: Qwen/Qwen3-8B
  target_model_backend: sglang    # or "remote"
  draft_model_config: /path/to/config.json

training:
  num_epochs: 1
  micro_batch_size: 2
  learning_rate: 1e-4
  ttt_length: 7                   # Speculative depth
  train_backend: fsdp
  fsdp_strategy: REPLICATE
  placement_strategy: training_first  # or inference_first/custom
  training_node_ips: null             # custom placement only

inference:
  inference_engine_type: hf       # or "sgl"
  inference_batch_size: 1
  inference_num_gpus: 4
  inference_node_ips: null        # custom placement only
  sglang:                         # nested under inference
    tp_size: 8
    extra_args:                   # power-user passthrough to sgl.Engine
      attention_backend: flashinfer

mooncake:
  master_addr: null
  protocol: rdma                  # or "tcp"

logging:
  report_to: wandb
  wandb_project: torchspec

debug:
  use_pytorch_profiler: false
```

--------------------------------

### Change Learning Rate in HF Quickstart

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/hf-quickstart/README.md

Alter the learning rate for the HF Quickstart example by specifying the training.learning_rate argument.

```bash
./examples/hf-quickstart/run.sh training.learning_rate=5e-5
```

--------------------------------

### Install Flash Attention

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Optionally, install Flash Attention for full feature support by installing the package with the 'fa' extra.

```bash
pip install -e ".[fa]"
```

--------------------------------

### Install TorchSpec in Current Environment

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Alternatively, install TorchSpec into your current environment using the provided build script.

```bash
./tools/build_conda.sh current
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Install pre-commit hooks to automatically enforce code style and linting rules before committing.

```bash
pre-commit install
```

--------------------------------

### Start Ray Head Node (Multi-Node Local Cluster)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

This command starts the head node for a local Ray cluster. Ensure this is run first on the designated head node.

```bash
ray start --head \
  --port 6379 \
  --node-ip-address <HEAD_IP> \
  --num-gpus <N> \
  --temp-dir /tmp/ray_$(id -u) \
  --disable-usage-stats
```

--------------------------------

### Install SGLang with FlashInfer

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Installs SGLang version 0.5.9 with all features, including PyTorch 2.9.1 and FlashInfer, from a specific wheel URL.

```bash
pip install "sglang[all]==0.5.9" --find-links https://flashinfer.ai/whl/cu124/torch2.9/flashinfer-python
```

--------------------------------

### Install into Current Environment

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Install TorchSpec into the current active environment, specifying 'sglang', 'vllm', or 'both' for backend support.

```bash
./tools/build_conda.sh current sglang  # or 'vllm' or 'both'
```

--------------------------------

### Install System Libraries for Mooncake

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Installs necessary libraries for the Mooncake transfer engine on Ubuntu/Debian and RHEL/CentOS systems.

```bash
sudo apt-get install -y libibverbs-dev librdmacm-dev libnuma-dev
```

```bash
sudo yum install -y libibverbs-devel librdmacm-devel numactl-devel
```

--------------------------------

### Start Ray Head Node

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Initiates the Ray cluster head node on the designated head machine. This command should be executed on Node 0.

```bash
NODE_ROLE=head bash examples/kimi-k25-3node-h100/setup_ray_cluster.sh
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Install the development dependencies for TorchSpec using pip.

```bash
pip install -e ".[dev]"
```

--------------------------------

### Scenario 2: Training Bottleneck Example

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

This scenario depicts a training bottleneck where training capacity is lower than inference capacity, resulting in near-zero dispatch wait and a growing sample pool. Further analysis of data loading versus GPU compute time is recommended.

```text
thru=12.1, I=25.0, T=12.3, wait=0.0s, pool=48
```

--------------------------------

### DFlash Data Format Example

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Example of a single conversation entry in the JSONL format expected by DFlash. Each entry must have an ID and a list of conversation turns with roles and content.

```json
{"id": "conv_001", "conversations": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
```

--------------------------------

### Start Ray Worker Node (Multi-Node Local Cluster)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

This command starts a worker node for a local Ray cluster. It must be run after the head node is active and configured to connect to it.

```bash
ray start \
  --address <HEAD_IP>:6379 \
  --num-gpus <N> \
  --temp-dir /tmp/ray_$(id -u) \
  --disable-usage-stats
```

--------------------------------

### Verify TorchSpec Installation

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Checks if TorchSpec is installed correctly and prints the installed PyTorch and CUDA versions, along with the number of available GPUs.

```python
import torchspec
import torch
print(f'TorchSpec installed')
print(f'PyTorch {torch.__version__}, CUDA {torch.version.cuda}')
print(f'GPUs: {torch.cuda.device_count()}')

```

--------------------------------

### Start Ray Worker Node

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Execute this script on the worker node, providing the head node's IP address, to join the Ray cluster.

```bash
HEAD_IP=<node0_ip> NODE_ROLE=worker bash examples/kimi-k25-2node-h200/setup_ray_cluster.sh
```

--------------------------------

### Scenario 1: Inference Bottleneck Example

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

This scenario illustrates an inference bottleneck where inference capacity is significantly lower than training capacity, leading to high dispatch wait times and an empty sample pool. Consider optimizing inference engines or batch size.

```text
thru=8.2, I=8.5, T=15.3, wait=2.1s, pool=0
```

--------------------------------

### Continual Training Configuration

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Configure continual training to start a new run from existing weights. Set `load_path` to the source weights, enable `continual_training`, and specify new parameters like `learning_rate` and `output_dir` for the new run.

```yaml
training:
  load_path: /path/to/old_run/checkpoints
  continual_training: true
  learning_rate: 1e-5
  warmup_ratio: 0.01
  num_epochs: 1

output_dir: /path/to/new_run
```

--------------------------------

### Scenario 3: Mooncake Transfer Bottleneck Example

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

This scenario highlights a bottleneck in Mooncake transfer or data loading, where data loading time constitutes a large portion of the total step time. Optimizing Mooncake buffer sizes or reducing sequence length may help.

```text
step_time=1.2s, data_time=0.9s, compute_time=0.25s
```

--------------------------------

### Start Ray Worker Node

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Connects a worker node to the existing Ray cluster. Replace `<node0_ip>` with the actual IP address of the head node.

```bash
HEAD_IP=<node0_ip> NODE_ROLE=worker bash examples/kimi-k25-3node-h100/setup_ray_cluster.sh
```

--------------------------------

### Force Local Ray Instance (Single-Node)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

When running on shared machines, 'auto' might connect to another user's cluster. Use 'RAY_ADDRESS=local' to ensure a fresh local instance is started.

```bash
RAY_ADDRESS=local CUDA_VISIBLE_DEVICES=4,5,6,7 ./examples/qwen3-8b-single-node/run.sh
```

--------------------------------

### Custom Node Placement with Ray Label Selectors

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Configure custom node placement for training and inference roles using Ray label selectors. This requires Ray nodes to be started with specific labels. Ensure that only one of `*_node_ips` or `*_node_selectors` is set for each role.

```yaml
training:
  placement_strategy: custom
  training_num_nodes: 2
  training_num_gpus_per_node: 8
  training_node_selectors:
    - {"torchspec/node": "trainer-0"}
    - {"torchspec/node": "trainer-1"}
inference:
  inference_node_selectors:
    - {"torchspec/node": "infer-0"}
    - {"torchspec/node": "infer-1"}
```

--------------------------------

### Apply SGLang Patch

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Applies a specific patch to the SGLang installation to enable speculative training hooks. Ensure the patch path is correct.

```bash
SGLANG_DIR=$(python -c "import sglang; print(sglang.__path__[0])")
cd "$(dirname "$SGLANG_DIR")"
git apply /path/to/TorchSpec/patches/sglang/v0.5.8.post1/sglang.patch
cd /path/to/TorchSpec
```

--------------------------------

### Run SGLang Demo with Custom Config

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/qwen3-8b-single-node/README.md

Run the demo using a specified configuration file.

```bash
./examples/qwen3-8b-single-node/run.sh configs/sglang_qwen3_8b.yaml
```

--------------------------------

### Launch Training Script

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Run this script on the head node after the Ray cluster is successfully set up to begin the training process.

```bash
bash examples/kimi-k25-2node-h200/run.sh
```

--------------------------------

### Run SGLang Demo with Overridden Settings

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/qwen3-8b-single-node/README.md

Execute the demo with specific training parameters overridden in the command line.

```bash
./examples/qwen3-8b-single-node/run.sh configs/sglang_qwen3_8b.yaml training.num_train_steps=10
```

--------------------------------

### Launch Training Script

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Executes the main training script on the head node after the Ray cluster has been successfully set up.

```bash
bash examples/kimi-k25-3node-h100/run.sh
```

--------------------------------

### Prepare PerfectBlend 50K Dataset

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Command to prepare the PerfectBlend 50K dataset using the provided script. Specify the output path and the maximum number of samples to include.

```bash
python scripts/tools/prepare_perfectblend.py \
  --output /path/to/data/perfectblend_50k.jsonl \
  --max_samples 50000
```

--------------------------------

### Run with vLLM Backend

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Execute the training script using the vLLM backend, specifying the configuration file for vLLM.

```bash
./examples/qwen3-8b-single-node/run.sh --config configs/vllm_qwen3_8b.yaml
```

--------------------------------

### Train Model with SGLang and FSDP

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Command to initiate training using DFlash with SGLang for inference and FSDP for distributed training. This command specifies the configuration file, training data path, and output directory.

```bash
python -m torchspec.train_entry \
  --config configs/sglang_qwen3_8b_dflash.yaml \
  dataset.train_data_path=/path/to/data/perfectblend_50k.jsonl \
  output_dir=./outputs/qwen3-8b-dflash
```

--------------------------------

### Install Mooncake Transfer Engine for Specific CUDA Version

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

If a 'libcudart.so.NN => not found' error occurs, this command uninstalls the default mooncake transfer engine and installs a version specific to the CUDA major version (e.g., CUDA 11 or 12).

```bash
uv pip uninstall mooncake-transfer-engine && uv pip install mooncake-transfer-engine-cudaNN
```

--------------------------------

### Build vLLM Docker Image

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Builds a Docker image with a specified vLLM version. Ensure you are in the 'docker' directory and set the necessary environment variables.

```bash
cd docker && BACKEND=vllm VLLM_VERSION=vX.Y.Z IMAGE_REPO=local/torchspec \
  IMAGE_TAG=test-vllm-vX.Y.Z just build
```

--------------------------------

### Command-Line Configuration Overrides

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/code_architecture.md

Demonstrates how to merge multiple YAML configuration files and override specific parameters directly from the command line using OmegaConf.

```bash
python train.py --config base.yaml --config experiment.yaml training.learning_rate=1e-5
```

--------------------------------

### Clone Repository and Set Up Conda Environment

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Clone the TorchSpec repository and set up the necessary conda environment for development.

```bash
git clone https://github.com/torchspec-project/TorchSpec.git
cd torchspec
./tools/build_conda.sh
micromamba activate torchspec
```

--------------------------------

### Override Configuration via CLI

Source: https://github.com/lightseekorg/torchspec/blob/main/configs/README.md

Demonstrates how to override specific training parameters, such as learning rate, directly from the command line when launching the training script.

```bash
python -m torchspec.train_entry --config configs/sglang_qwen3_8b.yaml training.learning_rate=5e-5
```

--------------------------------

### Override Configuration File

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Specifies a custom configuration file for the training process, overriding the default `sglang_kimi_k25_3node.yaml`. Use this to apply different training parameters or settings.

```bash
# Override config
CONFIG_FILE=path/to/custom.yaml bash examples/kimi-k25-3node-h100/run.sh
```

--------------------------------

### Resume Training Configuration

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Use this configuration to resume an interrupted training run. Ensure `load_path` points to the checkpoints of the original run and `output_dir` is set to the same location.

```yaml
training:
  load_path: /path/to/old_run/checkpoints

output_dir: /path/to/old_run
```

--------------------------------

### Sample Multimodal Dataset Entry

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Illustrates the JSONL format for multimodal conversations, including image and text content. Ensure assistant responses contain `<think>...</think>` blocks for proper processing.

```json
{
  "id": "kimi_mm_001",
  "conversations": [
    {
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/...jpg"}},
        {"type": "text", "text": "What animal is in this image?"}
      ]
    },
    {
      "role": "assistant",
      "content": "<think>The image shows a domestic cat.</think>This is a cat."
    }
  ]
}
```

--------------------------------

### TorchSpec Training Flow Overview

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/code_architecture.md

Outlines the multi-stage training process including data loading, inference, and training steps. It details the interaction between different components like the controller, inference managers, and data fetchers.

```text
1. DATASET LOADING (train_entry.py)
   ├── Parse YAML config with OmegaConf
   ├── Preprocess prompts with chat template
   ├── Tokenize to input_ids + loss_mask
   ├── Auto-generate vocab mapping if needed
   └── Add to controller's prompt_buffer

2. INFERENCE (Inference GPUs, async)
   ├── InferenceManager fetches prompts from controller
   ├── Dispatches to HFEngine / SglEngine Ray actors
   ├── Target model produces hidden_states + logits
   ├── Store tensors in EagleMooncakeStore
   ├── Return mooncake keys to controller
   └── Backpressure: pause if sample_pool exceeds limit

3. TRAINING (Training GPUs, synchronous per step)
   ├── Controller dispatches dispatch_batch_size samples to per-DP-rank queues
   ├── MooncakeDataFetcher retrieves tensors from Mooncake
   ├── Forward pass through Eagle3Model (TTT loop over ttt_length positions)
   ├── Compute loss (forward KL divergence against target distribution)
   ├── Backward pass with gradient accumulation
   ├── Optimizer step, LR scheduling
   └── Periodic checkpointing
```

--------------------------------

### Custom Ray Placement with IP Addresses

Source: https://github.com/lightseekorg/torchspec/blob/main/configs/README.md

Configure custom Ray node placement using IP addresses for training and inference. Ensure that for each role, either `*_node_ips` or `*_node_selectors` is set, but not both.

```yaml
training:
  placement_strategy: custom
  training_num_nodes: 1
  training_num_gpus_per_node: 8
  training_node_ips:
    - 10.0.0.1

inference:
  inference_num_gpus: 16
  inference_num_gpus_per_node: 8
  inference_node_ips:
    - 10.0.0.2
    - 10.0.0.3
```

--------------------------------

### Override Configuration File

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Specify a custom configuration file for the training process by setting the CONFIG_FILE environment variable.

```bash
# Override config
CONFIG_FILE=path/to/custom.yaml bash examples/kimi-k25-2node-h200/run.sh
```

--------------------------------

### Run Full Test Suite

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Execute the entire test suite for TorchSpec using the provided script.

```bash
./tools/run_all_tests.sh
```

--------------------------------

### Sanity Check vLLM Docker Image

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Runs a sanity check on the built Docker image to ensure vLLM, torchspec, and flash_attn are importable. Requires '--gpus all' for mooncake import.

```bash
# Sanity — mooncake import needs --gpus all even without training (libcuda.so.1):
docker run --rm --gpus all --entrypoint python3 local/torchspec:test-vllm-vX.Y.Z -c \
  "import vllm, torchspec, flash_attn; from mooncake.store import MooncakeDistributedStore; print('ok')"
```

--------------------------------

### Set Required Environment Variables

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Configures essential environment variables for correct training behavior, including Hugging Face cache location and PyTorch memory allocation settings.

```bash
export HF_HOME=/path/to/your/huggingface/cache
export PYTORCH_ALLOC_CONF=expandable_segments:True
export TORCHINDUCTOR_MAX_AUTOTUNE_GEMM_BACKENDS=ATEN,TRITON
```

--------------------------------

### Custom Ray Placement with Label Selectors

Source: https://github.com/lightseekorg/torchspec/blob/main/configs/README.md

Configure custom Ray node placement using label selectors for training and inference. This method is useful when Ray version supports placement group `bundle_label_selector`.

```yaml
training:
  placement_strategy: custom
  training_node_selectors:
    - {"torchspec/node": "trainer-0"}

inference:
  inference_node_selectors:
    - {"torchspec/node": "infer-0"}
    - {"torchspec/node": "infer-1"}
```

--------------------------------

### Customize GPU Allocation for SGLang Demo

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/qwen3-8b-single-node/README.md

Adjust the number of GPUs used for training and inference by setting environment variables and script arguments.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./examples/qwen3-8b-single-node/run.sh \
    training.training_num_gpus_per_node=4 \
    inference.inference_num_gpus=4
```

--------------------------------

### Resume Training from Checkpoint

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Command to resume training from a previously saved checkpoint. The trainer automatically detects the latest checkpoint and restores the training state.

```bash
python -m torchspec.train_entry \
  --config configs/sglang_qwen3_8b_dflash.yaml \
  dataset.train_data_path=/path/to/data/perfectblend_50k.jsonl \
  training.load_path=./outputs/qwen3-8b-dflash/checkpoints
```

--------------------------------

### Run TorchSpec on Head Node (Multi-Node Local Cluster)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Execute this command on the head node after the Ray cluster is set up. TorchSpec will automatically detect the cluster.

```bash
./examples/kimi-k25-3node-h100/run.sh
```

--------------------------------

### Enable Performance Metrics via CLI

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

Enable performance metrics by passing the CLI override when running the training script. This is an alternative to setting it in the YAML configuration.

```bash
python train.py --config configs/default.yaml debug.enable_perf_metrics=true
```

--------------------------------

### Run Script with Specific GPUs (Single-Node)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Use this command to run a script on a single node while pinning specific GPUs. This is useful on shared machines to avoid conflicts.

```bash
CUDA_VISIBLE_DEVICES=4,5,6,7 ./examples/qwen3-8b-single-node/run.sh
```

--------------------------------

### Custom Node Placement with IP Addresses

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Configure custom node placement for training and inference roles using IP addresses. Ensure that only one of `*_node_ips` or `*_node_selectors` is set for each role.

```yaml
training:
  placement_strategy: custom
  training_num_nodes: 2
  training_num_gpus_per_node: 8
  training_node_ips:
    - 10.0.0.1
    - 10.0.0.3
inference:
  inference_num_gpus: 16
  inference_num_gpus_per_node: 8
  inference_node_ips:
    - 10.0.0.2
    - 10.0.0.4
```

--------------------------------

### Run Ruff Linter and Formatter

Source: https://github.com/lightseekorg/torchspec/blob/main/CONTRIBUTING.md

Manually run the Ruff linter to check for code style issues and the formatter to check formatting.

```bash
ruff check .
ruff format --check .
```

--------------------------------

### Monitor GPU Utilization

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Displays current GPU utilization, memory usage, and total memory. Use `watch` for continuous monitoring to observe GPU activity over time.

```bash
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv
# or watch continuously:
watch -n 2 nvidia-smi
```

--------------------------------

### Pass Extra Overrides to Training Script

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-3node-h100/README.md

Allows passing additional configuration overrides directly to the training script, such as modifying the learning rate. This is useful for fine-tuning parameters without creating a new config file.

```bash
# Pass extra overrides
bash examples/kimi-k25-3node-h100/run.sh training.learning_rate=1e-5
```

--------------------------------

### List Recent Training Logs

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Lists the 5 most recently modified log files in the running_logs directory. Useful for quickly identifying the latest training output.

```bash
ls -lt running_logs/ | head -5
```

--------------------------------

### Convert FSDP Checkpoint to HuggingFace Format

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Use this command to convert a checkpoint saved in FSDP format to the HuggingFace format. Specify the input directory containing the FSDP checkpoints.

```bash
python tools/convert_to_hf.py --input-dir ./outputs/my_experiment/iter_0010000/
```

--------------------------------

### Run vLLM End-to-End Test

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Execute the end-to-end test for a vLLM model, capturing all output to a log file. Ensure CUDA_VISIBLE_DEVICES and MC_STORE_MEMCPY are set correctly.

```bash
export CUDA_VISIBLE_DEVICES=<the node's actual GPU ids>
export MC_STORE_MEMCPY=0
./examples/qwen3-8b-single-node/run.sh configs/vllm_qwen3_8b.yaml \
    training.num_train_steps=5 > /tmp/bump_e2e.log 2>&1
```

--------------------------------

### End-to-End Test with FA4 Patch

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Executes an end-to-end test using the Docker image, applying a Flash Attention 4 patch. This is the only scenario where the FA4 patch is active. Requires '--gpus all', '--network=host', and '--shm-size=16g'. Ensure CUDA_VISIBLE_DEVICES matches Phase 3.

```bash
# e2e — the ONLY place the FA4 patch executes (flash-attn-4 is installed here, not bare metal):
docker run --rm --gpus all --network=host --shm-size=16g --entrypoint bash \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  -e MC_STORE_MEMCPY=0 -e CUDA_VISIBLE_DEVICES=<same ids as Phase 3> \
  local/torchspec:test-vllm-vX.Y.Z \
  examples/qwen3-8b-single-node/run.sh configs/vllm_qwen3_8b.yaml training.num_train_steps=5
```

--------------------------------

### Pass Extra Training Overrides

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Provide additional training parameters as overrides directly when executing the training script.

```bash
# Pass extra overrides
bash examples/kimi-k25-2node-h200/run.sh training.learning_rate=1e-5
```

--------------------------------

### Reference Draft Model Config in Training

Source: https://github.com/lightseekorg/torchspec/blob/main/configs/draft_models/README.md

Use this command to reference a draft model configuration file within your training configuration or as a command-line override.

```bash
python -m torchspec.train_entry --config configs/sglang_qwen3_8b.yaml \
    model.draft_model_config=configs/draft_models/qwen3_8b_eagle3.json
```

--------------------------------

### Configure Per-Rank File Logging

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Set the TORCHSPEC_LOG_DIR environment variable to an absolute path on a shared filesystem to enable per-rank log files for Ray actors.

```bash
export TORCHSPEC_LOG_DIR=/my_project/running_logs
```

--------------------------------

### Training Bottleneck: Data Loading vs. GPU Compute

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

Within the training stage, compare data loading time against GPU compute time. If data loading time is much larger, the bottleneck is likely data transfer or loading. If GPU compute time is larger, the bottleneck is likely the GPU computation itself.

```python
if data_time >> compute_time:
    bottleneck = "mooncake transfer / data loading"
elif compute_time >> data_time:
    bottleneck = "GPU compute (forward/backward/optimizer)"
```

--------------------------------

### Convert Trained Checkpoint to Hugging Face Format

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Convert the trained vLLM model checkpoint to the Hugging Face format for further use or analysis.

```bash
python tools/convert_to_hf.py --input-dir outputs/.../iter_*/ --target-model-path Qwen/Qwen3-8B
```

--------------------------------

### Verify vLLM Docker Image Compatibility

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Check if the new vLLM Docker image tag exists and if its Python version is compatible. This involves inspecting Docker Hub tags and the base image's Dockerfile.

```bash
docker run --rm --entrypoint bash vllm/vllm-openai:vX.Y.Z -c "ldd /usr/local/lib/python3*/dist-packages/mooncake/engine.so | grep cudart"
```

--------------------------------

### Clone TorchSpec Repository

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Clones the TorchSpec project from GitHub and navigates into the project directory.

```bash
git clone https://github.com/torchspec-project/TorchSpec.git
cd TorchSpec
```

--------------------------------

### Convert Checkpoint with Vocabulary Pruning

Source: https://github.com/lightseekorg/torchspec/blob/main/README.md

Convert a checkpoint and prune the vocabulary using `--prune-vocab`. This requires specifying the dataset path, draft vocabulary size, tokenizer, and chat template. Optionally, use `--cache-dir` to reuse tokenized dataset caches.

```bash
python tools/convert_to_hf.py \
    --input-dir ./outputs/my_experiment/iter_0010000/ \
    --prune-vocab \
    --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \
    --draft-vocab-size 32000 \
    --tokenizer Qwen/Qwen3-8B \
    --chat-template qwen \
    --prompt-key conversations
```

--------------------------------

### Tail Latest Training Log

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Continuously displays new output from the most recent training log file. Useful for real-time monitoring of the driver's stdout/stderr.

```bash
tail -f running_logs/<latest>.log
```

--------------------------------

### Upload Checkpoint to Hugging Face

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/dflash/README.md

Optionally upload your extracted checkpoint to a Hugging Face model repository. Replace 'YOUR_ORG/dflash-qwen3-8b' with your organization and model name.

```bash
huggingface-cli upload YOUR_ORG/dflash-qwen3-8b \
  outputs/qwen3-8b-dflash/checkpoints/iter_NNNNNNN/ \
  --repo-type model
```

--------------------------------

### View Monitor Log

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Displays the Ray monitor log, which contains cluster-level events, autoscaling information, and errors. Essential for understanding cluster-wide behavior.

```bash
RAY_TEMP_DIR="/tmp/ray_torchspec_kimi25_$(id -u)"  # adjust per recipe

# Monitor log (cluster-level events, autoscaling, errors)
tail "$RAY_TEMP_DIR/session_latest/logs/monitor.log"
```

--------------------------------

### Connect TorchSpec to Kubernetes Ray Cluster

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Use this command to point TorchSpec to a Ray cluster managed by KubeRay on Kubernetes. Replace placeholders with your cluster's service name and port.

```bash
RAY_ADDRESS=ray://<kuberay-head-svc>:10001 ./examples/kimi-k25-3node-h100/run.sh
```

--------------------------------

### SGLang Extra Arguments via CLI

Source: https://github.com/lightseekorg/torchspec/blob/main/configs/README.md

Configure SGLang engine's `extra_args` for `watchdog_timeout` using the command-line interface. This allows for dynamic configuration without modifying YAML files.

```bash
python -m torchspec.train_entry --config my.yaml inference.sglang.extra_args.watchdog_timeout=1800
```

--------------------------------

### Analyze End-to-End Test Log for Success Criteria

Source: https://github.com/lightseekorg/torchspec/blob/main/skills/vllm-version-bump/SKILL.md

Analyze the captured log file to determine if the end-to-end test passed. Success requires specific log messages, a successful training completion, and a decrease in training loss.

```bash
grep -oE "loss=[0-9.]+" /tmp/bump_e2e.log
```

--------------------------------

### Set Network Interface for NCCL/Gloo

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/ray.md

Configure NCCL and Gloo to use a specific network interface on multi-NIC machines. Find your interface name using the 'ip' command.

```bash
export NCCL_SOCKET_IFNAME=<iface>   # e.g. eth0
export GLOO_SOCKET_IFNAME=<iface>
export TP_SOCKET_IFNAME=<iface>
```

--------------------------------

### Tqdm Progress Bar Metrics (Enabled)

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/performance_metrics.md

When performance metrics are enabled, the progress bar displays detailed information including loss, accuracy, throughput, inference capacity, training capacity, dispatch wait time, sample pool size, and current epoch.

```text
Training: 10%|== | loss=0.350, acc=0.830, thru=12.1, I=18.5, T=13.2, wait=0.1s, pool=24, epoch=1/10
```

--------------------------------

### Override GPU Counts for Training

Source: https://github.com/lightseekorg/torchspec/blob/main/examples/kimi-k25-2node-h200/README.md

Customize the number of GPUs used for training and inference by setting environment variables before running the main training script.

```bash
# Override GPU counts
TRAIN_GPUS=8 INFERENCE_GPUS=8 bash examples/kimi-k25-2node-h200/run.sh
```

--------------------------------

### Check Ray Cluster Status

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Use this command to verify if the Ray cluster is running and accessible. It's useful for initial connectivity checks.

```bash
ray status   # or: ray status --address 127.0.0.1:6380
```

--------------------------------

### AsyncTrainingController Definition

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/code_architecture.md

Defines the structure and key methods of the asynchronous training controller, a Ray actor managing data flow buffers and training queues.

```python
import ray
from collections import deque
from typing import List
from queue import Queue

# Assuming InferenceInput and InferenceOutput are defined elsewhere
# class InferenceInput: ...
# class InferenceOutput: ...

@ray.remote
class AsyncTrainingController:
    # Data flow buffers
    prompt_buffer: deque[InferenceInput]   # Samples waiting for inference
    sample_pool: deque[InferenceOutput]    # Completed inferences (mooncake keys)
    train_queues: List[Queue]              # Per-DP-rank Ray queues

    # Key methods
    def add_dataset(dataset)
    def get_prompts(n)
    def push_inference_results(results)
    def try_dispatch_batch()
```

--------------------------------

### Find Specific Actor Type Logs

Source: https://github.com/lightseekorg/torchspec/blob/main/docs/debugging_ray_jobs.md

Searches for log files containing specific Ray actor types (TrainerActor, SglEngine, MooncakeMaster). Useful for isolating logs from particular components.

```bash
RAY_TEMP_DIR="/tmp/ray_torchspec_kimi25_$(id -u)"  # adjust per recipe

# Find a specific actor type
grep -l "TrainerActor\|SglEngine\|MooncakeMaster" "$RAY_TEMP_DIR/session_latest/logs/worker-*.err"
```