### Setup and Launch Claude-Code

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install Node.js and the Claude-Code package, then configure environment variables and launch the Claude-Code service. This setup is for using Claude-Code with a local service endpoint.

```sh
curl -sL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs && sudo npm install -g @anthropic-ai/claude-code@2.1.83

mkdir -p config/
export ANTHROPIC_BASE_URL="http://0.0.0.0:8000"
export MOCK="api00-local-mock-key"
export ANTHROPIC_API_KEY="sk-ant-${MOCK}"
export CLAUDE_CONFIG_DIR="$(pwd)/config"
echo '{"customApiKeyResponses": {"approved": ["'${MOCK}'"]}}' > config/.claude.json
claude
```

--------------------------------

### Dynamic Configuration Switching Example

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launches the `helloworld_switch` example to demonstrate dynamic configuration switching with PyTorch distributed. This example uses a batch size of 16.

```sh
  >> Example:
    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld_switch --batch_size=16
```

--------------------------------

### Run Custom Gate/Expert Example

Source: https://github.com/microsoft/tutel/blob/main/README.md

Executes the `helloworld_custom_gate_expert` example with a specified batch size. This example is for demonstrating custom gate and expert configurations.

```sh
  >> Example of custom gates/experts:
    python3 -m tutel.examples.helloworld_custom_gate_expert --batch_size=16
```

--------------------------------

### Quick Test on Single-GPU

Source: https://github.com/microsoft/tutel/blob/main/README.md

Run various Tutel examples on a single GPU to verify installation and basic functionality. These tests cover different distribution strategies and features like AMP.

```bash
python3 -m tutel.examples.helloworld --batch_size=16
```

```bash
python3 -m tutel.examples.helloworld_ddp --batch_size=16
```

```bash
python3 -m tutel.examples.helloworld_ddp_tutel --batch_size=16
```

```bash
python3 -m tutel.examples.helloworld_amp --batch_size=16
```

```bash
python3 -m tutel.examples.helloworld_custom_gate_expert --batch_size=16
```

```bash
python3 -m tutel.examples.helloworld_from_scratch
```

```bash
python3 -m tutel.examples.moe_mnist
```

```bash
python3 -m tutel.examples.moe_cifar10
```

```bash
python3 ./tutel/examples/helloworld.py --batch_size=16
```

--------------------------------

### Quick Test for Multiple GPUs across Machines (Tutel Launcher - MPI)

Source: https://github.com/microsoft/tutel/blob/main/README.md

Run Tutel examples across multiple nodes and GPUs using the Tutel launcher with MPI. This requires the `openmpi-bin` package and proper MPI setup.

```bash
mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld_ddp_tutel --batch_size=16
```

```bash
mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.moe_mnist
```

```bash
mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.moe_cifar10
```

```bash
mpiexec -bind-to none -host <node-ip-0>,<node-ip-1>,.. -x MASTER_ADDR=<node-ip-0> -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16
```

```bash
mpiexec -bind-to none -host localhost -x LOCAL_SIZE=1 -x OMP_NUM_THREADS=1024 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16 --device cpu
```

--------------------------------

### Run Tutel Examples with TensorCore Option

Source: https://github.com/microsoft/tutel/blob/main/README.md

Demonstrates running the `helloworld` example with different data types (float32, float16) and toggling the `use_tensorcore` option. This is useful for benchmarking tensor core performance.

```sh
  >> Example of using tensorcore:

    python3 -m tutel.examples.helloworld --dtype=float32
    python3 -m tutel.examples.helloworld --dtype=float32 --use_tensorcore

    python3 -m tutel.examples.helloworld --dtype=float16
    python3 -m tutel.examples.helloworld --dtype=float16 --use_tensorcore
```

--------------------------------

### Install Hugging Face Hub CLI

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install the Hugging Face Hub command-line interface with support for the CLI.

```bash
pip3 install -U "huggingface_hub[cli]" --upgrade
```

--------------------------------

### Megablocks MoE Examples with Different Block Sizes

Source: https://github.com/microsoft/tutel/blob/main/README.md

Demonstrates running the `helloworld` example with Megablocks enabled for improved decoder inference. `capacity_factor=0` is required for dropless-MoE. Examples include BatchMatmul, Megablocks with block_size=1, and block_size=2.

```sh
  >> Example (capacity_factor=0 required by dropless-MoE):
    # Using BatchMatmul:
    python3 -m tutel.examples.helloworld --megablocks_size=0 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0
    # Using Megablocks with block_size = 1:
    python3 -m tutel.examples.helloworld --megablocks_size=1 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0
    # Using Megablocks with block_size = 2:
    python3 -m tutel.examples.helloworld --megablocks_size=2 --batch_size=1 --num_tokens=32 --top=1 --eval --num_local_experts=128 --capacity_factor=0
```

--------------------------------

### Qwen3 Fast MoE Gating Example

Source: https://github.com/microsoft/tutel/blob/main/README.md

Demonstrates the use of Qwen3's fast MoE scaled top-k gating function from the tutel.ops module. This example normalizes routed weights to 1.0 and prints the resulting weights, IDs, and their sum.

```python
import torch
from tutel import ops

# Qwen3 Fast MoE Gating for 128 Experts, with Routed Weights normalized to 1.0
logits_fp32 = torch.softmax(torch.randn([32, 128]), -1, dtype=torch.float32).cuda()
topk_weights, topk_ids = ops.qwen3_moe_scaled_topk(logits_fp32)
print(topk_weights, topk_ids, topk_weights.sum(-1))
```

--------------------------------

### Build Tutel from Source

Source: https://github.com/microsoft/tutel/blob/main/README.md

Clone the Tutel repository and install it locally. This method is useful for development or when direct online installation is not feasible.

```bash
git clone https://github.com/microsoft/tutel --branch main
python3 -m pip uninstall tutel -y
python3 ./tutel/setup.py install --user
```

--------------------------------

### Run NCCL Timeout Settings Example

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launches the `helloworld` example with a custom NCCL timeout setting. The `TUTEL_GLOBAL_TIMEOUT_SEC` environment variable controls the timeout duration.

```sh
  >> Example of NCCL timeout settings:
    TUTEL_GLOBAL_TIMEOUT_SEC=60 python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --use_tensorcore
```

--------------------------------

### Run NCCL All-to-All_v and All_Gather_v Examples

Source: https://github.com/microsoft/tutel/blob/main/README.md

Executes examples for NCCL's `all_to_all_v` and `all_gather_v` operations using PyTorch distributed. Requires at least 2 processes per node.

```sh
  >> Example:
    # All_to_All_v:
    python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_to_all_v
    # All_Gather_v:
    python3 -m torch.distributed.run --nproc_per_node=2 --master_port=7340 -m tutel.examples.nccl_all_gather_v
```

--------------------------------

### Prepare PyTorch for CPU

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install PyTorch for CPU-only execution. This is suitable when GPUs are not available.

```bash
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Install Tutel Online

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install Tutel directly from the GitHub repository. This command ensures you get the latest development version.

```bash
python3 -m pip uninstall tutel -y
python3 -m pip install -v -U --no-build-isolation git+https://github.com/microsoft/tutel@main
```

--------------------------------

### Download Pretrained SWIN-Transformer Checkpoint

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Download a pretrained SWIN-Transformer checkpoint file specifically for a 32-GPU setup with 32 experts.

```sh
curl -LO https://github.com/SwinTransformer/storage/releases/download/v2.0.2/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.zip
```

```sh
unzip swin_moe_small_patch4_window12_192_32expert_32gpu_22k.zip
```

--------------------------------

### Single-Node Training with Tutel's MPI Launcher

Source: https://context7.com/microsoft/tutel/llms.txt

Launches a distributed training job on a single node using Tutel's MPI-based launcher. This requires `openmpi-bin` to be installed.

```bash
# Using Tutel's MPI-based launcher (requires openmpi-bin)
mpiexec -bind-to none -host localhost -x LOCAL_SIZE=8 \
    python3 -m tutel.launcher.run \
    -m tutel.examples.helloworld_ddp_tutel --batch_size=16
```

--------------------------------

### Training Log Output

Source: https://github.com/microsoft/tutel/blob/main/tutel/examples/modded-nanogpt-moe/README.md

Example log output during training, showing validation loss and step time.

```txt
...
[DEBUG(weights-init)] Validating all parameters are in proper states..
[DEBUG(weights-init)] Validation passed. ✅
[DEBUG(gradients)] Validating all parameters are in proper states..
[DEBUG(gradients)] Validation passed. ✅
[DEBUG(weights-upd)] Validating all parameters are in proper states..
[DEBUG(weights-upd)] Validation passed. ✅
step:50/1750 val_loss:7.6209 val_balance_loss:8.5625 step_time:153.01ms
step:100/1750 val_loss:7.1094 val_balance_loss:7.7812 step_time:137.49ms
step:150/1750 val_loss:6.7864 val_balance_loss:7.7188 step_time:138.65ms
step:200/1750 val_loss:6.6302 val_balance_loss:7.5625 step_time:139.24ms
step:250/1750 val_loss:6.4994 val_balance_loss:6.7812 step_time:139.54ms
step:300/1750 val_loss:6.4303 val_balance_loss:6.7500 step_time:139.80ms
step:350/1750 val_loss:6.3354 val_balance_loss:4.5000 step_time:140.29ms
step:400/1750 val_loss:6.2024 val_balance_loss:4.2812 step_time:140.41ms
step:450/1750 val_loss:6.1186 val_balance_loss:4.2812 step_time:140.39ms
step:500/1750 val_loss:6.0563 val_balance_loss:4.2812 step_time:140.42ms
step:550/1750 val_loss:5.9913 val_balance_loss:4.2812 step_time:144.79ms
step:600/1750 val_loss:5.9157 val_balance_loss:4.2812 step_time:140.01ms
step:650/1750 val_loss:5.8420 val_balance_loss:4.2812 step_time:139.76ms
step:700/1750 val_loss:5.7899 val_balance_loss:3.9219 step_time:139.85ms
...
```

--------------------------------

### Quick Test for Multiple GPUs across Machines (Torch Launcher)

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch Tutel examples in a multi-node, multi-GPU configuration using PyTorch's distributed run. This requires SSH access and proper network configuration between nodes.

```bash
ssh <node-ip-0> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=2 --node_rank=0 --master_addr=<node-ip-0> -m tutel.examples.helloworld --batch_size=16
ssh <node-ip-1> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=2 --node_rank=1 --master_addr=<node-ip-0> -m tutel.examples.helloworld --batch_size=16
```

--------------------------------

### Download Models with Hugging Face CLI

Source: https://github.com/microsoft/tutel/blob/main/README.md

Use the Hugging Face CLI to download model weights locally. Ensure you have the latest version of `huggingface_hub` installed.

```sh
pip3 install -U "huggingface_hub[cli]" --upgrade
hf download --local-dir lukealonso/GLM-5.1-NVFP4 lukealonso/GLM-5.1-NVFP4
hf download --local-dir nvidia/GLM-5-NVFP4 nvidia/GLM-5-NVFP4
```

```sh
pip3 install -U "huggingface_hub[cli]" --upgrade
hf download nvidia/Kimi-K2.5-NVFP4 --local-dir nvidia/Kimi-K2.5-NVFP4
hf download nvidia/Kimi-K2-Thinking-NVFP4 --local-dir nvidia/Kimi-K2-Thinking-NVFP4
hf download nvidia/DeepSeek-V3.2-NVFP4 --local-dir nvidia/DeepSeek-V3.2-NVFP4
```

--------------------------------

### Quick Test for 8 GPUs within 1 Machine

Source: https://github.com/microsoft/tutel/blob/main/README.md

Execute Tutel examples across 8 GPUs on a single machine using PyTorch's distributed run utility.

```bash
python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16
```

--------------------------------

### Run All-to-All Benchmark

Source: https://github.com/microsoft/tutel/blob/main/README.md

Executes a bandwidth test for all-to-all communication using PyTorch distributed. Ensure `tutel` is installed and accessible.

```sh
  >> Example:

    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.bandwidth_test --size_mb=256
```

--------------------------------

### Distributed Training on 32 GPUs with Adapted Checkpoints

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Launch distributed training across multiple nodes using checkpoints adapted for 32 GPUs. This example demonstrates launching training on 4 nodes, each with 8 GPUs.

```sh
ssh <node-ip-0> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=4 --node_rank=0 --master_addr=<node-ip-0> -m tutel.examples.helloworld --num_local_experts=-2 --checkpoint=./adapted-for-32-gpus/{rank}-of-{size}.ckpt --device=cuda
```

```sh
ssh <node-ip-1> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=4 --node_rank=1 --master_addr=<node-ip-0> -m tutel.examples.helloworld --num_local_experts=-2 --checkpoint=./adapted-for-32-gpus/{rank}-of-{size}.ckpt --device=cuda
```

```sh
ssh <node-ip-2> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=4 --node_rank=2 --master_addr=<node-ip-0> -m tutel.examples.helloworld --num_local_experts=-2 --checkpoint=./adapted-for-32-gpus/{rank}-of-{size}.ckpt --device=cuda
```

```sh
ssh <node-ip-3> python3 -m torch.distributed.run --nproc_per_node=8 --nnodes=4 --node_rank=3 --master_addr=<node-ip-0> -m tutel.examples.helloworld --num_local_experts=-2 --checkpoint=./adapted-for-32-gpus/{rank}-of-{size}.ckpt --device=cuda
```

--------------------------------

### Import Tutel-optimized MoE in PyTorch

Source: https://github.com/microsoft/tutel/blob/main/README.md

Example demonstrating how to import and initialize a Tutel-optimized MoE layer within a PyTorch model. This includes setting up the MoE layer with specific configurations and casting it to the appropriate device.

```python
import torch
x = torch.ones([6, 1024], device='cuda:0')

from tutel import moe as tutel_moe
moe_layer = tutel_moe.moe_layer(
    gate_type={'type': 'top', 'k': 2},
    model_dim=x.shape[-1],
    experts={
        'num_experts_per_device': 2,
        'type': 'ffn', 'hidden_size_per_expert': 2048, 'activation_fn': lambda x: torch.nn.functional.relu(x)
    },
    scan_expert_func = lambda name, param: setattr(param, 'skip_allreduce', True),
)

moe_layer = moe_layer.to('cuda:0')
```

--------------------------------

### Single-Node Multi-GPU Training with torch.distributed.run

Source: https://context7.com/microsoft/tutel/llms.txt

Launches a distributed training job on a single node with multiple GPUs using torch.distributed.run. This is a common setup for development and smaller clusters.

```bash
python3 -m torch.distributed.run --nproc_per_node=8 \
    -m tutel.examples.helloworld \
    --batch_size=16 \
    --num_tokens=512 \
    --model_dim=2048 \
    --hidden_size=2048 \
    --num_local_experts=4 \
    --top=2 \
    --dtype=bfloat16 \
    --capacity_factor=1.5
```

--------------------------------

### Prepare PyTorch for NVIDIA CUDA

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install PyTorch with CUDA support for NVIDIA GPUs. Ensure your CUDA version is compatible.

```bash
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
```

--------------------------------

### Prepare PyTorch for AMD ROCm

Source: https://github.com/microsoft/tutel/blob/main/README.md

Install PyTorch with ROCm support for AMD GPUs. Ensure your ROCm version is compatible.

```bash
python3 -m pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2.2
```

--------------------------------

### 2DH Option for All-to-All at Scale

Source: https://github.com/microsoft/tutel/blob/main/README.md

Example demonstrating the use of the `--use_2dh` flag for optimizing All-to-All communication at scale. Requires the number of processes per node to match the total physical GPU count.

```sh
  >> Example (suggest enabling 2DH only at scale, note that the value of --nproc_per_node MUST equal to total physical GPU counts per node, e.g. 8 for A100x8):
    python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --batch_size=16 --use_2dh
```

--------------------------------

### Direct Request to Tutel Service

Source: https://github.com/microsoft/tutel/blob/main/README.md

Send a POST request to the Tutel service endpoint for chat interactions. This example shows how to request a Python Quicksort algorithm implementation.

```sh
curl -N -X POST http://0.0.0.0:8000/chat -d '{"text": "Write a Python code of the Quicksort algorithm."}'
```

--------------------------------

### Adapt All-in-One Checkpoint for 32 GPUs

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Convert the All-in-One checkpoint file for a distributed setup with 32 GPUs. This command supports conversion where the number of GPUs is a multiple of the number of global experts.

```sh
python3 -m tutel.checkpoint.scatter --input=./model-all-in-one.ckpt --output_size=32 --outputs=./adapted-for-32-gpus/{rank}-of-{size}.ckpt
```

--------------------------------

### Get Local Session Information

Source: https://context7.com/microsoft/tutel/llms.txt

Retrieves the current session and prints its global rank. This can be called from anywhere in the code.

```python
session = system.get_local_session()
print(f"Current session rank: {session.global_rank}")
```

--------------------------------

### Convert Distributed Checkpoints to All-in-One

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Gather checkpoint files from a distributed training setup into a single file. This is useful for testing on a single device or preparing for different distributed configurations.

```sh
python3 -m tutel.checkpoint.gather --inputs=./states/{rank}-of-{size}.ckpt --input_size=2 --output ./model-all-in-one.ckpt
```

--------------------------------

### Training Loop with MoE Auxiliary Loss

Source: https://context7.com/microsoft/tutel/llms.txt

Example of a standard PyTorch training loop that includes calculating and adding the MoE auxiliary loss. Ensure your model has the 'moe.l_aux' attribute.

```python
x = torch.randint(0, 10000, (32, 128), device=device)
labels = torch.randint(0, 10000, (32, 128), device=device)

for step in range(100):
    optimizer.zero_grad()

    logits = model(x)
    loss = F.cross_entropy(logits.view(-1, 10000), labels.view(-1))

    # Add MoE auxiliary loss
    if model.moe.l_aux is not None:
        loss = loss + 0.01 * model.moe.l_aux

    loss.backward()
    optimizer.step()  # Handles gradient chunking, local step, and parameter restore

    if step % 10 == 0:
        parallel_env.dist_print(f"Step {step}, Loss: {loss.item():.4f}")
```

--------------------------------

### Copy Remaining SWIN-Transformer Parameters

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

After converting the distributed SWIN-Transformer checkpoint, copy the master parameters file to the new directory to complete the checkpoint setup for the target GPU configuration.

```sh
cp ./swin_moe_small_patch4_window12_192_32expert_32gpu_22k/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.pth.master ./new_swin_moe_small_for_2_gpus/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.pth.master
```

--------------------------------

### Stream Request with OAI Example

Source: https://github.com/microsoft/tutel/blob/main/README.md

Execute a Python script to make a streaming chat request to the Tutel service using an OpenAI-compatible endpoint. This is useful for interactive applications where responses are received chunk by chunk.

```python
python3 -m tutel.examples.oai_request_stream --url '0.0.0.0:8000' --prompt 'Write a Python code of the Quicksort algorithm.'
```

--------------------------------

### Multi-Node Training with Tutel's MPI Launcher

Source: https://context7.com/microsoft/tutel/llms.txt

Configures a multi-node distributed training job using Tutel's MPI launcher. Specify all participating hosts and the master node address.

```bash
# Multi-node with MPI launcher
mpiexec -bind-to none \
    -host node0,node1,node2,node3 \
    -x MASTER_ADDR=node0 \
    -x LOCAL_SIZE=8 \
    python3 -m tutel.launcher.run \
    -m tutel.examples.helloworld --batch_size=16
```

--------------------------------

### Run VibeVoice Inference with Docker (MI300)

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container for VibeVoice inference on MI300 GPUs. Includes specific device mappings for AMD GPUs and necessary security options. Mounts host directories and sets environment variables for model paths.

```bash
docker run -e LOCAL_SIZE=1 -it --rm -p 8001:8000 --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/kfd --device=/dev/dri --group-add=video \
      --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      -e VOICES="https://homepages.inf.ed.ac.uk/htang2/notes/speech-samples/103-1240-0000.wav" \
      tutelgroup/deepseek-671b:mi300x8-chat-20251222 --serve=core \
        --try_path ./microsoft/VibeVoice-1.5B \
        --try_path ./microsoft/VibeVoice-Large
```

--------------------------------

### Download Models with Hugging Face Hub CLI

Source: https://github.com/microsoft/tutel/blob/main/README.md

Download specific models from the Hugging Face Hub to a local directory using the CLI.

```bash
hf download microsoft/VibeVoice-1.5B --local-dir microsoft/VibeVoice-1.5B
```

```bash
hf download Qwen/Qwen2.5-1.5B --local-dir Qwen/Qwen2.5-1.5B
```

```bash
hf download microsoft/VibeVoice-Large --local-dir aoi-ot/VibeVoice-Large
```

```bash
hf download Qwen/Qwen2.5-7B --local-dir Qwen/Qwen2.5-7B
```

--------------------------------

### Run VibeVoice Inference with Docker (A100/H100/B200)

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container for VibeVoice inference on A100/H100/B200 GPUs. Mounts host directories and sets environment variables for model paths and CUDA. Ensure the host has the necessary libcuda.so.1.

```bash
docker run -e LOCAL_SIZE=1 -it --rm -p 8001:8000 --shm-size=8g \
      --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
      -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
      -e VOICES="https://homepages.inf.ed.ac.uk/htang2/notes/speech-samples/103-1240-0000.wav" \
      tutelgroup/deepseek-671b:a100x8-chat-20251222 --serve=core \
        --try_path ./microsoft/VibeVoice-1.5B \
        --try_path ./microsoft/VibeVoice-Large
```

--------------------------------

### Download and Serve Deepseek R1 Model with Docker

Source: https://github.com/microsoft/tutel/blob/main/README.md

Steps to download the Deepseek R1 671B model and serve it using Docker on AMD MI300 GPUs. Includes instructions for issuing a prompt request via curl.

```sh
  >> Example:

    # Step-1: Download Deepseek R1 671B Model
    huggingface-cli download deepseek-ai/DeepSeek-R1 --local-dir ./deepseek-ai/DeepSeek-R1

    # Step-2: Using 8 MI300 GPUs to Serve Deepseek R1 Chat on Local Port :8000
    docker run -it --rm --ipc=host --privileged -p 8000:8000 \
        -v /:/host -w /host$(pwd) tutelgroup/deepseek-671b:mi300x8-chat-20250224 \
        --model_path ./deepseek-ai/DeepSeek-R1

    # Step-3: Issue a Prompt Request with curl
    curl -X POST http://0.0.0.0:8000/chat -d '{"text": "Calculate the result of: 1 / (sqrt(5) - sqrt(3))"}'
```

--------------------------------

### Multi-Node Training with torch.distributed.run (Node 0)

Source: https://context7.com/microsoft/tutel/llms.txt

Initiates a multi-node distributed training job from the master node (node 0) using torch.distributed.run. Requires specifying the number of nodes and master address.

```bash
# On node 0:
python3 -m torch.distributed.run \
    --nproc_per_node=8 \
    --nnodes=2 \
    --node_rank=0 \
    --master_addr=192.168.1.1 \
    -m tutel.examples.helloworld --batch_size=16
```

--------------------------------

### Initialize Distributed Environment

Source: https://context7.com/microsoft/tutel/llms.txt

Initializes the distributed data-parallel environment using the NCCL backend. This is a prerequisite for distributed training.

```python
parallel_env = system.init_data_model_parallel(backend='nccl')
device = parallel_env.local_device
```

--------------------------------

### Forward Pass with MOELayer

Source: https://github.com/microsoft/tutel/blob/main/README.md

Demonstrates a basic forward pass using the MOELayer. Ensure the MOELayer is properly initialized before use.

```python
y = moe_layer(x)

print(y)
```

--------------------------------

### Run MoE or Dense Training

Source: https://github.com/microsoft/tutel/blob/main/tutel/examples/modded-nanogpt-moe/README.md

Use these commands to execute the training script, toggling between MoE and Dense layer configurations.

```sh
# run MoE training with 8 local GPUs
USE_MOE=1 ./run.sh
```

```sh
# run Dense training with 8 local GPUs
USE_MOE=0 ./run.sh
```

--------------------------------

### Dropless MoE with Megablocks Acceleration

Source: https://context7.com/microsoft/tutel/llms.txt

Configures a training job for dropless MoE by setting `capacity_factor=0` and enables Megablocks acceleration. This is suitable for scenarios where expert capacity is not a bottleneck.

```bash
# Dropless MoE (capacity_factor=0) with Megablocks acceleration
python3 -m torch.distributed.run --nproc_per_node=8 \
    -m tutel.examples.helloworld \
    --capacity_factor=0 \
    --megablocks_size=1 \
    --num_local_experts=128 \
    --eval
```

--------------------------------

### Dynamic Configuration Parameters for MoE Layer

Source: https://github.com/microsoft/tutel/blob/main/README.md

Illustrates how to dynamically switch configurations in the MoE layer's forward pass. Parameters include `a2a_ffn_overlap_degree`, `adaptive_r`, `capacity_factor`, and `top_k`.

```python
    self._moe_layer.forward(x, .., a2a_ffn_overlap_degree=2)  # Control the switch of overlap granularity (1 for no overlapping)
    self._moe_layer.forward(x, .., adaptive_r=1)              # Control the switch of parallelism (0 for DP, 1 for DP + EP, W / E for MP + EP, else for DP + MP + EP)
    self._moe_layer.forward(x, .., capacity_factor=1)         # Control the switch of capacity_volume (positive for padding, negative for no-padding, 0 for dropless)
    self._moe_layer.forward(x, .., top_k=1)                   # Control the switch of top_k sparsity
```

--------------------------------

### Multi-Node Training with torch.distributed.run (Node 1)

Source: https://context7.com/microsoft/tutel/llms.txt

Configures a worker node (node 1) to join a multi-node distributed training job initiated by torch.distributed.run. Ensure the master address matches the master node.

```bash
# On node 1:
python3 -m torch.distributed.run \
    --nproc_per_node=8 \
    --nnodes=2 \
    --node_rank=1 \
    --master_addr=192.168.1.1 \
    -m tutel.examples.helloworld --batch_size=16
```

--------------------------------

### Initialize Distributed Training Environment with init_data_model_parallel

Source: https://context7.com/microsoft/tutel/llms.txt

Initialize the distributed training environment for MoE workloads using init_data_model_parallel. This function sets up communication groups, assigns devices, and configures NUMA affinity.

```python
import torch
from tutel import system

# Basic initialization for single-node multi-GPU training
parallel_env = system.init_data_model_parallel(
    group_count=1,           # Number of model parallel groups (1 = pure expert parallelism)
    backend='nccl'           # Communication backend ('nccl' for GPU, 'gloo' for CPU)
)

# Access distributed properties
print(f"Global rank: {parallel_env.global_rank}")
print(f"Global size: {parallel_env.global_size}")
print(f"Local device: {parallel_env.local_device}")
print(f"Data rank: {parallel_env.data_rank}")
print(f"Model rank: {parallel_env.model_rank}")
print(f"Is distributed: {parallel_env.is_distributed}")

# Use the distributed print helper (only rank 0 prints)
parallel_env.dist_print("This message appears only once across all processes")

# Access communication groups
model_group = parallel_env.model_group  # For expert parallelism
data_group = parallel_env.data_group    # For data parallelism

# Timing utility that handles CUDA synchronization
t_start = system.record_time()
```

--------------------------------

### Create MoE Model and Mark Expert Parameters

Source: https://context7.com/microsoft/tutel/llms.txt

Initializes a Mixture-of-Experts (MoE) model and marks parameters that should be skipped during all-reduce operations in distributed training.

```python
model = MoEModel(model_dim=512, hidden_size=2048, num_local_experts=4, top_k=2).to(device)

for name, param in model.named_parameters():
    if hasattr(param, 'skip_allreduce'):
        model.add_param_to_skip_allreduce(name)

# Wrap with DDP
if dist.is_initialized():
    model = torch.nn.parallel.DistributedDataParallel(
        model, device_ids=[parallel_env.local_rank]
    )

# Training loop
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
x = torch.randn(32, 128, 512, device=device)
labels = torch.randint(0, 10, (32,), device=device)

for step in range(100):
    optimizer.zero_grad()
    logits = model(x)
    loss = F.cross_entropy(logits, labels)

    # Add auxiliary load balancing loss
    moe_module = model.module._moe_layer if hasattr(model, 'module') else model._moe_layer
    if moe_module.l_aux is not None:
        loss = loss + 0.01 * moe_module.l_aux

    loss.backward()
    optimizer.step()

    if step % 10 == 0:
        parallel_env.dist_print(f"Step {step}, Loss: {loss.item():.4f}")
```

--------------------------------

### Run DeepSeek V3.2 Long-Context on A100/H100/B200

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container to serve DeepSeek V3.2 models with long-context capabilities on NVIDIA A100, H100, or B200 GPUs. This command configures the service for web UI access, specifies multiple model paths, and sets a maximum sequence length.

```sh
docker run -e LOCAL_SIZE=8 -e WORKER=1 -it --rm --ipc=host --net=host --shm-size=8g \
    --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
    -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
    tutelgroup/deepseek-671b:a100x8-chat-20260327 --serve=webui --listen_port 8000 \
      --try_path nvidia/Kimi-K2.5-NVFP4 \
      --try_path nvidia/Kimi-K2-Thinking-NVFP4 \
      --try_path nvidia/DeepSeek-V3.2-NVFP4 \
      --try_path nvidia/DeepSeek-R1-NVFP4 \
      --max_seq_len 32768
```

--------------------------------

### Run GLM-5/5.1 Chat Service on A100/H100/B200

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container to serve GLM-5/5.1 models with NVFP4 quantization on NVIDIA A100, H100, or B200 GPUs. This command configures the service for web UI access and specifies model paths and maximum sequence length.

```sh
docker run -e LOCAL_SIZE=8 -p 8000:8000 -it --rm --ipc=host --shm-size=8g \
    --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
    -v /usr/lib/x86_64-linux-gnu/libcuda.so.1:/usr/lib/x86_64-linux-gnu/libcuda.so.1 --privileged \
    tutelgroup/deepseek-671b:a100x8-chat-20260411 --serve=webui \
      --try_path lukealonso/GLM-5.1-NVFP4 \
      --try_path nvidia/GLM-5-NVFP4 \
      --max_seq_len 128000
```

--------------------------------

### Run DeepSeek V3.2 Long-Context on MI300

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container to serve DeepSeek V3.2 models with long-context capabilities on AMD MI300 GPUs. This command configures the service for web UI access, specifies multiple model paths, and sets a very large maximum sequence length, including necessary device and capability flags for ROCm.

```sh
docker run -e LOCAL_SIZE=8 -e WORKER=1 -it --rm --ipc=host --net=host --shm-size=8g \
    --ulimit memlock=-1 --ulimit stack=67108864 --device=/dev/kfd --device=/dev/dri --group-add=video \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v /:/host -w /host$(pwd) -v /tmp:/tmp \
    tutelgroup/deepseek-671b:mi300x8-chat-20260327 --serve=webui --listen_port 8000 \
      --try_path nvidia/Kimi-K2.5-NVFP4 \
      --try_path nvidia/Kimi-K2-Thinking-NVFP4 \
      --try_path nvidia/DeepSeek-V3.2-NVFP4 \
      --try_path nvidia/DeepSeek-R1-NVFP4 \
      --max_seq_len 1000000
```

--------------------------------

### Enable 2D Hierarchical All-to-All

Source: https://context7.com/microsoft/tutel/llms.txt

Enables 2D hierarchical all-to-all communication for large-scale distributed training. This can improve performance by optimizing communication patterns.

```bash
LOCAL_SIZE=8 python3 -m torch.distributed.run --nproc_per_node=8 \
    -m tutel.examples.helloworld \
    --batch_size=16 \
    --use_2dh
```

--------------------------------

### Train/Eval with Adapted Checkpoints for 8 GPUs

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Use the checkpoint files adapted for 8 GPUs to train or evaluate a model. The number of local experts will be adjusted accordingly.

```sh
python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.helloworld --num_local_experts=2 --checkpoint=./adapted-for-8-gpus/{rank}-of-{size}.ckpt --device=cuda
```

--------------------------------

### Run GLM-5/5.1 Chat Service on MI300

Source: https://github.com/microsoft/tutel/blob/main/README.md

Launch a Docker container to serve GLM-5/5.1 models with NVFP4 quantization on AMD MI300 GPUs. This command configures the service for web UI access and specifies model paths and maximum sequence length, including necessary device and capability flags for ROCm.

```sh
docker run -e LOCAL_SIZE=8 -p 8000:8000 -it --rm --ipc=host --shm-size=8g \
    --ulimit memlock=-1 --ulimit stack=67108864 -v /:/host -w /host$(pwd) -v /tmp:/tmp \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device=/dev/kfd --device=/dev/dri --group-add=video \
    tutelgroup/deepseek-671b:mi300x8-chat-20260411 --serve=webui \
      --try_path lukealonso/GLM-5.1-NVFP4 \
      --try_path nvidia/GLM-5-NVFP4 \
      --max_seq_len 128000
```

--------------------------------

### Train Model with 2 GPUs and Save Checkpoints

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Use this command to train a model with a specified number of local experts and save checkpoint files. Ensure the output path uses the {rank} and {size} placeholders for distributed saving.

```sh
python3 -m torch.distributed.run --nproc_per_node=2 -m tutel.examples.helloworld --num_local_experts=8 --checkpoint=./states/{rank}-of-{size}.ckpt --device=cuda
```

--------------------------------

### Adapt All-in-One Checkpoint for 8 GPUs

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Convert the All-in-One checkpoint file to be compatible with distributed training using a different number of GPUs. Specify the desired output size (number of GPUs).

```sh
python3 -m tutel.checkpoint.scatter --input=./model-all-in-one.ckpt --output_size=8 --outputs=./adapted-for-8-gpus/{rank}-of-{size}.ckpt
```

--------------------------------

### Define Model with Tutel MoE Layer and Initialize Distributed Optimizer

Source: https://context7.com/microsoft/tutel/llms.txt

Defines a PyTorch model incorporating Tutel's MoE layer and initializes TutelDistributedOptimizer for ZeRO-style optimization, sharding optimizer states and gradients across GPUs.

```python
import torch
import torch.nn.functional as F
from tutel import system
from tutel import net
from tutel import moe as tutel_moe

parallel_env = system.init_data_model_parallel(backend='nccl')
device = parallel_env.local_device

# Create model with MoE layer
class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.embed = torch.nn.Embedding(10000, 512)
        self.moe = tutel_moe.moe_layer(
            gate_type={'type': 'top', 'k': 2},
            experts={'type': 'ffn', 'num_experts_per_device': 4, 'hidden_size_per_expert': 2048},
            model_dim=512,
            scan_expert_func=lambda n, p: setattr(p, 'skip_allreduce', True)
        )
        self.output = torch.nn.Linear(512, 10000)

    def forward(self, x):
        x = self.embed(x)
        x = self.moe(x)
        return self.output(x)

model = Model().to(device)

# Create ZeRO-style distributed optimizer
# Wraps a local optimizer with parameter and gradient sharding
tutel_optim = net.TutelDistributedOptimizer(
    model.parameters(),
    group=None,           # Use default world group
    average_shared=True   # Average shared gradients across workers
)
```

--------------------------------

### Test All-in-One Checkpoint on Single CPU Device

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Evaluate the All-in-One checkpoint file on a single CPU device. Note that the number of local experts will be adjusted based on the total global experts.

```sh
python3 -m tutel.examples.helloworld --num_local_experts=16 --checkpoint=./model-all-in-one.ckpt --device=cpu --eval
```

--------------------------------

### Create MoE Layer with Custom Components

Source: https://context7.com/microsoft/tutel/llms.txt

Instantiates an MoE layer using custom gate and expert modules. It configures the layer with specific parameters and a scan function to skip all-reduce for expert parameters.

```python
moe_layer = tutel_moe.moe_layer(
    gate_type={
        'type': 'custom',
        'module': CustomGate,
        'top_k': 2
    },
    experts={
        'type': 'custom',
        'module': CustomExpert,
        'num_experts_per_device': num_local_experts,
        'hidden_dim': model_dim * 4
    },
    model_dim=model_dim,
    scan_expert_func=lambda name, param: setattr(param, 'skip_allreduce', True)
).to(parallel_env.local_device)
```

--------------------------------

### Generate Audio with VibeVoice API

Source: https://github.com/microsoft/tutel/blob/main/README.md

Send a POST request to the VibeVoice inference server to generate audio from text. The request body contains the text to be synthesized, and the output is saved to sound_output.mp3.

```bash
curl -X POST http://0.0.0.0:8001/chat -d '{"text": "VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text."}' > sound_output.mp3
```

--------------------------------

### DeepSeek V3/R1 Fast MoE Gating

Source: https://github.com/microsoft/tutel/blob/main/README.md

Demonstrates the use of `deepseek_moe_sigmoid_scaled_topk` for fast MoE gating with normalized routed weights. Requires CUDA-enabled PyTorch tensors.

```python
logits_bf16 = torch.randn([32, 256], dtype=torch.bfloat16).cuda()
correction_bias_bf16 = torch.randn([logits_bf16.size(-1)], dtype=torch.bfloat16).cuda()
topk_weights, topk_ids = ops.deepseek_moe_sigmoid_scaled_topk(logits_bf16, correction_bias_bf16, None, None)
print(topk_weights, topk_ids, topk_weights.sum(-1))
```

--------------------------------

### Custom Expert Module Implementation

Source: https://context7.com/microsoft/tutel/llms.txt

Defines a custom expert module using a GLU-style activation. This module allows for flexible expert computation logic.

```python
class CustomExpert(torch.nn.Module):
    def __init__(self, model_dim, num_experts_per_device, sharded_count, **options):
        super().__init__()
        self.model_dim = model_dim
        self.num_experts = num_experts_per_device
        hidden_dim = options.get('hidden_dim', model_dim * 4)

        # GLU-style expert with gating
        self.W_gate = torch.nn.Parameter(torch.randn(num_experts_per_device, model_dim, hidden_dim) * 0.01)
        self.W_up = torch.nn.Parameter(torch.randn(num_experts_per_device, model_dim, hidden_dim) * 0.01)
        self.W_down = torch.nn.Parameter(torch.randn(num_experts_per_device, hidden_dim, model_dim) * 0.01)

    def forward(self, x, ctx):
        # x shape: [num_experts, capacity, model_dim]
        gate = torch.sigmoid(torch.matmul(x, self.W_gate))
        up = torch.matmul(x, self.W_up)
        hidden = gate * up  # GLU activation
        return torch.matmul(hidden, self.W_down)
```

--------------------------------

### Convert SWIN-Transformer Checkpoint to All-in-One for Single-GPU

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Convert a SWIN-Transformer checkpoint, originally for distributed training, into an All-in-One format suitable for single-GPU use. This involves gathering parameters from multiple rank-specific files.

```sh
python3 -m tutel.checkpoint.gather --namespace model --default_num_global_experts=32 --inputs=./swin_moe_small_patch4_window12_192_32expert_32gpu_22k/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.pth.rank{rank} --input_size=32 --output ./swin-moe-small-all-in-one.ckpt
```

--------------------------------

### Convert SWIN-Transformer All-in-One Checkpoint for 2 GPUs

Source: https://github.com/microsoft/tutel/blob/main/doc/CHECKPOINT.md

Adapt an All-in-One SWIN-Transformer checkpoint to a distributed format for 2 GPUs. This command scatters the parameters into new rank-specific files.

```sh
python3 -m tutel.checkpoint.scatter --namespace model --input=./swin-moe-small-all-in-one.ckpt --output_size=2 --outputs=./new_swin_moe_small_for_2_gpus/swin_moe_small_patch4_window12_192_32expert_32gpu_22k.pth.rank{rank}
```