### Setup development environment

Source: https://github.com/pytorch/torchtitan/blob/main/CONTRIBUTING.md

Installs the necessary dependencies for the project, including both runtime and development requirements.

```bash
pip install -r requirements.txt -r requirements-dev.txt
```

--------------------------------

### Install TorchFT

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/ft/torchft.md

Instructions for installing the TorchFT library. This can be done by cloning the repository and following its README or by using pip.

```bash
pip install torchft-nightly
```

--------------------------------

### Quick Start Training with GraphTrainer

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/graph_trainer/README.md

Launches the training process for specified models using GraphTrainer. Requires setting MODULE and CONFIG environment variables.

```bash
MODULE=graph_trainer.llama3 CONFIG=graph_trainer_llama3_8b ./run_train.sh
```

```bash
MODULE=graph_trainer.deepseek_v3 CONFIG=graph_trainer_deepseek_v3_16b ./run_train.sh
```

--------------------------------

### Example Torchtitan Checkpoint Configuration

Source: https://github.com/pytorch/torchtitan/blob/main/docs/checkpoint.md

A comprehensive example configuration for Torchtitan checkpointing, demonstrating settings for interval, loading a specific step, saving only the model, and export data type.

```python
checkpoint=CheckpointManager.Config(
    interval=10,
    load_step=5,
    last_save_model_only=True,
    export_dtype="bfloat16",
)
```

--------------------------------

### Configure and Start Multi-Node Training with Torchrun

Source: https://github.com/pytorch/torchtitan/blob/main/README.md

This section details how to configure and start multi-node training using `torchrun` on ParallelCluster/Slurm environments. Adjust `ntasks`, `nodes`, `nnodes`, `nproc_per_node`, and `gpus-per-task` according to your cluster setup.

```bash
#SBATCH --ntasks=2
#SBATCH --nodes=2
```

```bash
srun torchrun --nnodes 2
```

--------------------------------

### Quick Start Training Script (Bash)

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/gpt_oss/README.md

This snippet shows the command to initiate training for the GPT-OSS model in debug mode using the provided run_train.sh script. It sets the MODULE and CONFIG environment variables to specify the model and configuration.

```bash
MODULE=gpt_oss CONFIG=gpt_oss_debugmodel ./run_train.sh
```

--------------------------------

### Install TorchAO for Float8 support

Source: https://github.com/pytorch/torchtitan/blob/main/docs/float8.md

Command to install the latest version of TorchAO required for Float8 training functionality.

```bash
USE_CPP=0 python -m pip install git+https://github.com/pytorch/ao.git
```

--------------------------------

### Launch TorchFT Lighthouse

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/ft/torchft.md

Command to start the TorchFT lighthouse service, which is essential for coordinating fault tolerance among replica groups. It requires specifying minimum replicas and timeout settings.

```bash
RUST_BACKTRACE=1 torchft_lighthouse --min_replicas 1 --quorum_tick_ms 100 --join_timeout_ms 10000
```

--------------------------------

### Initiate Model Training Runs

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/deepseek_v3/README.md

Starts the training process for different Torchtitan models. Users can select model size and configuration by setting environment variables before executing the training script. This allows for quick debug runs or training of larger parameter models.

```bash
MODEL=deepseek_v3 CONFIG=deepseek_v3_debugmodel ./run_train.sh
```

```bash
MODEL=deepseek_v3 CONFIG=deepseek_v3_16b ./run_train.sh
```

```bash
MODEL=deepseek_v3 CONFIG=deepseek_v3_671b ./run_train.sh
```

--------------------------------

### Run Integration Tests with Specific Module and Config

Source: https://github.com/pytorch/torchtitan/blob/main/tests/README.md

Demonstrates how to run integration tests with a custom module and configuration. This allows testing different model architectures or training setups.

```python
python -m tests.integration_tests.run_tests test_output --module llama3 --config llama3_8b
```

--------------------------------

### Run All Torchtitan Feature Integration Tests

Source: https://github.com/pytorch/torchtitan/blob/main/tests/README.md

A specific example of running all feature integration tests using default module and configuration settings. This is useful for a quick check of core features.

```python
python -m tests.integration_tests.run_tests test_output
```

--------------------------------

### Start Local Training Run with Llama 3 8B

Source: https://github.com/pytorch/torchtitan/blob/main/README.md

This command initiates a local training run for the Llama 3 8B model on 8 GPUs. It sets the MODULE and CONFIG environment variables to specify the model and configuration, then executes the training script.

```bash
MODULE=llama3 CONFIG=llama3_8b ./run_train.sh
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/pytorch/torchtitan/blob/main/tests/README.md

Installs the necessary development dependencies for the Torchtitan project using pip. This is a prerequisite for running tests and contributing to the project.

```bash
pip install -r requirements-dev.txt
pip install -r requirements.txt
```

--------------------------------

### Install PyTorch Nightly for GraphTrainer

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/graph_trainer/README.md

Installs the latest PyTorch nightly build required for GraphTrainer. Ensure to select the correct CUDA version or specify AMD GPU support.

```bash
pip3 install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu128 --force-reinstall
```

--------------------------------

### Configure Multi-Dimensional Parallelism with ParallelismConfig

Source: https://context7.com/pytorch/torchtitan/llms.txt

This section details the configuration of various parallelism strategies including FSDP, Tensor Parallel, Pipeline Parallel, Context Parallel, and Expert Parallel using the ParallelismConfig class. It allows for fine-grained control over distributed training setups.

```python
from torchtitan.config import ParallelismConfig

# Configure FSDP + Tensor Parallel
parallelism = ParallelismConfig(
    data_parallel_shard_degree=-1,      # FSDP: -1 means auto (remaining ranks)
    data_parallel_replicate_degree=1,   # DDP/HSDP replicate dimension
    tensor_parallel_degree=8,           # Tensor parallelism degree
    enable_async_tensor_parallel=True,  # Async TP for better overlap
    disable_loss_parallel=False,        # Enable loss parallel with TP
    fsdp_reshard_after_forward="default",  # Reshard policy: default/always/never
)

# Configure Pipeline Parallel
parallelism_pp = ParallelismConfig(
    pipeline_parallel_degree=4,         # Number of PP ranks
    pipeline_parallel_schedule="1F1B",  # Schedule: 1F1B, Interleaved1F1B, etc.
    pipeline_parallel_microbatch_size=1,
)

# Configure Context Parallel for long sequences
parallelism_cp = ParallelismConfig(
    context_parallel_degree=2,          # CP degree
    context_parallel_load_balancer="headtail",  # headtail or ptrr
    context_parallel_rotate_method="allgather", # allgather or alltoall
)

# Configure Expert Parallel for MoE models
parallelism_moe = ParallelismConfig(
    expert_parallel_degree=4,           # EP degree
    expert_tensor_parallel_degree=1,    # ETP within experts
    expert_parallel_comm_backend="standard",  # standard/deepep/hybridep
)
```

--------------------------------

### Run Model Generation Check on Single GPU

Source: https://github.com/pytorch/torchtitan/blob/main/scripts/generate/README.md

Executes the model generation script on a single GPU. It requires specifying the number of GPUs (NGPU), the model module, configuration, and checkpoint directory. The prompt and generation parameters like max new tokens, temperature, and seed are also provided. This is a basic sanity check for runtime setup.

```bash
NGPU=1 MODULE=llama3 CONFIG=llama3_8b CHECKPOINT_DIR=./outputs/checkpoint/ \
PROMPT="What is the meaning of life?" \
./scripts/generate/run_llama_generate.sh --max_new_tokens=32 --temperature=0.8 --seed=3
```

--------------------------------

### Compare FSDP1 and FSDP2 Meta-Device Initialization

Source: https://github.com/pytorch/torchtitan/blob/main/docs/fsdp.md

Demonstrates the shift from FSDP1's requirement for param_init_fn to FSDP2's post-sharding materialization approach using to_empty.

```python
# FSDP1 approach
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
with torch.device("meta"):
    model = Transformer()
policy = ModuleWrapPolicy({TransformerBlock})
def param_init_fn(module: nn.Module) -> None: ...
model = FSDP(model, auto_wrap_policy=policy, param_init_fn=param_init_fn)

# FSDP2 approach
with torch.device("meta"):
    model = Transformer()
for module in model.modules():
    if isinstance(module, TransformerBlock):
        fully_shard(module)
fully_shard(model)
model.to_empty(device="cuda")
model.init_weights()
```

--------------------------------

### Configure MXFP8 for Grouped GEMMs (MoE) in PyTorch

Source: https://github.com/pytorch/torchtitan/blob/main/docs/mxfp8.md

This Python code snippet demonstrates how to configure MXFP8 training for grouped GEMMs, typically used in Mixture-of-Experts (MoE) models, within a PyTorch setup using TorchTitan. It utilizes MXGroupedMMConverter for this purpose. Ensure PyTorch nightly and TorchAO v0.14.0+ are installed.

```python
from torchtitan.components.quantization.mx import MXGroupedMMConverter
from torchtitan.protocols.model_converter import ModelConvertersContainer

# In your config_registry function:
# model_converters=ModelConvertersContainer.Config(
#     converters=[
#         MXGroupedMMConverter.Config(recipe_name="mxfp8_cublas"),
#     ],
# ),
# compile=CompileConfig(enable=True),

```

--------------------------------

### Configure and Manage Model Checkpointing

Source: https://context7.com/pytorch/torchtitan/llms.txt

Demonstrates how to initialize the CheckpointManager with specific settings like interval, async mode, and Hugging Face compatibility. It also shows how to save, load, and manage resources during the training lifecycle.

```python
checkpoint_config = CheckpointManager.Config(enable=True, folder="checkpoint", interval=500, async_mode="async", keep_latest_k=10, last_save_model_only=True, last_save_in_hf=True, export_dtype="bfloat16")
checkpointer = checkpoint_config.build(dataloader=dataloader, model_parts=model_parts, optimizers=optimizers, lr_schedulers=lr_schedulers, states={"train_state": trainer}, base_folder="./outputs")
loaded = checkpointer.load(step=-1)
checkpointer.save(curr_step=100, last_step=False)
checkpointer.maybe_wait_for_staging()
checkpointer.close()
```

--------------------------------

### Configure and Run HuggingFace Transformers Backend

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/transformers_modeling_backend/README.md

This snippet demonstrates how to modify the configuration registry to use the transformers modeling backend and the command to initiate training with torch.compile enabled.

```diff
- --module llama3
+ --module transformers_modeling_backend
--config transformers_modeling_backend_debugmodel
```

```bash
LOG_RANK=7 MODEL=transformers_modeling_backend CONFIG=transformers_modeling_backend_debugmodel ./run_train.sh --compile.enable
```

--------------------------------

### Execute Custom Training Configuration

Source: https://github.com/pytorch/torchtitan/blob/main/docs/extension.md

Shows the command-line interface usage to run a custom training experiment by specifying the module and configuration function.

```bash
MODULE=your_folder CONFIG=my_experiment_debugmodel ./run_train.sh
```

--------------------------------

### Initialize Trainer Configuration

Source: https://context7.com/pytorch/torchtitan/llms.txt

This snippet shows the import of necessary components for initializing the Trainer.Config, which aggregates all training and parallelism configurations. It includes components for checkpointing, optimizers, learning rate scheduling, metrics, and data loading.

```python
from torchtitan.trainer import Trainer
from torchtitan.config import TrainingConfig, ParallelismConfig, ActivationCheckpointConfig, CompileConfig
from torchtitan.components.checkpoint import CheckpointManager
from torchtitan.components.optimizer import OptimizersContainer
from torchtitan.components.lr_scheduler import LRSchedulersContainer
from torchtitan.components.metrics import MetricsProcessor
from torchtitan.hf_datasets.text_datasets import HuggingFaceTextDataLoader
from torchtitan.models.llama3 import model_registry
```

--------------------------------

### Run Specific Integration Test with Custom GPU Count

Source: https://github.com/pytorch/torchtitan/blob/main/tests/README.md

An example of running a specific integration test function ('gradient_accumulation') with a defined number of GPUs (2). This is useful for performance testing or debugging specific scenarios.

```python
python -m tests.integration_tests.run_tests test_output --test_suite features --test_name gradient_accumulation --ngpu 2
```

--------------------------------

### Programmatic Training Loop Execution in PyTorch

Source: https://context7.com/pytorch/torchtitan/llms.txt

Execute the training loop programmatically using the Trainer class in PyTorch. This approach allows for direct control over training setup, execution, and cleanup, bypassing command-line argument parsing.

```python
from torchtitan.train import main
from torchtitan.config import ConfigManager
from torchtitan.trainer import Trainer
import torch

# Option 1: Use main entry point
# main()  # Parses CLI args

# Option 2: Programmatic training
config_manager = ConfigManager()
config = config_manager.parse_args()

# Build and run trainer
trainer = config.build()

try:
    # Load checkpoint if available
    trainer.checkpointer.load(step=-1)

    # Run training loop
    trainer.train()

finally:
    # Clean up
    trainer.close()
    if torch.distributed.is_initialized():
        torch.distributed.destroy_process_group()
```

--------------------------------

### Download Qwen3 Tokenizer Assets

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/qwen3/README.md

Uses the download_hf_assets script to fetch tokenizer files from the Hugging Face repository. Requires the repository ID as an input argument.

```bash
python scripts/download_hf_assets.py --repo_id Qwen/Qwen3-0.6B --assets tokenizer
```

--------------------------------

### Apply Compiler Optimizations with GraphTrainer (JIT Mode)

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/graph_trainer/README.md

Demonstrates applying full Inductor compilation in JIT mode for GraphTrainer. Switches to JIT mode and specifies the Inductor backend.

```bash
MODULE=graph_trainer.llama3 CONFIG=graph_trainer_llama3_8b ./run_train.sh --compile.mode jit --compile.backend inductor
```

--------------------------------

### Enable Torchtitan Configuration Printing

Source: https://github.com/pytorch/torchtitan/blob/main/benchmarks/README.md

This command-line argument is used to output the complete configuration of a training run. It is the preferred method for documenting training configs in performance reports to ensure all default values are captured.

```bash
--print_config
```

--------------------------------

### Evaluate Model using lm_eval with vLLM Backend

Source: https://github.com/pytorch/torchtitan/blob/main/docs/evaluation.md

This bash command demonstrates how to use the lm_eval library with the vLLM backend for evaluating a model. It specifies the model to use (via a local checkpoint path), tensor parallel size, data type, GPU memory utilization, and the evaluation tasks. Ensure lm-eval is installed with the vLLM extra and that HuggingFace assets like config.json are available.

```bash
pip install "lm-eval[vllm]"
lm_eval --model vllm \
    --model_args pretrained=./outputs/checkpoint/step-1000,tensor_parallel_size=8,dtype=auto,gpu_memory_utilization=0.8, \
    --tasks mmlu \
    --batch_size auto
```

--------------------------------

### Launch TorchTitan Replica Group 0

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/ft/torchft.md

Command to launch the first TorchTitan instance, configured as replica group 0. This command specifies GPU visibility, training file, model, configuration, and enables fault tolerance with replica ID 0 and group size 2.

```bash
NGPU=4 CUDA_VISIBLE_DEVICES=0,1,2,3 TRAIN_FILE=torchtitan.experiments.ft.train MODEL=llama3 CONFIG=llama3_8B ./run_train.sh --fault_tolerance.enable --fault_tolerance.replica_id=0 --fault_tolerance.group_size=2 --parallelism.data_parallel_shard_degree=4
```

--------------------------------

### Download HuggingFace Tokenizer Assets

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/deepseek_v3/README.md

Downloads tokenizer configuration and data files from HuggingFace repositories. It supports different model sizes by specifying the repository ID. This script is crucial for setting up the necessary tokenizer files before training or inference.

```bash
python scripts/download_hf_assets.py --repo_id deepseek-ai/DeepSeek-V3.1-Base --assets tokenizer
```

```bash
python scripts/download_hf_assets.py --repo_id deepseek-ai/deepseek-moe-16b-base --assets tokenizer
```

--------------------------------

### Configure Parallel Device Meshes with ParallelDims API

Source: https://context7.com/pytorch/torchtitan/llms.txt

Demonstrates how to create and manage multi-dimensional device meshes for distributed training using the ParallelDims API. It shows how to initialize ParallelDims with various parallelism degrees, build the world mesh, access specific sub-meshes (like tensor parallel or FSDP), and check the status of enabled parallelism features.

```python
from torchtitan.distributed import ParallelDims

# Create parallel dimensions
parallel_dims = ParallelDims(
    dp_replicate=2,    # DDP/HSDP replicate dimension
    dp_shard=4,        # FSDP shard dimension
    cp=1,              # Context parallel
    tp=2,              # Tensor parallel
    pp=1,              # Pipeline parallel
    ep=1,              # Expert parallel
    etp=1,             # Expert tensor parallel
    world_size=16,     # Total world size
)

# Build device mesh
world_mesh = parallel_dims.build_mesh()

# Access specific meshes
tp_mesh = parallel_dims.get_mesh("tp")           # Tensor parallel mesh
fsdp_mesh = parallel_dims.get_mesh("fsdp")       # FSDP mesh (dp_shard * cp)
batch_mesh = parallel_dims.get_mesh("batch")     # Data loading mesh
loss_mesh = parallel_dims.get_optional_mesh("loss")  # Loss reduction mesh (may be None)

# Check parallelism status
if parallel_dims.tp_enabled:
    print(f"Tensor Parallel enabled with degree {parallel_dims.tp}")
if parallel_dims.fsdp_enabled:
    print(f"FSDP enabled with degree {parallel_dims.dp_shard}")

# Get all active meshes
active_meshes = parallel_dims.get_all_one_dimensional_meshes()
print(f"Active meshes: {list(active_meshes.keys())}")
```

--------------------------------

### Initialize FSDP2 with fully_shard

Source: https://github.com/pytorch/torchtitan/blob/main/docs/fsdp.md

The fully_shard function is the core entry point for FSDP2. It applies sharding to an nn.Module by dynamically swapping its class and attaching FSDPState, replacing the traditional nn.Module wrapper approach used in FSDP1.

```python
@contract(state_cls=FSDPState)
def fully_shard(
  module: nn.Module,
  *,
  mesh: Optional[DeviceMesh] = None,
  reshard_after_forward: Union[bool, int] = True,
  mp_policy: MixedPrecisionPolicy = MixedPrecisionPolicy(),
  offload_policy: OffloadPolicy = OffloadPolicy(),
) -> nn.Module:  # returns `module` for `contract` checks
```

--------------------------------

### Launch Semi-Synchronous Replica Groups

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/ft/torchft.md

Commands to launch two distinct replica groups for semi-synchronous training. These commands specify the fault tolerance group size and individual replica IDs to coordinate training across different GPU sets.

```bash
TRAIN_FILE=torchtitan.experiments.ft.train MODEL=llama3_ft CONFIG=llama3_ft_debugmodel CUDA_VISIBLE_DEVICES=0,1,2,3 NGPU=4 ./run_train.sh --parallelism.data_parallel_shard_degree=4 --fault_tolerance.enable --fault_tolerance.group_size=2 --fault_tolerance.replica_id=0
```

```bash
TRAIN_FILE=torchtitan.experiments.ft.train MODEL=llama3_ft CONFIG=llama3_ft_debugmodel CUDA_VISIBLE_DEVICES=4,5,6,7 NGPU=4 ./run_train.sh --parallelism.data_parallel_shard_degree=4 --fault_tolerance.enable --fault_tolerance.group_size=2 --fault_tolerance.replica_id=1
```

--------------------------------

### Load HuggingFace Checkpoint Directly in Torchtitan Training

Source: https://github.com/pytorch/torchtitan/blob/main/docs/checkpoint.md

Enable direct loading of HuggingFace safetensors files during Torchtitan training. Use `--checkpoint.initial_load_in_hf` along with `--hf_assets_path` or `--checkpoint.initial_load_path`.

```bash
--checkpoint.initial_load_in_hf --hf_assets_path <path_to_hf_checkpoint>
```

```bash
--checkpoint.initial_load_in_hf --checkpoint.initial_load_path <path_to_hf_checkpoint>
```

--------------------------------

### Launch TorchTitan Replica Group 1

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/ft/torchft.md

Command to launch the second TorchTitan instance, configured as replica group 1. Similar to replica group 0, this command sets up GPU visibility, training parameters, and enables fault tolerance with replica ID 1 and group size 2.

```bash
NGPU=4 CUDA_VISIBLE_DEVICES=4,5,6,7 TRAIN_FILE=torchtitan.experiments.ft.train MODEL=llama3 CONFIG=llama3_8B ./run_train.sh --fault_tolerance.enable --fault_tolerance.replica_id=1 --fault_tolerance.group_size=2 --parallelism.data_parallel_shard_degree=4
```

--------------------------------

### Launch Distributed Training with run_train.sh

Source: https://context7.com/pytorch/torchtitan/llms.txt

This script initiates distributed training runs for models like Llama 3. It supports various configurations for GPU count, communication modes, and dry-runs for validation. Additional command-line arguments can be passed for fine-tuning training parameters.

```bash
# Train Llama 3 8B on 8 GPUs
MODULE=llama3 CONFIG=llama3_8b ./run_train.sh

# Train with custom GPU count
NGPU=4 MODULE=llama3 CONFIG=llama3_8b ./run_train.sh

# Dry-run for configuration validation (no GPU)
NGPU=32 COMM_MODE="fake_backend" MODULE=llama3 CONFIG=llama3_8b ./run_train.sh

# Debug mode with simulated multi-GPU on single GPU
NGPU=32 COMM_MODE="local_tensor" MODULE=llama3 CONFIG=llama3_debugmodel ./run_train.sh

# Pass additional CLI overrides
MODULE=llama3 CONFIG=llama3_8b ./run_train.sh --training.steps 500 --checkpoint.enable
```

--------------------------------

### Execute Distributed Training with torchrun

Source: https://github.com/pytorch/torchtitan/blob/main/benchmarks/llama3-8b_h200_202506_trainy-whitefiber.md

This command initiates a distributed training job using torchrun. It configures the rendezvous backend, specifies the number of nodes and processes per node, and applies specific training configurations including FP8 quantization settings.

```bash
torchrun \
  --nnodes $NUM_NODES  \
  --nproc_per_node 8 \
  --rdzv_id 101 \
  --rdzv_backend c10d \
  --rdzv_endpoint "$MASTER_ADDR:29500" \
  torchtitan/train.py \
  --job.config-file torchtitan/models/llama3/train_configs/llama3_8b.toml \
  --metrics.enable_wandb \
  --training.local_batch_size=2 \
  --training.compile \
  --model.converters="quantize.linear.float8" \
  --quantize.linear.float8.enable_fsdp_float8_all_gather \
  --quantize.linear.float8.precompute_float8_dynamic_scale_for_fsdp \
  --quantize.linear.float8.force_recompute_fp8_weight_in_bwd \
  --profiling.profile_freq 1000000 \
  --training.steps 2000
```

--------------------------------

### Enable Memory Profiling in Torchtitan

Source: https://github.com/pytorch/torchtitan/blob/main/docs/debugging.md

Launches a training job with memory profiling enabled. Memory snapshots are saved to a specified folder, with specific paths for OOMs and regular snapshots. Visualizes snapshots using a web tool and provides a link for more details.

```bash
MODULE=llama3 CONFIG=llama3_debugmodel ./run_train.sh --profiling.enable_memory_snapshot --profiling.save_memory_snapshot_folder memory_snapshot
```

--------------------------------

### Configure 2D Parallelism for GraphTrainer

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/graph_trainer/README.md

Sets up 2D parallelism (FSDP and TP) for training Llama3-8B with GraphTrainer. Specifies the number of GPUs and parallelism degrees.

```bash
NGPU=8 MODULE=graph_trainer.llama3 CONFIG=graph_trainer_llama3_8b ./run_train.sh --parallelism.data_parallel_shard_degree=4 --parallelism.tensor_parallel_degree=2
```

--------------------------------

### Configure Trainer for Llama 3 8B

Source: https://context7.com/pytorch/torchtitan/llms.txt

Sets up a comprehensive trainer configuration for Llama 3 8B, specifying model assets, output directories, training parameters, parallelism degrees, optimizer details, learning rate scheduler, dataloader settings, checkpointing behavior, activation checkpointing mode, torch.compile options, and metrics logging.

```python
config = Trainer.Config(
    hf_assets_path="./assets/hf/Llama-3.1-8B",  # Path to HF assets
    dump_folder="./outputs",                     # Output folder

    # Model specification
    model_spec=model_registry("8B"),

    # Training parameters
    training=TrainingConfig(
        local_batch_size=1,
        seq_len=8192,
        steps=1000,
        dtype="bfloat16",
    ),

    # Parallelism
    parallelism=ParallelismConfig(
        tensor_parallel_degree=8,
    ),

    # Optimizer
    optimizer=OptimizersContainer.Config(
        name="AdamW",
        lr=3e-4,
        beta1=0.9,
        beta2=0.95,
        weight_decay=0.1,
        implementation="fused",  # fused/foreach/for-loop
    ),

    # Learning rate scheduler
    lr_scheduler=LRSchedulersContainer.Config(
        warmup_steps=200,
        decay_ratio=0.8,
        decay_type="linear",  # linear/sqrt/cosine
        min_lr_factor=0.0,
    ),

    # Dataset
    dataloader=HuggingFaceTextDataLoader.Config(
        dataset="c4",
        infinite=True,
    ),

    # Checkpointing
    checkpoint=CheckpointManager.Config(
        enable=True,
        folder="checkpoint",
        interval=500,
        async_mode="async",  # disabled/async/async_with_pinned_mem
        keep_latest_k=10,
    ),

    # Activation checkpointing
    activation_checkpoint=ActivationCheckpointConfig(
        mode="selective",  # none/selective/full/memory_budget
    ),

    # torch.compile
    compile=CompileConfig(
        enable=True,
        components=["model", "loss"],
        backend="inductor",
    ),

    # Metrics
    metrics=MetricsProcessor.Config(
        log_freq=10,
        enable_tensorboard=True,
    ),
)

```

--------------------------------

### Configure Optimizers and Learning Rate Schedulers

Source: https://context7.com/pytorch/torchtitan/llms.txt

Sets up an AdamW optimizer with fused implementation and a warmup-stable-decay learning rate scheduler. This configuration is essential for stable model convergence during training.

```python
optimizer_config = OptimizersContainer.Config(name="AdamW", lr=3e-4, weight_decay=0.1, implementation="fused")
optimizers = optimizer_config.build(model_parts=[model])
lr_scheduler_config = LRSchedulersContainer.Config(warmup_steps=200, decay_ratio=0.8, decay_type="cosine", min_lr_factor=0.1)
lr_schedulers = lr_scheduler_config.build(optimizers=optimizers, training_steps=10000)
```

--------------------------------

### Extend Trainer Configuration for Custom Experiments

Source: https://github.com/pytorch/torchtitan/blob/main/docs/extension.md

Demonstrates how to subclass Trainer.Config to add custom fields and register them in the config_registry. This allows users to inject experiment-specific parameters into the training pipeline.

```python
from dataclasses import dataclass, field
from torchtitan.trainer import Trainer

@dataclass
class CustomConfig:
    how_is_your_day: str = "good"

class MyTrainer(Trainer):
    @dataclass(kw_only=True, slots=True)
    class Config(Trainer.Config):
        custom_config: CustomConfig = field(default_factory=CustomConfig)
```

```python
from .trainer import MyTrainer, CustomConfig

def my_experiment_debugmodel() -> MyTrainer.Config:
    return MyTrainer.Config(
        custom_config=CustomConfig(how_is_your_day="great"),
        training=TrainingConfig(steps=100)
    )
```

--------------------------------

### View Available Arguments for Model Generation Script

Source: https://github.com/pytorch/torchtitan/blob/main/scripts/generate/README.md

Displays the help message for the `test_generate` Python module, listing all available command-line arguments and their descriptions. This command is essential for understanding the full range of options for customizing model generation, including parameters for token generation, sampling, and output formatting.

```bash
python -m scripts.generate.test_generate --help
```

--------------------------------

### Optimizer and LR Scheduler Configuration

Source: https://context7.com/pytorch/torchtitan/llms.txt

Configure optimizers and learning rate schedulers with warmup-stable-decay schedule.

```APIDOC
## Optimizer and LR Scheduler Configuration

### Description
Configure optimizers and learning rate schedulers with warmup-stable-decay schedule.

### Method
N/A (Configuration Classes)

### Endpoint
N/A

### Parameters
#### OptimizersContainer.Config
- **name** (str) - Required - Optimizer name ('AdamW', 'Adam').
- **lr** (float) - Required - Learning rate.
- **beta1** (float) - Optional - Adam beta1 parameter.
- **beta2** (float) - Optional - Adam beta2 parameter.
- **eps** (float) - Optional - Adam epsilon parameter.
- **weight_decay** (float) - Optional - Weight decay.
- **implementation** (str) - Optional - Optimizer implementation ('fused', 'foreach', 'for-loop').

#### LRSchedulersContainer.Config
- **warmup_steps** (int) - Required - Number of linear warmup steps.
- **total_steps** (int) - Optional - Total training steps. Defaults to training.steps.
- **decay_ratio** (float) - Optional - Decay during the last X% of training.
- **decay_type** (str) - Optional - Decay type ('linear', 'sqrt', 'cosine').
- **min_lr_factor** (float) - Optional - Minimum LR as a fraction of initial LR.

### Usage Example
```python
from torchtitan.components.optimizer import OptimizersContainer
from torchtitan.components.lr_scheduler import LRSchedulersContainer

# Optimizer configuration
optimizer_config = OptimizersContainer.Config(
    name="AdamW",
    lr=3e-4,
    beta1=0.9,
    beta2=0.95,
    eps=1e-8,
    weight_decay=0.1,
    implementation="fused",
)
optimizers = optimizer_config.build(model_parts=[model])

# LR Scheduler configuration
lr_scheduler_config = LRSchedulersContainer.Config(
    warmup_steps=200,
    total_steps=None,
    decay_ratio=0.8,
    decay_type="cosine",
    min_lr_factor=0.1,
)
lr_schedulers = lr_scheduler_config.build(
    optimizers=optimizers,
    training_steps=10000,
)

# Training loop usage
for step in range(training_steps):
    optimizers.zero_grad()
    loss.backward()
    optimizers.step()
    lr_schedulers.step()

    current_lr = lr_schedulers.schedulers[0].get_last_lr()[0]
```
```

--------------------------------

### Float8 Quantization Configuration

Source: https://context7.com/pytorch/torchtitan/llms.txt

Enable Float8 training for improved performance on supported hardware.

```APIDOC
## Float8 Quantization Configuration

### Description
Enable Float8 training for improved performance on supported hardware.

### Method
N/A (Configuration Class)

### Endpoint
N/A

### Parameters
#### ModelConvertersContainer.Config
- **converters** (list) - Required - List of model converter configurations.

##### Float8LinearConverter.Config
- **enable_fsdp_float8_all_gather** (bool) - Optional - Enable Float8 all-gather for FSDP.
- **precompute_float8_dynamic_scale_for_fsdp** (bool) - Optional - Precompute dynamic scale for FSDP.
- **recipe_name** (str) - Optional - Quantization recipe ('tensorwise', 'rowwise').
- **filter_fqns** (list[str]) - Optional - Fully Qualified Names (FQNs) to exclude from quantization.
- **emulate** (bool) - Optional - Enable emulation mode for testing.

### Usage Example
```python
from torchtitan.components.quantization.float8 import Float8LinearConverter
from torchtitan.protocols.model_converter import ModelConvertersContainer

# Configure Float8 training
model_converters_config = ModelConvertersContainer.Config(
    converters=[
        Float8LinearConverter.Config(
            enable_fsdp_float8_all_gather=True,
            precompute_float8_dynamic_scale_for_fsdp=True,
            recipe_name="tensorwise",
            filter_fqns=["output"],
            emulate=False,
        ),
    ],
)

# Use in trainer config
config = Trainer.Config(
    model_converters=model_converters_config,
    # ... other config
)
```
```

--------------------------------

### Configure 3D Parallelism for GraphTrainer

Source: https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/graph_trainer/README.md

Sets up 3D parallelism (FSDP, TP, and EP) for training DeepSeek-v3-16B with GraphTrainer. Specifies the number of GPUs and parallelism degrees.

```bash
NGPU=8 MODULE=graph_trainer.deepseek_v3 CONFIG=graph_trainer_deepseek_v3_16b ./run_train.sh --parallelism.data_parallel_shard_degree=4 --parallelism.tensor_parallel_degree=2 --parallelism.expert_parallel_degree=2
```

--------------------------------

### Download Llama 3.1 Tokenizer using Python

Source: https://github.com/pytorch/torchtitan/blob/main/README.md

This script downloads the Llama 3.1 tokenizer assets from Hugging Face. It requires a Hugging Face token and specifies the repository ID and assets to download. Ensure you have access to the Llama model weights before running.

```bash
# Get your HF token from https://huggingface.co/settings/tokens

# Llama 3.1 tokenizer
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
```