### Install Dependencies

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Install the necessary libraries, timm and torchvision, before running the example.

```bash
pip install timm torchvision
```

--------------------------------

### Launch Example Script on Intel Gaudi

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/gaudi.md

Use this command to run an example script on Intel Gaudi devices. Ensure you have the 'accelerate' library installed and configured for Gaudi.

```bash
accelerate launch /examples/cv_example.py --data_dir images
```

--------------------------------

### Install Dependencies

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Install the necessary libraries for the NLP example, including datasets, evaluate, and transformers.

```bash
pip install datasets evaluate transformers
```

--------------------------------

### Install minGPT and huggingface_hub

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/quantization.md

Install minGPT and huggingface_hub libraries to run the provided examples.

```bash
git clone https://github.com/karpathy/minGPT.git
pip install minGPT/
pip install huggingface_hub
```

--------------------------------

### Launch GLUE Example with Accelerate

Source: https://github.com/huggingface/accelerate/blob/main/README.md

Example of launching a specific NLP example script using the Accelerate CLI.

```bash
accelerate launch examples/nlp_example.py
```

--------------------------------

### Self-Contained Gradient Accumulation Example Setup

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/gradient_accumulation.md

This self-contained example demonstrates gradient accumulation in action. It sets up a toy dataset, dataloader, and defines `gradient_accumulation_steps` and `per_device_batch_size`.

```python
import torch
import copy
from accelerate import Accelerator
from accelerate.utils import set_seed
from torch.utils.data import TensorDataset, DataLoader

# seed
set_seed(0)

# define toy inputs and labels
x = torch.tensor([1., 2., 3., 4., 5., 6., 7., 8.])
y = torch.tensor([2., 4., 6., 8., 10., 12., 14., 16.])
gradient_accumulation_steps = 4
per_device_batch_size = len(x) // gradient_accumulation_steps

# define dataset and dataloader
dataset = TensorDataset(x, y)
dataloader = DataLoader(dataset, batch_size=per_device_batch_size)
```

--------------------------------

### Run NLP Example on TPUs with Accelerate Config

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Configure Accelerate for TPU training and launch the NLP example. This setup is suitable for single or multi-TPU configurations.

```bash
accelerate config  # This will create a config file on your TPU server
```

```bash
accelerate launch ./nlp_example.py  # This will run the script on each server
```

--------------------------------

### Launch Base NLP Example with Mixed Precision and CPU

Source: https://github.com/huggingface/accelerate/blob/main/examples/by_feature/README.md

Launches the base NLP example script using Accelerate, specifying mixed precision and CPU usage. Ensure Accelerate is installed and configured.

```bash
accelerate launch ../nlp_example.py --mixed_precision fp16 --cpu 0
```

--------------------------------

### Basic Accelerator Initialization

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/deepspeed.md

Initializes the Accelerator and prints its current state. This is a fundamental example for starting with Accelerate.

```python
from accelerate.state import AcceleratorState


def main():
    accelerator = Accelerator()
    accelerator.print(f"{AcceleratorState()}")


if __name__ == "__main__":
    main()

```

--------------------------------

### Launch Gradient Accumulation Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/by_feature/README.md

Launches the gradient accumulation example script, setting the number of steps before gradients are accumulated and the optimizer is stepped. This script demonstrates preventing gradient averaging in distributed setups.

```bash
accelerate launch ./gradient_accumulation.py --gradient_accumulation_steps 5
```

--------------------------------

### Install Watchdog for Preview

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Install the watchdog module, which is required for the live preview functionality of the documentation.

```bash
pip install watchdog
```

--------------------------------

### Launch Training Loop with notebook_launcher (TPU)

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/notebook.md

Launch a training loop on TPUs using `notebook_launcher`. This example includes passing the model as an argument, which is common for TPU setups.

```python
model = create_model("resnet50d", pretrained=True, num_classes=len(label_to_id))

args = (model, "fp16", 42, 64)
notebook_launcher(training_loop, args, num_processes=8)
```

--------------------------------

### Install bitsandbytes

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/quantization.md

Install the bitsandbytes library to enable model quantization. For non-CUDA devices, refer to the bitsandbytes installation guide.

```bash
pip install bitsandbytes
```

--------------------------------

### Preview Documentation

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Start a local server to preview the documentation. This command requires the package name and the path to the documentation source files.

```bash
doc-builder preview accelerate docs/source/
```

--------------------------------

### Basic Distributed Execution Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Demonstrates a simple distributed execution pattern using `Accelerator` for synchronized operations. Ensure `accelerate` is installed. This example shows how to make the main process wait and then synchronize all processes.

```python
>>> import time
>>> from accelerate import Accelerator
>>> accelerator = Accelerator()
>>> if accelerator.is_main_process:
...     time.sleep(2)
... else:
...     print("I'm waiting for the main process to finish its sleep...")
>>> accelerator.wait_for_everyone()
>>> # Should print on every process at the same time
>>> print("Everyone is here")
```

--------------------------------

### Run Example Script with Context Parallelism

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/context_parallelism.md

Execute an example script demonstrating context parallelism, specifying CP size and sequence length for large context handling.

```bash
accelerate launch --use-fsdp --fsdp-activation-checkpointing=TRUE examples/fsdp2/nd_parallel.py --cp-size=8 --sequence-length=128000
```

--------------------------------

### Self-contained Causal LM Example Setup

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/gradient_accumulation.md

This snippet sets up a self-contained causal language modeling example using Accelerate. It includes defining a custom dataset, a collate function for padding, and initializing the Accelerator with gradient accumulation.

```python
import torch
import copy
from accelerate import Accelerator
from accelerate.utils import set_seed
from accelerate.logging import  get_logger
from torch.utils.data import Dataset, DataLoader
import math
import contexlib

# seed
set_seed(0)
logger = get_logger(__name__)

class MyDataset(Dataset):
    def __init__(self, num_samples):
        super().__init__()
        self.len = num_samples

    def __getitem__(self, index):
        input_ids = torch.arange(1, index+2, dtype=torch.float32)
        labels = torch.remainder(input_ids, 2)
        return {"input_ids": input_ids, "labels": labels}

    def __len__(self):
        return self.len
    
def collate_fn(features):
    input_ids = torch.nn.utils.rnn.pad_sequence([f["input_ids"] for f in features], batch_first=True, padding_value=-100)
    labels = torch.nn.utils.rnn.pad_sequence([f["labels"] for f in features], batch_first=True, padding_value=-100)
    return {"input_ids": input_ids[..., None], "labels": labels[..., None]}

# define toy inputs and labels
gradient_accumulation_steps = 2
per_device_batch_size = 4

# define accelerator
accelerator = Accelerator(gradient_accumulation_steps=gradient_accumulation_steps)

# define dataset and dataloader
# for this toy example, we'll compute gradient descent over one single global batch
dataset = MyDataset(per_device_batch_size*gradient_accumulation_steps*accelerator.num_processes)
dataloader = DataLoader(dataset, batch_size=per_device_batch_size, collate_fn=collate_fn)
```

--------------------------------

### Configure and Launch Multi-GPU Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Configure Accelerate for multi-GPU training and launch the script. This creates a `config.yaml` file for the Accelerate launcher.

```bash
accelerate config --config_file config.yaml  # This will create a config file on your server to `config.yaml`
accelerate launch --config_file config.yaml ./cv_example.py --data_dir path_to_data  # This will run the script on your server
```

--------------------------------

### Run Single GPU Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Run the vision example on a single GPU. Ensure a GPU is available on the server.

```bash
python ./cv_example.py  # from a server with a GPU
```

--------------------------------

### Install minGPT

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/big_model_inference.md

Clone the minGPT repository and install it to use its model definitions.

```bash
git clone https://github.com/karpathy/minGPT.git
pip install minGPT/
```

--------------------------------

### Launch Experiment Tracking Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/by_feature/README.md

Launches the experiment tracking example script, enabling all available experiment trackers from the environment. This script shows how to integrate Accelerate with tools like Weights and Biases, TensorBoard, or CometML.

```bash
accelerate launch ./tracking.py --with_tracking
```

--------------------------------

### Launch Checkpointing Example with Output Directory and Resume

Source: https://github.com/huggingface/accelerate/blob/main/examples/by_feature/README.md

Launches the checkpointing example script, setting the number of steps for saving states, the output directory, and the checkpoint to resume from. This assumes a previous run with the same checkpointing configuration.

```bash
accelerate launch ./checkpointing.py --checkpointing_steps epoch output_dir "checkpointing_tutorial" --resume_from_checkpoint "checkpointing_tutorial/epoch_0"
```

--------------------------------

### Launch Benchmark with Accelerate

Source: https://github.com/huggingface/accelerate/blob/main/benchmarks/fsdp2/README.md

Use `accelerate launch` to run the main benchmark script. This is the recommended way to start the benchmark process.

```bash
accelerate launch main.py
```

--------------------------------

### Install Accelerate from source

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/quantization.md

Install the latest version of the Accelerate library from its GitHub repository.

```bash
pip install git+https://github.com/huggingface/accelerate.git
```

--------------------------------

### Install huggingface_hub

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/big_model_inference.md

Install the huggingface_hub library to download model checkpoints.

```bash
pip install huggingface_hub
```

--------------------------------

### Example of launching a SageMaker training script

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/sagemaker.md

This command demonstrates how to launch an example training script (`sagemaker_example.py`) using Accelerate on SageMaker. Ensure `accelerator.save('/opt/ml/model')` is included in the script.

```bash
accelerate launch ./examples/sagemaker_example.py
```

--------------------------------

### Install Doc-Builder Tool

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Install the specialized tool used for building and managing documentation.

```bash
pip install git+https://github.com/huggingface/doc-builder
```

--------------------------------

### Run NLP Example on Multi-GPU with Accelerate Config

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Configure Accelerate for multi-GPU training and launch the NLP example. This creates a config file on the server.

```bash
accelerate config  # This will create a config file on your server
```

```bash
accelerate launch ./nlp_example.py  # This will run the script on your server
```

--------------------------------

### Run Mixed Precision Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the vision example using fp16 mixed precision. This can be configured via the Accelerator or the Accelerate launcher.

```bash
python ./cv_example.py --data_dir path_to_data --mixed_precison fp16
```

```bash
accelerate launch --mixed_precison fp16 ./cv_example.py --data_dir path_to_data
```

--------------------------------

### Install Accelerate and PyTorch

Source: https://github.com/huggingface/accelerate/blob/main/examples/inference/distributed/README.md

Install the necessary libraries for distributed training and inference.

```bash
pip install accelerate torch
```

--------------------------------

### Full Experiment Tracking Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/tracking.md

A comprehensive example demonstrating initialization, training loop logging, and finalization of experiment tracking with Accelerate.

```python
from accelerate import Accelerator

accelerator = Accelerator(log_with="all")
config = {
    "num_iterations": 5,
    "learning_rate": 1e-2,
    "loss_function": str(my_loss_function),
}

accelerator.init_trackers("example_project", config=config)

my_model, my_optimizer, my_training_dataloader = accelerator.prepare(my_model, my_optimizer, my_training_dataloader)
device = accelerator.device
my_model.to(device)

for iteration in range(config["num_iterations"]):
    for step, batch in enumerate(my_training_dataloader):
        my_optimizer.zero_grad()
        inputs, targets = batch
        inputs = inputs.to(device)
        targets = targets.to(device)
        outputs = my_model(inputs)
        loss = my_loss_function(outputs, targets)
        accelerator.backward(loss)
        my_optimizer.step()
        accelerator.log({"training_loss": loss}, step=step)
accelerator.end_training()
```

--------------------------------

### Install Accelerate with SageMaker support

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/sagemaker.md

Install the `sagemaker` SDK for Accelerate. This command ensures you have the necessary components for SageMaker integration.

```bash
pip install "accelerate[sagemaker]" --upgrade
```

--------------------------------

### Run NLP Example on Single GPU

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Run the NLP example script on a server equipped with a GPU.

```bash
python ./nlp_example.py  # from a server with a GPU
```

--------------------------------

### Run Single CPU Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the vision example on a single CPU. This can be done directly or by passing `cpu=True` to the Accelerator.

```bash
python ./cv_example.py --data_dir path_to_data
```

```bash
python ./cv_example.py --data_dir path_to_data --cpu
```

```bash
accelerate launch --cpu ./cv_example.py --data_dir path_to_data
```

--------------------------------

### Example FSDP Configuration

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/fsdp.md

This is an example of a generated Accelerate configuration file, demonstrating FSDP (Fully Sharded Data Parallel) settings for distributed training.

```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

```

--------------------------------

### Run NLP Example on Multi-CPU with Intel MPI

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Launch the NLP example on multiple CPUs using Intel MPI. Ensure `MASTER_ADDR` is set to the IP of node 0.

```bash
export CCL_WORKER_COUNT=1
export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
mpirun -f hostfile -n 16 -ppn 4 python ./nlp_example.py
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Install the necessary packages for building documentation, including the project itself in editable mode and the docs extra.

```bash
pip install -e ".[docs]"
```

--------------------------------

### Run NLP Example on Multi-CPU with Accelerate Config

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Configure Accelerate for multi-CPU training and launch the NLP example. This requires MPI (Open MPI, Intel MPI, or MVAPICH).

```bash
accelerate config  # Select to have accelerate launch mpirun
```

```bash
accelerate launch ./nlp_example.py  # This will run the script on each server
```

--------------------------------

### Run NLP Example on Multi-Node Multi-GPU with Accelerate Config

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Configure Accelerate for multi-node, multi-GPU training and launch the NLP example. This configuration should be applied on each machine.

```bash
accelerate config  # This will create a config file on each server
```

```bash
accelerate launch ./nlp_example.py  # This will run the script on each server
```

--------------------------------

### Run Multi-CPU Example with Intel MPI

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the vision example on multiple CPUs using Intel MPI. Ensure to set the master address and worker count.

```bash
export CCL_WORKER_COUNT=1
export MASTER_ADDR=xxx.xxx.xxx.xxx #node0 ip
mpirun -f hostfile -n 16 -ppn 4 python ./cv_example.py --data_dir path_to_data
```

--------------------------------

### Run NLP Example on Single CPU

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the NLP example script on a single CPU. This can be done directly or by passing `--cpu` to the Accelerator.

```bash
python ./nlp_example.py
```

```bash
python ./nlp_example.py --cpu
```

```bash
accelerate launch --cpu ./nlp_example.py
```

--------------------------------

### Megatron-LM Plugin Configuration Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/megatron_lm.md

Example of setting the output directory for the Megatron-LM plugin. This is a common configuration option.

```bash
--output_dir "awesome_model"
```

--------------------------------

### Install Transformers Library

Source: https://github.com/huggingface/accelerate/blob/main/benchmarks/big_model_inference/README.md

Install the necessary transformers library for running model inference benchmarks. This is a prerequisite for the benchmark script.

```bash
pip install transformers
```

--------------------------------

### Install Accelerate from GitHub (latest features)

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/install.md

Install Accelerate directly from the GitHub repository to access the newest features not yet released.

```bash
pip install git+https://github.com/huggingface/accelerate
```

--------------------------------

### Set Up Pre-commit Hooks

Source: https://github.com/huggingface/accelerate/blob/main/CONTRIBUTING.md

Install and install pre-commit to automatically run checks as Git commit hooks. This ensures code quality before committing.

```bash
pip install pre-commit
```

```bash
pre-commit install
```

--------------------------------

### Custom Configuration YAML Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/launch.md

An example YAML file defining a custom configuration for launching a script on two GPUs with fp16 mixed precision.

```yaml
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
fsdp_config: {}
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 2
use_cpu: false
```

--------------------------------

### Install Accelerate from local source (editable)

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/install.md

Clone the repository and install Accelerate in editable mode for development or testing with live code changes.

```bash
git clone https://github.com/huggingface/accelerate
cd accelerate
pip install -e .
```

--------------------------------

### Launch Local SGD Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/by_feature/README.md

Launches the local SGD example script, specifying the number of local SGD steps. This script demonstrates preventing gradient averaging without altering the effective batch size, and can be combined with gradient accumulation.

```bash
accelerate launch ./local_sgd.py --local_sgd_steps 4
```

--------------------------------

### Run NLP Example on Multi-GPU with PyTorch Distributed Launcher

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the NLP example using PyTorch's distributed launcher for multi-GPU training on a single node.

```bash
torchrun --nproc_per_node 2 ./nlp_example.py
```

--------------------------------

### Run Multi-GPU Example with PyTorch Launcher

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the vision example on multiple GPUs using the traditional PyTorch distributed launcher. Specify the number of processes per node.

```bash
torchrun --nproc_per_node 2 ./cv_example.py --data_dir path_to_data
```

--------------------------------

### Configure and Launch Multi-CPU Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Set up Accelerate configuration for multi-CPU training and launch the script using the Accelerate launcher. Requires Open MPI, Intel MPI, or MVAPICH.

```bash
accelerate config --config_file config.yaml  # Select to have accelerate launch mpirun
accelerate launch ./cv_example.py --data_dir path_to_data # This will run the script on each server
```

--------------------------------

### Run ALST/Ulysses Sequence Parallelism Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/alst_ulysses_sequence_parallelism/README.md

Execute the provided bash script to run the ALST/Ulysses sequence parallelism example. Ensure you have at least 2 GPUs available.

```bash
bash ./sp-alst.sh
```

--------------------------------

### Run Multi-Node Multi-GPU Example with PyTorch Launcher

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Execute the vision example across multiple nodes and GPUs using the PyTorch distributed launcher. Requires specifying node count, process count, and rendezvous information.

```bash
torchrun --nproc_per_node 2 \
    --nnodes 2 \
    --rdzv_id 2299 \ # A unique job id 
    --rdzv_backend c10d \
    --rdzv_endpoint master_node_ip_address:29500 \
    ./cv_example.py --data_dir path_to_data
```

--------------------------------

### Configure and Launch Multi-Node Multi-GPU Example

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Set up Accelerate for multi-node, multi-GPU training and launch the script. This involves creating a `config.yaml` file and running the launch command on each machine.

```bash
accelerate config --config_file config.yaml  # This will create a config file on your server to `config.yaml`
accelerate launch --config_file config.yaml ./cv_example.py --data_dir path_to_data  # This will run the script on each server
```

--------------------------------

### Example Output

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/gradient_accumulation.md

This output shows the initial and final model weights, demonstrating the effect of gradient accumulation compared to standard training.

```text
initial model weight is 0.00000
initial model weight is 0.00000
0 tensor([1., 2.])
1 tensor([3., 4.])
2 tensor([5., 6.])
3 tensor([7., 8.])
w/ accumulation, the final model weight is 2.04000
w/o accumulation, the final model weight is 2.04000
```

--------------------------------

### Run NLP Example on Multi-Node Multi-GPU with PyTorch Launcher

Source: https://github.com/huggingface/accelerate/blob/main/examples/README.md

Launch the NLP example across multiple nodes and GPUs using PyTorch's distributed launcher. This command should be executed on each node.

```bash
torchrun \
    --nproc_per_node 2 \
    --nnodes 2 \
    --rdzv_id 2299 \
    --rdzv_backend c10d \
    --rdzv_endpoint master_node_ip_address:29500 \
    ./nlp_example.py
```

--------------------------------

### Example of optimized max_memory for BLOOM-176B

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/big_model_inference.md

An example configuration of `max_memory` for a large model (BLOOM-176B) on an 8x80 A100 setup, allocating more memory to GPUs 1-7 than GPU 0 to optimize batch size.

```python
max_memory = {0: "30GIB", 1: "46GIB", 2: "46GIB", 3: "46GIB", 4: "46GIB", 5: "46GIB", 6: "46GIB", 7: "46GIB"}
```

--------------------------------

### Display Help for Benchmark Script

Source: https://github.com/huggingface/accelerate/blob/main/benchmarks/fsdp2/README.md

Run the main script with the `--help` argument to see all configurable options and understand how to customize the benchmark.

```bash
python3 main.py --help
```

--------------------------------

### Implement a Custom Weights and Biases Tracker

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/tracking.md

Example of a custom tracker for Weights and Biases, logging only on the main process. Ensure wandb is installed.

```python
from accelerate.tracking import GeneralTracker, on_main_process
from typing import Optional

import wandb


class MyCustomTracker(GeneralTracker):
    name = "wandb"
    requires_logging_directory = False

    @on_main_process
    def __init__(self, run_name: str):
        self.run_name = run_name
        run = wandb.init(self.run_name)

    @property
    def tracker(self):
        return self.run.run

    @on_main_process
    def store_init_configuration(self, values: dict):
        wandb.config(values)

    @on_main_process
    def log(self, values: dict, step: Optional[int] = None):
        wandb.log(values, step=step)
```

--------------------------------

### Get Accelerator Device

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/quicktour.md

Retrieve the appropriate device for your training setup using `accelerator.device`. This is recommended for manual device placement if automatic placement is disabled.

```python
device = accelerator.device
```

--------------------------------

### Get Accelerate Logger

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/troubleshooting.md

Use `accelerate.logging.get_logger` instead of the standard Python logging module for synchronized logs in distributed setups. Set the verbosity level with `log_level`.

```python
from accelerate.logging import get_logger

logger = get_logger(__name__, log_level="DEBUG")
```

--------------------------------

### Sequence Parallelism Configuration via YAML

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/sequence_parallelism.md

Configure sequence parallelism using a YAML configuration file, suitable for Accelerate's setup. This example sets up SP for 2 GPUs.

```yaml
distributed_type: DEEPSPEED
deepspeed_config:
  deepspeed_config_file: path/to/ds_config.json
machine_rank: 0
num_machines: 1
num_processes: 2
parallelism_config:
  parallelism_config_dp_replicate_size: 1
  parallelism_config_dp_shard_size: 1  # Must satisfy: 1 × 1 × 2 = 2 num_processes
  parallelism_config_sp_size: 2
  parallelism_config_sp_backend: deepspeed
  parallelism_config_sp_seq_length_is_variable: true
  parallelism_config_sp_attn_implementation: sdpa
```

--------------------------------

### Add Accelerate to requirements.txt

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/sagemaker.md

When using Accelerate within Amazon SageMaker Deep Learning Containers (DLCs), add `accelerate` to your `requirements.txt` file to ensure it's installed during the training job setup.

```text
accelerate
```

--------------------------------

### Configure 2D Parallelism (SP and DP)

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/concept_guides/sequence_parallelism.md

Example of configuring 2D parallelism combining sequence parallelism (SP) and data parallelism (DP) for 4 GPUs. This setup uses 2 sequence parallelism replicas.

```python
# Example: 4 GPUs with 2D parallelism (SP=2, DP=2)
# Ensure: dp_replicate_size × dp_shard_size × sp_size = 2 × 1 × 2 = 4 GPUs
parallelism_config = ParallelismConfig(
    dp_replicate_size=2,
    dp_shard_size=1,  # Explicit: no sharding within replicas
    sp_size=2,
    sp_backend="deepspeed",
    sp_handler=DeepSpeedSequenceParallelConfig(...),
)
```

--------------------------------

### SageMaker Training Job Output Log

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/sagemaker.md

This is an example output log from an Accelerate training job launched on Amazon SageMaker. It shows the progress from starting the job to completion, including metrics and the final model location.

```log
Configuring Amazon SageMaker environment
Converting Arguments to Hyperparameters
Creating Estimator
2021-04-08 11:56:50 Starting - Starting the training job...
2021-04-08 11:57:13 Starting - Launching requested ML instancesProfilerReport-1617883008: InProgress
.........
2021-04-08 11:58:54 Starting - Preparing the instances for training.........
2021-04-08 12:00:24 Downloading - Downloading input data
2021-04-08 12:00:24 Training - Downloading the training image..................
2021-04-08 12:03:39 Training - Training image download completed. Training in progress..
........
epoch 0: {'accuracy': 0.7598039215686274, 'f1': 0.8178438661710037}
epoch 1: {'accuracy': 0.8357843137254902, 'f1': 0.882249560632689}
epoch 2: {'accuracy': 0.8406862745098039, 'f1': 0.8869565217391304}
........
2021-04-08 12:05:40 Uploading - Uploading generated training model
2021-04-08 12:05:40 Completed - Training job completed
Training seconds: 331
Billable seconds: 331
You can find your model data at: s3://your-bucket/accelerate-sagemaker-1-2021-04-08-11-56-47-108/output/model.tar.gz
```

--------------------------------

### Test Accelerate Setup

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/quicktour.md

Use this command to launch a short script that tests your distributed environment after configuration. You can specify a custom config file location with --config_file.

```bash
accelerate test
```

--------------------------------

### Launch Training Loop with notebook_launcher (Basic)

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/notebook.md

Launch a training loop function with specified arguments and number of processes using `notebook_launcher`. This is suitable for single-node, multi-GPU setups.

```python
args = ("fp16", 42, 64)
notebook_launcher(training_loop, args, num_processes=2)
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/megatron_lm.md

Installs PyTorch, Torchvision, and Torchaudio with CUDA 11.3 support using conda. Ensure your system has the matching CUDA version installed.

```bash
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
```

--------------------------------

### Display Help for Accelerate Launch

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/launch.md

Run this command to view all available parameters and options for the accelerate launch command.

```bash
accelerate launch -h
```

--------------------------------

### Launch Script with Torchrun Equivalent

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/launch.md

This example shows how a multi-GPU launch with Accelerate might look when using torchrun, including mixed precision settings.

```bash
MIXED_PRECISION="fp16" torchrun --nproc_per_node=2 --nnodes=1 {script_name.py} {--arg1} {--arg2} ...
```

--------------------------------

### Install Accelerate with conda

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/install.md

Use this command to install Accelerate from the conda-forge channel.

```bash
conda install -c conda-forge accelerate
```

--------------------------------

### Import notebook_launcher

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/notebook.md

Import the `notebook_launcher` utility from the Accelerate library.

```python
from accelerate import notebook_launcher
```

--------------------------------

### Install Accelerate with pip

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/install.md

Use this command to install the latest stable version of Accelerate from PyPI.

```bash
pip install accelerate
```

--------------------------------

### Install PiPPy and Accelerate

Source: https://github.com/huggingface/accelerate/blob/main/examples/inference/pippy/README.md

Install the necessary libraries for PiPPy and Accelerate. Ensure you have at least Python 3.9.

```bash
pip install 'accelerate>=0.27.0' 'torchpippy>=0.2.0'
```

--------------------------------

### Install latest PyTorch

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/distributed_inference.md

Ensure you have the latest PyTorch version installed to use pipeline parallelism features.

```bash
pip install torch
```

--------------------------------

### Multi-line Code Block Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/README.md

Use triple backticks to define multi-line code blocks for displaying examples.

```python
# first line of code
# second line
# etc
```

--------------------------------

### Launch Benchmark with Torchrun

Source: https://github.com/huggingface/accelerate/blob/main/benchmarks/fsdp2/README.md

Alternatively, use `torchrun` for launching the benchmark, specifying the number of processes per node. Example shown for two GPUs.

```bash
torchrun --nproc_per_node 2 main.py
```

--------------------------------

### Launch Training with accelerate CLI

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/basic_tutorials/notebook.md

Use the `accelerate launch` command in the terminal to launch training based on the Accelerate configuration file, bypassing `notebook_launcher`'s behavior of ignoring the config.

```bash
accelerate launch
```

--------------------------------

### Launch Script for ZeRO Stage-3 Offload Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/deepspeed.md

Command to launch a script using Accelerate with a DeepSpeed ZeRO Stage-3 configuration that includes CPU offloading. It specifies model, dataset, and training parameters.

```bash
accelerate launch examples/by_feature/deepspeed_with_config_support.py \
--config_name "gpt2-large" \
--tokenizer_name "gpt2-large" \
--dataset_name "wikitext" \
--dataset_config_name "wikitext-2-raw-v1" \
--block_size 128 \
--output_dir "./clm/clm_deepspeed_stage3_offload_accelerate" \
--learning_rate 5e-4
```

--------------------------------

### Full Big Model Inference Example

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/big_modeling.md

Combines initialization, checkpoint loading, and inference for large models. This example demonstrates the complete workflow.

```python
import torch
from accelerate import init_empty_weights, load_checkpoint_and_dispatch

with init_empty_weights():
    model = MyModel(...)

model = load_checkpoint_and_dispatch(
    model, checkpoint=checkpoint_file, device_map="auto"
)

input = torch.randn(2,3)
device_type = next(iter(model.parameters())).device.type
input = input.to(device_type)
output = model(input)
```

--------------------------------

### Install Megatron-LM

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/megatron_lm.md

Clones the Megatron-LM repository, checks out a specific commit, modifies pyproject.toml for build configuration, and installs the package in editable mode.

```bash
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout 9a1c0d05c992c8a241da384ab27dce2021bb56dd
you need to manually move gpt_builders.py to megatron/training and update
include = [
    "megatron.core", 
    "megatron.core.*",
    "megatron.training",
    "megatron.training.*",
    "megatron.legacy",
    "megatron.legacy.*",
```

```bash
in pyproject.toml file to unblock yourself from using Megatron
pip install --no-use-pep517 -e .
```

--------------------------------

### Install Nvidia APEX

Source: https://github.com/huggingface/accelerate/blob/main/docs/source/usage_guides/megatron_lm.md

Clones the APEX repository, navigates into its directory, and installs it with C++ and CUDA extensions. This is required for certain Megatron-LM functionalities.

```bash
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
cd ..
```