### Install minGPT and huggingface_hub

Source: https://huggingface.co/docs/accelerate/main/usage_guides/quantization

Clone the minGPT repository and install it along with the huggingface_hub library to run examples.

```bash
git clone https://github.com/karpathy/minGPT.git
pip install minGPT/ 
pip install huggingface_hub
```

--------------------------------

### Run Example Script on Intel Gaudi

Source: https://huggingface.co/docs/accelerate/main/usage_guides/gaudi

Use this command to directly test Accelerate functionality on Intel Gaudi devices. Ensure you have the necessary environment setup.

```bash
accelerate launch /examples/cv_example.py --data_dir images
```

--------------------------------

### Self-Contained Causal LM Example Setup

Source: https://huggingface.co/docs/accelerate/main/usage_guides/gradient_accumulation

This code sets up a minimal causal language modeling example with a custom dataset and dataloader, demonstrating how to integrate Accelerate for gradient accumulation.

```python
import torch
import copy
from accelerate import Accelerator
from accelerate.utils import set_seed
from accelerate.logging import  get_logger
from torch.utils.data import Dataset, DataLoader
import math
import contexlib

# seed
set_seed(0)
logger = get_logger(__name__)

class MyDataset(Dataset):
    def __init__(self, num_samples):
        super().__init__()
        self.len = num_samples

    def __getitem__(self, index):
        input_ids = torch.arange(1, index+2, dtype=torch.float32)
        labels = torch.remainder(input_ids, 2)
        return {"input_ids": input_ids, "labels": labels}

    def __len__(self):
        return self.len
    
def collate_fn(features):
    input_ids = torch.nn.utils.rnn.pad_sequence([f["input_ids"] for f in features], batch_first=True, padding_value=-100)
    labels = torch.nn.utils.rnn.pad_sequence([f["labels"] for f in features], batch_first=True, padding_value=-100)
    return {"input_ids": input_ids[..., None], "labels": labels[..., None]}

# define toy inputs and labels
gradient_accumulation_steps = 2
per_device_batch_size = 4

# define accelerator
accelerator = Accelerator(gradient_accumulation_steps=gradient_accumulation_steps)
```

--------------------------------

### Install bitsandbytes

Source: https://huggingface.co/docs/accelerate/main/usage_guides/quantization

Install the bitsandbytes library to enable quantization features. For non-CUDA devices, refer to the bitsandbytes installation guide.

```bash
pip install bitsandbytes
```

--------------------------------

### Example SageMaker Training Launch

Source: https://huggingface.co/docs/accelerate/main/usage_guides/sagemaker

An example command to launch a SageMaker training job using an example script. Ensure your script includes `accelerator.save('/opt/ml/model')` for model artifact upload.

```bash
accelerate launch ./examples/sagemaker_example.py
```

--------------------------------

### Launch Accelerate Configuration

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Use this command to start an interactive process that guides you through creating and saving a `default_config.yml` file for your training system. It's recommended to run this first on your machine.

```bash
accelerate config [arguments]
```

--------------------------------

### Running Example Script for Context Parallelism

Source: https://huggingface.co/docs/accelerate/main/concept_guides/context_parallelism

Launch an example script demonstrating context parallelism, FSDP2, and checkpointing on multiple GPUs with a specified sequence length. This command is used to reproduce performance benchmarks.

```bash
accelerate launch --use-fsdp --fsdp-activation-checkpointing=TRUE examples/fsdp2/nd_parallel.py --cp-size=8 --sequence-length=128000
```

--------------------------------

### Install huggingface_hub

Source: https://huggingface.co/docs/accelerate/main/concept_guides/big_model_inference

Install the huggingface_hub library to download model checkpoints.

```bash
pip install huggingface_hub
```

--------------------------------

### Install Nvidia APEX

Source: https://huggingface.co/docs/accelerate/main/usage_guides/megatron_lm

Clone the APEX repository, navigate into it, and install it with C++ and CUDA extensions enabled. This is necessary for mixed-precision training and other optimizations.

```bash
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
cd ..
```

--------------------------------

### Full Experiment Tracking Example

Source: https://huggingface.co/docs/accelerate/main/usage_guides/tracking

A complete example demonstrating initialization, training loop logging, and ending the tracking session.

```python
from accelerate import Accelerator

accelerator = Accelerator(log_with="all")
config = {
    "num_iterations": 5,
    "learning_rate": 1e-2,
    "loss_function": str(my_loss_function),
}

accelerator.init_trackers("example_project", config=config)

my_model, my_optimizer, my_training_dataloader = accelerator.prepare(my_model, my_optimizer, my_training_dataloader)
device = accelerator.device
my_model.to(device)

for iteration in range(config["num_iterations"]):
    for step, batch in enumerate(my_training_dataloader):
        my_optimizer.zero_grad()
        inputs, targets = batch
        inputs = inputs.to(device)
        targets = targets.to(device)
        outputs = my_model(inputs)
        loss = my_loss_function(outputs, targets)
        accelerator.backward(loss)
        my_optimizer.step()
        accelerator.log({"training_loss": loss}, step=step)
accelerator.end_training()
```

--------------------------------

### Install minGPT

Source: https://huggingface.co/docs/accelerate/main/concept_guides/big_model_inference

Clone the minGPT repository and install it to use its GPT model implementation.

```bash
git clone https://github.com/karpathy/minGPT.git
pip install minGPT/
```

--------------------------------

### Install Accelerate from source

Source: https://huggingface.co/docs/accelerate/main/usage_guides/quantization

Install the latest version of Accelerate directly from its GitHub repository.

```bash
pip install git+https://github.com/huggingface/accelerate.git
```

--------------------------------

### Basic Script Launch with Accelerate

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/launch

Use `accelerate launch` followed by your script name and its arguments to start your distributed training.

```bash
accelerate launch {script_name.py} --arg1 --arg2 ...
```

--------------------------------

### Install Accelerate with pip

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Use this command to install the Accelerate library from PyPI.

```bash
pip install accelerate
```

--------------------------------

### Install Accelerate with SageMaker SDK

Source: https://huggingface.co/docs/accelerate/main/usage_guides/sagemaker

Install the Accelerate library with the SageMaker integration. This command ensures you have the necessary components for running Accelerate on SageMaker.

```bash
pip install "accelerate[sagemaker]" --upgrade
```

--------------------------------

### Self-contained Causal LM Example with Gradient Accumulation

Source: https://huggingface.co/docs/accelerate/main/usage_guides/gradient_accumulation

This example demonstrates how to implement gradient accumulation for a causal language model. It sets up a dataset, dataloader, model, and optimizer, then iterates through batches, accumulating gradients before stepping the optimizer. It also includes logic for handling multi-device setups and calculating loss correctly.

```python
import copy
import math
import contextlib

import torch
from torch.utils.data import DataLoader, Dataset

from accelerate import Accelerator


class MyDataset(Dataset):
    def __init__(self, num_samples):
        self.num_samples = num_samples

    def __len__(self):
        return self.num_samples

    def __getitem__(self, idx):
        return {"input_ids": torch.randint(0, 100, (1, 10)), "labels": torch.randint(0, 100, (1, 10))}

def collate_fn(batch):
    return {"input_ids": torch.cat([item["input_ids"] for item in batch]), "labels": torch.cat([item["labels"] for item in batch])}


accelerator = Accelerator()

per_device_batch_size = 1
gradient_accumulation_steps = 4

# define dataset and dataloader
# for this toy example, we'll compute gradient descent over one single global batch
dataset = MyDataset(per_device_batch_size*gradient_accumulation_steps*accelerator.num_processes)
dataloader = DataLoader(dataset, batch_size=per_device_batch_size, collate_fn=collate_fn)

# define model, model_optimizer and loss function
model = torch.nn.Linear(1, 2, bias=False)
model_clone = copy.deepcopy(model)
criterion = torch.nn.CrossEntropyLoss(reduction="sum") # must sum over samples rather than averaging
model_optimizer = torch.optim.SGD(model.parameters(), lr=0.08)

logger.warning(f"initial model weight is {model.weight.detach().cpu().squeeze()}")
logger.warning(f"initial model clone weight is {model_clone.weight.detach().cpu().squeeze()}")

# prepare artifacts - accelerator handles device placement and dataloader splitting
model, model_optimizer = accelerator.prepare(model, model_optimizer)
dataloader = accelerator.prepare_data_loader(dataloader, device_placement=True)
training_iterator = iter(dataloader)

num_samples_in_epoch = len(dataloader)
remainder = num_samples_in_epoch % gradient_accumulation_steps
remainder = remainder if remainder != 0 else gradient_accumulation_steps
total_gradient_updates = math.ceil(num_samples_in_epoch / gradient_accumulation_steps)

total_batched_samples = 0
for update_step in range(total_gradient_updates):
        # In order to correctly the total number of non-padded tokens on which we'll compute the cross-entropy loss
        # we need to pre-load the full local batch - i.e the next per_device_batch_size * accumulation_steps samples
        batch_samples = []
        num_batches_in_step = gradient_accumulation_steps if update_step != (total_gradient_updates - 1) else remainder
        for _ in range(num_batches_in_step):
            batch_samples += [next(training_iterator)]
            
        # get local num items in batch 
        local_num_items_in_batch = sum([(batch["labels"].ne(-100)).sum() for batch in batch_samples])
        logger.warning(f"Step {update_step} - Device {accelerator.process_index} - num items in the local batch {local_num_items_in_batch}", main_process_only=False)

        # to compute it correctly in a multi-device DDP training, we need to gather the total number of items in the full batch.
        num_items_in_batch = accelerator.gather(local_num_items_in_batch).sum().item()
        logger.warning(f"Total num items {num_items_in_batch}")

        for i, batch in enumerate(batch_samples):
            inputs, labels = batch["input_ids"], batch["labels"]
            total_batched_samples += 1
            # if we perform gradient accumulation in a multi-devices set-up, we want to avoid unnecessary communications when accumulating
            # cf: https://muellerzr.github.io/blog/gradient_accumulation.html
            if (i < len(batch_samples) - 1 and accelerator.num_processes > 1):
                ctx = model.no_sync
            else:
                ctx = contextlib.nullcontext
            with ctx():

                outputs = model(inputs)
                loss = criterion(outputs.view(-1, 2), labels.view(-1).to(torch.int64))
                
                # We multiply by num_processes because the DDP calculates the average gradient across all devices whereas dividing by num_items_in_batch already takes into account all devices
                # Same reason for gradient_accumulation_steps, but this times it's Accelerate that calculate the average gradient across the accumulated steps 
                loss = (loss * gradient_accumulation_steps * accelerator.num_processes) / num_items_in_batch
                accelerator.backward(loss)
        model_optimizer.step()
        model_optimizer.zero_grad()
                

logger.warning(f"Device {accelerator.process_index} - w/ accumulation, the final model weight is {accelerator.unwrap_model(model).weight.detach().cpu().squeeze()}", main_process_only=False)

# We know do the same operation but on a single device and without gradient accumulation

if accelerator.is_main_process:
    # prepare one single entire batch
    dataloader = DataLoader(dataset, batch_size=len(dataset), collate_fn=collate_fn)
    full_batch_without_accum = next(iter(dataloader))
    total_inputs, total_labels = full_batch_without_accum["input_ids"], full_batch_without_accum["labels"]
    model_clone_optimizer = torch.optim.SGD(model_clone.parameters(), lr=0.08)
    
    # train the cloned model
    loss = torch.nn.CrossEntropyLoss(reduction="mean")(model_clone(total_inputs).view(-1, 2), total_labels.view(-1).to(torch.int64))
    model_clone_optimizer.zero_grad()
    loss.backward()
    model_clone_optimizer.step()
    
    # We should have the same final weights.
    logger.warning(f"w/o accumulation, the final model weight is {model_clone.weight.detach().cpu().squeeze()}")

```

--------------------------------

### Create Example Inputs for Model Tracing

Source: https://huggingface.co/docs/accelerate/main/usage_guides/distributed_inference

Generate example inputs, including batch size and sequence length, to help `torch.distributed.pipelining` trace the model. The size of these inputs determines the relative batch size used during inference.

```python
input = torch.randint(
    low=0,
    high=config.vocab_size,
    size=(2, 1024),  # bs x seq_len
    device="cpu",
    dtype=torch.int64,
    requires_grad=False,
)
```

--------------------------------

### Install Accelerate from local source (editable)

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Install an editable version of Accelerate from a locally cloned repository, useful for development and contributing.

```bash
git clone https://github.com/huggingface/accelerate
cd accelerate
pip install -e .
```

--------------------------------

### Install Accelerate from GitHub

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Install the latest development version of Accelerate directly from its GitHub repository.

```bash
pip install git+https://github.com/huggingface/accelerate
```

--------------------------------

### Full Big Model Inference Example

Source: https://huggingface.co/docs/accelerate/main/usage_guides/big_modeling

A complete example demonstrating the workflow: initializing an empty model, loading and dispatching weights, and performing inference. This approach allows running large models by distributing layers across available devices.

```python
import torch
from accelerate import init_empty_weights, load_checkpoint_and_dispatch

with init_empty_weights():
    model = MyModel(...)

model = load_checkpoint_and_dispatch(
    model, checkpoint=checkpoint_file, device_map="auto"
)

input = torch.randn(2,3)
device_type = next(iter(model.parameters())).device.type
input = input.to(device_type)
output = model(input)
```

--------------------------------

### Example FSDP Configuration

Source: https://huggingface.co/docs/accelerate/main/usage_guides/fsdp

This is an example configuration generated by `accelerate config` when FSDP is enabled. It specifies distributed type, mixed precision, and FSDP-specific settings.

```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch_policy: BACKWARD_PRE
  fsdp_forward_prefetch: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_transformer_layer_cls_to_wrap: BertLayer
  fsdp_use_orig_params: true
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Get Help for Accelerate Launch

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/launch

View all available parameters and options for `accelerate launch` by running the command with the `-h` flag.

```bash
accelerate launch -h
```

--------------------------------

### accelerate.commands.config.default.write_basic_config

Source: https://huggingface.co/docs/accelerate/main/package_reference/utilities

Creates and saves a basic configuration file for Accelerate, suitable for local multi-GPU or CPU-only setups. This function can be used as an alternative to `accelerate config` for quick setup.

```APIDOC
## write_basic_config

### Description
Creates and saves a basic cluster config to be used on a local machine with potentially multiple GPUs. Will also set CPU if it is a CPU-only machine.

When setting up 🤗 Accelerate for the first time, rather than running `accelerate config` [~utils.write_basic_config] can be used as an alternative for quick configuration.

### Parameters
* **mixed_precision** (`str`, *optional*, defaults to "no") : Mixed Precision to use. Should be one of "no", "fp16", or "bf16"
* **save_location** (`str`, *optional*, defaults to `default_json_config_file`) : Optional custom save location. Should be passed to `--config_file` when using `accelerate launch`. Default location is inside the huggingface cache folder (`~/.cache/huggingface`) but can be overridden by setting the `HF_HOME` environmental variable, followed by `accelerate/default_config.yaml`.
```

--------------------------------

### Example Accelerate Configuration YAML

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/launch

This YAML file defines a custom configuration for Accelerate, specifying multi-GPU usage with mixed precision.

```yaml
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: MULTI_GPU
fsdp_config: {}
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 2
use_cpu: false
```

--------------------------------

### Example Max Memory for BLOOM-176B

Source: https://huggingface.co/docs/accelerate/main/concept_guides/big_model_inference

An example of a `max_memory` dictionary optimized for BLOOM-176B on an 8x80 A100 setup, allocating more memory to GPUs 1-7 than GPU 0.

```python
max_memory = {0: "30GIB", 1: "46GIB", 2: "46GIB", 3: "46GIB", 4: "46GIB", 5: "46GIB", 6: "46GIB", 7: "46GIB"}
```

--------------------------------

### SageMaker Training Job Output Log

Source: https://huggingface.co/docs/accelerate/main/usage_guides/sagemaker

Example output log from a SageMaker training job launched with Accelerate, showing the progression from setup to completion and model upload.

```log
Configuring Amazon SageMaker environment
Converting Arguments to Hyperparameters
Creating Estimator
2021-04-08 11:56:50 Starting - Starting the training job...
2021-04-08 11:57:13 Starting - Launching requested ML instancesProfilerReport-1617883008: InProgress
.........
2021-04-08 11:58:54 Starting - Preparing the instances for training.........
2021-04-08 12:00:24 Downloading - Downloading input data
2021-04-08 12:00:24 Training - Downloading the training image..................
2021-04-08 12:03:39 Training - Training image download completed. Training in progress..
........
epoch 0: {'accuracy': 0.7598039215686274, 'f1': 0.8178438661710037}
epoch 1: {'accuracy': 0.8357843137254902, 'f1': 0.882249560632689}
epoch 2: {'accuracy': 0.8406862745098039, 'f1': 0.8869565217391304}
........
2021-04-08 12:05:40 Uploading - Uploading generated training model
2021-04-08 12:05:40 Completed - Training job completed
Training seconds: 331
Billable seconds: 331
You can find your model data at: s3://your-bucket/accelerate-sagemaker-1-2021-04-08-11-56-47-108/output/model.tar.gz
```

--------------------------------

### Test Accelerate Configuration

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Runs a verification script to ensure 🤗 Accelerate is correctly configured on your system. Use this command to check your setup before starting distributed training.

```bash
accelerate test [arguments]
```

--------------------------------

### Initialize Accelerator and Prepare Training Components

Source: https://huggingface.co/docs/accelerate/main/package_reference/accelerator

Instantiate the Accelerator with gradient accumulation steps and prepare dataloaders, models, optimizers, and schedulers for distributed training.

```python
from accelerate import Accelerator

accelerator = Accelerator(gradient_accumulation_steps=1)
dataloader, model, optimizer, scheduler = accelerator.prepare(dataloader, model, optimizer, scheduler)
```

--------------------------------

### Show Help Message

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Display the help message for the `accelerate launch` command and exit.

```bash
-h, --help
```

--------------------------------

### Teacher-Student Model Training Loop

Source: https://huggingface.co/docs/accelerate/main/usage_guides/deepspeed_multiple_model

Example training loop for a teacher-student model setup. The teacher model is set to evaluation mode and its outputs are used without gradients, while the student model is trained.

```python
teacher_model.eval()
student_model.train()
for batch in train_dataloader:
    with torch.no_grad():
        output_teacher = teacher_model(**batch)
    output_student = student_model(**batch)
    # Combine the losses or modify it in some way
    loss = output_teacher.loss + student_model.loss
    accelerator.backward(loss)
    optimizer.step()
    scheduler.step()
    optimizer.zero_grad()
```

--------------------------------

### Initialize Trackers and Log Hyperparameters

Source: https://huggingface.co/docs/accelerate/main/usage_guides/tracking

Call `Accelerator.init_trackers()` at the start of your experiment to set up the project and log initial hyperparameters. The project name is required, and a configuration dictionary is optional.

```python
hps = {"num_iterations": 5, "learning_rate": 1e-2}
accelerator.init_trackers("my_project", config=hps)
```

--------------------------------

### Add Accelerate to SageMaker requirements.txt

Source: https://huggingface.co/docs/accelerate/main/usage_guides/sagemaker

When using Accelerate within SageMaker's Deep Learning Containers (DLCs), add 'accelerate' to your `requirements.txt` file to ensure it's installed during the training job setup.

```text
accelerate
```

--------------------------------

### TPU Configuration with Accelerate Installation

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Configure a TPU environment and ensure Accelerate is installed on the pod. You can specify a particular version or install the latest development version from GitHub.

```bash
accelerate tpu-config --install_accelerate --accelerate_version dev
```

--------------------------------

### Launch Training Function with notebook_launcher

Source: https://huggingface.co/docs/accelerate/main/concept_guides/training_tpu

Use `notebook_launcher` to start the training process defined in `training_function`. Accelerate defaults to 8 processes if configured for TPU.

```python
from accelerate import notebook_launcher

notebook_launcher(training_function)
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://huggingface.co/docs/accelerate/main/usage_guides/megatron_lm

Install PyTorch, torchvision, and torchaudio with the specified CUDA toolkit version (e.g., 11.3). Ensure your system has the corresponding CUDA version installed.

```bash
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
```

--------------------------------

### Basic notebook_launcher usage

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/notebook

Launch a training loop with specified arguments and number of processes.

```python
args = ("fp16", 42, 64)
notebook_launcher(training_loop, args, num_processes=2)
```

--------------------------------

### Initialize Accelerator with Multiple Trackers

Source: https://huggingface.co/docs/accelerate/main/usage_guides/tracking

Combine your custom tracker with existing trackers, including 'all', by passing a list to log_with.

```python
tracker = MyCustomTracker("some_run_name")
accelerator = Accelerator(log_with=[tracker, "all"])
```

--------------------------------

### Install PyTorch XLA Wheels

Source: https://huggingface.co/docs/accelerate/main/package_reference/utilities

Use this helper function to install the appropriate PyTorch XLA wheels on Google Colaboratory. Set `upgrade=True` to upgrade PyTorch and install the latest `torch_xla` wheels.

```python
from accelerate.utils import install_xla

install_xla(upgrade=True)
```

--------------------------------

### Instantiate Optimizer and LR Scheduler

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/notebook

Set up the optimizer and learning rate scheduler for the training process.

```python
optimizer = torch.optim.Adam(params=model.parameters(), lr=3e-2 / 25)
lr_scheduler = OneCycleLR(optimizer=optimizer, max_lr=3e-2, epochs=5, steps_per_epoch=len(train_dataloader))
```

--------------------------------

### Configure Accelerate

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Run this command to initiate the Accelerate configuration process, answering prompts to set up your training environment.

```bash
accelerate config
```

--------------------------------

### Accelerate CLI Help

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Displays help information for the Accelerate CLI, listing available commands and their options.

```bash
accelerate --help
```

--------------------------------

### Install Accelerate with conda

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Use this command to install the Accelerate library using conda from the conda-forge channel.

```bash
conda install -c conda-forge accelerate
```

--------------------------------

### Install Latest PyTorch

Source: https://huggingface.co/docs/accelerate/main/usage_guides/distributed_inference

Ensure you have the latest PyTorch version installed for optimal performance and compatibility with distributed features.

```bash
pip install torch
```

--------------------------------

### Initialize Accelerator with a Custom Tracker

Source: https://huggingface.co/docs/accelerate/main/usage_guides/tracking

Instantiate your custom tracker and pass it to the Accelerator's log_with argument.

```python
tracker = MyCustomTracker("some_run_name")
accelerator = Accelerator(log_with=tracker)
```

--------------------------------

### Launch Training Script with Accelerate

Source: https://huggingface.co/docs/accelerate/main/quicktour

Launch your training script using Accelerate after configuring your environment. Pass any necessary script arguments after the script path.

```bash
accelerate launch path_to_script.py --args_for_the_script
```

--------------------------------

### Install Megatron-LM

Source: https://huggingface.co/docs/accelerate/main/usage_guides/megatron_lm

Clone the Megatron-LM repository, checkout a specific commit, modify the pyproject.toml file to include necessary packages, and install it in editable mode.

```bash
git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout 9a1c0d05c992c8a241da384ab27dce2021bb56dd
you need to manually move gpt_builders.py to megatron/training and update
include = [
    "megatron.core", 
    "megatron.core.*",
    "megatron.training",
    "megatron.training.*",
    "megatron.legacy",
    "megatron.legacy.*",
```

```bash
in pyproject.toml file to unblock yourself from using Megatron
pip install --no-use-pep517 -e .
```

--------------------------------

### Specify Configuration File

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Provide a configuration file to use for default values in the launching script.

```bash
--config_file CONFIG_FILE
```

--------------------------------

### Profiler Table Output Example

Source: https://huggingface.co/docs/accelerate/main/usage_guides/profiler

Example of the tabular output generated by PyTorch Profiler, showing operator names, CPU times, and call counts. This output helps identify performance bottlenecks.

```text
---------------------------------  ------------  ------------  ------------  ------------  
                             Name      Self CPU     CPU total  CPU time avg    # of Calls  
---------------------------------  ------------  ------------  ------------  ------------  
                     aten::conv2d     171.000us      52.260ms       2.613ms            20  
                aten::convolution     227.000us      52.089ms       2.604ms            20  
               aten::_convolution     270.000us      51.862ms       2.593ms            20  
         aten::mkldnn_convolution      51.273ms      51.592ms       2.580ms            20  
                 aten::batch_norm     118.000us       7.059ms     352.950us            20  
     aten::_batch_norm_impl_index     315.000us       6.941ms     347.050us            20  
          aten::native_batch_norm       6.305ms       6.599ms     329.950us            20  
                 aten::max_pool2d      40.000us       4.008ms       4.008ms             1  
    aten::max_pool2d_with_indices       3.968ms       3.968ms       3.968ms             1  
                       aten::add_     780.000us     780.000us      27.857us            28  
---------------------------------  ------------  ------------  ------------  ------------  
Self CPU time total: 67.016ms
```

--------------------------------

### Import notebook_launcher

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/notebook

Import the necessary function from the accelerate library.

```python
from accelerate import notebook_launcher
```

--------------------------------

### Multi-node training with notebook_launcher

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/notebook

Configure multi-node training by specifying master address, node rank, total nodes, and processes per node.

```python
notebook_launcher(training_loop, args, master_addr="172.31.43.8", node_rank=0, num_nodes=2, num_processes=8)
```

```python
notebook_launcher(training_loop, args, master_addr="172.31.43.8", node_rank=1, num_nodes=2, num_processes=8)
```

--------------------------------

### Test Accelerate Distributed Environment

Source: https://huggingface.co/docs/accelerate/main/quicktour

Use this command to launch a short script that tests your distributed environment setup. You can specify a configuration file with `--config_file` if it's not in the default location.

```bash
accelerate test
```

--------------------------------

### Self-Contained Gradient Accumulation Example

Source: https://huggingface.co/docs/accelerate/main/usage_guides/gradient_accumulation

A runnable example demonstrating gradient accumulation. It compares the final model weights trained with and without accumulation, showing identical results when batch size and total steps are equivalent.

```python
import torch
import copy
from accelerate import Accelerator
from accelerate.utils import set_seed
from torch.utils.data import TensorDataset, DataLoader

# seed
set_seed(0)

# define toy inputs and labels
x = torch.tensor([1., 2., 3., 4., 5., 6., 7., 8.])
y = torch.tensor([2., 4., 6., 8., 10., 12., 14., 16.])
gradient_accumulation_steps = 4
per_device_batch_size = len(x) // gradient_accumulation_steps

# define dataset and dataloader
dataset = TensorDataset(x, y)
dataloader = DataLoader(dataset, batch_size=per_device_batch_size)

# define model, optimizer and loss function
class SimpleLinearModel(torch.nn.Module):
    def __init__(self):
        super(SimpleLinearModel, self).__init__()
        self.weight = torch.nn.Parameter(torch.zeros((1, 1)))

    def forward(self, inputs):
        return inputs @ self.weight

model = SimpleLinearModel()
model_clone = copy.deepcopy(model)
criterion = torch.nn.MSELoss()
model_optimizer = torch.optim.SGD(model.parameters(), lr=0.02)
accelerator = Accelerator(gradient_accumulation_steps=gradient_accumulation_steps)
model, model_optimizer, dataloader = accelerator.prepare(model, model_optimizer, dataloader)
model_clone_optimizer = torch.optim.SGD(model_clone.parameters(), lr=0.02)
print(f"initial model weight is {model.weight.mean().item():.5f}")
print(f"initial model weight is {model_clone.weight.mean().item():.5f}")
for i, (inputs, labels) in enumerate(dataloader):
    with accelerator.accumulate(model):
        inputs = inputs.view(-1, 1)
        print(i, inputs.flatten())
        labels = labels.view(-1, 1)
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        accelerator.backward(loss)
        model_optimizer.step()
        model_optimizer.zero_grad()
loss = criterion(x.view(-1, 1) @ model_clone.weight, y.view(-1, 1))
model_clone_optimizer.zero_grad()
loss.backward()
model_clone_optimizer.step()
print(f"w/ accumulation, the final model weight is {model.weight.mean().item():.5f}")
print(f"w/o accumulation, the final model weight is {model_clone.weight.mean().item():.5f}")
```

```text
initial model weight is 0.00000
initial model weight is 0.00000
0 tensor([1., 2.])
1 tensor([3., 4.])
2 tensor([5., 6.])
3 tensor([7., 8.])
w/ accumulation, the final model weight is 2.04000
w/o accumulation, the final model weight is 2.04000
```

--------------------------------

### Enable TPU Training

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Enable training on a TPU.

```bash
--tpu
```

--------------------------------

### Offline Consolidation of ZeRO Checkpoint

Source: https://huggingface.co/docs/accelerate/main/usage_guides/deepspeed

This command-line example shows how to use the `zero_to_fp32.py` script to consolidate ZeRO Stage-3 partitions into a single FP32 model checkpoint. This process requires no configuration files or GPUs.

```bash
$ cd /path/to/checkpoint_dir
$ ./zero_to_fp32.py . pytorch_model.bin
```

--------------------------------

### accelerate.utils.is_npu_available

Source: https://huggingface.co/docs/accelerate/main/package_reference/utilities

Checks if `torch_npu` is installed and potentially if a NPU is in the environment.

```APIDOC
## accelerate.utils.is_npu_available

### Description
Checks if `torch_npu` is installed and potentially if a NPU is in the environment.

### Parameters:
None

### Returns:
- bool: True if NPU is available, False otherwise.
```

--------------------------------

### MLflowTracker

Source: https://huggingface.co/docs/accelerate/main/package_reference/tracking

A Tracker class that supports `mlflow`. Should be initialized at the start of your script.

```APIDOC
## MLflowTracker

### Description
A `Tracker` class that supports `mlflow`. Should be initialized at the start of your script.

### Parameters
- **experiment_name** (str, *optional*) - Name of the experiment. Environment variable MLFLOW_EXPERIMENT_NAME has priority over this argument.
- **logging_dir** (str or os.PathLike, defaults to ".") - Location for mlflow logs to be stored.
- **run_id** (str, *optional*) - If specified, get the run with the specified UUID and log parameters and metrics under that run. The run’s end time is unset and its status is set to running, but the run’s other attributes (source_version, source_type, etc.) are not changed. Environment variable MLFLOW_RUN_ID has priority over this argument.
- **tags** (Dict[str, str], *optional*) - An optional `dict` of `str` keys and values, or a `str` dump from a `dict`, to set as tags on the run. If a run is being resumed, these tags are set on the resumed run. If a new run is being created, these tags are set on the new run. Environment variable MLFLOW_TAGS has priority over this argument.
- **nested_run** (bool, *optional*, defaults to `False`) - Controls whether run is nested in parent run. True creates a nested run. Environment variable MLFLOW_NESTED_RUN has priority over this argument.
- **run_name** (str, *optional*) - Name of new run (stored as a mlflow.runName tag). Used only when `run_id` is unspecified.
- **description** (str, *optional*) - An optional string that populates the description box of the run. If a run is being resumed, the description is set on the resumed run. If a new run is being created, the description is set on the new run.
```

--------------------------------

### AimTracker

Source: https://huggingface.co/docs/accelerate/main/package_reference/tracking

A Tracker class that supports `aim`. Should be initialized at the start of your script.

```APIDOC
## AimTracker

### Description
A `Tracker` class that supports `aim`. Should be initialized at the start of your script.

### Parameters
- **run_name** (str) - The name of the experiment run.
- ****kwargs** (additional keyword arguments, *optional*) - Additional key word arguments passed along to the `Run.__init__` method.
```

--------------------------------

### TrackioTracker

Source: https://huggingface.co/docs/accelerate/main/package_reference/tracking

A Tracker class that supports trackio. Should be initialized at the start of your script.

```APIDOC
## TrackioTracker

#### accelerate.tracking.TrackioTracker

[Source](https://github.com/huggingface/accelerate/blob/main/src/accelerate/tracking.py#L418)

A `Tracker` class that supports `trackio`. Should be initialized at the start of your script.

### __init__

accelerate.tracking.TrackioTracker.__init__

[Source](https://github.com/huggingface/accelerate/blob/main/src/accelerate/tracking.py#L435)

**Parameters:**

- **run_name** (`str`) : The name of the experiment run. Will be used as the `project` name when instantiating trackio.
- ****kwargs** (additional keyword arguments, *optional*) : Additional key word arguments passed along to the `trackio.init` method. Refer to this [init](https://github.com/gradio-app/trackio/blob/814809552310468b13f84f33764f1369b4e5136c/trackio/__init__.py#L22) to see all supported key word arguments.
```

--------------------------------

### DataLoaderShard

Source: https://huggingface.co/docs/accelerate/main/package_reference/torch_wrappers

Subclass of DataLoaderAdapter that handles device placement and current distributed setup for DataLoaders.

```APIDOC
## DataLoaderShard

#### accelerate.data_loader.DataLoaderShard

Subclass of `DataLoaderAdapter` that will deal with device placement and current distributed setup.

**Available attributes:**

- **total_batch_size** (`int`) -- Total batch size of the dataloader across all processes.
  Equal to the original batch size when `split_batches=True`; otherwise the original batch size * the total
  number of processes

- **total_dataset_length** (`int`) -- Total length of the inner dataset across all processes.

**Parameters:**

dataset (`torch.utils.data.dataset.Dataset`) : The dataset to use to build this dataloader.

device (`torch.device`, *optional*) : If passed, the device to put all batches on.

rng_types (list of `str` or [RNGType](/docs/accelerate/main/en/package_reference/utilities#accelerate.utils.RNGType)) : The list of random number generators to synchronize at the beginning of each iteration. Should be one or several of:  - "torch": the base torch random number generator - "cuda": the CUDA random number generator (GPU only) - "xla": the XLA random number generator (TPU only) - "generator": an optional `torch.Generator`

synchronized_generator (`torch.Generator`, *optional*) : A random number generator to keep synchronized across processes.

skip_batches (`int`, *optional*, defaults to 0) : The number of batches to skip at the beginning.

use_stateful_dataloader (`bool`, *optional*, defaults to `False`) : Whether to have this class adapt `StatefulDataLoader` from `torchdata` instead of the regular `DataLoader`.

- ****kwargs** (additional keyword arguments, *optional*) : All other keyword arguments to pass to the regular `DataLoader` initialization.
```

--------------------------------

### Launching with Context Parallelism Flags

Source: https://huggingface.co/docs/accelerate/main/concept_guides/context_parallelism

Enable context parallelism directly from the command line using `accelerate launch` flags. This is an alternative to programmatic configuration.

```bash
accelerate launch --parallelism-config-cp-size 8 --parallelism-config-cp-comm-strategy [allgather|alltoall] ...
```

--------------------------------

### SwanLabTracker

Source: https://huggingface.co/docs/accelerate/main/package_reference/tracking

A Tracker class that supports SwanLab for experiment tracking. It should be initialized at the start of your script.

```APIDOC
## SwanLabTracker

### Description
A `Tracker` class that supports `swanlab`. Should be initialized at the start of your script.

### Method Signature
`__init__(run_name: str, **kwargs)`

### Parameters
#### Keyword Arguments
- **run_name** (`str`) : The name of the experiment run.
- ****kwargs** (additional keyword arguments, *optional*) : Additional key word arguments passed along to the `swanlab.init` method.
```

--------------------------------

### Configure Accelerate for Single-Node CPU Training

Source: https://huggingface.co/docs/accelerate/main/usage_guides/intel_cpu

Run `accelerate config` and select 'This machine', 'No distributed training', and 'yes' for CPU-only training to generate a configuration file for optimizing single-node CPU training.

```bash
$ accelerate config
-----------------------------------------------------------------------------------------------------------------------------------------------------------
In which compute environment are you running?
This machine
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Which type of machine are you using?
No distributed training
Do you want to run your training on CPU only (even if a GPU / Apple Silicon device is available)? [yes/NO]:yes
Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
Do you want to use DeepSpeed? [yes/NO]: NO
-----------------------------------------------------------------------------------------------------------------------------------------------------------
Do you wish to use FP16 or BF16 (mixed precision)?
bf16

```

```yaml
compute_environment: LOCAL_MACHINE
distributed_type: 'NO'
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: true

```

```bash
accelerate launch my_script.py --args_to_my_script
```

```bash
accelerate launch examples/nlp_example.py
```

--------------------------------

### Check Accelerate Environment Configuration

Source: https://huggingface.co/docs/accelerate/main/basic_tutorials/install

Verify your Accelerate setup and configuration by running this command. It displays details about your environment, hardware, and Accelerate's current settings.

```bash
accelerate env
```

--------------------------------

### ClearMLTracker

Source: https://huggingface.co/docs/accelerate/main/package_reference/tracking

A Tracker class that supports ClearML for experiment tracking. It should be initialized at the start of your script.

```APIDOC
## ClearMLTracker

### Description
A `Tracker` class that supports `clearml`. Should be initialized at the start of your script.

### Method Signature
`__init__(run_name: typing.Optional[str] = None, **kwargs)`

### Parameters
#### Keyword Arguments
- **run_name** (`str`, *optional*) : Name of the experiment. Environment variables `CLEARML_PROJECT` and `CLEARML_TASK` have priority over this argument.
- ****kwargs** (additional keyword arguments, *optional*) : Kwargs passed along to the `Task.__init__` method.
```

--------------------------------

### Select Training Paradigm

Source: https://huggingface.co/docs/accelerate/main/package_reference/cli

Choose the distributed training paradigm by enabling DeepSpeed, FSDP, or Megatron-LM.

```bash
--use_deepspeed
```

```bash
--use_fsdp
```

```bash
--use_megatron_lm
```

--------------------------------

### prepare_scheduler

Source: https://huggingface.co/docs/accelerate/main/package_reference/accelerator

Prepares a PyTorch Scheduler for training in any distributed setup. It is recommended to use Accelerator.prepare() instead.

```APIDOC
## prepare_scheduler

### Description
Prepares a PyTorch Scheduler for training in any distributed setup. It is recommended to use [Accelerator.prepare()](/docs/accelerate/main/en/package_reference/accelerator#accelerate.Accelerator.prepare) instead.

### Parameters
- **scheduler** (torch.optim.lr_scheduler.LRScheduler) - Required - A vanilla PyTorch scheduler to prepare

### Example
```python
>>> import torch
>>> from accelerate import Accelerator

>>> accelerator = Accelerator()
>>> optimizer = torch.optim.Adam(...)
>>> scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, ...)
>>> scheduler = accelerator.prepare_scheduler(scheduler)
```
```