### Run Activation Quantization Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Commands to set up and run the activation quantization example for BERT models. Ensure all requirements are installed before execution.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
```

```shell
DeepSpeedExamples/compression/bert$ bash bash_script/quant_activation.sh
```

--------------------------------

### Run Sparse Pruning Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Commands to install dependencies and execute the sparse pruning script.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
DeepSpeedExamples/compression/bert$ bash bash_script/pruning_sparse.sh
```

--------------------------------

### Install GPT-2 ZeroQuant Requirements

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Install the necessary Python packages for running the GPT-2 ZeroQuant example.

```shell
DeepSpeedExamples/compression/gpt2$ pip install -r requirements.txt
```

--------------------------------

### Install DeepSpeed and Clone Examples

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/deepspeed-chat/README.md

Installs the DeepSpeed library and clones the DeepSpeedExamples repository. Ensure you have DeepSpeed version 0.9.0 or higher.

```bash
pip install deepspeed>=0.9.0

git clone https://github.com/deepspeedai/DeepSpeedExamples.git
cd DeepSpeedExamples/applications/DeepSpeed-Chat/
pip install -r requirements.txt
```

--------------------------------

### Install Hugging Face Transformers from source

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/autotuning.md

Required setup for running Hugging Face examples with DeepSpeed.

```bash
    git clone https://github.com/huggingface/transformers.git
    cd transformers
    pip install .
```

--------------------------------

### Run Weight Quantization Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Commands to install dependencies and execute the weight quantization script for BERT models.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
DeepSpeedExamples/compression/bert$ bash bash_script/quant_weight.sh
```

--------------------------------

### Run Row Pruning Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Commands to install dependencies and execute the row pruning script.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
DeepSpeedExamples/compression/bert$ bash bash_script/pruning_row.sh
```

--------------------------------

### Install FlashAttention

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/ds-sequence.md

Clone and install the FlashAttention library. Navigate to the cloned directory and run the installation command.

```bash
cd ${WORK_DIR}
git clone -b v1.0.4 https://github.com/HazyResearch/flash-attention
cd flash-attention
python -m pip install .
```

--------------------------------

### Run ZenFlow Fine-Tuning Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/zenflow.md

This bash script navigates to the DeepSpeedExamples ZenFlow directory and executes the fine-tuning script for GPT models on the GLUE benchmark. Ensure you have followed the setup instructions in the README.md.

```bash
cd DeepSpeedExamples/training/DeepSpeed-ZenFlow
bash finetune_gpt_glue.sh
```

--------------------------------

### Run Channel Pruning Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Install required packages and execute the channel pruning script for CIFAR models.

```shell
pip install torch torchvision
DeepSpeedExamples/compression/cifar$ bash run_compress.sh
```

--------------------------------

### Run Head Pruning Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Install dependencies and execute the head pruning script for BERT models.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
DeepSpeedExamples/compression/bert$ bash bash_script/pruning_head.sh
```

--------------------------------

### Install BERT ZeroQuant Requirements

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Install the necessary Python packages for running the BERT ZeroQuant example.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
```

--------------------------------

### Run BERT Layer Reduction Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Commands to install dependencies and execute the layer reduction script for BERT fine-tuning.

```shell
DeepSpeedExamples/compression/bert$ pip install -r requirements.txt
DeepSpeedExamples/compression/bert$ bash bash_script/layer_reduction.sh
```

--------------------------------

### Install PyTorch with XPU Support

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Install the XPU-specific variant of PyTorch from the official repository.

```bash
pip install torch --index-url https://download.pytorch.org/whl/xpu
```

--------------------------------

### Activation Quantization Example Output

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Example output showing the training progress and final accuracy after running the activation quantization script.

```shell
Epoch: 02 | Time: 28m 50s
Clean the best model, and the accuracy of the clean model is acc/mm-acc:0.8375955170657158/0.8422497965825875
```

--------------------------------

### Install Intel Extension for PyTorch

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Install the extension to improve LLM inference performance on CPUs.

```bash
pip install intel-extension-for-pytorch
```

--------------------------------

### Clone and Install DeepSpeed-VisualChat

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/deepspeed-visualchat/10-03-2023/README.md

Clone the DeepSpeedExamples repository, navigate to the DeepSpeed-VisualChat directory, install dependencies, and run the training script.

```bash
git clone https://github.com/deepspeedai/DeepSpeedExamples.git
cd DeepSpeedExamples/applications/DeepSpeed-VisualChat/
pip install -r requirements.txt
cd training
bash training_scripts/run_7b.sh
```

--------------------------------

### Install PyTorch and DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Install the NPU-compatible PyTorch and the DeepSpeed library.

```bash
pip install torch torch_npu
```

```bash
pip install deepspeed
```

--------------------------------

### Load HuggingFace Model Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/bert-finetuning.md

Example command to load a HuggingFace pretrained model. Ensure `--deepspeed_transformer_kernel` is used and `--preln` is not.

```shell
[HuggingFace]

--model_file test/huggingface/bert-large-uncased-whole-word-masking-pytorch_model.bin \
--ckpt_type HF \
--origin_bert_config_file test/huggingface/bert-large-uncased-whole-word-masking-config.json \

```

--------------------------------

### Install numactl on Ubuntu

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Install the numactl package to enable fine-grained CPU core allocation.

```bash
sudo apt-get install numactl
```

--------------------------------

### Install Huawei Ascend NPU Driver and Firmware

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Commands to install the necessary driver and firmware for the Ascend NPU.

```bash
./Ascend-hdk-<soc_version>-npu-driver_x.x.x_linux-{arch}.run --full --install-for-all
```

```bash
./Ascend-hdk-<soc_version>-npu-firmware_x.x.x.x.X.run --full
```

--------------------------------

### Install Triton for FlashAttention

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/ds-sequence.md

Install Triton with the legacy backend for FlashAttention compatibility. Ensure you are in the cloned triton/python directory.

```bash
git clone -b legacy-backend https://github.com/openai/triton
cd triton/python/
pip install cmake
pip install .
```

--------------------------------

### Start BingBertSQuAD Training with DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-adam.md

Use this script to start training with DeepSpeed. Provide the number of GPUs, path to checkpoint, data directory, and output directory.

```shell
bash run_squad_deepspeed.sh <NUM_GPUS> <PATH_TO_CHECKPOINT> <PATH_TO_DATA_DIR> <PATH_TO_OUTPUT_DIR>
```

--------------------------------

### Install Pre-built DeepSpeed Wheel

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/advanced-install.md

Install a DeepSpeed binary wheel that has been previously built. This is useful for deploying DeepSpeed on multiple machines without rebuilding the extensions.

```bash
pip install dist/deepspeed-0.3.13+8cd046f-cp38-cp38-linux_x86_64.whl
```

--------------------------------

### Load TensorFlow Model Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/bert-finetuning.md

Example command to load a TensorFlow pretrained model. Ensure `--deepspeed_transformer_kernel` is used and `--preln` is not.

```shell
[TensorFlow]

--model_file /test/tensorflow/bert_model.ckpt \
--ckpt_type TF \
--origin_bert_config_file /test/tensorflow/bert_config.json \

```

--------------------------------

### Install DeepSpeed-MII

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/deepspeed-fastgen/chinese/README.md

Install the latest DeepSpeed-MII distribution to get started with DeepSpeed-FastGen.

```bash
pip install deepspeed-mii
```

--------------------------------

### Configure MPAM Partition Settings

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/deepspeed-superoffload/README.md

Example configuration for CPU core lists and memory/cache schemata for partitions.

```text
/sys/fs/resctrl/p1/cpus_list:
0-6
/sys/fs/resctrl/p2/cpus_list:
7-71
/sys/fs/resctrl/p1/schemata:
MB:1=100
L3:1=ff0
/sys/fs/resctrl/p2/schemata:
MB:1=20
L3:1=f
```

--------------------------------

### Example: Offload FP32 Parameters and Optimizer States

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/code-docs/source/zero3.md

Demonstrates how to selectively offload FP32 parameters and optimizer states to CPU memory after forward, backward, and step operations. States are reloaded back to the device when needed.

```python
# Offload after forward, backward, and step
ds_engine.offload_states(include=[OffloadStateTypeEnum.hp_params, OffloadStateTypeEnum.optim_states])

# Do something requiring a lot of device memory
...
# Load states back to device memory
ds_engine.reload_states()

```

--------------------------------

### Validate DeepSpeed Installation

Source: https://github.com/deepspeedai/deepspeed/blob/master/README.md

Run this command after installation to check your DeepSpeed setup and compatibility with various extensions and operations on your machine.

```bash
ds_report
```

--------------------------------

### Setup Model and Optimizer with DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/megatron.md

Initialize the model, optimizer, and scheduler using DeepSpeed within the setup_model_and_optimizer function.

```python
def setup_model_and_optimizer(args):
    """Setup model and optimizer."""

    model = get_model(args)
    optimizer = get_optimizer(model, args)
    lr_scheduler = get_learning_rate_scheduler(optimizer, args)

    if args.deepspeed:
        import deepspeed

        print_rank_0("DeepSpeed is enabled.")

        model, optimizer, _, lr_scheduler = deepspeed.initialize(
            model=model,
            optimizer=optimizer,
            args=args,
            lr_scheduler=lr_scheduler,
            mpu=mpu,
            dist_init_required=False
       )
```

--------------------------------

### Example mpirun command for 32 GPUs with InfiniBand

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-adam.md

This is an example of an 'mpirun' command for launching BERT pre-training with DeepSpeed and 1-bit Adam on 32 GPUs using MVAPICH2 with InfiniBand support.

```bash
mpirun -np 32 -ppn 4 -hostfile hosts -env MV2_USE_CUDA=1 -env MV2_SUPPORT_DL=1 -env MV2_ENABLE_AFFINITY=0 -env MV2_SMP_USE_CMA=0 bash ds_train_bert_onebit_bsz4k_seq128.sh
```

--------------------------------

### Initialize DeepSpeed Submodules

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/cifar-10.md

Download the required DeepSpeed examples repository using git submodules.

```bash
git submodule update --init --recursive
```

--------------------------------

### Run BERT Pre-training with DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-adam.md

Use this script to start BERT pre-training with DeepSpeed. Ensure DeepSpeed is installed and configured.

```bash
bash ds_train_bert_bsz64k_seq128.sh
```

--------------------------------

### View ds_nvme_tune Help

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/deepnvme.md

Display the command-line options and usage instructions for the ds_nvme_tune tool.

```bash
usage: ds_nvme_tune [-h] --nvme_dir NVME_DIR [NVME_DIR ...] [--sweep_config SWEEP_CONFIG] [--no_read] [--no_write] [--io_size IO_SIZE] [--gpu] [--gds] [--flush_page_cache] [--log_dir LOG_DIR] [--loops LOOPS] [--verbose]

options:
  -h, --help            show this help message and exit
  --nvme_dir NVME_DIR [NVME_DIR ...]
                        Directory in which to perform I/O tests. A writeable directory on a NVMe device.
  --sweep_config SWEEP_CONFIG
                        Performance sweep configuration json file.
  --no_read             Disable read performance measurements.
  --no_write            Disable write performance measurements.
  --io_size IO_SIZE     Number of I/O bytes to read/write for performance measurements.
  --gpu                 Test tensor transfers between GPU device and NVME device.
  --gds                 Run the sweep over NVIDIA GPUDirectStorage operator
  --flush_page_cache    Page cache will not be flushed and reported read speeds may be higher than actual ***Requires sudo access***.
  --log_dir LOG_DIR     Output directory for performance log files. Default is ./_aio_bench_logs
  --loops LOOPS         Count of operation repetitions
  --verbose             Print debugging information.
```

--------------------------------

### Hugging Face Text Classification Example with DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/MoQ-tutorial.md

This script runs the Hugging Face text classification example (MRPC task) using DeepSpeed with a specified JSON configuration file. Ensure DeepSpeed is installed and the configuration file is accessible.

```bash
TSK=mrpc
TEST_JSON=test.json

python text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TSK \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TSK/ \
  --fp16 \
  --warmup_steps 2 \
  --deepspeed test.json

```

--------------------------------

### Example mpirun Command for 1-bit Adam Training with InfiniBand

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-adam.md

An example `mpirun` command for launching 1-bit Adam training using 32 GPUs with InfiniBand support, leveraging MVAPICH2 library flags.

```shell
mpirun -np 32 -ppn 4 -hostfile hosts -env MV2_USE_CUDA=1 -env MV2_SUPPORT_DL=1 -env MV2_ENABLE_AFFINITY=0 -env MV2_SMP_USE_CMA=0 bash run_squad_mpi_onebitadam.sh
```

--------------------------------

### Configure 1-bit LAMB Optimizer

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-lamb.md

Example JSON configuration for enabling the 1-bit LAMB optimizer within a DeepSpeed training setup.

```json
{
  "train_batch_size": 65536,
  "train_micro_batch_size_per_gpu": 64,
  "optimizer": {
    "type": "OneBitLamb",
    "params": {
      "lr": 11e-3,
      "max_coeff": 0.3,
      "min_coeff": 0.01,
      "freeze_step": 1000,
      "cuda_aware": false,
      "comm_backend_name": "nccl",
      "coeff_beta": 0.9,
      "factor_max": 4.0,
      "factor_min": 0.5,
      "factor_threshold": 0.1
    }
  },
  "gradient_clipping": 1.0,
  "fp16": {
    "enabled": true,
    "loss_scale": 0,
    "initial_scale_power": 16
  }
}
```

--------------------------------

### Training Large Models with DeepSpeed-Chat

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/deepspeed-chat/README.md

Example of training a 66 billion parameter ChatGPT model using DeepSpeed-Chat on a multi-node setup.

```APIDOC
## POST /train/large_model

### Description
Initiates the training process for a large language model (e.g., 66B parameters) using DeepSpeed-Chat on a multi-node GPU cluster.

### Method
POST

### Endpoint
/train

### Parameters
#### Query Parameters
- **actor-model** (string) - Required - The name or path of the actor model to train (e.g., `facebook/opt-66b`).
- **reward-model** (string) - Required - The name or path of the reward model to use (e.g., `facebook/opt-350m`).
- **deployment-type** (string) - Required - Specifies the deployment type, typically `multi_node` for large-scale training.

### Request Example
```python
python train.py --actor-model facebook/opt-66b --reward-model facebook/opt-350m --deployment-type multi_node
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation that the training job has started.
- **estimated_completion_time** (string) - Estimated time for the training process (e.g., '9 hours').

#### Response Example
{
  "message": "Training job initiated for facebook/opt-66b.",
  "estimated_completion_time": "9 hours"
}
```

--------------------------------

### Install pre-commit hooks

Source: https://github.com/deepspeedai/deepspeed/blob/master/CONTRIBUTING.md

Initialize the pre-commit hooks to ensure consistent code formatting before commits.

```bash
pre-commit install
```

--------------------------------

### Pre-install Specific DeepSpeed Op (FusedLamb)

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/advanced-install.md

Use specific DS_BUILD environment variables to pre-compile only certain C++/CUDA extensions. This example shows how to install DeepSpeed with only the FusedLamb op.

```bash
DS_BUILD_FUSED_LAMB=1 pip install deepspeed
```

--------------------------------

### Run GPT-2 ZeroQuant Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/model-compression.md

Execute the bash script to perform ZeroQuant quantization on a GPT-2 model.

```shell
DeepSpeedExamples/compression/gpt2$ bash bash_script/run_zero_quant.sh
```

--------------------------------

### Get Accelerator Device and Load Model with DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/blogs/intel-inference/README.md

Use the device-agnostic interface `deepspeed.accelerator.get_accelerator()` to obtain the accelerator device name and load a model. Ensure necessary software like PyTorch, Intel Extension for PyTorch, and DeepSpeed are installed.

```python
import deepspeed
from deepspeed.accelerator import get_accelerator
...
# load model checkpoint into model
model = model.eval().to(get_accelerator().device_name())

ds_world_size = int(os.getenv('WORLD_SIZE', '0'))

engine = deepspeed.init_inference(model=model, mp_size=ds_world_size, \
  dtype=torch.bfloat16, replace_method="auto", \
  replace_with_kernel_inject=False)

model = engine.module
...
# evaluate model
```

--------------------------------

### Launch 1-bit LAMB with mpirun Launcher (MPI)

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/onebit-lamb.md

Alternatively, use the standard mpirun command to launch a training script with the MPI-based 1-bit LAMB implementation. Configure process count, GPUs per node, and host files.

```shell
mpirun -np [num processes] -ppn [num GPUs on each node] -hostfile [hostfile] [MPI flags] python [training_script.py]
```

--------------------------------

### Profile AlexNet Model Inference with DeepSpeed FLOPs Profiler

Source: https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/profiling/flops_profiler/README.md

Use the `get_model_profile` function to profile a trained model during inference. This example demonstrates profiling AlexNet with specified batch size and input shape. Ensure DeepSpeed is installed and import the necessary modules.

```python
import torchvision.models as models
import torch
from deepspeed.profiling.flops_profiler import get_model_profile
from deepspeed.accelerator import get_accelerator

with get_accelerator().device(0):
    model = models.alexnet()
    batch_size = 256
    flops, macs, params = get_model_profile(model=model, # model
                                    input_shape=(batch_size, 3, 224, 224), # input shape to the model. If specified, the model takes a tensor with this shape as the only positional argument.
                                    args=None, # list of positional arguments to the model.
                                    kwargs=None, # dictionary of keyword arguments to the model.
                                    print_profile=True, # prints the model graph with the measured profile attached to each module
                                    detailed=True, # print the detailed profile
                                    module_depth=-1, # depth into the nested modules, with -1 being the inner most modules
                                    top_modules=1, # the number of top modules to print aggregated profile
                                    warm_up=10, # the number of warm-ups before measuring the time of each module
                                    as_string=True, # print raw numbers (e.g. 1000) or as human-readable strings (e.g. 1k)
                                    output_file=None, # path to the output file. If None, the profiler prints to stdout.
                                    ignore_modules=None)
```

--------------------------------

### Sparse Attention Configuration Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_pages/config-json.md

This JSON snippet illustrates how to configure sparse attention parameters. It includes settings for attention mode, block size, global and local attention configurations, and windowing strategies. Use this as a template for your DeepSpeed sparse attention setup.

```json
{
  "sparse_attention": {
    "mode": "fixed",
    "block": 16,
    "different_layout_per_head": true,
    "num_local_blocks": 4,
    "num_global_blocks": 1,
    "attention": "bidirectional",
    "horizontal_global_attention": false,
    "num_different_global_patterns": 4,
    "num_random_blocks": 0,
    "local_window_blocks": [4],
    "global_block_indices": [0],
    "global_block_end_indices": null,
    "num_sliding_window_blocks": 3
  }
}
```

--------------------------------

### Install CIFAR-10 Requirements

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/cifar-10.md

Install the necessary dependencies for the CIFAR-10 model.

```bash
cd DeepSpeedExamples/cifar
pip install -r requirements.txt
```

--------------------------------

### Install DeepSpeed from source

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/advanced-install.md

Perform a JIT-mode installation of DeepSpeed using pip.

```bash
pip install .
```

--------------------------------

### FP16 Training Configuration Example

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_pages/config-json.md

This JSON snippet shows the structure and available parameters for configuring FP16 training. It is used when FP16 training is enabled and ZeRO is required.

```json
"fp16": {
    "enabled": true,
    "auto_cast": false,
    "loss_scale": 0,
    "initial_scale_power": 16,
    "loss_scale_window": 1000,
    "hysteresis": 2,
    "consecutive_hysteresis": false,
    "min_loss_scale": 1,
    "fp16_master_weights_and_grads": false
}
```

--------------------------------

### View Autotuning Summary Table

Source: https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/autotuning/README.md

Example of the summary table printed to the terminal upon completion of autotuning.

```text
| tuning_space | num_exps | best_metric_val | best_exp_name   |
| :----------- | -------: | --------------: | :-------------- |
| z0           |        2 |         90.1269 | z0_tmbspg2_gas1 |
| z1           |        2 |         187.298 | z1_tmbspg3_gas1 |
| z2           |        2 |         148.154 | z2_tmbspg3_gas1 |
| global       |        6 |         187.298 | z1_tmbspg3_gas1 |

Tuning completed in 0:00:03.602291
```

--------------------------------

### Install PyTorch and DeepSpeed

Source: https://github.com/deepspeedai/deepspeed/blob/master/docs/_tutorials/accelerator-setup-guide.md

Standard installation commands for PyTorch and DeepSpeed via pip.

```bash
pip install torch
```

```bash
pip install deepspeed
```