### Full Recipe Lifecycle Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/recipe_deepdive.md

Demonstrates the complete lifecycle of a distributed fine-tuning recipe, including setup, training, and cleanup.

```python
recipe = FullFinetuneRecipeDistributed(cfg=cfg)
recipe.setup(cfg=cfg)
recipe.train()
recipe.cleanup()
```

--------------------------------

### Build Specific Documentation Examples

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Build a subset of documentation examples using a regex pattern to specify which examples to include.

```bash
EXAMPLES_PATTERN="plot_the_best_example*" make html
```

--------------------------------

### Recipe Setup Process

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/recipe_deepdive.md

Loads checkpoints, sets up the model, tokenizer, optimizer, loss function, and data loaders.

```python
def setup(self, cfg: DictConfig):

    ckpt_dict = self.load_checkpoint(cfg.checkpointer)

    # Setup the model, including FSDP wrapping, setting up activation checkpointing and
    # loading the state dict
    self._model = self._setup_model(...)
    self._tokenizer = self._setup_tokenizer(...)

    # Setup Optimizer, including transforming for FSDP when resuming training
    self._optimizer = self._setup_optimizer(...)
    self._loss_fn = self._setup_loss(...)
    self._sampler, self._dataloader = self._setup_data(...)
```

--------------------------------

### Install torchtune with development dependencies

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Navigate to the torchtune directory and install the package with all development dependencies.

```shell
cd torchtune
pip install -e ".[dev]"
```

--------------------------------

### List Default Model Configs and Copy Qwen2 Config

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/custom_components.md

Lists all default model configurations available for recipes and copies a Qwen2 full finetune configuration to a specified project directory. This is useful for starting with a pre-defined setup and making modifications.

```bash
# Show all the default model configs for each recipe
tune ls
# This makes a copy of a Qwen2 full finetune config
tune cp qwen2/0.5B_full_single_device ~/my_project/config/qwen_config.yaml
```

--------------------------------

### Download Qwen2.5-1.5B-Instruct

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/api_ref_models.md

Use this command to download the Qwen2.5-1.5B-Instruct model. Specify an output directory. This example demonstrates downloading a model from the Qwen2.5 family.

```bash
tune download Qwen/Qwen2.5-1.5B-Instruct --output-dir /tmp/Qwen2_5-1_5B-Instruct
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Install the necessary Python packages for building documentation from the docs directory.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install torchtune via PyPI

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/install.md

Install the latest stable version of torchtune from PyPI. This is the recommended method for most users.

```bash
pip install torchtune
```

--------------------------------

### Download Qwen2-1.5B-Instruct

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/api_ref_models.md

Use this command to download the Qwen2-1.5B-Instruct model. Specify an output directory. This example demonstrates downloading a model from the Qwen2 family.

```bash
tune download Qwen/Qwen2-1.5B-Instruct --output-dir /tmp/Qwen2-1.5B-Instruct
```

--------------------------------

### Serve Documentation Locally via HTTP Server

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Start a simple Python HTTP server in the build/html directory to serve documentation files locally.

```bash
python -m http.server 8000 # or any free port
```

--------------------------------

### Summarize Template Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/chat.md

A basic string formatting example for a summarization prompt template.

```python
f"Summarize this dialogue:\n{dialogue}\n---\nSummary:\n"
```

--------------------------------

### List Available QAT Configurations

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Use these commands to list available configurations for QAT recipes, either in a distributed or single-device setup. This is useful for exploring different QAT options.

```bash
tune ls qat_distributed
```

```bash
tune ls qat_single_device
```

--------------------------------

### List Available DPO Configurations

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Use this command to list all available configurations for the full DPO recipe. This helps in selecting the appropriate setup for your needs.

```bash
tune ls full_dpo_distributed
```

--------------------------------

### Developer Installation via git clone

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/install.md

Install torchtune locally from a git clone with development dependencies included. This is recommended for contributors.

```bash
pip install -e ".[dev]"
```

--------------------------------

### List Available Recipes and Configurations

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/e2e_flow.md

Use the `tune ls` command to view all available fine-tuning recipes and their corresponding configurations. This helps in selecting the appropriate setup for your model and hardware.

```text
$ tune ls
RECIPE                                  CONFIG
full_finetune_single_device             llama2/7B_full_low_memory
                                        llama3/8B_full_single_device
                                        llama3_1/8B_full_single_device
                                        llama3_2/1B_full_single_device
                                        llama3_2/3B_full_single_device
                                        mistral/7B_full_low_memory
                                        phi3/mini_full_low_memory
                                        qwen2/7B_full_single_device
                                        ...


full_finetune_distributed               llama2/7B_full
                                        llama2/13B_full
                                        llama3/8B_full
                                        llama3_1/8B_full
                                        llama3_2/1B_full
                                        llama3_2/3B_full
                                        mistral/7B_full
                                        gemma2/9B_full
                                        gemma2/27B_full
                                        phi3/mini_full
                                        qwen2/7B_full
                                        ...

lora_finetune_single_device             llama2/7B_lora_single_device
                                        llama2/7B_qlora_single_device
                                        llama3/8B_lora_single_device
...
```

--------------------------------

### QLoRA Training Iteration Log Example with Compile

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qlora_finetune.md

Sample log output from a QLoRA training run after enabling torch.compile, showing improved iterations per second.

```python
1|228|Loss: 0.8158286809921265:   1%|          | 228/25880 [11:59<1:48:16,  3.95it/s
```

--------------------------------

### Example Config with Optimizer Parameters

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/configs.md

Shows a sample configuration snippet for an optimizer, including parameters like `lr` and `foreach`, which might need to be removed when overriding with a different optimizer.

```yaml
# In configs/llama3/8B_full.yaml
optimizer:
  _component_: torch.optim.AdamW
  lr: 2e-5
  foreach: False
```

--------------------------------

### Launch Finetuning Run with Tune CLI

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/first_finetune_tutorial.md

Initiate a finetuning run using the tune CLI, specifying the recipe and configuration. The output shows module initialization and the start of the training process, including loss and progress.

```bash
$ tune run lora_finetune_single_device --config llama2/7B_lora_single_device epochs=1
INFO:torchtune.utils.logging:Running LoRAFinetuneRecipeSingleDevice with resolved config:
Writing logs to /tmp/lora_finetune_output/log_1713194212.txt
INFO:torchtune.utils.logging:Model is initialized with precision torch.bfloat16.
INFO:torchtune.utils.logging:Tokenizer is initialized from file.
INFO:torchtune.utils.logging:Optimizer and loss are initialized.
INFO:torchtune.utils.logging:Loss is initialized.
INFO:torchtune.utils.logging:Dataset and Sampler are initialized.
INFO:torchtune.utils.logging:Learning rate scheduler is initialized.
1|52|Loss: 2.3697006702423096:   0%|▏                     | 52/25880 [00:24<3:55:01,  1.83it/s]
```

--------------------------------

### Install Torchtune and Dependencies

Source: https://github.com/meta-pytorch/torchtune/blob/main/recipes/dev/async_grpo.md

Set up a dedicated conda environment and install Torchtune with the necessary dependencies for asynchronous RL training.

```bash
conda create --name tunerl python=3.10
conda activate tunerl
git clone https://github.com/pytorch/torchtune.git
cd torchtune
pip install torch torchvision torchao
pip install -e .[async_rl]
```

--------------------------------

### Verify Torchtune Installation

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Runs the torchtune CLI with the help flag to confirm successful installation and display available commands.

```bash
tune --help
```

--------------------------------

### Running QAT Finetuning Workload

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qat_finetune.md

Executes the QAT finetuning workload using the specified configuration. This command requires a distributed setup with multiple GPUs and sufficient VRAM.

```bash
tune run --nnodes 1 --nproc_per_node 6 qat_distributed --config custom_8B_qat_full.yaml
```

--------------------------------

### Llama3 Instruct Chat Prompt Template Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/chat.md

This example demonstrates the Llama3 Instruct chat prompt template, highlighting its distinct tags and structure for multiturn conversations.

```text
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful, respectful, and honest assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Hi! I am a human.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hello there! Nice to meet you! I'm Meta AI, your friendly AI assistant<|eot_id|>
```

--------------------------------

### Install torchtune via git clone

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/install.md

Clone the torchtune repository and install it locally using pip. This method is useful for contributors or for accessing the latest development features.

```bash
git clone https://github.com/pytorch/torchtune.git
cd torchtune
pip install -e .
```

--------------------------------

### List Available Recipes and Configs

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tune_cli.md

Display all built-in recipes and their associated configurations available within the Torchtune library. This helps in choosing the right setup for your task.

```bash
$ tune ls
RECIPE                                   CONFIG
full_finetune_single_device              llama2/7B_full_low_memory
                                         llama3/8B_full_single_device
                                         mistral/7B_full_low_memory
                                         phi3/mini_full_low_memory
full_finetune_distributed                llama2/7B_full
                                         llama2/13B_full
                                         llama3/8B_full
                                         llama3/70B_full
...

```

--------------------------------

### Example Instruct Dataset Usage (CSV)

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/instruct_datasets.md

Demonstrates loading and tokenizing an instruct dataset from a local CSV file for grammar correction. Includes options for including user input in training and prepending a system prompt. Shows how to decode the tokenized output and view the labels, noting that the system message is masked.

```bash
head data/my_data.csv
# incorrect,correct
# This are a cat,This is a cat.
```

```python
from torchtune.models.gemma import gemma_tokenizer
from torchtune.datasets import instruct_dataset

g_tokenizer = gemma_tokenizer(
    path="/tmp/gemma-7b/tokenizer.model",
    prompt_template="torchtune.data.GrammarErrorCorrectionTemplate",
    max_seq_len=8192,
)
ds = instruct_dataset(
    tokenizer=g_tokenizer,
    source="csv",
    data_files="data/my_data.csv",
    split="train",
    # By default, user prompt is ignored in loss. Set to True to include it
    train_on_input=True,
    # Prepend a system message to every sample
    new_system_prompt="You are an AI assistant. ",
    # Use columns in our dataset instead of default
    column_map={"input": "incorrect", "output": "correct"},
)
tokenized_dict = ds[0]
tokens, labels = tokenized_dict["tokens"], tokenized_dict["labels"]
print(g_tokenizer.decode(tokens))
# You are an AI assistant. Correct this to standard English:This are a cat---\nCorrected:This is a cat.
print(labels)  # System message is masked out, but not user message
# [-100, -100, -100, -100, -100, -100, 27957, 736, 577, ...]
```

```yaml
# In config
tokenizer:
  _component_: torchtune.models.gemma.gemma_tokenizer
  path: /tmp/gemma-7b/tokenizer.model
  prompt_template: torchtune.data.GrammarErrorCorrectionTemplate
  max_seq_len: 8192

dataset:
  source: csv
  data_files: data/my_data.csv
  split: train
  train_on_input: True
  new_system_prompt: You are an AI assistant.
  column_map:
    input: incorrect
    output: correct
```

--------------------------------

### Example JSON Data for Instruct Dataset

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/instruct_datasets.md

This JSON file demonstrates the expected format for an instruct dataset, using custom column names 'prompt' and 'response'.

```json
# data/my_data.json
[
    {"prompt": "hello world", "response": "bye world"},
    {"prompt": "are you a robot", "response": "no, I am an AI assistant"},
    ...
]
```

--------------------------------

### Run QAT Recipe with LoRA/QLoRA

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Execute the Quantization-Aware Training (QAT) recipe using LoRA/QLoRA on a distributed setup. Specify the model configuration for Llama3.1 8B.

```bash
tune run qat_distributed --config llama3_1/8B_qat_lora
```

--------------------------------

### Launch Fine-tuning Job

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/chat.md

Execute this command to start a fine-tuning process using a LoRA single-device recipe. Customize the configuration file and specify the number of epochs for training.

```bash
$ tune run lora_finetune_single_device --config custom_8B_lora_single_device.yaml epochs=15
```

--------------------------------

### Airtight Config Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/configs.md

Demonstrates the recommended practice of including only necessary fields in a config file for clarity and easier debugging, contrasting it with an overly flexible but less clear approach.

```yaml
# dont do this
alpaca_dataset:
  _component_: torchtune.datasets.alpaca_dataset
slimorca_dataset:
  ...

# do this
dataset:
  # change this in config or override when needed
  _component_: torchtune.datasets.alpaca_dataset
```

--------------------------------

### Loading Local/Remote Instruct Dataset (JSON)

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/instruct_datasets.md

Demonstrates loading an instruct dataset from a local or remote JSON file using the `instruct_dataset` builder. This example specifies the source as 'json', provides the data file path, and sets the split.

```python
from torchtune.models.gemma import gemma_tokenizer
from torchtune.datasets import instruct_dataset

g_tokenizer = gemma_tokenizer("/tmp/gemma-7b/tokenizer.model")
ds = instruct_dataset(
    tokenizer=g_tokenizer,
    source="json",
    data_files="data/my_data.json",
    split="train",
)
```

```yaml
# In config
tokenizer:
  _component_: torchtune.models.gemma.gemma_tokenizer
  path: /tmp/gemma-7b/tokenizer.model

# Tokenizer is passed into the dataset in the recipe
dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: json
  data_files: data/my_data.json
  split: train
```

--------------------------------

### Download Model and Launch LoRA Finetune

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/custom_components.md

Downloads a specified model and launches a LoRA finetuning run using the default single device configuration. Ensure torchtune is installed and the `tune` command is available.

```bash
mkdir ~/my_project
cd ~/my_project
# This downloads the Llama 3.2 1B Instruct model
tune download meta-llama/Llama-3.2-1B-Instruct --output-dir /tmp/Llama-3.2-1B-Instruct --ignore-patterns "original/consolidated.00.pth"
# This launches a lora finetuning run with the default single device config
tune run lora_finetune_single_device --config llama3_2/1B_lora_single_device
```

--------------------------------

### Custom QAT Finetuning Configuration

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qat_finetune.md

A sample YAML configuration for QAT finetuning, specifying dataset parameters, training epochs, and steps for fake quantization. This configures the training process, including when to start applying quantization noise.

```yaml
dataset:
  _component_: torchtune.datasets.text_completion_dataset
  source: allenai/c4
  column: text
  name: en
  split: train

...

epochs: 1
max_steps_per_epoch: 2000
fake_quant_after_n_steps: 1000
```

--------------------------------

### Run LoRA Fine-tuning on Single Device

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/e2e_flow.md

Execute the LoRA fine-tuning process using the `tune run` command with a specified recipe and configuration. This example uses the `lora_finetune_single_device` recipe with the `llama3_2/3B_lora_single_device` configuration.

```text
$ tune run lora_finetune_single_device --config llama3_2/3B_lora_single_device
Setting manual seed to local seed 3977464327. Local seed is seed + rank = 3977464327 + 0
Hint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.
Writing logs to /tmp/torchtune/llama3_2_3B/lora_single_device/logs/log_1734708879.txt
Model is initialized with precision torch.bfloat16.
Memory stats after model init:
        GPU peak memory allocation: 6.21 GiB
        GPU peak memory reserved: 6.27 GiB
        GPU peak memory active: 6.21 GiB
Tokenizer is initialized from file.
Optimizer and loss are initialized.
Loss is initialized.
Dataset and Sampler are initialized.
Learning rate scheduler is initialized.
Profiling disabled.
Profiler config after instantiation: {'enabled': False}
1|3|Loss: 1.943998098373413:   0%|                    | 3/1617 [00:21<3:04:47,  6.87s/it]
```

--------------------------------

### Install Nightly torchtune Build

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/install.md

Install the latest nightly build of torchtune without cloning the repository. The --no-cache-dir option ensures the latest version is installed, overwriting any existing installation.

```bash
pip install --pre torchtune --extra-index-url https://download.pytorch.org/whl/nightly/cpu --no-cache-dir
```

--------------------------------

### Loading Hugging Face Instruct Dataset

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/instruct_datasets.md

Shows how to load an instruct dataset directly from a Hugging Face repository using the `instruct_dataset` builder. This example uses the 'liweili/c4_200m' dataset and specifies the training split.

```python
from torchtune.models.gemma import gemma_tokenizer
from torchtune.datasets import instruct_dataset

g_tokenizer = gemma_tokenizer("/tmp/gemma-7b/tokenizer.model")
ds = instruct_dataset(
    tokenizer=g_tokenizer,
    source="liweili/c4_200m",
    split="train"
)
```

```yaml
# In config
tokenizer:
  _component_: torchtune.models.gemma.gemma_tokenizer
  path: /tmp/gemma-7b/tokenizer.model

# Tokenizer is passed into the dataset in the recipe
dataset:
  _component_: torchtune.datasets.instruct_dataset
  source: liweili/c4_200m
  split: train
```

--------------------------------

### Install comet_ml package

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/comet_logging.md

Install the necessary package for Comet logging using pip.

```bash
pip install comet_ml
```

--------------------------------

### Configure LoRA Rank and Alpha for Finetuning

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/memory_optimizations.md

This command-line example extends the previous configuration by setting the LoRA rank and alpha parameters. These parameters control the scale and magnitude of LoRA updates, influencing memory usage and training stability.

```bash
tune run lora_finetune_single_device --config llama3/8B_lora_single_device  \
model.apply_lora_to_mlp=True \
model.lora_attn_modules=["q_proj","k_proj","v_proj","output_proj"] \
model.lora_rank=32 \
model.lora_alpha=64
```

--------------------------------

### Install Stable Torchtune and Dependencies

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Installs the latest stable releases of PyTorch, torchvision, torchao, and torchtune.

```bash
pip install torch torchvision torchao
pip install torchtune
```

--------------------------------

### Copy and Launch Custom Recipe

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/custom_components.md

Copies a default single-device full finetune recipe locally and then launches a custom recipe with a custom configuration. This allows for using customized training logic. Ensure the recipe file extension is specified when launching.

```bash
mkdir ~/my_project/recipes
# Show all the default recipes
tune ls
# This makes a copy of the full finetune single device recipe locally
tune cp full_finetune_single_device ~/my_project/recipes/single_device.py
# Launch custom recipe with custom config from project directory
tune run recipes/single_device.py --config config/qwen_config.yaml
```

--------------------------------

### Download and Run LoRA Finetuning Recipe

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/recipes/lora_finetune_single_device.md

Download a model and then run the LoRA finetuning recipe using a specified configuration. Ensure you have access to gated repositories if necessary.

```bash
# download the model
tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
--output-dir /tmp/Meta-Llama-3.1-8B-Instruct \
--ignore-patterns "original/consolidated.00.pth"

# run the recipe
tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device
```

--------------------------------

### Install wandb package

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/wandb_logging.md

Install the wandb package using pip. This is a prerequisite for using W&B logging.

```bash
pip install wandb
```

--------------------------------

### Install Stable PyTorch Libraries

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/install.md

Install the stable versions of PyTorch, torchvision, and torchao using pip. These are prerequisites for torchtune.

```bash
pip install torch torchvision torchao
```

--------------------------------

### Finetune with QLoRA using Command Line

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/memory_optimizations.md

Use this command to finetune a model with QLoRA using a single device. It specifies the configuration file and essential LoRA parameters.

```bash
tune run lora_finetune_single_device --config llama3/8B_qlora_single_device \
model.apply_lora_to_mlp=True \
model.lora_attn_modules=["q_proj","k_proj","v_proj"] \
model.lora_rank=32 \
model.lora_alpha=64
```

--------------------------------

### Launch Custom Config with Absolute and Relative Paths

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/custom_components.md

Launches a full finetuning run using a custom configuration file. Demonstrates launching with both an absolute path and a relative path to the configuration file within the project directory. Ensure the model is downloaded and the recipe matches the config.

```bash
mkdir ~/my_project/config
tune run full_finetune_single_device --config ~/my_project/config/qwen_config.yaml
# Or launch directly from the project directory with a relative path
tune run full_finetune_single_device --config config/qwen_config.yaml
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Install pre-commit hooks to automatically run style checks and prevent common mistakes before each commit.

```bash
pre-commit install
```

--------------------------------

### Configuring Tokenizer and Dataset with `instantiate`

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/configs.md

Shows how to configure both a tokenizer and a dataset, passing the instantiated tokenizer as an argument to the dataset. Demonstrates overriding optional keyword arguments.

```yaml
# Tokenizer is needed for the dataset, configure it first
tokenizer:
  _component_: torchtune.models.llama2.llama2_tokenizer
  path: /tmp/tokenizer.model

dataset:
  _component_: torchtune.datasets.alpaca_dataset
```

--------------------------------

### JSON Data Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/text_completion_datasets.md

Example of a JSON file containing text data for training. Each entry should have an 'input' field with the text content.

```json
# odyssey.json
[
    {
        "input": "After we were clear of the river Oceanus, and had got out into the open sea, we went on till we reached the Aeaean island where there is dawn and sunrise as in other places. We then drew our ship on to the sands and got out of her on to the shore, where we went to sleep and waited till day should break."
    },
    {
        "input": "Then, when the child of morning, rosy-fingered Dawn, appeared, I sent some men to Circe's house to fetch the body of Elpenor. We cut firewood from a wood where the headland jutted out into the sea, and after we had wept over him and lamented him we performed his funeral rites. When his body and armour had been burned to ashes, we raised a cairn, set a stone over it, and at the top of the cairn we fixed the oar that he had been used to row with."
    }
]
```

--------------------------------

### Example Chat Dataset Structure

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/chat_datasets.md

This is an example of a local JSON file structure for a chat dataset. Each entry contains a list of conversations.

```json
[
    {
        "conversations": [
            {
                "from": "human",
                "value": "What is the answer to life?"
            },
            {
                "from": "gpt",
                "value": "The answer is 42."
            },
            {
                "from": "human",
                "value": "That's ridiculous"
            },
            {
                "from": "gpt",
                "value": "Oh I know."
            }
        ]
    }
]
```

--------------------------------

### Run Finetuning with Local Configuration

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Executes a finetuning recipe using a locally modified configuration file.

```bash
tune run full_finetune_distributed --config ./my_custom_config.yaml
```

--------------------------------

### TXT Data Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/text_completion_datasets.md

Example of a plain text file containing concatenated text data for training. Each line is treated as a separate sample.

```text
# odyssey.txt

After we were clear of the river Oceanus, and had got out into the open sea, we went on till we reached the Aeaean island where there is dawn and sunrise as in other places. We then drew our ship on to the sands and got out of her on to the shore, where we went to sleep and waited till day should break.
Then, when the child of morning, rosy-fingered Dawn, appeared, I sent some men to Circe's house to fetch the body of Elpenor. We cut firewood from a wood where the headland jutted out into the sea, and after we had wept over him and lamented him we performed his funeral rites. When his body and armour had been burned to ashes, we raised a cairn, set a stone over it, and at the top of the cairn we fixed the oar that he had been used to row with.
```

--------------------------------

### Run QLoRA Finetuning with torch.compile

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qlora_finetune.md

Execute a QLoRA finetuning job with performance optimizations enabled by torch.compile.

```bash
tune run lora_finetune_single_device --config llama2/7B_qlora_single_device compile=True
```

--------------------------------

### Build Documentation Locally

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Build the HTML documentation locally from the docs directory. Open build/html/index.html to view.

```bash
make html
# Now open build/html/index.html
```

--------------------------------

### Llama2 Chat Prompt Template Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/chat.md

This example shows the structure of the Llama2 chat prompt template, including system and user messages, and the expected assistant response.

```text
<s>[INST] <<SYS>>
You are a helpful, respectful, and honest assistant.
<</SYS>>

Hi! I am a human. [/INST] Hello there! Nice to meet you! I'm Meta AI, your friendly AI assistant </s>
```

--------------------------------

### Run QAT Finetuning

Source: https://github.com/meta-pytorch/torchtune/blob/main/recipes/quantization.md

Execute Quantization-Aware Training (QAT) for finetuning a model. This command initiates the QAT process using a specified configuration.

```bash
tune run --nproc_per_node 4 qat_distributed --config llama3/8B_qat_full
```

--------------------------------

### Install PyTorch, torchvision, and torchao Nightlies

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Installs the latest nightly builds of PyTorch, torchvision, and torchao. Supports various hardware backends like CPU, CUDA, XPU, ROCm, and others.

```bash
pip install --pre --upgrade torch torchvision torchao --index-url https://download.pytorch.org/whl/nightly/cu126
```

```bash
pip install --pre --upgrade torchtune --extra-index-url https://download.pytorch.org/whl/nightly/cpu
```

--------------------------------

### Quantize Model with PyTorch AO

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/e2e_flow.md

Applies post-training quantization to a model using the int4_weight_only technique from torchao. Requires torchao to be installed.

```python
from torchao.quantization.quant_api import quantize_, int4_weight_only
quantize_(model, int4_weight_only())
```

--------------------------------

### Launching Training with Custom Components

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/custom_components.md

Command to launch a Torchtune training recipe using a custom configuration file. Assumes the project is launched from its root directory.

```bash
cd ~/my_project/
tune run recipes/single_device.py --config config/custom_finetune.yaml
```

--------------------------------

### Build Documentation Without Plots

Source: https://github.com/meta-pytorch/torchtune/blob/main/CONTRIBUTING.md

Build the HTML documentation locally, skipping the execution of examples that generate plots to save time.

```bash
make html-noplot
```

--------------------------------

### Instantiating Objects with `instantiate` API

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/configs.md

Demonstrates how to use the `config.instantiate` function to create object instances from configuration, passing required and optional arguments.

```python
from torchtune import config

# Access the dataset field and create the object instance
dataset = config.instantiate(cfg.dataset)
```

--------------------------------

### Configuring Multimodal Chat Dataset

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/multimodal_datasets.md

Example configuration for loading a multimodal dataset using `multimodal_chat_dataset`. Specifies the source, split, and image directory.

```yaml
dataset:
  _component_: torchtune.datasets.multimodal.multimodal_chat_dataset
  source: Lin-Chen/ShareGPT4V
  split: train
  name: ShareGPT4V
  image_dir: /home/user/dataset/
  image_tag: "<image>"
```

--------------------------------

### Run Finetuning on Custom Device (XPU)

Source: https://github.com/meta-pytorch/torchtune/blob/main/README.md

Overrides the default device configuration to run finetuning on an XPU device.

```bash
tune run lora_finetune_single_device --config llama3_1/8B_lora_single_device device=xpu
```

--------------------------------

### Chat Dataset Format Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/chat_datasets.md

Illustrates the typical structure of a chat dataset, showing a table with a 'conversations' column containing lists of messages.

```text
|  conversations                                               |
|--------------------------------------------------------------|
| [{"role": "user", "content": "What day is today?"},          |
|  {"role": "assistant", "content": "It is Tuesday."} ]         |
| [{"role": "user", "content": "What about tomorrow?"},        |
|  {"role": "assistant", "content": "Tomorrow is Wednesday."} ] |
```

--------------------------------

### Run Recipe with CPU Offload Optimizer

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/memory_optimizations.md

Use this command to enable CPU offloading for optimizer states and gradients when running a finetuning recipe.

```bash
tune run <RECIPE> --config <CONFIG> \
optimizer=optimizer=torchao.prototype.low_bit_optim.CPUOffloadOptimizer \
optimizer.offload_gradients=True \
lr=4e-5
```

--------------------------------

### Get Llama2 Special Token IDs

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/chat.md

Retrieves the token IDs for the beginning-of-sequence (BOS) and end-of-sequence (EOS) tokens used by the Llama2 tokenizer.

```python
print(tokenizer._spm_model.spm_model.piece_to_id("<s>"))
# 1
print(tokenizer._spm_model.spm_model.piece_to_id("</s>"))
# 2
```

--------------------------------

### QAT Quantized Model Evaluation Results

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qat_finetune.md

Example evaluation results for a model quantized using QAT, showing metrics like perplexity and accuracy.

```bash
# QAT quantized model evaluation results (int8 activations + int4 weights)

|  Tasks  |Version|Filter|n-shot|    Metric     |Value |   |Stderr|
|---------|------:|------|-----:|---------------|-----:|---|------|
|wikitext |      2|none  |     0|word_perplexity|9.9148|±  |N/A   |
|         |       |none  |     0|byte_perplexity|1.5357|±  |N/A   |
|         |       |none  |     0|bits_per_byte  |0.6189|±  |N/A   |
|hellaswag|      1|none  |     0|acc            |0.5687|±  |0.0049|
|         |       |none  |     0|acc_norm       |0.7536|±  |0.0043|
```

--------------------------------

### Download Llama3.1-70B-Instruct

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/api_ref_models.md

Use this command to download the Llama3.1-70B-Instruct model. Specify an output directory and provide your HF_TOKEN. The --ignore-patterns flag excludes specific files.

```bash
tune download meta-llama/Meta-Llama-3.1-70B-Instruct --output-dir /tmp/Meta-Llama-3.1-70B-Instruct --ignore-patterns "original/consolidated*" --hf-token <HF_TOKEN>
```

--------------------------------

### Customize Special Tokens for Llama3 Tokenizer

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/tokenizers.md

Example of customizing special tokens for the Llama3 tokenizer by providing a JSON file with added token mappings.

```json
# tokenizer/special_tokens.json
{
    "added_tokens": [
        {
            "id": 128257,
            "content": "<|begin_of_text|>",
        },
        {
            "id": 128258,
            "content": "<|end_of_text|>",
        },
        # Remaining required special tokens
        ...
    ]
}
```

```python
# In code
from torchtune.models.llama3 import llama3_tokenizer

tokenizer = llama3_tokenizer(
    path="/tmp/Meta-Llama-3-8B-Instruct/original/tokenizer.model",
    special_tokens_path="tokenizer/special_tokens.json",
)
print(tokenizer.special_tokens)
```

--------------------------------

### Run Single-Device QLoRA Fine-tuning

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/llama3.md

Initiate a QLoRA fine-tuning job on a single device using a specific configuration for reduced memory usage. This recipe is suitable for consumer GPUs with limited VRAM.

```bash
tune run lora_finetune_single_device --config llama3/8B_qlora_single_device
```

--------------------------------

### Download Llama3-8B-Instruct Weights

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/llama3.md

Use the `tune download` command to fetch the Llama3-8B-Instruct model weights and tokenizer from Hugging Face. Ensure you have your Hugging Face access token ready.

```bash
tune download meta-llama/Meta-Llama-3-8B-Instruct \
    --output-dir <checkpoint_dir> \
    --hf-token <ACCESS TOKEN>
```

--------------------------------

### Resuming Full Finetuning Configuration

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/deep_dives/checkpointer.md

This YAML configuration snippet demonstrates how to set up for resuming full finetuning. Key parameters include enabling `resume_from_checkpoint` and updating `checkpoint_files` to point to the desired epoch.

```yaml
checkpointer:
    # [... rest of the config...]

    # checkpoint files. Note that you will need to update this
    # section of the config with the intermediate checkpoint files
    checkpoint_files: [
        epoch_{YOUR_EPOCH}/model-00001-of-00002.safetensors,
        epoch_{YOUR_EPOCH}/model-00001-of-00002.safetensors,
    ]

# set to True if restarting training
resume_from_checkpoint: True
```

--------------------------------

### Download Model and Log in to Weights and Biases

Source: https://github.com/meta-pytorch/torchtune/blob/main/recipes/dev/async_grpo.md

Download the specified model file and authenticate with Weights and Biases for experiment tracking before running the fine-tuning job.

```bash
conda activate tunerl
tune download Qwen/Qwen2.5-3B --output-dir /tmp/Qwen2.5-3B --ignore-patterns "original/consolidated.00.pth"
wandb login
```

--------------------------------

### PTQ Quantized Model Evaluation Results

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qat_finetune.md

Example evaluation results for a model quantized using Post-Training Quantization (PTQ), used for comparison with QAT results.

```bash
# PTQ quantized model evaluation results (int8 activations + int4 weights)

|  Tasks  |Version|Filter|n-shot|    Metric     | Value |   |Stderr|
|---------|------:|------|-----:|---------------|------:|---|------|
|wikitext |      2|none  |     0|word_perplexity|10.7735|±  |N/A   |
|         |       |none  |     0|byte_perplexity| 1.5598|±  |N/A   |
|         |       |none  |     0|bits_per_byte  | 0.6413|±  |N/A   |
|hellaswag|      1|none  |     0|acc            | 0.5481|±  |0.0050|
|         |       |none  |     0|acc_norm       | 0.7390|±  |0.0044|
```

--------------------------------

### LoRA Training Iteration Log Example

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/tutorials/qlora_finetune.md

Sample log output from a LoRA training run, indicating the current iteration, loss, and iterations per second.

```python
1|149|Loss: 0.9157477021217346:   1%|          | 149/25880 [02:08<6:14:19,  1.15it/s
```

--------------------------------

### List Available Recipes

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/recipes/recipes_overview.md

Run this command to see a full list of available recipes in torchtune.

```bash
tune ls
```

--------------------------------

### Configure Quantizer for Quantization Step

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/recipes/qat_distributed.md

When running the 'tune run quantize' command after QAT, specify the quantizer component and its parameters, such as 'groupsize'. This example uses 'Int8DynActInt4WeightQATQuantizer'.

```yaml
quantizer:
  _component_: torchtune.training.quantization.Int8DynActInt4WeightQATQuantizer
  groupsize: 256
```

--------------------------------

### Interleaving Multiple Images in Text

Source: https://github.com/meta-pytorch/torchtune/blob/main/docs/source/basics/multimodal_datasets.md

Example of creating a `Message` with multiple images interleaved with text. Demonstrates the `contains_media`, `get_media`, and `text_content` properties of the `Message` object.

```python
import PIL
from torchtune.data import Message

image_dog = PIL.Image.new(mode="RGB", size=(4, 4))
image_cat = PIL.Image.new(mode="RGB", size=(4, 4))
image_bird = PIL.Image.new(mode="RGB", size=(4, 4))

user_message = Message(
    role="user",
    content=[
        {"type": "image", "content": image_dog},
        {"type": "text", "content": "This is an image of a dog. "},
        {"type": "image", "content": image_cat},
        {"type": "text", "content": "This is an image of a cat. "},
        {"type": "image", "content": image_bird},
        {"type": "text", "content": "This is a bird, the best pet of the three."},
    ]
)
print(user_message.contains_media)
# True
print(user_message.get_media())
# [<PIL.Image.Image>, <PIL.Image.Image>, <PIL.Image.Image>]
print(user_message.text_content)
# This is an image of a dog. This is an image of a cat. This is a bird, the best pet of the three.
```