### Install Project

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/README.md

Install the project in editable mode. This is a prerequisite for building the documentation.

```bash
pip install -e .
```

--------------------------------

### Setup AutoTrain in a Conda Environment

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart.mdx

Recommended setup procedure including dependency installation for PyTorch, CUDA, and optional acceleration libraries.

```bash
$ conda create -n autotrain python=3.10
$ conda activate autotrain
$ pip install autotrain-advanced
$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
$ conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc
$ conda install xformers -c xformers
$ python -m nltk.downloader punkt
$ pip install flash-attn --no-build-isolation # if you want to use flash-attn
$ pip install deepspeed # if you want to use deepspeed
```

--------------------------------

### Install Documentation Builder Dependencies

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/README.md

Install the `hf-doc-builder` for building documentation and `watchdog` for live reloads. These are required in addition to the project itself.

```bash
# `hf-doc-builder` to build the docs
pip install git+https://github.com/huggingface/doc-builder@main
# `watchdog` for live reloads
pip install watchdog
```

--------------------------------

### Install and Setup AutoTrain

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

Installs the AutoTrain advanced library and sets up the Colab environment. It prints the installed AutoTrain version.

```python
#@title 🤗 AutoTrain LLM
#@markdown In order to use this colab
#@markdown - upload train.csv to a folder named `data/`
#@markdown - train.csv must contain a `text` column
#@markdown - choose a project name if you wish
#@markdown - change model if you wish, you can use most of the text-generation models from Hugging Face Hub
#@markdown - add huggingface information (token) if you wish to push trained model to huggingface hub
#@markdown - update hyperparameters if you wish
#@markdown - click `Runtime > Run all` or run each cell individually
#@markdown - report issues / feature requests here: https://github.com/huggingface/autotrain-advanced/issues


import os
!pip install -U autotrain-advanced > install_logs.txt 2>&1
!autotrain setup --colab > setup_logs.txt
from autotrain import __version__
print(f'AutoTrain version: {__version__}')
```

--------------------------------

### Start AutoTrain Application

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/autotrain_api.mdx

Launch the AutoTrain server on a specified host and port.

```bash
$ autotrain app --port 8000 --host 127.0.0.1
```

--------------------------------

### Launch AutoTrain Application

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Starts the local web interface on the specified host and port.

```bash
autotrain app --port 8080 --host 127.0.0.1
```

--------------------------------

### SmolLM2 Finetuning Configuration

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Example YAML configuration for fine-tuning the SmolLM2 model.

```yaml
task: llm-sft
base_model: HuggingFaceTB/SmolLM2-1.7B-Instruct
project_name: autotrain-smollm2-finetune
log: tensorboard
backend: local

data:
  path: HuggingFaceH4/no_robots
  train_split: train
  valid_split: null
  chat_template: tokenizer
  column_mapping:
    text_column: messages

params:
  block_size: 2048
  model_max_length: 4096
  epochs: 2
  batch_size: 1
  lr: 1e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: paged_adamw_8bit
  scheduler: linear
  gradient_accumulation: 8
  mixed_precision: bf16
  merge_adapter: true

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true
```

--------------------------------

### Initialize and create local training project

Source: https://github.com/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb

Instantiate the AutoTrainProject with the defined parameters and specify 'local' as the backend. Set `process=True` to start the training process immediately.

```python
# this will train the model locally
project = AutoTrainProject(params=params, backend="local", process=True)
project.create()
```

--------------------------------

### Install AutoTrain Advanced via pip

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart.mdx

Standard installation command for the AutoTrain Advanced package.

```bash
$ pip install autotrain-advanced
```

--------------------------------

### Launch AutoTrain UI

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart.mdx

Commands to set the Hugging Face token and start the local web application.

```bash
$ export HF_TOKEN=your_hugging_face_write_token
$ autotrain app --host 127.0.0.1 --port 8000
```

--------------------------------

### Local Training with Local Data

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/llm_finetuning.mdx

Configuration example for training with a local dataset file.

```APIDOC
## Local Training with Local Data

### Description
This configuration snippet shows how to specify a local training file (CSV or JSONL) within the `data` section of the `config.yaml` file.

### Method
Command Line Interface (CLI)

### Endpoint
`autotrain --config config.yaml`

### Parameters
#### Data Configuration (Local File)
- **data.path** (string) - Required - Path to the directory containing the training file (e.g., `path/to/training/file`).
- **data.train_split** (string) - Required - Name of the training file (e.g., `train.csv` or `train.jsonl`).
- **data.valid_split** (string) - Optional - Name of the validation file.
- **data.chat_template** (string) - Optional - Chat template to use for formatting data (e.g., `chatml`).
- **data.column_mapping** (object) - Optional - Maps dataset columns to trainer-specific fields.
  - **text_column** (string) - Required if used - Name of the text column.
  - **rejected_text_column** (string) - Required if used - Name of the rejected text column.
  - **prompt_text_column** (string) - Required if used - Name of the prompt text column.

### Request Example
```yaml
data:
  path: path/to/training/file
  train_split: train # name of the training file (e.g., train.csv or train.jsonl)
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt
```

### Response
Similar to the general local training, the output will be console logs indicating the training process and completion status.
```

--------------------------------

### Local Training Configuration

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/llm_finetuning.mdx

Example configuration file for local LLM ORPO training.

```APIDOC
## Local Training Configuration

### Description
This section details the `config.yaml` file used for local training with AutoTrain Advanced, specifically for the LLM ORPO task. It outlines the necessary parameters for model, data, training, and Hugging Face Hub integration.

### Method
Command Line Interface (CLI)

### Endpoint
`autotrain --config config.yaml`

### Parameters
#### Configuration File (`config.yaml`)

- **task** (string) - Required - The training task (e.g., `llm-orpo`).
- **base_model** (string) - Required - The base model to fine-tune (e.g., `meta-llama/Meta-Llama-3-8B-Instruct`).
- **project_name** (string) - Required - Name of the project.
- **log** (string) - Optional - Logging method (e.g., `tensorboard`).
- **backend** (string) - Optional - Training backend (e.g., `local`).

#### Data Configuration
- **data.path** (string) - Required - Path to the dataset (local directory or Hugging Face dataset ID).
- **data.train_split** (string) - Required - Name of the training split.
- **data.valid_split** (string) - Optional - Name of the validation split.
- **data.chat_template** (string) - Optional - Chat template to use for formatting data (e.g., `chatml`).
- **data.column_mapping** (object) - Optional - Maps dataset columns to trainer-specific fields.
  - **text_column** (string) - Required if used - Name of the text column.
  - **rejected_text_column** (string) - Required if used - Name of the rejected text column.
  - **prompt_text_column** (string) - Required if used - Name of the prompt text column.

#### Training Parameters (`params`)
- **params.block_size** (integer) - Optional - Maximum sequence length. Default: -1 (auto).
- **params.model_max_length** (integer) - Optional - Maximum length for the model to process. Default: 1024.
- **params.max_prompt_length** (integer) - Optional - Maximum length for prompts. Used for `orpo` and `dpo` trainers.
- **params.max_completion_length** (integer) - Optional - Completion length for encoder-decoder models (ORPO) or completion text length (DPO).
- **params.epochs** (integer) - Optional - Number of training epochs. Default: 3.
- **params.batch_size** (integer) - Optional - Training batch size. Default: 2.
- **params.lr** (float) - Optional - Learning rate. Default: 3e-5.
- **params.peft** (boolean) - Optional - Whether to use Parameter-Efficient Fine-Tuning. Default: true.
- **params.quantization** (string) - Optional - Quantization method (e.g., `int4`).
- **params.target_modules** (string) - Optional - Target modules for PEFT (e.g., `all-linear`).
- **params.padding** (string) - Optional - Padding side (e.g., `right`).
- **params.optimizer** (string) - Optional - Optimizer to use (e.g., `adamw_torch`).
- **params.scheduler** (string) - Optional - Learning rate scheduler (e.g., `linear`).
- **params.gradient_accumulation** (integer) - Optional - Gradient accumulation steps. Default: 4.
- **params.mixed_precision** (string) - Optional - Mixed precision training (e.g., `fp16`).

#### Hugging Face Hub Configuration (`hub`)
- **hub.username** (string) - Required if `push_to_hub` is true - Hugging Face username.
- **hub.token** (string) - Required if `push_to_hub` is true - Hugging Face Hub token.
- **hub.push_to_hub** (boolean) - Optional - Whether to push the trained model to the Hub. Default: true.

### Request Example
```yaml
task: llm-orpo
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-orpo
log: tensorboard
backend: local

data:
  path: argilla/distilabel-capybara-dpo-7k-binarized
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt

params:
  block_size: 1024
  model_max_length: 8192
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: fp16

hub:
  username: "${HF_USERNAME}"
  token: "${HF_TOKEN}"
  push_to_hub: true
```

### Response
This section describes the expected output or status after running the command. Since this is a CLI command, there isn't a direct HTTP response. The output will be logged to the console and potentially to TensorBoard if configured.

#### Success Response
Training completes successfully, and the model is pushed to the Hugging Face Hub if `push_to_hub` is true. Logs are available in the specified log directory (e.g., `tensorboard`).

#### Response Example
Console output indicating training progress, completion, and any errors encountered. Example:
```
Training completed successfully.
Model pushed to Hugging Face Hub.
```
```

--------------------------------

### Install AutoTrain Advanced

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain.ipynb

Installs the autotrain-advanced package using pip. Logs are redirected to install_logs.txt. Ensure you have a stable internet connection.

```python
!pip install -U autotrain-advanced > install_logs.txt 2>&1
```

--------------------------------

### CLI Training with Config File

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Sets environment variables for Hugging Face username and token, then starts model training using a YAML configuration file. Recommended for local training.

```bash
export HF_USERNAME=your_hugging_face_username
export HF_TOKEN=your_hugging_face_write_token
autotrain --config path/to/config.yaml
```

--------------------------------

### CSV Data Format Example

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/token_classification.mdx

Example of how to format your data in CSV for token classification. Ensure tokens and tags are stringified lists.

```csv
tokens,tags
"['I', 'love', 'Paris']","['O', 'O', 'B-LOC']"
"['I', 'live', 'in', 'New', 'York']","['O', 'O', 'O', 'B-LOC', 'I-LOC']"
.
.
.
```

--------------------------------

### Install AutoTrain Advanced

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Installs AutoTrain Advanced and its dependencies, including PyTorch with CUDA support. Ensure your environment has Python 3.10.

```bash
conda create -n autotrain python=3.10
conda activate autotrain
pip install autotrain-advanced
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc
pip install flash-attn --no-build-isolation
pip install deepspeed
```

--------------------------------

### Setup Conda Environment for AutoTrain

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Creates a dedicated environment with PyTorch and CUDA dependencies.

```bash
conda create -n autotrain python=3.10
conda activate autotrain
pip install autotrain-advanced
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc
```

--------------------------------

### JSONL Data Format Example

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/token_classification.mdx

Example of how to format your data in JSONL for token classification. Each line is a JSON object with 'tokens' and 'tags' keys.

```json
{"tokens": ["I", "love", "Paris"],"tags": ["O", "O", "B-LOC"]}
{"tokens": ["I", "live", "in", "New", "York"],"tags": ["O", "O", "O", "B-LOC", "I-LOC"]}
.
.
.
```

--------------------------------

### Launch AutoTrain UI

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Starts the AutoTrain web interface locally. Access the UI at http://127.0.0.1:8000 and API documentation at http://127.0.0.1:8000/docs.

```bash
export HF_TOKEN=your_hugging_face_write_token
autotrain app --host 127.0.0.1 --port 8000
```

--------------------------------

### Install AutoTrain Advanced via PIP

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Requires Python 3.10 or higher.

```bash
pip install autotrain-advanced
```

--------------------------------

### Image Regression Metadata Example

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/image_classification_regression.mdx

Example JSON Lines format for metadata.jsonl used in image regression. Each line maps a file name to its numerical target value.

```json
{"file_name": "0001.png", "target": 0.5}
{"file_name": "0002.png", "target": 0.7}
{"file_name": "0003.png", "target": 0.3}
```

--------------------------------

### Multi-line Manually Formatted Dataset

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/llm_finetuning.mdx

A JSONL format example where each line contains a pre-formatted text field.

```json
[{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nhello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nhi nice to meet you<|eot_id|>"}]
```

--------------------------------

### Run Local Extractive QA Training

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/extractive_qa.mdx

Execute this bash command to start the local training process for an Extractive QA model using the specified configuration file.

```bash
$ autotrain --config config.yaml
```

--------------------------------

### Multi-line Chatbot Dataset

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/llm_finetuning.mdx

A JSONL format example containing multiple conversation samples.

```json
[{"content": "hello", "role": "user"}, {"content": "hi nice to meet you", "role": "assistant"}]
[{"content": "how are you", "role": "user"}, {"content": "I am fine", "role": "assistant"}]
[{"content": "What is your name?", "role": "user"}, {"content": "My name is Mary", "role": "assistant"}]
[{"content": "Which is the best programming language?", "role": "user"}, {"content": "Python", "role": "assistant"}]
.
.
.
```

--------------------------------

### Configure and Launch AutoTrain in Colab

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_ngrok.ipynb

Sets required environment variables for Hugging Face and ngrok authentication before installing and launching the application.

```python
#@title 🤗 AutoTrain
#@markdown In order to use this colab
#@markdown - Enter your [Hugging Face Write Token](https://huggingface.co/settings/tokens)
#@markdown - Enter your [ngrok auth token](https://dashboard.ngrok.com/get-started/your-authtoken)
huggingface_token = '' # @param {type:"string"}
ngrok_token = "" # @param {type:"string"}

#@markdown
#@markdown - Attach appropriate accelerator `Runtime > Change runtime type > Hardware accelerator`
#@markdown - click `Runtime > Run all`
#@markdown - Follow the link to access the UI
#@markdown - Training happens inside this Google Colab
#@markdown - report issues / feature requests [here](https://github.com/huggingface/autotrain-advanced/issues)

import os
os.environ["HF_TOKEN"] = str(huggingface_token)
os.environ["NGROK_AUTH_TOKEN"] = str(ngrok_token)
os.environ["AUTOTRAIN_LOCAL"] = "1"

!pip install -U autotrain-advanced > install_logs.txt 2>&1
!autotrain app --share
```

--------------------------------

### Example LLM Fine-tuning Configuration

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/config.mdx

This YAML configuration defines parameters for fine-tuning a Llama 3 8B Instruct model using the ORPO trainer. It specifies the base model, dataset, training parameters, and Hugging Face Hub integration settings.

```yaml
task: llm
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-orpo
log: tensorboard
backend: local

data:
  path: argilla/distilabel-capybara-dpo-7k-binarized
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected

params:
  trainer: orpo
  block_size: 1024
  model_max_length: 2048
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: bf16

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true
```

--------------------------------

### Manually Formatted Chatbot Sample

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/llm_finetuning.mdx

An example of a sample manually formatted with special tokens for models that do not use --chat-template.

```text
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 03 Oct 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHelp write a letter of 100 -200 words to my future self for Kyra, reflecting on her goals and aspirations.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nDear Future Self,\n\nI hope you're happy and proud of what you've achieved. As I write this, I'm excited to think about our goals and how far you've come. One goal was to be a machine learning engineer. I hope you've worked hard and become skilled in this field. Keep learning and innovating. Traveling was important to us. I hope you've seen different places and enjoyed the beauty of our world. Remember the memories and lessons. Starting a family mattered to us. If you have kids, treasure every moment. Be patient, loving, and grateful for your family.\n\nTake care of yourself. Rest, reflect, and cherish the time you spend with loved ones. Remember your dreams and celebrate what you've achieved. Your determination brought you here. I'm excited to see the person you've become, the impact you've made, and the love and joy in your life. Embrace opportunities and keep dreaming big.\n\nWith love,\nKyra<|eot_id|>
```

--------------------------------

### Tabular Dataset CSV Format

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/tabular.mdx

Example of the required CSV structure for tabular training, including id, feature, and target columns.

```csv
id,category1,category2,feature1,target
1,A,X,0.3373961604172684,1
2,B,Z,0.6481718720511972,0
3,A,Y,0.36824153984054797,1
4,B,Z,0.9571551589530464,1
5,B,Z,0.14035078041264515,1
6,C,X,0.8700872583584364,1
7,A,Y,0.4736080452737105,0
8,C,Y,0.8009107519796442,1
9,A,Y,0.5204774795512048,0
10,A,Y,0.6788795301189603,0
.
.
.
```

--------------------------------

### Example Dataset Format for Extractive QA

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/extractive_qa.mdx

This JSONL format is required for training Extractive QA models. Ensure 'text', 'question', and 'answers' (with 'text' and 'answer_start') columns are present.

```json
{"context":"Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.","question":"To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?","answers":{"text":["Saint Bernadette Soubirous"],"answer_start":[515]}}
```

```json
{"context":"Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.","question":"What is in front of the Notre Dame Main Building?","answers":{"text":["a copper statue of Christ"],"answer_start":[188]}}
```

```json
{"context":"Architecturally, the school has a Catholic character. Atop the Main Building's gold dome is a golden statue of the Virgin Mary. Immediately in front of the Main Building and facing it, is a copper statue of Christ with arms upraised with the legend \"Venite Ad Me Omnes\". Next to the Main Building is the Basilica of the Sacred Heart. Immediately behind the basilica is the Grotto, a Marian place of prayer and reflection. It is a replica of the grotto at Lourdes, France where the Virgin Mary reputedly appeared to Saint Bernadette Soubirous in 1858. At the end of the main drive (and in a direct line that connects through 3 statues and the Gold Dome), is a simple, modern stone statue of Mary.","question":"The Basilica of the Sacred heart at Notre Dame is beside to which structure?","answers":{"text":["the Main Building"],"answer_start":[279]}}
```

--------------------------------

### Preview Documentation

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/README.md

Serve the documentation locally for preview using `doc-builder preview`. This command requires existing doc files. For new files, update `_toctree.yml` and restart the preview.

```bash
doc-builder preview autotrain docs/source/
```

--------------------------------

### Create and run training project with Python

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Initializes and executes a training project using the AutoTrain SDK.

```python
backend = "local"
project = AutoTrainProject(params=params, backend=backend, process=True)
project.create()
```

--------------------------------

### Build Documentation

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/README.md

Generate the documentation files using `doc-builder`. Specify the source directory and the build output directory. The build directory will be created if it doesn't exist.

```bash
doc-builder build autotrain docs/source/ --build_dir ~/tmp/test-build
```

--------------------------------

### Run Training Script

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart_py.mdx

Set environment variables and execute the training script.

```bash
$ export HF_USERNAME=<your-hf-username>
$ export HF_TOKEN=<your-hf-write-token>
$ python train.py
```

--------------------------------

### Execute Training with Environment Variables

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Sets authentication credentials before running the training command.

```bash
$ export HF_USERNAME=<your_hugging_face_username>
$ export HF_TOKEN=<your_hugging_face_write_token>
$ autotrain --config <path_to_config_file>
```

--------------------------------

### Create Training Project via API

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/autotrain_api.mdx

Send a POST request to the AutoTrain API to initiate a model training project with specific task and hardware configurations.

```bash
curl -X POST "http://127.0.0.1:8000/api/create_project" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer hf_XXXXX" \
     -d '{
           "username": "abhishek",
           "project_name": "my-autotrain-api-model",
           "task": "llm:orpo",
           "base_model": "meta-llama/Meta-Llama-3-8B-Instruct",
           "hub_dataset": "argilla/distilabel-capybara-dpo-7k-binarized",
           "train_split": "train",
           "hardware": "spaces-a10g-large",
           "column_mapping": {
               "text_column": "chosen",
               "rejected_text_column": "rejected",
               "prompt_text_column": "prompt"
           },
           "params": {
               "block_size": 1024,
               "model_max_length": 4096,
               "max_prompt_length": 512,
               "epochs": 1,
               "batch_size": 2,
               "lr": 0.00003,
               "peft": true,
               "quantization": "int4",
               "target_modules": "all-linear",
               "padding": "right",
               "optimizer": "adamw_torch",
               "scheduler": "linear",
               "gradient_accumulation": 4,
               "mixed_precision": "fp16",
               "chat_template": "chatml"
           }
         }'
```

--------------------------------

### Run Training via CLI Config

Source: https://github.com/huggingface/autotrain-advanced/blob/main/README.md

Executes training using a specified configuration file.

```bash
autotrain --config <path_to_config_file>
```

--------------------------------

### CSV Data Format for Regression

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/text_classification_regression.mdx

Example CSV structure for scoring tasks on a numerical scale.

```csv
text,target
"this movie is great",4.9
"this movie is bad",1.5
.
.
.
```

--------------------------------

### Configure AutoTrain LLM Project and Hyperparameters

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

Sets up project configuration, Hugging Face Hub integration, and training hyperparameters. It generates a 'conf.yaml' file based on user inputs.

```python
#@markdown --- 
#@markdown #### Project Config
#@markdown Note: if you are using a restricted/private model, you need to enter your Hugging Face token in the next step.
project_name = 'my-autotrain-llm' # @param {type:"string"} 
model_name = 'abhishek/llama-2-7b-hf-small-shards' # @param {type:"string"}

#@markdown --- 
#@markdown #### Push to Hub?
#@markdown Use these only if you want to push your trained model to a private repo in your Hugging Face Account
#@markdown If you dont use these, the model will be saved in Google Colab and you are required to download it manually.
#@markdown Please enter your Hugging Face write token. The trained model will be saved to your Hugging Face account.
#@markdown You can find your token here: https://huggingface.co/settings/tokens
push_to_hub = False # @param ["False", "True"] {type:"raw"}
hf_token = "hf_XXX" #@param {type:"string"}
hf_username = "abc" #@param {type:"string"}

#@markdown --- 
#@markdown #### Hyperparameters
unsloth = False # @param ["False", "True"] {type:"raw"}
learning_rate = 2e-4 # @param {type:"number"}
num_epochs = 1 #@param {type:"number"}
batch_size = 1 # @param {type:"slider", min:1, max:32, step:1}
block_size = 1024 # @param {type:"number"}
trainer = "sft" # @param ["generic", "sft"] {type:"string"}
warmup_ratio = 0.1 # @param {type:"number"}
weight_decay = 0.01 # @param {type:"number"}
gradient_accumulation = 4 # @param {type:"number"}
mixed_precision = "fp16" # @param ["fp16", "bf16", "none"] {type:"string"}
peft = True # @param ["False", "True"] {type:"raw"}
quantization = "int4" # @param ["int4", "int8", "none"] {type:"string"}
lora_r = 16 #@param {type:"number"}
lora_alpha = 32 #@param {type:"number"}
lora_dropout = 0.05 #@param {type:"number"}

os.environ["HF_TOKEN"] = hf_token
os.environ["HF_USERNAME"] = hf_username

conf = f"""
task: llm-{trainer}
base_model: {model_name}
project_name: {project_name}
log: tensorboard
backend: local

data:
  path: data/
  train_split: train
  valid_split: null
  chat_template: null
  column_mapping:
    text_column: text

params:
  block_size: {block_size}
  lr: {learning_rate}
  warmup_ratio: {warmup_ratio}
  weight_decay: {weight_decay}
  epochs: {num_epochs}
  batch_size: {batch_size}
  gradient_accumulation: {gradient_accumulation}
  mixed_precision: {mixed_precision}
  peft: {peft}
  quantization: {quantization}
  lora_r: {lora_r}
  lora_alpha: {lora_alpha}
  lora_dropout: {lora_dropout}
  unsloth: {unsloth}

hub:
  username: ${{HF_USERNAME}}
  token: ${{HF_TOKEN}}
  push_to_hub: {push_to_hub}
"""

with open("conf.yaml", "w") as f:
  f.write(conf)
```

--------------------------------

### CSV Data Format for Classification

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/tasks/text_classification_regression.mdx

Example CSV structure for sentiment classification with text and label columns.

```csv
text,target
"this movie is great",positive
"this movie is bad",negative
.
.
.
```

--------------------------------

### Finetune an LLM with AutoTrain

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart_py.mdx

Configure and execute an LLM training project using LLMTrainingParams and AutoTrainProject.

```python
import os

from autotrain.params import LLMTrainingParams
from autotrain.project import AutoTrainProject


params = LLMTrainingParams(
    model="meta-llama/Llama-3.2-1B-Instruct",
    data_path="HuggingFaceH4/no_robots",
    chat_template="tokenizer",
    text_column="messages",
    train_split="train",
    trainer="sft",
    epochs=3,
    batch_size=1,
    lr=1e-5,
    peft=True,
    quantization="int4",
    target_modules="all-linear",
    padding="right",
    optimizer="paged_adamw_8bit",
    scheduler="cosine",
    gradient_accumulation=8,
    mixed_precision="bf16",
    merge_adapter=True,
    project_name="autotrain-llama32-1b-finetune",
    log="tensorboard",
    push_to_hub=True,
    username=os.environ.get("HF_USERNAME"),
    token=os.environ.get("HF_TOKEN"),
)


backend = "local"
project = AutoTrainProject(params=params, backend=backend, process=True)
project.create()
```

--------------------------------

### Image Classification Data Structure

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Example directory structure for image classification datasets. Classes are represented by subfolders.

```tree
train.zip
├── cats/
│   ├── cat.1.jpg
│   ├── cat.2.jpg
│   └── ...
└── dogs/
    ├── dog.1.jpg
    ├── dog.2.jpg
    └── ...

```

--------------------------------

### Access AutoTrain CLI

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/quickstart.mdx

Commands to initialize the CLI and display available help options.

```bash
$ export HF_TOKEN=your_hugging_face_write_token
$ autotrain --help
```

```text
usage: autotrain <command> [<args>]

positional arguments:
{
    app,
    llm,
    setup,
    api,
    text-classification,
    text-regression,
    image-classification,
    tabular,
    spacerunner,
    seq2seq,
    token-classification
}
    
    commands

options:
  -h, --help            show this help message and exit
  --version, -v         Display AutoTrain version
  --config CONFIG       Optional configuration file
```

--------------------------------

### Launch AutoTrain Colab App

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain.ipynb

Imports necessary modules and launches the AutoTrain Colab application. This requires the autotrain-advanced package to be installed.

```python
from IPython.display import display
from autotrain.app.colab import colab_app
elements = colab_app()
display(elements)
```

--------------------------------

### Initialize TextRegressionParams for Hub datasets

Source: https://github.com/huggingface/autotrain-advanced/blob/main/notebooks/text_regression.ipynb

Configure training parameters for a dataset hosted on the Hugging Face Hub.

```python
params = TextRegressionParams(
    model="google-bert/bert-base-uncased",
    data_path="lewtun/drug-reviews", # path to the dataset on huggingface hub
    text_column="review", # the column in the dataset that contains the text
    target_column="rating", # the column in the dataset that contains the labels
    train_split="train",
    valid_split="test",
    epochs=3,
    batch_size=8,
    max_seq_length=512,
    lr=1e-5,
    optimizer="adamw_torch",
    scheduler="linear",
    gradient_accumulation=1,
    mixed_precision="fp16",
    project_name="autotrain-model",
    log="tensorboard",
    push_to_hub=True,
    username=HF_USERNAME,
    token=HF_TOKEN,
)
# tip: you can use `?TextClassificationParams` to see the full list of allowed parameters
```

--------------------------------

### Create training project via REST API

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Sends a POST request to the AutoTrain API to initiate a training job with specific model and dataset parameters.

```bash
curl -X POST "http://127.0.0.1:8000/api/create_project" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer hf_XXXXX" \
     -d '{
           "username": "your-username",
           "project_name": "my-autotrain-api-model",
           "task": "llm:orpo",
           "base_model": "meta-llama/Meta-Llama-3-8B-Instruct",
           "hub_dataset": "argilla/distilabel-capybara-dpo-7k-binarized",
           "train_split": "train",
           "hardware": "spaces-a10g-large",
           "column_mapping": {
               "text_column": "chosen",
               "rejected_text_column": "rejected",
               "prompt_text_column": "prompt"
           },
           "params": {
               "block_size": 1024,
               "model_max_length": 4096,
               "max_prompt_length": 512,
               "epochs": 1,
               "batch_size": 2,
               "lr": 0.00003,
               "peft": true,
               "quantization": "int4",
               "target_modules": "all-linear",
               "padding": "right",
               "optimizer": "adamw_torch",
               "scheduler": "linear",
               "gradient_accumulation": 4,
               "mixed_precision": "fp16",
               "chat_template": "chatml"
           }
         }'
```

--------------------------------

### POST /api/create_project

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Creates a new training project on the AutoTrain server.

```APIDOC
## POST /api/create_project

### Description
Creates and initializes a new training project with specified task, model, dataset, and training parameters.

### Method
POST

### Endpoint
http://127.0.0.1:8000/api/create_project

### Request Body
- **username** (string) - Required - The Hugging Face username.
- **project_name** (string) - Required - The name of the project.
- **task** (string) - Required - The training task (e.g., llm:orpo).
- **base_model** (string) - Required - The base model identifier from the Hub.
- **hub_dataset** (string) - Required - The dataset identifier from the Hub.
- **train_split** (string) - Required - The dataset split to use for training.
- **hardware** (string) - Required - The hardware configuration (e.g., spaces-a10g-large).
- **column_mapping** (object) - Required - Mapping of dataset columns to task-specific roles.
- **params** (object) - Required - Training hyperparameters.

### Request Example
{
  "username": "your-username",
  "project_name": "my-autotrain-api-model",
  "task": "llm:orpo",
  "base_model": "meta-llama/Meta-Llama-3-8B-Instruct",
  "hub_dataset": "argilla/distilabel-capybara-dpo-7k-binarized",
  "train_split": "train",
  "hardware": "spaces-a10g-large",
  "column_mapping": {
    "text_column": "chosen",
    "rejected_text_column": "rejected",
    "prompt_text_column": "prompt"
  },
  "params": {
    "block_size": 1024,
    "model_max_length": 4096,
    "max_prompt_length": 512,
    "epochs": 1,
    "batch_size": 2,
    "lr": 0.00003,
    "peft": true,
    "quantization": "int4",
    "target_modules": "all-linear",
    "padding": "right",
    "optimizer": "adamw_torch",
    "scheduler": "linear",
    "gradient_accumulation": 4,
    "mixed_precision": "fp16",
    "chat_template": "chatml"
  }
}
```

--------------------------------

### Sentence Transformer Data Formats

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Examples of data formats for sentence transformer training, including pair, triplet, pair_score, and qa formats.

```csv
# pair format
anchor,positive
"hello","hi"
"how are you","I am fine"

```

```csv
# triplet format
anchor,positive,negative
"hello","hi","goodbye"
"how are you","I am fine","I am not fine"

```

```csv
# pair_score format
sentence1,sentence2,score
"hello","hi",0.8
"how are you","goodbye",0.2

```

```csv
# qa format
query,answer
"What is Python?","Python is a programming language"

```

--------------------------------

### Image Regression Data Format

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Example data format for image regression, consisting of image files and a metadata JSONL file with target scores.

```json
{"file_name": "0001.png", "target": 0.5}
{"file_name": "0002.png", "target": 0.7}
{"file_name": "0003.png", "target": 0.3}

```

--------------------------------

### Object Detection Data Format

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Example JSON format for object detection annotations. Bounding boxes must be in COCO format [x, y, width, height].

```json
{"file_name": "0001.png", "objects": {"bbox": [[302.0, 109.0, 73.0, 52.0]], "category": [0]}}
{"file_name": "0002.png", "objects": {"bbox": [[810.0, 100.0, 57.0, 28.0]], "category": [1]}}
{"file_name": "0003.png", "objects": {"bbox": [[160.0, 31.0, 248.0, 616.0], [741.0, 68.0, 202.0, 401.0]], "category": [2, 2]}}

```

--------------------------------

### Run AutoTrain with a Configuration File

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/config.mdx

Execute the AutoTrain command with the `--config` flag, pointing to your YAML configuration file. This command initiates the model training process based on the specified settings.

```bash
autotrain --config path/to/config.yaml
```

--------------------------------

### Run AutoTrain Training

Source: https://github.com/huggingface/autotrain-advanced/blob/main/colabs/AutoTrain_LLM.ipynb

Executes the AutoTrain training process using the configuration specified in 'conf.yaml'.

```bash
!autotrain --config conf.yaml
```

--------------------------------

### Triplet Data Mapping for AutoTrain

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/col_map.mdx

Use this mapping for triplet tasks, specifying columns for anchor, positive, and negative examples. Ensure these column names match your dataset.

```json
{"sentence1_column": "anchor", "sentence2_column": "positive", "sentence3_column": "negative"}
```

--------------------------------

### Import necessary libraries for AutoTrain

Source: https://github.com/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb

Import the required classes for LLM training parameters and project creation from the autotrain library.

```python
from autotrain.params import LLMTrainingParams
from autotrain.project import AutoTrainProject
```

--------------------------------

### Basic Text Classification Column Mapping

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/col_map.mdx

Map your dataset's text and target columns to AutoTrain's expected 'text' and 'label' fields. This is a common setup for text classification tasks.

```json
{"text": "text", "label": "target"}
```

--------------------------------

### Configure LLM DPO with QLoRA

Source: https://context7.com/huggingface/autotrain-advanced/llms.txt

Use this configuration for Direct Preference Optimization (DPO) tasks using QLoRA.

```yaml
task: llm-dpo
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-dpo-qlora
log: tensorboard
backend: local

data:
  path: mlabonne/orpo-dpo-mix-40k
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt

params:
  block_size: 1024
  model_max_length: 2048
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: fp16

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: false
```

--------------------------------

### Extractive Question Answering Column Mapping

Source: https://github.com/huggingface/autotrain-advanced/blob/main/docs/source/col_map.mdx

Configure column mappings for extractive question answering. 'text' maps to context, 'question' to the question, and 'answers' to a dictionary containing the answer text and its start position. Ensure these keys match your dataset structure.

```json
{"text": "context", "question": "question", "answer": "answers"}
```

--------------------------------

### Configure parameters for local CSV/JSONL datasets

Source: https://github.com/huggingface/autotrain-advanced/blob/main/notebooks/llm_finetuning.ipynb

Adjust the `data_path`, `text_column`, and `train_split` parameters when using local CSV or JSONL files for training. Ensure `data_path` points to the directory containing your dataset file.

```python
params = LLMTrainingParams(
    data_path="data/", # this is the path to folder where train.jsonl/train.csv is located
    text_column="text", # this is the column name in the CSV/JSONL file which contains the text
    train_split = "train" # this is the filename without extension
    .
    .
    .
)
```