### Developer Install: macOS, Linux, WSL

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Clone the Unsloth repository, navigate to the directory, and run the local installation script. Then, start Unsloth Studio. To update, repeat the process after pulling the latest changes.

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -p 8888
```

```bash
cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888
```

--------------------------------

### Clone Unsloth Repo and Setup Studio

Source: https://github.com/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb

Clones the Unsloth repository and executes the setup script for local installation on Colab.

```python
!git clone --depth 1 --branch main https://github.com/unslothai/unsloth.git
%cd /content/unsloth
!chmod +x studio/setup.sh && ./studio/setup.sh --local
```

--------------------------------

### Nightly Build: Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Clone the Unsloth repository, switch to the nightly branch, set the execution policy, and run the local installation script. Then, start Unsloth Studio. To launch subsequently, simply run the studio command.

```powershell
git clone https://github.com/unslothai/unsloth.git
cd unsloth
git checkout nightly
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888
```

```powershell
unsloth studio -p 8888
```

--------------------------------

### Nightly Build: macOS, Linux, WSL

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Clone the Unsloth repository, switch to the nightly branch, and run the local installation script. Then, start Unsloth Studio. To launch subsequently, simply run the studio command.

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
./install.sh --local
unsloth studio -p 8888
```

```bash
unsloth studio -p 8888
```

--------------------------------

### Developer Install: Windows PowerShell

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Clone the Unsloth repository, navigate to the directory, set the execution policy, and run the local installation script. Then, start Unsloth Studio. To update, repeat the process after pulling the latest changes.

```powershell
git clone https://github.com/unslothai/unsloth.git
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -p 8888
```

```powershell
cd unsloth && git pull
./install.sh --local
unsloth studio -p 8888
```

--------------------------------

### Start Unsloth Studio Backend

Source: https://github.com/unslothai/unsloth/blob/main/studio/Unsloth_Studio_Colab.ipynb

Imports the necessary module and starts the Unsloth Studio backend service.

```python
import sys
sys.path.insert(0, "/content/unsloth/studio/backend")
from colab import start
start()
```

--------------------------------

### Advanced Launch: Install to Custom Location

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided script, setting the UNSLOTH_STUDIO_HOME environment variable to specify a custom installation directory. This is for macOS, Linux, and WSL.

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_STUDIO_HOME=/abs/path sh
```

--------------------------------

### Advanced Launch: Install to Custom Location - Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided PowerShell command, setting the UNSLOTH_STUDIO_HOME environment variable to specify a custom installation directory.

```powershell
$env:UNSLOTH_STUDIO_HOME='C:\path'; irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Install Unsloth Core on Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Installs Unsloth Core on Windows using winget for Python and uv. Sets up a Python 3.13 environment and installs Unsloth with auto torch backend. Note: 'pip install unsloth' requires PyTorch to be pre-installed.

```powershell
winget install -e --id Python.Python.3.13
winget install --id=astral-sh.uv  -e
uv venv unsloth_env --python 3.13
.\unsloth_env\Scripts\activate
uv pip install unsloth --torch-backend=auto
```

--------------------------------

### Dockerfile for Unsloth Environment Setup

Source: https://github.com/unslothai/unsloth/wiki/Home

This Dockerfile defines the environment for Unsloth. It starts with a CUDA-enabled base image, installs system dependencies, sets up Miniconda, creates a Conda environment from a YAML file, installs Unsloth from its GitHub repository, and configures the environment to run Jupyter Lab by default. It ensures GPU support through CUDA and Conda.

```dockerfile
# Stage 1: Base image with system dependencies
FROM nvidia/cuda:11.8.0-devel-ubuntu22.04 as base

# Install system dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    git \
    vim \
    curl \
    wget \
    && rm -rf /var/lib/apt/lists/*

# Install Miniconda only if it's not already installed
RUN if [ ! -d "/opt/conda" ]; then \
        wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh && \
        bash miniconda.sh -b -p /opt/conda && \
        rm miniconda.sh; \
    fi

# Set path to conda
ENV PATH /opt/conda/bin:$PATH

# Set path to conda
ENV PATH /opt/conda/bin:$PATH

# Stage 2: Python environment setup
FROM base as python-env

COPY unsloth_env_file.yml unsloth_env_file.yml

RUN conda env create -f unsloth_env_file.yml

SHELL ["conda", "run", "-n", "unsloth_env", "/bin/bash", "-c"]

# Stage 3: Final image
FROM python-env as final

# Install Unsloth (This step is separate because it's likely to change more frequently)
RUN pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

ENV PATH /usr/local/cuda/bin:$PATH
ENV LD_LIBRARY_PATH /usr/local/cuda/lib64:$LD_LIBRARY_PATH

# Set the working directory
WORKDIR /workspace

# Set the default command to run Jupyter Lab
CMD ["conda", "run", "--no-capture-output", "-n", "unsloth_env", "jupyter", "lab", "--ip=0.0.0.0", "--no-browser", "--allow-root", "--NotebookApp.token='',"--NotebookApp.password=''"]

```

--------------------------------

### Install Unsloth Core on Linux, WSL

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Installs Unsloth Core using uv for environment and package management. Sets up a Python 3.13 environment and installs Unsloth with auto torch backend.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv unsloth_env --python 3.13
source unsloth_env/bin/activate
uv pip install unsloth --torch-backend=auto
```

--------------------------------

### Install Unsloth on Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Use this PowerShell command to install Unsloth on Windows.

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Install Plugin for Development

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/plugins/data-designer-github-repo-seed/README.md

Command to install the data-designer-github-repo-seed plugin in editable mode for development purposes.

```bash
pip install -e .
```

--------------------------------

### Install Unsloth on macOS, Linux, WSL

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Use this command to install Unsloth on macOS, Linux, or Windows Subsystem for Linux environments.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

--------------------------------

### Advanced Launch: Skip PyTorch (GGUF-only) - Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided PowerShell command, setting the UNSLOTH_NO_TORCH environment variable to enable GGUF-only mode.

```powershell
$env:UNSLOTH_NO_TORCH=1; irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Advanced Launch: Skip PyTorch (GGUF-only)

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided script, setting the UNSLOTH_NO_TORCH environment variable to enable GGUF-only mode. This is for macOS, Linux, and WSL.

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_NO_TORCH=1 sh
```

--------------------------------

### Install llama.cpp Dependencies

Source: https://github.com/unslothai/unsloth/wiki/Home

This command installs the Python dependencies required for llama.cpp's Python scripts. It reads the dependencies from the `requirements.txt` file within the llama.cpp directory.

```shell
pip install -r requirements.txt
```

--------------------------------

### Advanced Launch: Pin Python Version

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided script, setting the UNSLOTH_PYTHON environment variable to specify the desired Python version. This is for macOS, Linux, and WSL.

```bash
curl -fsSL https://unsloth.ai/install.sh | UNSLOTH_PYTHON=3.12 sh
```

--------------------------------

### Remove Unsloth Install Directory (Mac/Linux/WSL)

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Manually remove only the Unsloth Studio installation directory if you wish to keep launcher data and shortcuts for a future reinstall.

```bash
rm -rf ~/.unsloth/studio
```

--------------------------------

### Advanced Launch: Pin Python Version - Windows

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Install Unsloth using the provided PowerShell command, setting the UNSLOTH_PYTHON environment variable to specify the desired Python version.

```powershell
$env:UNSLOTH_PYTHON='3.12'; irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Example ModelMeta Instantiation for Llama 3.1

Source: https://github.com/unslothai/unsloth/blob/main/unsloth/registry/REGISTRY.md

An example of instantiating the ModelMeta dataclass for a specific model version, Llama 3.1. It sets the organization, base name, version, instruct tags, model sizes, the corresponding ModelInfo class, and supported quantization types. This demonstrates how to configure metadata for a particular model.

```python
# Assuming LlamaModelInfo and QuantType are defined
class LlamaModelInfo(ModelInfo):
    pass

LlamaMeta_3_1 = ModelMeta(
    org="meta-llama",
    base_name="Llama",
    instruct_tags=[None, "Instruct"],
    model_version="3.1",
    model_sizes=["8"],
    model_info_cls=LlamaModelInfo,
    is_multimodal=False,
    quant_types=[QuantType.NONE, QuantType.BNB, QuantType.UNSLOTH],
)

```

--------------------------------

### Example Output of Model Verification Script

Source: https://github.com/unslothai/unsloth/blob/main/unsloth/registry/REGISTRY.md

Sample output from running a model registration file as a script. It lists the verified model paths, including different versions, quantization types (e.g., 'bnb-4bit', 'unsloth-bnb-4bit'), and instruct variants, confirming their presence on Hugging Face Hub.

```text
✓ unsloth/Llama-3.1-8B
✓ unsloth/Llama-3.1-8B-bnb-4bit
✓ unsloth/Llama-3.1-8B-unsloth-bnb-4bit
✓ meta-llama/Llama-3.1-8B
✓ unsloth/Llama-3.1-8B-Instruct
✓ unsloth/Llama-3.1-8B-Instruct-bnb-4bit
✓ unsloth/Llama-3.1-8B-Instruct-unsloth-bnb-4bit
✓ meta-llama/Llama-3.1-8B-Instruct
✓ unsloth/Llama-3.2-1B
✓ unsloth/Llama-3.2-1B-bnb-4bit
✓ unsloth/Llama-3.2-1B-unsloth-bnb-4bit
✓ meta-llama/Llama-3.2-1B
...

```

--------------------------------

### GitHub Repo Seed Configuration

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/plugins/data-designer-github-repo-seed/README.md

Example configuration for the GitHub repository seed source within an Unsloth Studio recipe. It specifies repositories, token, item types, limits, and comment inclusion.

```json
{
  "seed_config": {
    "source": {
      "seed_type": "github_repo",
      "repos": ["unslothai/unsloth", "unslothai/unsloth-zoo"],
      "token": "",
      "item_types": ["issues", "pulls"],
      "limit": 100,
      "include_comments": true,
      "max_comments_per_item": 30
    },
    "sampling_strategy": "shuffle",
    "selection_strategy": null
  }
}
```

--------------------------------

### Remove Unsloth Install Directory (Windows)

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Use this command in Windows to delete the Unsloth Studio install directory while preserving launcher data and shortcuts for potential reinstallation.

```powershell
Remove-Item -Recurse -Force "$HOME\.unsloth\studio"
```

--------------------------------

### Save Models to GGUF Format

Source: https://github.com/unslothai/unsloth/wiki/Home

These Python commands demonstrate how to save models in the GGUF format, both locally and to the Hugging Face Hub. It shows examples for different quantization methods like `q4_k_m`, `q8_0`, and `f16`.

```python
model.save_pretrained_gguf("dir", tokenizer, quantization_method = "q4_k_m")
model.save_pretrained_gguf("dir", tokenizer, quantization_method = "q8_0")
model.save_pretrained_gguf("dir", tokenizer, quantization_method = "f16")

model.push_to_hub_gguf("hf_username/dir", tokenizer, quantization_method = "q4_k_m")
model.push_to_hub_gguf("hf_username/dir", tokenizer, quantization_method = "q8_0")
```

--------------------------------

### Convert Model to GGUF using llama.cpp (Bash)

Source: https://github.com/unslothai/unsloth/wiki/Home

Installs necessary dependencies, clones the llama.cpp repository, builds the quantization tools, and converts a Hugging Face model to GGUF format with f16 output type using the terminal.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggerganov/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=ON -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-quantize llama-cli llama-gguf-split llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp

python llama.cpp/convert_hf_to_gguf.py FOLDER --outfile OUTPUT --outtype f16
```

--------------------------------

### AIME Dataset Problem Format Example (JSON)

Source: https://github.com/unslothai/unsloth/blob/main/tests/utils/aime_eval.md

Illustrates the expected format for AIME dataset problems, including the question, answer, solution, and source information. This structure is automatically handled by the evaluator.

```json
{
    "global_id": 0,
    "original_id": "problem_1",
    "source_dataset": "test2024",
    "problem": "Find the number of...",
    "answer": "123",
    "solution": "Step-by-step solution...",
    "prompt": [
        {"role": "system", "content": "You are a mathematical problem solver..."},
        {"role": "user", "content": "Problem: Find the number of..."}
    ]
}
```

--------------------------------

### Launch Unsloth Studio

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Launches the Unsloth Studio web UI. Add -H 0.0.0.0 for cloud or global access.

```bash
unsloth studio -p 8888
```

--------------------------------

### Run Unsloth Studio with Docker

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Deploys Unsloth Studio using a Docker container, mapping ports and mounting a workspace volume. Ensures GPU access is enabled.

```bash
docker run -d -e JUPYTER_PASSWORD="mypassword" \
  -p 8888:8888 -p 8000:8000 -p 2222:22 \
  -v $(pwd)/work:/workspace/work \
  --gpus all \
  unsloth/unsloth
```

--------------------------------

### Advanced Launch: Cap CPU Threads

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Launch Unsloth Studio with a capped number of native CPU threads by setting the UNSLOTH_CPU_THREADS environment variable. This is useful on high-core hosts.

```bash
UNSLOTH_CPU_THREADS=8 unsloth studio -p 8888
```

--------------------------------

### Unsloth Parameter Flow for GPU Selection

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/utils/hardware/VRAM_ESTIMATION.md

Illustrates the data flow from frontend requests to backend GPU selection and device mapping for training and inference. Covers explicit and automatic GPU ID selection, including VRAM estimation for the latter.

```python
Frontend -> routes/{training,inference}.py
         -> prepare_gpu_selection(gpu_ids, model_name, ...)
            |
            +-- gpu_ids is explicit (e.g. [5,6,7])
            |     -> resolve_requested_gpu_ids: validate against parent-visible set
            |     -> return all requested GPUs (model sharded across all of them)
            |
            +-- gpu_ids is None or []
                  -> auto_select_gpu_ids: estimate VRAM, pick minimum GPUs needed
                  -> estimate_required_model_memory_gb -> estimate_training_vram
                  -> greedy selection: rank GPUs by free VRAM, add until model fits

         -> get_device_map(resolved_gpu_ids)
            -> "balanced" if >1 GPU, "sequential" otherwise

         -> worker subprocess: apply_gpu_ids(resolved_gpu_ids)
            -> sets CUDA_VISIBLE_DEVICES before torch/CUDA init
```

--------------------------------

### Finetune from Last Checkpoint

Source: https://github.com/unslothai/unsloth/wiki/Home

This section explains how to resume finetuning from the last saved checkpoint. It requires configuring the `Trainer` with `save_strategy` and `save_steps`, then calling `trainer.train()` with `resume_from_checkpoint = True`.

```python
trainer = SFTTrainer(
    ....
    args = TrainingArguments(
        ....
        output_dir = "outputs",
        save_strategy = "steps",
        save_steps = 50,
    ),
)
trainer_stats = trainer.train(resume_from_checkpoint = True)
```

--------------------------------

### Executing Model Registration File for Verification

Source: https://github.com/unslothai/unsloth/blob/main/unsloth/registry/REGISTRY.md

Demonstrates how to execute a model registration file (e.g., '_llama.py') as a Python script. This action verifies that all registered models are correctly available on Hugging Face Hub, providing a confirmation of the registry's integrity.

```bash
python unsloth.registry._llama.py

```

--------------------------------

### Multi-GPU VRAM Estimation Formula

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/utils/hardware/VRAM_ESTIMATION.md

This formula calculates the usable VRAM in a multi-GPU setup, accounting for an overhead discount on additional GPUs beyond the first. It's calibrated for 2-8 GPU systems with NVLink and PCIe.

```python
usable_gb = free[gpu_0] + sum(free[gpu_i] * 0.85 for i in 1..N)
```

--------------------------------

### Train on Completions/Responses Only

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python snippet enables training exclusively on the completion or response part of the data, ignoring the input prompts. It utilizes the `train_on_responses_only` utility from Unsloth within a `SFTTrainer` setup.

```python
from trl import SFTTrainer
from transformers import TrainingArguments, DataCollatorForSeq2Seq
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),
    ...
    args = TrainingArguments(
        ...
    ),
)
from unsloth.chat_templates import train_on_responses_only
trainer = train_on_responses_only(trainer)
```

--------------------------------

### Build llama.cpp (CPU)

Source: https://github.com/unslothai/unsloth/wiki/Home

This command compiles the llama.cpp project for CPU execution. It first cleans any previous builds and then builds the project using all available cores (`-j`) for faster compilation.

```shell
make clean && make all -j
```

--------------------------------

### Load LoRA Adapters for Continued Finetuning

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python code snippet demonstrates how to load a previously saved LoRA adapter using Unsloth's FastLanguageModel. It prepares the model and tokenizer for continued training, resetting the optimizer state. For loading optimizer states, refer to subsequent sections.

```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "LORA_MODEL_NAME",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)
trainer = Trainer(...)
trainer.train()
```

--------------------------------

### Gradient Memory Floor (Full Fine-Tuning)

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/utils/hardware/VRAM_ESTIMATION.md

Sets a minimum VRAM requirement for gradients during full fine-tuning, ensuring it's at least 15% of the model weight memory to account for overheads.

```text
gradient_bytes = max(computed_gradient_bytes, weights * 0.15)
```

--------------------------------

### Uninstall Unsloth Studio (MacOS, WSL, Linux)

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Use this command to fully remove Unsloth Studio from your system. It stops servers, removes directories, and cleans up shortcuts.

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.sh | sh
```

--------------------------------

### Gradient Memory Floor (LoRA/QLoRA)

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/utils/hardware/VRAM_ESTIMATION.md

Sets a minimum VRAM requirement for gradients during LoRA/QLoRA training. This floor is capped by the scale of trainable states and live activations, preventing the frozen model's weight size from dominating overhead calculations.

```text
raw_gradient_bytes = trainable_params * 2
gradient_floor = min(weights * 0.15, max(computed_activation_bytes, optimizer_bytes))
gradient_bytes = max(raw_gradient_bytes, gradient_floor)
```

--------------------------------

### Uninstall Unsloth Studio (Windows PowerShell)

Source: https://github.com/unslothai/unsloth/blob/main/README.md

Execute this command in PowerShell to completely uninstall Unsloth Studio. It handles server stops, directory removals, and shortcut cleanup.

```powershell
irm https://raw.githubusercontent.com/unslothai/unsloth/main/scripts/uninstall.ps1 | iex
```

--------------------------------

### Verify Hugging Face CLI Login

Source: https://github.com/unslothai/unsloth/wiki/Home

This shell command verifies that your Hugging Face CLI token is correctly set up and authenticated. It should output your Hugging Face username if the login was successful.

```shell
!huggingface-cli whoami
```

--------------------------------

### Enable 2x Faster Inference with Unsloth

Source: https://github.com/unslothai/unsloth/wiki/Home

Shows how to enable Unsloth's native 2x faster inference for QLoRA, LoRA, and non-LoRA models. This feature requires no code changes or new dependencies beyond loading the model and calling `FastLanguageModel.for_inference`.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 64)
```

--------------------------------

### Add New Tokens to Finetune with Unsloth

Source: https://github.com/unslothai/unsloth/wiki/Home

Demonstrates how to add new tokens to a model for finetuning using the `add_new_tokens` function. This function must be called before `FastLanguageModel.get_peft_model`. It takes the model, tokenizer, and a list of new tokens as input.

```python
from unsloth import add_new_tokens

model, tokenizer = FastLanguageModel.from_pretrained(...)
add_new_tokens(model, tokenizer, new_tokens = ["<CHARACTER_1>", "<THINKING>", "<SCRATCH_PAD>"])
model = FastLanguageModel.get_peft_model(...)
```

--------------------------------

### Implement Early Stopping for Model Training with Transformers

Source: https://github.com/unslothai/unsloth/wiki/Home

This snippet demonstrates how to configure and use the `EarlyStoppingCallback` from the `transformers` library to halt model training when the evaluation loss plateaus. It requires setting up the `SFTTrainer` with specific evaluation and saving strategies, and then adding the callback to the trainer instance. Ensure `load_best_model_at_end` is set to `True` and `metric_for_best_model` is configured to `eval_loss` with `greater_is_better` as `False`.

```python
from trl import SFTConfig, SFTTrainer
from transformers import EarlyStoppingCallback

# Assuming 'model', 'tokenizer', and 'new_dataset' are already defined

trainer = SFTTrainer(
    args = SFTConfig(
        fp16_full_eval = True,
        per_device_eval_batch_size = 2,
        eval_accumulation_steps = 4,
        output_dir = "training_checkpoints", # location of saved checkpoints for early stopping
        save_strategy = "steps",             # save model every N steps
        save_steps = 10,                     # how many steps until we save the model
        save_total_limit = 3,                # keep ony 3 saved checkpoints to save disk space
        eval_strategy = "steps",             # evaluate every N steps
        eval_steps = 10,                     # how many steps until we do evaluation
        load_best_model_at_end = True,       # MUST USE for early stopping
        metric_for_best_model = "eval_loss", # metric we want to early stop on
        greater_is_better = False,           # the lower the eval loss, the better
    ),
    model = model,
    tokenizer = tokenizer,
    train_dataset = new_dataset["train"],
    eval_dataset = new_dataset["test"],
)

early_stopping_callback = EarlyStoppingCallback(
    early_stopping_patience = 3,     # How many steps we will wait if the eval loss doesn't decrease
                                     # For example the loss might increase, but decrease after 3 steps
    early_stopping_threshold = 0.0,  # Can set higher - sets how much loss should decrease by until
                                     # we consider early stopping. For eg 0.01 means if loss was
                                     # 0.02 then 0.01, we consider to early stop the run.
)

trainer.add_callback(early_stopping_callback)

# Then train the model as usual: trainer.train()

```

--------------------------------

### Configure Trainer for Evaluation Loop

Source: https://github.com/unslothai/unsloth/wiki/Home

Sets up `TrainingArguments` for an evaluation loop to prevent OOM errors and potentially improve performance. Key parameters include `fp16_full_eval`, `per_device_eval_batch_size`, `eval_accumulation_steps`, `eval_strategy`, and `eval_steps`.

```python
new_dataset = dataset.train_test_split(test_size = 0.01)
SFTTrainer(
    args = TrainingArguments(
        fp16_full_eval = True,
        per_device_eval_batch_size = 2,
        eval_accumulation_steps = 4,
        eval_strategy = "steps",
        eval_steps = 1,
    ),
    train_dataset = new_dataset["train"],
    eval_dataset = new_dataset["test"],
)
```

--------------------------------

### Docker Compose Configuration for Unsloth Environment

Source: https://github.com/unslothai/unsloth/wiki/Home

This configuration sets up a Docker service named 'unsloth-env' designed for GPU acceleration. It maps local cache and workspace directories, exposes port 8888 for Jupyter Lab, and builds the environment using a Dockerfile.

```yaml
version: '3.8'

services:
  unsloth-env:
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ./cache:/root/.cache
      - ./workspace:/workspace
    working_dir: /workspace
    ports:
      - "8888:8888"  # For Jupyter Lab
    tty: true
    stdin_open: true
    build:
      context: .
      dockerfile: Dockerfile

```

--------------------------------

### Load Model with Unsloth FastLanguageModel

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python code demonstrates loading a pre-trained language model using Unsloth's FastLanguageModel. It configures parameters like sequence length, data type, and 4-bit quantization for efficient loading.

```python
from unsloth import FastLanguageModel
import torch
max_seq_length = 2048
dtype = None
load_in_4bit = True

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/tinyllama", # "unsloth/tinyllama" for 16bit loading
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)
```

--------------------------------

### Login to Hugging Face CLI

Source: https://github.com/unslothai/unsloth/wiki/Home

This shell command logs you into the Hugging Face CLI using an environment variable containing your HF token. Ensure the HF_TOKEN environment variable is set before running this command.

```shell
!huggingface-cli login --token $HF_TOKEN
```

--------------------------------

### Model Quantization Options

Source: https://github.com/unslothai/unsloth/wiki/Home

Defines a dictionary of allowed quantization methods for models. Each key represents a quantization type, and the value describes its characteristics, recommendations, and performance trade-offs.

```python
ALLOWED_QUANTS = {
    "not_quantized"  : "Recommended. Fast conversion. Slow inference, big files.",
    "fast_quantized" : "Recommended. Fast conversion. OK inference, OK file size.",
    "quantized"      : "Recommended. Slow conversion. Fast inference, small files.",
    "f32"     : "Not recommended. Retains 100% accuracy, but super slow and memory hungry.",
    "f16"     : "Fastest conversion + retains 100% accuracy. Slow and memory hungry.",
    "q8_0"    : "Fast conversion. High resource use, but generally acceptable.",
    "q4_k_m"  : "Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K",
    "q5_k_m"  : "Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K",
    "q2_k"    : "Uses Q4_K for the attention.vw and feed_forward.w2 tensors, Q2_K for the other tensors.",
    "q3_k_l"  : "Uses Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else Q3_K",
    "q3_k_m"  : "Uses Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else Q3_K",
    "q3_k_s"  : "Uses Q3_K for all tensors",
    "q4_0"    : "Original quant method, 4-bit.",
    "q4_1"    : "Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models.",
    "q4_k_s"  : "Uses Q4_K for all tensors",
    "q4_k"    : "alias for q4_k_m",
    "q5_k"    : "alias for q5_k_m",
    "q5_0"    : "Higher accuracy, higher resource usage and slower inference.",
    "q5_1"    : "Even higher accuracy, resource usage and slower inference.",
    "q5_k_s"  : "Uses Q5_K for all tensors",
    "q6_k"    : "Uses Q8_K for all tensors",
    "iq2_xxs" : "2.06 bpw quantization",
    "iq2_xs"  : "2.31 bpw quantization",
    "iq3_xxs" : "3.06 bpw quantization",
    "q3_k_xs" : "3-bit extra small quantization",
}
```

--------------------------------

### Continued Pretraining with lm_head and embed_tokens

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python code configures a model for continued pretraining or finetuning by enabling the `lm_head` and `embed_tokens` matrices. It includes a note about potential memory issues with Llama-3 8b in Colab, suggesting to only add `lm_head` if necessary. It also shows how to set different learning rates for these specific matrices.

```python
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",
                      "lm_head", "embed_tokens",],
    lora_alpha = 16,
)

from unsloth import UnslothTrainer, UnslothTrainingArguments

trainer = UnslothTrainer(
    ....
    args = UnslothTrainingArguments(
        ....
        learning_rate = 5e-5,
        embedding_learning_rate = 5e-6, # 2-10x smaller than learning_rate
    ),
)
```

--------------------------------

### Test Ollama Model

Source: https://github.com/unslothai/unsloth/wiki/Home

This command runs a test prompt against a created Ollama model. It's used to verify the model's functionality and response generation.

```shell
ollama run my-model "<s>\nQ: \nWhat is the capital of France?\nA:\n"
```

--------------------------------

### Build llama.cpp (NVIDIA GPU)

Source: https://github.com/unslothai/unsloth/wiki/Home

This command compiles the llama.cpp project with support for NVIDIA GPUs. It enables CUDA by setting the `LLAMA_CUDA=1` flag during the build process, utilizing all available cores (`-j`).

```shell
make clean && LLAMA_CUDA=1 make all -j
```

--------------------------------

### Convert LoRA Adapters for GGML

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python command converts LoRA adapter files into a format compatible with GGML. It processes the specified model directory and outputs a `ggml-adapter-model.bin` file.

```python
python convert-lora-to-ggml.py my-model
```

--------------------------------

### Perform Inference with Unsloth FastLanguageModel

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python code demonstrates how to perform inference with a fine-tuned Unsloth model. It prepares input tokens, generates output using the model, and decodes the output back into human-readable text.

```python
FastLanguageModel.for_inference(model)
inputs = tokenizer(
[
"""
<s>
Q:
What is the capital of France?
A:
"""
], return_tensors = "pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens = 1000, use_cache = True)

print(tokenizer.batch_decode(outputs))
```

--------------------------------

### Fine-tune Model with Unsloth SFTTrainer

Source: https://github.com/unslothai/unsloth/wiki/Home

This Python snippet shows how to fine-tune a language model using the SFTTrainer from the `trl` library. It configures training arguments such as batch size, learning rate, and optimization, and requires `dataset` and `eval_dataset` to be defined.

```python
from trl import SFTTrainer
from transformers import TrainingArguments
from transformers.utils import logging
logging.set_verbosity_info()

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    eval_dataset = eval_dataset,
    dataset_text_field = "text",
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = True, # Packs short sequences together to save time!
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_ratio = 0.1,
        num_train_epochs = 2,
        learning_rate = 2e-5,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.1,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = "outputs",
    ),
)

trainer_stats = trainer.train()
```

--------------------------------

### Clone llama.cpp Repository

Source: https://github.com/unslothai/unsloth/wiki/Home

This shell command clones the llama.cpp repository from GitHub. llama.cpp is a C/C++ implementation for running LLaMA models.

```shell
git clone https://github.com/ggerganov/llama.cpp
```

--------------------------------

### Merge GGUF Base Model and LoRA Adapters

Source: https://github.com/unslothai/unsloth/wiki/Home

This command merges a GGUF base model with LoRA adapter weights. It takes the base GGUF model, the converted LoRA adapter file, and outputs a new merged GGUF model.

```shell
export-lora --model-base tinyllama.f16.gguf --model-out tinyllama-my-model.gguf --lora my-model/ggml-adapter-model.bin
```

--------------------------------

### Model Weights Memory Modes

Source: https://github.com/unslothai/unsloth/blob/main/studio/backend/utils/hardware/VRAM_ESTIMATION.md

Estimates the VRAM for model weights based on the training mode: QLoRA 4-bit, or LoRA/Full fp16. The calculation accounts for different quantization factors and potential use of double quantization or skipped modules.

```text
QLoRA 4-bit | Quantizable * 2 / 3.2 + Non-quantizable * 2
LoRA / Full fp16 | (Quantizable + Non-quantizable) * 2
```

--------------------------------

### Create Ollama Model

Source: https://github.com/unslothai/unsloth/wiki/Home

This command creates a new Ollama model using a specified Modelfile. Replace 'my-model' with the desired name for your Ollama model.

```shell
ollama create my-model -f Modelfile
```

--------------------------------

### Clone Hugging Face Base Model

Source: https://github.com/unslothai/unsloth/wiki/Home

This command clones a base language model from the Hugging Face Hub. Replace 'unsloth/tinyllama' with the desired model repository name.

```shell
git clone https://huggingface.co/unsloth/tinyllama
```