Unsloth

https://unsloth.ai/docs
Admin
Unsloth is an open-source framework for LLM fine-tuning and reinforcement learning that enables...
Tokens:632,215
Snippets:4,512
Trust Score:9.6
Update:1 week ago
Show doc for...
Tokens:
Raw
### Install Unsloth Desktop App

Source: https://unsloth.ai/docs/basics/codex

The easiest way to get started is by installing the Unsloth Desktop app. It supports MacOS, Linux, Windows, and WSL.

```text
1
```

--------------------------------

### Serve Qwen3.8-27B-NVFP4 with vLLM

Source: https://unsloth.ai/docs/models/qwen3.8

Starts a vLLM server for the Qwen3.8-27B-NVFP4 model. Requires the vLLM installation from the previous step.

```bash
vllm serve unsloth/Qwen3.8-27B-NVFP4
```

--------------------------------

### Unsloth AMD quickstart command

Source: https://unsloth.ai/docs/basics/amd

Quickstart command for running Unsloth on AMD hardware. The page links to a Quickstart tab for further setup details.

```text
pip install unsloth
```

--------------------------------

### Initialize and Run GRPOTrainer

Source: https://unsloth.ai/docs/get-started/install/intel

Creates a GRPOTrainer with the model, tokenizer, reward functions, and training arguments, then starts training. The reward functions are passed as a list to guide the model's output.

```python
trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[
        xmlcount_reward_func,
        soft_format_reward_func,
        strict_format_reward_func,
        int_reward_func,
        correctness_reward_func,
    ],
    args=training_args,
    train_dataset=dataset,
    dataset_num_proc=1,  # Recommended on Windows
)

trainer.train()
```

--------------------------------

### Install Unsloth, TorchAO, and Executorch

Source: https://unsloth.ai/docs/basics/inference-and-deployment/deploy-llms-phone

Install the required packages for phone deployment: update Unsloth and install TorchAO and Executorch.

```bash
pip install --upgrade unsloth unsloth_zoo
pip install torchao==0.14.0 executorch pytorch_tokenizers
```

--------------------------------

### Download model with HuggingFace snapshot_download

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune

Uses snapshot_download to download the Qwen3-VL-8B-Instruct GGUF model and its mmproj file. Requires pip install huggingface_hub hf_transfer. Set HF_HUB_ENABLE_HF_TRANSFER=1 for faster downloads.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id   = "unsloth/Qwen3-VL-8B-Instruct-GGUF", # Or "unsloth/Qwen3-VL-8B-Thinking-GGUF"
    local_dir = "unsloth/Qwen3-VL-8B-Instruct-GGUF", # Or "unsloth/Qwen3-VL-8B-Thinking-GGUF"
    allow_patterns = ["*UD-Q4_K_XL*", "*mmproj-F16*"],
)
```

--------------------------------

### Example prompt for image relationship

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune

A sample prompt asking how two images are related, demonstrating the model's ability to reason about both images. The response explains the connection between the Unsloth logo and a real sloth photo.

```text
The two images are directly related because they both feature the **tree sloth**, which is the central subject of the "made with unsloth" project.

- The first image is the **official logo** for the "made with unsloth" project. It features a stylized, cartoonish tree sloth character inside a green circle, with the text "made with unsloth" next to it. This is the visual identity of the project.
- The second image is a **photograph** of a real tree sloth in its natural habitat. This photo captures the animal's physical appearance and behavior in the wild.

The relationship between the two images is that the logo (image 1) is a digital representation or symbol used to promote the "made with unsloth" project, while the photograph (image 2) is a real-world depiction of the actual tree sloth. The project likely uses the character from the logo as an icon or mascot, and the photograph serves to illustrate what the tree sloth looks like in its natural environment.
```

--------------------------------

### Install OpenClaw on Windows PowerShell

Source: https://unsloth.ai/docs/integrations/openclaw

Use the official installer script to install OpenClaw on Windows PowerShell. This sets up OpenClaw and guides you through initial setup.

```powershell
iwr -useb https://openclaw.ai/install.ps1 | iex
```

--------------------------------

### Install uv and set up Python environment

Source: https://unsloth.ai/docs/models/qwen3-coder-next

Install uv if not present, create a Python 3.12 virtual environment, and activate it.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh

# Make a new Python environment - not needed if you want to change your whole system
uv venv unsloth_fp8 --python 3.12 --seed
source unsloth_fp8/bin/activate
```

--------------------------------

### Install OpenClaw on macOS, Linux, WSL

Source: https://unsloth.ai/docs/integrations/openclaw

Use the official installer script to install OpenClaw on macOS, Linux, or WSL. This sets up OpenClaw and guides you through initial setup.

```bash
curl -fsSL https://openclaw.ai/install.sh | bash
```

--------------------------------

### Install Unsloth Studio from the main repo

Source: https://unsloth.ai/docs/get-started/install/linux

Clones the Unsloth repository, runs the local install script, and launches Unsloth Studio on host 0.0.0.0 port 8888.

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Start Unsloth Studio on port 8888

Source: https://unsloth.ai/docs/new/studio/install

Use this command to start the Unsloth app on your local 8888 port if you chose not to start it during installation.

```bash
unsloth studio -p 8888
```

--------------------------------

### Run Qwen3-Next-80B-A3B-Instruct with llama-cli

Source: https://unsloth.ai/docs/models/tutorials/qwen3-next

Directly pulls the Instruct GGUF from Hugging Face and runs it with recommended settings: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, presence_penalty 1.0.

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF:Q4_K_XL \
    --jinja -ngl 99 --ctx-size 32768 \
    --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0
```

--------------------------------

### Launch llama-server with --kv-unified for Kimi K2.5

Source: https://unsloth.ai/docs/models/tutorials/kimi-k2.5

Starts an OpenAI-compatible server for the Kimi K2.5 GGUF model. The --kv-unified flag can improve inference speed. Requires llama.cpp installed as per the Kimi K2.5 setup. The server listens on port 8001 with a context size of 16384.

```bash
LLAMA_SET_ROWS=1 ./llama.cpp/llama-server \
    --model unsloth/Kimi-K2.5-GGUF/UD-TQ1_0/Kimi-K2.5-UD-TQ1_0-00001-of-00005.gguf \
    --special \
    --alias "unsloth/Kimi-K2.5" \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --kv-unified
```

--------------------------------

### Launch Unsloth Studio

Source: https://unsloth.ai/docs/models/mtp

Starts Unsloth Studio on localhost port 8888. Open http://127.0.0.1:8888 in your browser.

```bash
unsloth studio -H 127.0.0.1 -p 8888
```

--------------------------------

### Install Unsloth with latest Torch, Triton, and dependencies

Source: https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss

Use this command to install Unsloth and its dependencies for gpt-oss fine-tuning. It installs the latest PyTorch, Triton, and related packages via uv. Note that `pip install unsloth` will not work for this setup.

```bash
# We're installing the latest Torch, Triton, OpenAI's Triton kernels, Transformers and Unsloth!
!pip install --upgrade -qqq uv
try: import numpy; install_numpy = f"numpy=={numpy.__version__}"
except: install_numpy = "numpy"
!uv pip install -qqq \
    "torch>=2.8.0" "triton>=3.4.0" {install_numpy} \
    "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \
    "unsloth[base] @ git+https://github.com/unslothai/unsloth" \
    torchvision bitsandbytes \
    git+https://github.com/huggingface/transformers \
    git+https://github.com/triton-lang/triton.git@05b2c186c1b6c9a08375389d5efe9cb4c401c075#subdirectory=python/triton_kernels
```

--------------------------------

### Run Qwen3.5-9B with llama-server for general tasks

Source: https://unsloth.ai/docs/models/qwen3.5

Starts llama-server with thinking enabled for general tasks. Uses UD-Q4_K_XL quantization. Set LLAMA_CACHE to force save location.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Run Qwen3-Coder-480B-A35B-Instruct with local model file

Source: https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally

Run the model using a locally downloaded GGUF file. Adjust --threads, --ctx-size, and --n-gpu-layers as needed.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/UD-Q2_K_XL/Qwen3-Coder-480B-A35B-Instruct-UD-Q2_K_XL-00001-of-00004.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    --min-p 0.0 \
    --top-p 0.8 \
    --top-k 20 \
    --repeat-penalty 1.05
```

--------------------------------

### Install OpenAI Python SDK

Source: https://unsloth.ai/docs/integrations/connect-python-sdk-to-unsloth

Install the official OpenAI Python SDK using pip. This is required for all OpenAI SDK examples on this page.

```bash
pip install openai
```

--------------------------------

### Install SDK components and accept licenses

Source: https://unsloth.ai/docs/basics/inference-and-deployment/deploy-llms-phone

Accept SDK licenses and install API 34, platform-tools, build-tools, and NDK 25 as required by ExecuTorch.

```bash
# Accept licenses
yes | sdkmanager --licenses

# Install API 34 and NDK 25
sdkmanager "platforms;android-34" "platform-tools" "build-tools;34.0.0" "ndk;25.0.8775105"

```

--------------------------------

### Install Codex on Linux/WSL

Source: https://unsloth.ai/docs/basics/codex

Install Codex on Linux or WSL using apt and npm. Run these commands in your terminal before starting the local model server.

```bash
apt update
sudo apt install nodejs npm -y
npm install -g @openai/codex
```

--------------------------------

### Start llama-server with Gemma 4 26B-A4B GGUF

Source: https://unsloth.ai/docs/basics/mcp

Starts llama-server with the larger unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M model, using the same server configuration as the E4B example.

```bash
llama-server \
  -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M \
  --alias local \
  --host 127.0.0.1 \
  --port 8080 \
  --no-ui \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --reasoning off
```

--------------------------------

### Launch Unsloth Studio

Source: https://unsloth.ai/docs/basics/diffusion-image

Starts the Unsloth studio server on all interfaces at port 8888. Open http://localhost:8888 if it does not open automatically.

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Clone and install Unsloth Studio (macOS/Linux/WSL)

Source: https://unsloth.ai/docs/new/studio/install

Clones the unsloth repository, runs the local install script, and launches the studio on port 8888.

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -p 8888
```

--------------------------------

### Create isolated Python environment and install Unsloth

Source: https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth

Alternative setup that creates a separate virtual environment before installing Unsloth. Note that the command may be `pip3` or `pip3.13` and `python3` or `python3.13` depending on your system.

```bash
python -m venv unsloth
source unsloth/bin/activate
pip install unsloth
```

--------------------------------

### Connect OpenCode with `unsloth start opencode`

Source: https://unsloth.ai/docs/integrations/opencode

Run this command to connect OpenCode to your loaded Unsloth model. It mints an API key, writes the config, and launches OpenCode.

```bash
unsloth start opencode
```

--------------------------------

### Install Unsloth and Train a Decision Model

Source: https://unsloth.ai/docs/basics/train-your-own-decision-model-with-unsloth

Install or update Unsloth with pip, then use FastDecisionModel and DecisionTrainer to train a decision model on the typed-decisions dataset. The example uses Qwen3.5-4B, but you can change model_name to unsloth/Llama-3.2-3B-Instruct or unsloth/gemma-4-E4B-it. It includes dataset building, splitting, and training with LoRA and 4-bit quantization.

```python
from unsloth import FastDecisionModel, DecisionTrainer, is_bfloat16_supported
from datasets import load_dataset
from transformers import TrainingArguments

model, tokenizer = FastDecisionModel.from_pretrained(
    model_name = "unsloth/Qwen3.5-4B",
    max_seq_length = 2048,
    load_in_4bit = True,
)

model = FastDecisionModel.get_peft_model(
    model,
    r = 16,
    lora_alpha = 16,
    lora_dropout = 0,
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
)

dataset = load_dataset("LocalLLaMA/typed-decisions", "all", split = "train")
items, report = FastDecisionModel.build_dataset(dataset, tokenizer, model)
print(f"Skipped {report['skipped']} of {report['total']} decisions")

train_items, eval_items = FastDecisionModel.split_holdout(items, seed = 3407)
print(f"{len(train_items)} training rows, {len(eval_items)} held out")

from transformers import TrainingArguments

trainer = DecisionTrainer(
    model = model,
    processing_class = tokenizer,
    train_dataset = train_items,
    eval_dataset = eval_items,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4, # Use GA to mimic batch size!
        warmup_steps = 10,
        # num_train_epochs = 2, # Set this for 1 full training run.
        max_steps = 240,
        learning_rate = 2e-5,
        lr_scheduler_type = "cosine",
        weight_decay = 0.01,
        bf16 = is_bfloat16_supported(),
        fp16 = not is_bfloat16_supported(),
        eval_strategy = "epoch",
        logging_steps = 1,
        output_dir = "outputs",
        report_to = "none", # Use TrackIO/WandB etc
        seed = 3407,
    ),
)
trainer_stats = trainer.train()
```

--------------------------------

### Download models (diffusion, text encoder, VAE)

Source: https://unsloth.ai/docs/models/tutorials/qwen-image-2512/stable-diffusion.cpp

Creates models and outputs directories, then downloads the Qwen-Image-2512 and Qwen-Image-Edit-2511 GGUF diffusion models, the Qwen2.5-VL-7B text encoder GGUF, and the Qwen Image VAE safetensors from Hugging Face.

```bash
cd .. 
mkdir models
mkdir outputs

## Diffusion Models
curl -L -C - -o models/qwen-image-2512-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-2512-GGUF/resolve/main/qwen-image-2512-Q4_K_M.gguf
curl -L -C - -o models/qwen-image-edit-2511-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen-Image-Edit-2511-GGUF/resolve/main/qwen-image-edit-2511-Q4_K_M.gguf
 
## Text Encoder + VAE   
curl -L -C - -o models/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf \
  https://huggingface.co/unsloth/Qwen2.5-VL-7B-Instruct-GGUF/resolve/main/Qwen2.5-VL-7B-Instruct-UD-Q4_K_XL.gguf
curl -L -C - -o models/qwen_image_vae.safetensors \
  https://huggingface.co/Comfy-Org/Qwen-Image_ComfyUI/resolve/main/split_files/vae/qwen_image_vae.safetensors
```

--------------------------------

### Pull and run an Ollama model

Source: https://unsloth.ai/docs/integrations/connections/ollama

Pulls and runs a native Ollama model (example: qwen3.6:35b-a3b). The pull command downloads the model, and the run command starts an interactive chat.

```bash
ollama pull qwen3.6:35b-a3b
ollama run qwen3.6:35b-a3b
```

--------------------------------

### Download and run Qwen3-235B-A22B-Instruct-2507 via llama-cli

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-2507

Directly downloads the Qwen3-235B-A22B-Instruct-2507 GGUF model using llama-cli with recommended sampling parameters and MoE layer offloading to CPU.

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-235B-A22B-Instruct-2507-GGUF:Q2_K_XL \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --temp 0.7 \
    --min-p 0.0 \
    --top-p 0.8 \
    --top-k 20 \
    --repeat-penalty 1.0
```

--------------------------------

### Chat template for tool calling

Source: https://unsloth.ai/docs/models/qwen3-coder-how-to-run-locally

Chat template for tool calling, showing how to request a function call and provide a tool response. The example gets the current temperature for San Francisco.

```text
<|im_start|>user
What's the temperature in San Francisco now? How about tomorrow?<|im_end|>
<|im_start|>assistant
<tool_call>\n<function=get_current_temperature>\n<parameter=location>\nSan Francisco, CA, USA
</parameter>\n</function>\n</tool_call><|im_end|>
<|im_start|>user
<tool_response>
{"temperature": 26.1, "location": "San Francisco, CA, USA", "unit": "celsius"}
</tool_response>\n<|im_end|>

```
Tokens:
Raw