### Install uv and Set Up Environment

Source: https://unsloth.ai/docs/blog/fine-tuning-llms-with-blackwell-rtx-50-series-and-unsloth

Install uv and create a new virtual environment for your project. This setup is recommended for managing complex dependencies.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh && source $HOME/.local/bin/env

mkdir 'unsloth-blackwell' && cd 'unsloth-blackwell'
uv venv .venv --python=3.12 --seed
source .venv/bin/activate
```

--------------------------------

### Install Unsloth on Windows (PowerShell)

Source: https://unsloth.ai/docs/basics/claude-code

Installs Unsloth using a PowerShell command. It automates the setup and prompts to start the Unsloth Studio.

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Run Qwen3-VL with Llama.cpp (Initial Setup)

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune

Use this command to start the llama.cpp CLI with the Qwen3-VL model, specifying various parameters for inference.

```bash
./llama.cpp/llama-mtmd-cli \
    -hf unsloth/Qwen3-VL-8B-Instruct-GGUF:UD-Q4_K_XL \
    --n-gpu-layers 99 \
    --jinja \
    --top-p 0.8 \
    --top-k 20 \
    --temp 0.7 \
    --min-p 0.0 \
    --flash-attn on \
    --presence-penalty 1.5 \
    --ctx-size 8192
```

--------------------------------

### Wandb Integration Setup

Source: https://unsloth.ai/docs/basics/finetuning-from-last-checkpoint

Install the Wandb library, log in with your token, and set environment variables for project name and model logging. Configure `TrainingArguments` to report to Wandb.

```shell
# Install library
!pip install wandb --upgrade

# Setting up Wandb
!wandb login <token>
```

```python
import os

os.environ["WANDB_PROJECT"] = "<name>"
os.environ["WANDB_LOG_MODEL"] = "checkpoint"
```

```python
report_to = "wandb",
logging_steps = 1, # Change if needed
save_steps = 100 # Change if needed
run_name = "<name>" # (Optional)
```

--------------------------------

### Various Unsloth Pip Installation Examples

Source: https://unsloth.ai/docs/get-started/install/pip-install

A collection of pip installation commands for Unsloth, covering different combinations of CUDA versions (with and without Ampere optimization) and PyTorch versions (2.3, 2.4, 2.5).

```bash
pip install "unsloth[cu121-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu118-ampere-torch240] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu121-torch240] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu118-torch240] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu121-ampere-torch230] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu121-torch250] @ git+https://github.com/unslothai/unsloth.git"
```

```bash
pip install "unsloth[cu124-ampere-torch250] @ git+https://github.com/unslothai/unsloth.git"
```

--------------------------------

### Install and Run Qwen3.6 MLX Models on MacOS

Source: https://unsloth.ai/docs/models/qwen3.6

Installs necessary scripts and activates the environment for running Qwen3.6 MLX models. This example uses the 4-bit quantized version of Qwen3.6-27B.

```bash
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/scripts/install_qwen3_6_mlx.sh | sh
source ~/.unsloth/unsloth_qwen3_6_mlx/bin/activate
python -m mlx_vlm.chat --model unsloth/Qwen3.6-27B-UD-MLX-4bit
```

--------------------------------

### Install Unsloth from Main Repo (Windows PowerShell)

Source: https://unsloth.ai/docs/get-started/install/pip-install

Developer installation from the main GitHub repository for Windows using PowerShell. Includes installing Python, uv, Git, cloning the repo, setting execution policy, running the local install script, and launching the studio.

```powershell
winget install -e --id Python.Python.3.13 --source winget
winget install --id=astral-sh.uv  -e --source winget
winget install --id Git.Git -e --source winget
git clone https://github.com/unslothai/unsloth
cd unsloth
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Serve Llama 3.3 70B Instruct with vLLM

Source: https://unsloth.ai/docs/blog/unsloth-amd-pytorch-synthetic-data-hackathon

Starts a vLLM server to load the Llama 3.3 70B Instruct model. Ensure vLLM is installed and configured correctly. This command utilizes specific ports and GPU memory utilization.

```bash
vllm serve Unsloth/Llama-3.3-70B-Instruct --port 8001 --max-model-len 48000 --gpu-memory-utilization 0.85
```

--------------------------------

### Install Nightly Build (Windows PowerShell)

Source: https://unsloth.ai/docs/get-started/install/pip-install

Installs the nightly build of Unsloth on Windows using PowerShell. Includes installing Python, uv, Git, cloning the repo, checking out the nightly branch, setting execution policy, and running the local install script.

```powershell
winget install -e --id Python.Python.3.13 --source winget
winget install --id=astral-sh.uv  -e --source winget
winget install --id Git.Git -e --source winget
git clone https://github.com/unslothai/unsloth
cd unsloth
git checkout nightly
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
.\install.ps1 --local
```

--------------------------------

### SGLang Installation Debug Hint

Source: https://unsloth.ai/docs/basics/inference-and-deployment/sglang-guide

A common installation error message indicating a potential issue with the 'outlines-core' package or the build environment. Update Rust and outlines-core as per the setup guide.

```shellscript
hint: This usually indicates a problem with the package or the build environment.
  help: `outlines-core` (v0.1.26) was included because `sglang` (v0.5.5.post2) depends on `outlines` (v0.1.11) which depends on `outlines-core`
```

--------------------------------

### Start llama-server with Qwen3.5 and recommended settings

Source: https://unsloth.ai/docs/basics/claude-code

Launches the llama-server to deploy the Qwen3.5 model, using recommended sampling parameters for thinking mode and KV cache quantization for reduced VRAM usage. Adjust --ctx-size if performance issues arise.

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --alias "unsloth/Qwen3.5-35B-A3B" \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001 \
    --kv-unified \
    --cache-type-k q8_0 --cache-type-v q8_0
```

--------------------------------

### Start Claude Code with a Different Model

Source: https://unsloth.ai/docs/basics/claude-code

Example of starting Claude Code with a different model, 'unsloth/Qwen3.6-35B-A3B'.

```bash
claude --model unsloth/Qwen3.6-35B-A3B
```

--------------------------------

### Install Claude Code on macOS, Linux, WSL

Source: https://unsloth.ai/docs/basics/claude-code

Use this command to install Claude Code via curl. After installation, navigate to your project directory and run 'claude' to start.

```bash
curl -fsSL https://claude.ai/install.sh | bash
```

```bash
cd ~/projects/my-project 
claude
```

--------------------------------

### Start Unsloth Studio

Source: https://unsloth.ai/docs/basics/codex

Start the Unsloth studio application on a specified port. Add '-H 0.0.0.0' to make it accessible externally.

```bash
unsloth studio -p 8888
```

--------------------------------

### Setup Virtual Environment and Install Unsloth

Source: https://unsloth.ai/docs/get-started/install/pip-install

Installs necessary venv packages for Python 3.10-3.13, creates and activates a virtual environment named 'unsloth_env', upgrades pip, installs uv, and finally installs Unsloth with the auto torch backend. This method ensures isolated installations.

```bash
apt install python3.10-venv python3.11-venv python3.12-venv python3.13-venv -y
python -m venv unsloth_env
source unsloth_env/bin/activate
pip install --upgrade pip && pip install uv
uv pip install unsloth --torch-backend=auto
```

--------------------------------

### Unsloth Studio API: User Authentication - Signup

Source: https://unsloth.ai/docs/new/studio/start

Creates a new user account. Requires a setup token on the first run. Authentication is not required for this endpoint.

```http
POST /api/auth/signup
```

--------------------------------

### Start Unsloth Studio

Source: https://unsloth.ai/docs/basics/claude-code

Command to launch the Unsloth studio on a specified port. Use -H 0.0.0.0 to make it accessible externally.

```bash
unsloth studio -p 8888
```

```bash
unsloth studio -p 8888 -H 0.0.0.0
```

--------------------------------

### Install Unsloth on MacOS

Source: https://unsloth.ai/docs/basics/claude-code

Installs Unsloth using a curl command. It sets up the environment and required packages, then prompts to start the Unsloth Studio on port 8888.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

--------------------------------

### Install Unsloth from Main Repo (macOS, Linux, WSL)

Source: https://unsloth.ai/docs/get-started/install/pip-install

Developer installation from the main GitHub repository for macOS, Linux, and WSL. Includes cloning the repo, running the local install script, and launching the studio.

```bash
git clone https://github.com/unslothai/unsloth
cd unsloth
./install.sh --local
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Install OpenClaw on macOS, Linux, or WSL

Source: https://unsloth.ai/docs/integrations/openclaw

Use this command to install OpenClaw on macOS, Linux, or Windows Subsystem for Linux. It sets up the application and guides through initial configuration.

```bash
curl -fsSL https://openclaw.ai/install.sh | bash
```

--------------------------------

### Run Kimi-K2-Instruct with llama.cpp (Local Path)

Source: https://unsloth.ai/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally

Executes the Kimi-K2-Instruct model from a local GGUF file path using llama.cpp. Configurable parameters include GPU layers, temperature, context size, and MoE CPU offloading.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Kimi-K2-Instruct-GGUF/UD-TQ1_0/Kimi-K2-Instruct-UD-TQ1_0-00001-of-00005.gguf \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

--------------------------------

### Install Unsloth and Load Magistral Model

Source: https://unsloth.ai/docs/models/tutorials/magistral-how-to-run-and-fine-tune

Installs the Unsloth library and loads the Magistral-Small-2509 model with 4-bit quantization for efficient fine-tuning. This setup is optimized for multi-GPU environments like dual T4s.

```python
!pip install --upgrade unsloth
from unsloth import FastLanguageModel
import torch
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Magistral-Small-2509-unsloth-bnb-4bit",
    max_seq_length = 2048,   # Context length - can be longer, but uses more memory
    load_in_4bit = True,     # 4bit uses much less memory
    load_in_8bit = False,    # A bit more accurate, uses 2x memory
    full_finetuning = False, # We have full finetuning now!
    device_map = "balanced", # Uses 2x Telsa T4s
    # token = "hf_...",      # use one if using gated models
)
```

--------------------------------

### Run Qwen3-30B-A3B-Instruct-2507 with Llama.cpp

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-2507

Executes the Qwen3 model using the Llama.cpp CLI with specified parameters for temperature, top-p, and context size. Ensure Llama.cpp is built.

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_XL \
    --jinja -ngl 99 --ctx-size 32768 \
    --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0
```

--------------------------------

### Install Unsloth and Dependencies Locally

Source: https://unsloth.ai/docs/models/gpt-oss-how-to-run-and-fine-tune/tutorial-how-to-fine-tune-gpt-oss

Installs the latest PyTorch, Triton, Transformers, and Unsloth packages using uv for local fine-tuning setups. Ensure your system meets Unsloth compatibility requirements before running.

```python
# We're installing the latest Torch, Triton, OpenAI's Triton kernels, Transformers and Unsloth!
!pip install --upgrade -qqq uv
try: import numpy; install_numpy = f"numpy=={numpy.__version__}"
except: install_numpy = "numpy"
!uv pip install -qqq \
    "torch>=2.8.0" "triton>=3.4.0" {install_numpy} \
    "unsloth_zoo[base] @ git+https://github.com/unslothai/unsloth-zoo" \
    "unsloth[base] @ git+https://github.com/unslothai/unsloth" \
    torchvision bitsandbytes \
    git+https://github.com/huggingface/transformers \
    git+https://github.com/triton-lang/triton.git@05b2c186c1b6c9a08375389d5efe9cb4c401c075#subdirectory=python/triton_kernels
```

--------------------------------

### Run Kimi-K2-Instruct with llama.cpp (Hugging Face)

Source: https://unsloth.ai/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally

Executes the Kimi-K2-Instruct model using llama.cpp, leveraging Hugging Face for model loading. Includes options for GPU offloading and MoE layer CPU offloading.

```bash
export LLAMA_CACHE="unsloth/Kimi-K2-Instruct-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Kimi-K2-Instruct-GGUF:TQ1_0 \
    --n-gpu-layers 99 \
    --temp 0.6 \
    --min-p 0.01 \
    --ctx-size 16384 \
    --seed 3407 \
    -ot ".ffn_.*_exps.=CPU"
```

--------------------------------

### Start LM Studio Local Server

Source: https://unsloth.ai/docs/basics/inference-and-deployment/lm-studio

Initiate the local OpenAI-compatible API server on a specified port.

```bash
lms server start --port 1234
```

--------------------------------

### Interact with DeepSeek V3.1 via OpenAI Python Library

Source: https://unsloth.ai/docs/models/tutorials/deepseek-v3.1-how-to-run-locally

After starting the llama-server, use this Python script to send chat completion requests to the model. Ensure the OpenAI library is installed (`pip install openai`).

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/DeepSeek-V3.1-Terminus",
    messages = [{"role": "user", "content": "What is 2+2?"}],
)
print(completion.choices[0].message.content)
```

--------------------------------

### Install uv and Set Up Python Environment

Source: https://unsloth.ai/docs/models/qwen3-coder-next

Installs uv for faster environment management and creates a new Python 3.12 virtual environment named 'unsloth_fp8'. Activate this environment before proceeding.

```bash
# Install uv if you don't have it for faster environment installs
curl -LsSf https://astral.sh/uv/install.sh | sh

# Make a new Python environment - not needed if you want to change your whole system
uv venv unsloth_fp8 --python 3.12 --seed
source unsloth_fp8/bin/activate

uv pip install --upgrade --force-reinstall vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly/cu130
uv pip install --upgrade --force-reinstall git+https://github.com/huggingface/transformers.git
uv pip install --force-reinstall numba
```

--------------------------------

### Run GLM-4.6 with Ollama

Source: https://unsloth.ai/docs/models/tutorials/glm-4.6-how-to-run-locally

Starts the Ollama server and runs the GLM-4.6 model. Use this to interact with the model after installation.

```bash
OLLAMA_MODELS=unsloth ollama serve &

OLLAMA_MODELS=unsloth ollama run hf.co/unsloth/GLM-4.6-GGUF:TQ1_0
```

--------------------------------

### Start Unsloth Studio Manually

Source: https://unsloth.ai/docs/basics/claude-code

Starts the Unsloth Studio application on a specified port. Use -H 0.0.0.0 to make the instance accessible externally.

```bash
unsloth studio -p 8888
```

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Prompt for Unsloth Installation

Source: https://unsloth.ai/docs/basics/claude-code

This prompt guides Claude Code to install Unsloth within a virtual environment using uv, and then perform a finetuning run. It specifies constraints like working only in the current directory and having access to one GPU.

```text
You can only work in the cwd project/. Do not search for CLAUDE.md - this is it. Install Unsloth via a virtual environment via uv. Use `python -m venv unsloth_env` then `source unsloth_env/bin/activate` if possible. See https://unsloth.ai/docs/get-started/install/pip-install on how (get it and read). Then do a simple Unsloth finetuning run described in https://github.com/unslothai/unsloth. You have access to 1 GPU.
```

--------------------------------

### Query Unsloth Documentation

Source: https://unsloth.ai/docs/get-started/install

Example of how to query the Unsloth documentation dynamically using a GET request with an 'ask' query parameter.

```http
GET https://unsloth.ai/docs/get-started/install.md?ask=<question>
```

--------------------------------

### NotImplementedError Example (Out of Memory)

Source: https://unsloth.ai/docs/blog/unsloth-amd-pytorch-synthetic-data-hackathon

This error indicates that the GPU is out of memory. Refer to the guide on freeing AMD GPU memory to resolve this issue.

```text
NotImplementedError: Cannot copy out of meta tensor; no data!
```

--------------------------------

### Full code example: llama-server and mcp-cli

Source: https://unsloth.ai/docs/basics/mcp

Demonstrates running llama-server and mcp-cli in separate terminals for an integrated experience.

```bash
# terminal 1
llama-server -hf unsloth/gemma-4-E4B-it-GGUF:UD-Q4_K_XL \
  --alias local --host 127.0.0.1 --port 8080 --no-ui \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --chat-template-kwargs '{"enable_thinking":false}'

# terminal 2, in the folder with server_config.json
uvx mcp-cli \
  --provider llamacpp \
  --api-base http://127.0.0.1:8080/v1 \
  --api-key none \
  --model local \
  --server filesystem \
  --config-file server_config.json
```

--------------------------------

### Launch llama-server for Kimi-K2-Thinking

Source: https://unsloth.ai/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally

Use this command to start an OpenAI-compatible server with the Kimi-K2-Thinking model. Ensure llama.cpp is installed and the model path is correct.

```bash
./llama.cpp/llama-server \
    --model unsloth/Kimi-K2-Thinking-GGUF/UD-TQ1_0/Kimi-K2-Thinking-UD-TQ1_0-00001-of-00006.gguf \
    --alias "unsloth/Kimi-K2-Thinking" \
    -fa on \
    --n-gpu-layers 999 \
    -ot ".ffn_.*=CPU" \
    --min-p 0.01 \
    --ctx-size 16384 \
    --port 8001 \
    --jinja
```

--------------------------------

### Install Unsloth from Source

Source: https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth/ddp

Clone the Unsloth repository from GitHub and install it using pip. It is recommended to use a virtual environment for this process.

```bash
git clone https://github.com/unslothai/unsloth.git
cd unsloth
pip install .
```

--------------------------------

### SGLang Interactive Offline Mode

Source: https://unsloth.ai/docs/basics/inference-and-deployment/sglang-guide

Runs a model in an interactive Python environment without starting a server. Requires the sglang library to be installed.

```python
import sglang as sgl
engine = sgl.Engine(model_path = "unsloth/Qwen3-0.6B", random_seed = 42)

prompt = "Today is a sunny day and I like"
sampling_params = {"temperature": 0, "max_new_tokens": 256}
outputs = engine.generate(prompt, sampling_params)["text"]
print(outputs)
engine.shutdown()
```

--------------------------------

### Qwen3-VL Instruct Settings for llama.cpp

Source: https://unsloth.ai/docs/models/tutorials/qwen3-how-to-run-and-fine-tune/qwen3-vl-how-to-run-and-fine-tune

Recommended environment variables for running Qwen3-VL Instruct models in llama.cpp.

```bash
export greedy='false'
export seed=3407
export top_p=0.8
export top_k=20
export temperature=0.7
export repetition_penalty=1.0
export presence_penalty=1.5
export out_seq_length=32768
```

--------------------------------

### Install Git on Linux/WSL

Source: https://unsloth.ai/docs/get-started/fine-tuning-for-beginners/unsloth-requirements

Installs Git on Linux or WSL using apt. This is a prerequisite for using Unsloth Studio on these systems.

```bash
sudo apt install git
```

--------------------------------

### Run llama-server with Preserve Thinking enabled

Source: https://unsloth.ai/docs/models/qwen3.6

Example command to start the llama-server with Qwen3.6-35B-A3B GGUF model and 'preserve_thinking' enabled. Adjust model path and port as needed.

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-BF16.gguf \
    --alias "unsloth/Qwen3.6-35B-A3B-GGUF" \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001 \
    --chat-template-kwargs '{"preserve_thinking":true}'
```

--------------------------------

### DPO Training Setup with Unsloth

Source: https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide/preference-dpo-orpo-and-kto

This snippet shows how to initialize a model for DPO training using Unsloth's optimized components. It includes model loading, PEFT configuration, and setting up the `DPOTrainer` with `DPOConfig`.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0" # Optional set GPU device ID

from unsloth import FastLanguageModel, PatchDPOTrainer
from unsloth import is_bfloat16_supported
PatchDPOTrainer()
import torch
from trl import DPOTrainer, DPOConfig  # Changed from TrainingArguments

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/zephyr-sft-bnb-4bit",
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = True,
)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = 64,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 64,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    max_seq_length = max_seq_length,
)

dpo_trainer = DPOTrainer(
    model = model,
    ref_model = None,
    args = DPOConfig( # Use DPOConfig
        per_device_train_batch_size = 4,
        gradient_accumulation_steps = 8,
        warmup_ratio = 0.1,
        num_train_epochs = 3,
        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        seed = 42,
        output_dir = "outputs",
    ),
    beta = 0.1,
    train_dataset = YOUR_DATASET_HERE,
    # eval_dataset = YOUR_DATASET_HERE,
    tokenizer = tokenizer,
    max_length = 1024,
    max_prompt_length = 512,
)

dpo_trainer.train()

```

--------------------------------

### Launch Unsloth Studio

Source: https://unsloth.ai/docs/models/mtp

Launches the Unsloth Studio application with specified host and port. Access the studio via the provided URL in your browser.

```bash
unsloth studio -H 127.0.0.1 -p 8888
```

--------------------------------

### Run GPT-OSS Model with Docker

Source: https://unsloth.ai/docs/models/tutorials/how-to-run-llms-with-docker

Use this command to quickly run the OpenAI GPT-OSS model via Docker. No prior setup is required beyond having Docker installed.

```bash
docker model run ai/gpt-oss:20B
```

--------------------------------

### Run Qwen3-Next-80B-A3B-Instruct with llama.cpp

Source: https://unsloth.ai/docs/models/tutorials/qwen3-next

Executes the Qwen3-Next-80B-A3B-Instruct model using llama.cpp, pulling directly from Hugging Face. Adjust context size and other parameters as needed.

```bash
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF:Q4_K_XL \
    --jinja -ngl 99 --ctx-size 32768 \
    --temp 0.7 --min-p 0.0 --top-p 0.80 --top-k 20 --presence-penalty 1.0
```

--------------------------------

### Start Unsloth Server with Custom Defaults

Source: https://unsloth.ai/docs/integrations/connect-python-sdk-to-unsloth

Use this command to start the Unsloth server with specific defaults for reasoning and temperature, and to set the port. This is useful for controlling model behavior.

```bash
unsloth run \
  --model unsloth/Qwen3-1.7B-GGUF \
  --reasoning off \
  --temp 0.6 \
  -p 8888
```

--------------------------------

### RuntimeError Example (bitsandbytes ROCm Binary Not Found)

Source: https://unsloth.ai/docs/blog/unsloth-amd-pytorch-synthetic-data-hackathon

This error signifies that the bitsandbytes ROCm binary was not found. Ensure bitsandbytes is installed correctly by following the Unsloth update instructions.

```text
RuntimeError: Configured ROCm binary not found at /usr/local/lib/python3.12/dist-packages/bitsandbytes/libbitsandbytes_rocm64.so
```

--------------------------------

### Interact with Deployed Qwen3.6 via OpenAI Client

Source: https://unsloth.ai/docs/models/qwen3.6

Connect to the locally deployed llama-server using the OpenAI client library to get model completions. Requires `pip install openai`.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.6-35B-A3B",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

--------------------------------

### Launch Unsloth Studio on MacOS/Linux/WSL

Source: https://unsloth.ai/docs/models/ibm-granite-4.1

Activate the Unsloth Studio environment and launch the studio on MacOS, Linux, or WSL.

```bash
source unsloth_studio/bin/activate
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Configure Hugging Face Hub Download Environment Variables

Source: https://unsloth.ai/docs/basics/troubleshooting-and-faqs/hugging-face-hub-xet-debugging

Use these environment variables to optimize Hugging Face Hub downloads, especially when they get stuck. Ensure `huggingface_hub` is installed.

```bash
pip install -U huggingface_hub

HF_HOME=".cache_new/huggingface" \
HF_XET_CACHE=".cache_new/huggingface/xet" \
HF_HUB_CACHE=".cache_new/huggingface/hub" \
HF_XET_HIGH_PERFORMANCE=1 \
HF_XET_CHUNK_CACHE_SIZE_BYTES=0 \
HF_XET_RECONSTRUCT_WRITE_SEQUENTIALLY=0 \
HF_XET_NUM_CONCURRENT_RANGE_GETS=64 \
hf download unsloth/Qwen3-Coder-Next-GGUF \
    --local-dir unsloth/Qwen3-Coder-Next-GGUF \
    --include "*UD-Q6_K_XL*"
```

--------------------------------

### Create Android SDK Directory

Source: https://unsloth.ai/docs/basics/inference-and-deployment/deploy-llms-phone

Set up the directory structure for the Android SDK command-line tools. This is the first step in manually installing the SDK.

```bash
mkdir -p ~/android-sdk/cmdline-tools
cd ~/android-sdk
```

--------------------------------

### Interact with Kimi-K2-Thinking using OpenAI Python library

Source: https://unsloth.ai/docs/models/tutorials/kimi-k2-thinking-how-to-run-locally

After starting the llama-server, use this Python code to send a chat completion request to the local server. Make sure to install the openai library.

```python
from openai import OpenAI
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Kimi-K2-Thinking",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
```

--------------------------------

### Run Qwen3.5-35B-A3B with llama.cpp Server (Reasoning Tasks)

Source: https://unsloth.ai/docs/models/qwen3.5

Starts llama.cpp server for reasoning tasks without 'thinking' mode enabled. Sets the cache directory, model quantization, and inference parameters, including chat template configuration.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Download DeepSeek-V3 GGUF Model

Source: https://unsloth.ai/docs/models/tutorials/deepseek-v3-0324-how-to-run-locally

Downloads a specific quantized version of the DeepSeek-V3 model using huggingface_hub. Ensure you have `huggingface_hub` and `hf_transfer` installed. This example downloads the UD-Q2_K_XL dynamic 2.7bit quant.

```python
# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/DeepSeek-V3-0324-GGUF-UD",
    local_dir = "unsloth/DeepSeek-V3-0324-GGUF-UD",
    allow_patterns = ["*UD-Q2_K_XL*"] # Dynamic 2.7bit (230GB) Use "*UD-IQ_S*" for Dynamic 1.78bit (151GB)
)
```

--------------------------------

### Run Deepseek V3 with Unsloth (Full Example)

Source: https://unsloth.ai/docs/models/tutorials/deepseek-v3-0324-how-to-run-locally

A complete example demonstrating how to load, prepare, and use the Deepseek V3 model for text generation.

```python
from unsloth import FastLanguageModel
import torch
from transformers import TextGenerationPipeline

# Load model
model, tokenizer = FastLanguageModel.from_pretrained(
    model="unsloth/deepseek-coder-v3-1-instruct",
    torch_dtype=torch.bfloat16, # or torch.float16
    load_in_4bit=True, # or load_in_8bit=True
)

# Prepare model for inference
model = FastLanguageModel.get_peft_model(model, {
    "lora_alpha": 16,
    "lora_dropout": 0.1,
    "r": 64,
    "bias": "none",
    "task_type": "CAUSAL_LM",
    "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
})

# Create text generation pipeline
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer, torch_dtype=torch.bfloat16, device_map="auto")

# Define prompt
pcs = "Write a short story about a robot learning to paint."

# Generate text
text = pipeline(npcs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(text)

```

--------------------------------

### Python OpenAI client for llama-server

Source: https://unsloth.ai/docs/models/qwen3.6

Example Python code using the OpenAI client to interact with a locally running llama-server. Assumes the server is started with the correct base URL and model alias.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.6-35B-A3B-GGUF",
    messages = [{"role": "user", "content": "What is 2+2?"}],
)
print(completion.choices[0].message.content)
print(completion.choices[0].message.reasoning_content)
```