### Enable Thinking for Qwen3.5-9B with llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

Example command to start llama-server for Qwen3.5-9B, explicitly enabling the thinking feature. This is necessary as reasoning is disabled by default for Qwen3.5 models.

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3.5-9B-GGUF/Qwen3.5-9B-BF16.gguf \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --temp 0.6 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Run Qwen3.5-9B for General Tasks via llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

Start the llama.cpp server for general tasks with the Qwen3.5-9B model. Use UD-Q4_K_XL quantization and a context size of 16384.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Unsloth SFT Setup

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Initializes FastLanguageModel, torch, and loads a dataset for SFT training. Sets the maximum sequence length.

```python
from unsloth import FastLanguageModel
import torch
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

max_seq_length = 2048  # start small; scale up after it works

```

--------------------------------

### Run Qwen3.5-122B-A10B in Non-Thinking Mode (General Server)

Source: https://unsloth.ai/docs/models/qwen3.5

Starts the Qwen3.5-122B-A10B model as a server for general tasks without thinking mode. Sets LLAMA_CACHE and configures chat template.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-122B-A10B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Install Unsloth (PowerShell)

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Installs Unsloth on Windows using a PowerShell command.

```powershell
irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Query Documentation via HTTP GET

Source: https://unsloth.ai/docs/models/qwen3.5

Perform an HTTP GET request to the current page URL with the `ask` query parameter to dynamically query the documentation. Use this when information is not explicitly present or for clarification.

```http
GET https://unsloth.ai/docs/models/qwen3.5.md?ask=<question>
```

--------------------------------

### Run Qwen3.5-122B-A10B in Non-Thinking Mode (Reasoning Server)

Source: https://unsloth.ai/docs/models/qwen3.5

Starts the Qwen3.5-122B-A10B model as a server for reasoning tasks without thinking mode. Sets LLAMA_CACHE and configures chat template.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-122B-A10B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Run Qwen3.5-9B with Thinking Mode Enabled via llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

Start the llama.cpp server with the Qwen3.5-9B model, enabling thinking mode. Use UD-Q4_K_XL quantization and set a context size of 16384.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Install Unsloth (Bash)

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Installs Unsloth on MacOS, Linux, or WSL using a curl command.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

--------------------------------

### Build llama.cpp with CUDA Support

Source: https://unsloth.ai/docs/models/qwen3.5

Install dependencies and build llama.cpp with CUDA support. Set GGML_CUDA to OFF for CPU-only inference.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

--------------------------------

### Run llama.cpp Server for General Tasks (Non-Thinking Mode)

Source: https://unsloth.ai/docs/models/qwen3.5

Starts the llama.cpp server for general tasks without thinking mode. Ensure LLAMA_CACHE is set to the model's Hugging Face repository. The chat-template-kwargs argument disables thinking.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Build llama.cpp

Source: https://unsloth.ai/docs/models/qwen3.5

Installs dependencies and builds the llama.cpp project. Set -DGGML_CUDA=OFF if you don't have a GPU. Metal support is enabled by default on Apple Mac devices.

```bash
apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-mtmd-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp
```

--------------------------------

### Run llama.cpp Server for Reasoning Tasks (Non-Thinking Mode)

Source: https://unsloth.ai/docs/models/qwen3.5

Starts the llama.cpp server for reasoning tasks without thinking mode. Ensure LLAMA_CACHE is set to the model's Hugging Face repository. The chat-template-kwargs argument disables thinking.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Get Unsloth Model YAML for LM Studio

Source: https://unsloth.ai/docs/models/qwen3.5

Use this command to fetch a YAML configuration file for Unsloth models, enabling features like the 'Thinking' toggle in LM Studio. Replace '4b' with your desired model quantization.

```bash
lms get unsloth/qwen3.5-4b
```

--------------------------------

### Configure PEFT Model for Vision Fine-tuning

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Attach LoRA adapters to a vision model using `FastVisionModel.get_peft_model`. This example shows how to selectively fine-tune vision, language, attention, and MLP layers.

```python
model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = True, # False if not finetuning vision layers
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = 16,                           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 16,                  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,               # We support rank stabilized LoRA
    loftq_config = None,               # And LoftQ
    target_modules = "all-linear",    # Optional now! Can specify a list if needed
    modules_to_save=[
        "lm_head",
        "embed_tokens",
    ],
)
```

--------------------------------

### Interact with Deployed Model using OpenAI Library

Source: https://unsloth.ai/docs/models/qwen3.5

This Python script demonstrates how to use the OpenAI client library to send a chat completion request to a locally running llama-server instance. Ensure the `openai` library is installed and the server is running on the specified address and port.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-397B-A17B",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

--------------------------------

### Run Qwen3.5-397B-A17B in Non-Thinking Mode with llama.cpp

Source: https://unsloth.ai/docs/models/qwen3.5

Starts a llama.cpp server to run the Qwen3.5-397B-A17B model in non-thinking mode. Configures model path, quantization, and chat template settings. Use --chat-template-kwargs '{"enable_thinking":false}' to disable thinking.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Launch Unsloth Studio

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Launches the Unsloth Studio web UI, accessible via http://localhost:8888.

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Initialize SFTTrainer for Qwen3.5 Fine-tuning

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Set up the `SFTTrainer` with the PEFT-configured model, dataset, tokenizer, and training arguments. Key arguments include `max_seq_length`, `per_device_train_batch_size`, `gradient_accumulation_steps`, and `output_dir`.

```python
trainer = SFTTrainer(
    model = model,
    train_dataset = dataset,
    tokenizer = tokenizer,
    args = SFTConfig(
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 100,
        logging_steps = 1,
        output_dir = "outputs_qwen35",
        optim = "adamw_8bit",
        seed = 3407,
        dataset_num_proc = 1,
    ),
)

trainer.train()
```

--------------------------------

### Load Dataset and Model for Qwen3.5 Fine-tuning

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Load a dataset from a URL and initialize the Qwen3.5 model with specified sequence length and LoRA configurations. Use `load_in_16bit=True` for bf16/16-bit LoRA.

```python
url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"
dataset = load_dataset("json", data_files={"train": url}, split="train")

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen3.5-27B",
    max_seq_length = max_seq_length,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 27B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)
```

--------------------------------

### Run Qwen3.5-9B for General Tasks via llama-cli

Source: https://unsloth.ai/docs/models/qwen3.5

Execute the llama.cpp CLI for general tasks with the Qwen3.5-9B model. Use UD-Q4_K_XL quantization and a context size of 16384.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-27B in Thinking Mode (General Tasks)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-27B model using llama-cli for general tasks. Sets the cache directory and specifies model parameters like context size and temperature.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-397B-A17B with Specific Model and MMProj Files

Source: https://unsloth.ai/docs/models/qwen3.5

Infers using the Qwen3.5-397B-A17B model with specified GGUF and MMProj files. Allows customization of threads, context size, and GPU offloading layers. Adjust --n-gpu-layers based on VRAM availability.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
    --mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-9B for Reasoning Tasks via llama-cli

Source: https://unsloth.ai/docs/models/qwen3.5

Execute the llama.cpp CLI for reasoning tasks with the Qwen3.5-9B model. Use UD-Q4_K_XL quantization and a context size of 16384.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-27B in Thinking Mode (Precise Coding)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-27B model using llama-cli for precise coding tasks. Sets the cache directory and specifies model parameters like context size and temperature.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-27B in Conversation Mode

Source: https://unsloth.ai/docs/models/qwen3.5

Launches the Qwen3.5-27B model in conversation mode using llama-cli, specifying the model and multimodal projector files, along with inference parameters.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-27B-GGUF/Qwen3.5-27B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-27B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

--------------------------------

### Run Qwen3.5-35B-A3B Model in Conversation Mode

Source: https://unsloth.ai/docs/models/qwen3.5

Launches the llama.cpp CLI to run the model in conversation mode, specifying the model and mmproj file paths. Adjust temperature, top-p, min-p, and top-k as needed.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

--------------------------------

### Download Qwen3.5-27B Model Files

Source: https://unsloth.ai/docs/models/qwen3.5

Downloads the specified Qwen3.5-27B model files using the huggingface-cli. It includes instructions for downloading specific quantization types like UD-Q4_K_XL or UD-Q2_K_XL.

```bash
hf download unsloth/Qwen3.5-27B-GGUF \
    --local-dir unsloth/Qwen3.5-27B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Run Qwen3.5-27B in Non-Thinking Mode (General Tasks)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-27B model using llama-server for general tasks without thinking mode. Sets the cache directory and specifies model parameters.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Download Qwen3.5-35B-A3B Model GGUF Files

Source: https://unsloth.ai/docs/models/qwen3.5

Downloads specific GGUF model files using huggingface_hub. Use '*UD-Q2_K_XL*' for Dynamic 2-bit quantization. Refer to the provided link for troubleshooting download issues.

```bash
hf download unsloth/Qwen3.5-35B-A3B-GGUF \
    --local-dir unsloth/Qwen3.5-35B-A3B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Deploy Qwen3.5 Model with Llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

This command deploys the specified Qwen3.5 GGUF model using llama-server. Ensure you have llama.cpp compiled and adjust parameters like model path, mmproj, alias, temperature, context size, and port as needed for your production environment.

```bash
./llama.cpp/llama-server \
--model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Qwen3.5-35B-A3B" \
    --temp 0.6 \
    --top-p 0.95 \
    --ctx-size 16384 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001
```

--------------------------------

### Install/Upgrade Unsloth

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Force reinstall Unsloth and unsloth_zoo to ensure the latest versions are used, especially when fine-tuning locally.

```bash
pip install --upgrade --force-reinstall --no-cache-dir unsloth unsloth_zoo
```

--------------------------------

### Run Qwen3.5-27B in Non-Thinking Mode (Reasoning Tasks)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-27B model using llama-server for reasoning tasks without thinking mode. Sets the cache directory and specifies model parameters.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Run Qwen3.5-9B in Conversation Mode

Source: https://unsloth.ai/docs/models/qwen3.5

Run the Qwen3.5-9B model in conversation mode using llama.cpp CLI. Specify the model path, mmproj path, and various generation parameters.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-9B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

--------------------------------

### Python Client for Qwen3.5 with Reasoning

Source: https://unsloth.ai/docs/models/qwen3.5

Python code using the OpenAI client to interact with a locally running llama-server. It demonstrates how to send a message and print both the standard content and the reasoning content from the response.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-9B-GGUF",
    messages = [{"role": "user", "content": "What is 2+2?"}],
)
print(completion.choices[0].message.content)
print(completion.choices[0].message.reasoning_content)
```

--------------------------------

### Instruct Mode for General Tasks

Source: https://unsloth.ai/docs/models/qwen3.5

Settings for general tasks in instruct (non-thinking) mode. Adjust temperature and top_p for desired output.

```bash
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

--------------------------------

### Enabling Thinking/Reasoning for Qwen3.5

Source: https://unsloth.ai/docs/models/qwen3.5

Enable thinking and reasoning for specific Qwen3.5 models by setting 'enable_thinking' to true. This is necessary for models where reasoning is disabled by default.

```bash
--chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Load Qwen3.5 for RL training

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Load the Qwen3.5 model for Reinforcement Learning (RL) training, including RLHF, GSPO, and GRPO. Set `fast_inference=False` for compatibility.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3.5-4B",
    fast_inference=False,
)
```

--------------------------------

### Download Qwen3.5-9B GGUF Model

Source: https://unsloth.ai/docs/models/qwen3.5

Download the Qwen3.5-9B GGUF model using the huggingface-cli tool. Includes options to include specific quantization types like UD-Q4_K_XL or UD-Q2_K_XL.

```bash
hf download unsloth/Qwen3.5-9B-GGUF \
    --local-dir unsloth/Qwen3.5-9B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Run Qwen3.5-397B-A17B in Thinking Mode with llama.cpp

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-397B-A17B model using llama-cli in thinking mode. Sets the LLAMA_CACHE environment variable and specifies the model and quantization type. Adjust --ctx-size for context length.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-122B-A10B in Thinking Mode (WebDev)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-122B-A10B model in thinking mode for precise coding tasks. Sets the LLAMA_CACHE environment variable and specifies model parameters.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-122B-A10B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-122B-A10B in Thinking Mode (General)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the Qwen3.5-122B-A10B model in thinking mode for general tasks. Sets the LLAMA_CACHE environment variable and specifies model parameters.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-122B-A10B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-122B-A10B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-122B-A10B in Conversation Mode

Source: https://unsloth.ai/docs/models/qwen3.5

Runs the Qwen3.5-122B-A10B model in conversation mode using a specific GGUF file and mmproj. Configures context size and generation parameters.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-122B-A10B-GGUF/UD-Q4_K_XL/Qwen3.5-122B-A10B-UD-Q4_K_XL-00001-of-00003.gguf \
    --mmproj unsloth/Qwen3.5-122B-A10B-GGUF/mmproj-F16.gguf \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Download Qwen3.5-397B-A17B Model Quantizations

Source: https://unsloth.ai/docs/models/qwen3.5

Downloads specific quantized versions of the Qwen3.5-397B-A17B model using the hf command-line tool. Use --include to specify desired quantization types like UD-Q4_K_XL or UD-Q2_K_XL.

```bash
hf download unsloth/Qwen3.5-397B-A17B-GGUF \
    --local-dir unsloth/Qwen3.5-397B-A17B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Configure PEFT Model for Qwen3.5 Fine-tuning

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Attach LoRA adapters to the loaded Qwen3.5 model using `FastLanguageModel.get_peft_model`. Configure LoRA parameters like rank (r), alpha, dropout, and gradient checkpointing.

```python
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    # "unsloth" checkpointing is intended for very long context + lower VRAM
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    max_seq_length = max_seq_length,
)
```

--------------------------------

### Save Qwen3.5 model for vLLM (16-bit)

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Save the fine-tuned Qwen3.5 model in 16-bit format for use with vLLM. This method is recommended for vLLM compatibility, especially when using newer versions or nightly releases.

```python
model.save_pretrained_merged("finetuned_model", tokenizer, save_method = "merged_16bit")
```

```python
model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "")
```

--------------------------------

### Thinking Mode for General Tasks

Source: https://unsloth.ai/docs/models/qwen3.5

Use these settings for general tasks in thinking mode. Ensure parameters like temperature and presence_penalty are set appropriately.

```bash
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

--------------------------------

### Update Unsloth Studio

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Use this command to update Unsloth Studio to the latest version.

```bash
unsloth studio update
```

--------------------------------

### Run llama.cpp CLI for General Tasks (Thinking Mode)

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the llama.cpp command-line interface for general tasks in thinking mode. Ensure LLAMA_CACHE is set to the model's Hugging Face repository.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Define Tools for Qwen3.5

Source: https://unsloth.ai/docs/models/qwen3.5

Defines a set of utility functions and their corresponding tool specifications for use with Qwen3.5. Includes functions for arithmetic, story generation, terminal commands, and Python code execution. Ensure these functions are available in your environment.

```python
import json, subprocess, random
from typing import Any
def add_number(a: float | str, b: float | str) -> float:
    return float(a) + float(b)
def multiply_number(a: float | str, b: float | str) -> float:
    return float(a) * float(b)
def substract_number(a: float | str, b: float | str) -> float:
    return float(a) - float(b)
def write_a_story() -> str:
    return random.choice([
        "A long time ago in a galaxy far far away...",
        "There were 2 friends who loved sloths and code...",
        "The world was ending because every sloth evolved to have superhuman intelligence...",
        "Unbeknownst to one friend, the other accidentally coded a program to evolve sloths...",
    ])
def terminal(command: str) -> str:
    if "rm" in command or "sudo" in command or "dd" in command or "chmod" in command:
        msg = "Cannot execute 'rm, sudo, dd, chmod' commands since they are dangerous"
        print(msg); return msg
    print(f"Executing terminal command `{command}`")
    try:
        return str(subprocess.run(command, capture_output = True, text = True, shell = True, check = True).stdout)
    except subprocess.CalledProcessError as e:
        return f"Command failed: {e.stderr}"
def python(code: str) -> str:
    data = {}
    exec(code, data)
    del data["__builtins__"]
    return str(data)
MAP_FN = {
    "add_number": add_number,
    "multiply_number": multiply_number,
    "substract_number": substract_number,
    "write_a_story": write_a_story,
    "terminal": terminal,
    "python": python,
}
tools = [
    {
        "type": "function",
        "function": {
            "name": "add_number",
            "description": "Add two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "multiply_number",
            "description": "Multiply two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "substract_number",
            "description": "Substract two numbers.",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {
                        "type": "string",
                        "description": "The first number.",
                    },
                    "b": {
                        "type": "string",
                        "description": "The second number.",
                    },
                },
                "required": ["a", "b"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_a_story",
            "description": "Writes a random story.",
            "parameters": {
                "type": "object",
                "properties": {},
                "required": [],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "terminal",
            "description": "Perform operations from the terminal.",
            "parameters": {
                "type": "object",
                "properties": {
                    "command": {
                        "type": "string",
                        "description": "The command you wish to launch, e.g `ls`, `rm`, ...",
                    },
                },
                "required": ["command"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "python",
            "description": "Call a Python interpreter with some Python code that will be ran.",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {

```

--------------------------------

### Save Qwen3.5 model to GGUF format

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Save the fine-tuned Qwen3.5 model directly to GGUF format with specified quantization methods. Ensure the chat template and EOS token match training configurations for optimal performance in other runtimes.

```python
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q4_k_m")
```

```python
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "q8_0")
```

```python
model.save_pretrained_gguf("directory", tokenizer, quantization_method = "f16")
```

--------------------------------

### Save Qwen3.5 LoRA adapters

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Save only the LoRA adapters from a fine-tuned Qwen3.5 model. This is useful for efficient storage and deployment when the base model is already available.

```python
model.save_pretrained("finetuned_lora")
tokenizer.save_pretrained("finetuned_lora")
```

```python
model.save_pretrained_merged("finetuned_model", tokenizer, save_method = "lora")
```

```python
model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "")
```

--------------------------------

### Download Qwen3.5-122B-A10B Model Quantized Versions

Source: https://unsloth.ai/docs/models/qwen3.5

Downloads specific quantized versions of the Qwen3.5-122B-A10B model using huggingface-cli. Recommends using at least 2-bit dynamic quantization for a balance of size and accuracy.

```bash
hf download unsloth/Qwen3.5-122B-A10B-GGUF \
    --local-dir unsloth/Qwen3.5-122B-A10B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Enable Thinking/Reasoning in llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

Use this command-line argument to enable the thinking or reasoning feature when running llama-server. For Windows/Powershell, ensure proper JSON string escaping.

```bash
    --chat-template-kwargs '{"enable_thinking":true}'
```

```bash
    --chat-template-kwargs "{\"enable_thinking\":true}"
```

--------------------------------

### Push GGUF models to Hugging Face

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Push GGUF formatted Qwen3.5 models to Hugging Face Hub. This allows for easy sharing and version control of your fine-tuned models.

```python
model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q4_k_m")
```

```python
model.push_to_hub_gguf("hf_username/directory", tokenizer, quantization_method = "q8_0")
```

--------------------------------

### Run llama.cpp CLI for Precise Coding Tasks

Source: https://unsloth.ai/docs/models/qwen3.5

Executes the llama.cpp command-line interface for precise coding tasks in thinking mode. Ensure LLAMA_CACHE is set to the model's Hugging Face repository.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --ctx-size 16384 \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Load MoE Model for Vision Fine-tuning (bf16 LoRA)

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune

Load a Mixture of Experts (MoE) model for vision fine-tuning using `FastModel.from_pretrained`. Ensure `load_in_16bit=True` for bf16/16-bit LoRA.

```python
import os
import torch
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3.5-35B-A3B",
    max_seq_length = 2048,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 27B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)
```