Qwen3.5

https://unsloth.ai/docs/models/qwen3.5
Admin
Qwen3.5 is Alibaba's advanced language model family with sizes from 0.8B to 397B parameters,...
Tokens:35,886
Snippets:236
Trust Score:9.6
Update:19 hours ago
Show doc for...
Tokens:
Raw
### Run llama-server with Qwen3.5-9B and thinking enabled

Source: https://unsloth.ai/docs/models/qwen3.5.md

Example command to start llama-server with the Qwen3.5-9B model, setting the alias and enabling thinking. This is the server setup for the subsequent Python client example.

```bash
./llama.cpp/llama-server \
    --model unsloth/Qwen3.5-9B-GGUF/Qwen3.5-9B-BF16.gguf \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'

```

--------------------------------

### Python example to enable thinking for Qwen3.5-9B

Source: https://unsloth.ai/docs/models/qwen3.5

Python code to enable thinking for Qwen3.5-9B using OpenAI library.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
response = client.chat.completions.create(
    model="qwen3.5-9b",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)
print(response.choices[0].message.content)
```

--------------------------------

### Launch Unsloth Studio

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune.md

Starts the Unsloth Studio server on all interfaces and port 8888. After running, open http://localhost:8888 in your browser.

```bash
unsloth studio -H 0.0.0.0 -p 8888
```

--------------------------------

### Run llama-server for general tasks with thinking enabled

Source: https://unsloth.ai/docs/models/qwen3.5.md

Start llama-server with different sampling parameters for general tasks, with thinking mode enabled.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Install Unsloth on MacOS, Linux, WSL

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune.md

Run this command in your terminal to install Unsloth. Installation is quick and takes approximately 1-2 minutes.

```bash
curl -fsSL https://unsloth.ai/install.sh | sh
```

--------------------------------

### Install Unsloth on Windows PowerShell

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune.md

Run this command in Windows PowerShell to install Unsloth. Installation is quick and takes approximately 1-2 minutes.

```bash
irm https://unsloth.ai/install.ps1 | iex
```

--------------------------------

### Deploy Qwen3.5 with llama-server

Source: https://unsloth.ai/docs/models/qwen3.5.md

Starts llama-server serving the Qwen3.5-35B-A3B GGUF model with multimodal projection, alias, and sampling parameters on port 8001. Use in a new terminal (e.g., via tmux) for production deployment.

```bash
./llama.cpp/llama-server \
--model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --alias "unsloth/Qwen3.5-35B-A3B" \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --port 8001
```

--------------------------------

### Download Qwen3.5-35B-A3B GGUF model with hf download

Source: https://unsloth.ai/docs/models/qwen3.5.md

Downloads the model and multimodal projector using hf download. Requires pip install huggingface_hub hf_transfer. Use --include "*UD-Q2_K_XL*" for Dynamic 2bit.

```bash
hf download unsloth/Qwen3.5-35B-A3B-GGUF \
    --local-dir unsloth/Qwen3.5-35B-A3B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Query Documentation with HTTP GET

Source: https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks.md

Use this HTTP GET request to query the documentation for additional information. The 'ask' parameter should contain a specific, self-contained question, and 'goal' should describe the user's ultimate objective. The response includes a direct answer and relevant excerpts.

```http
GET https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks.md?ask=<question>&goal=<user_goal>
```

--------------------------------

### Run llama-server with thinking mode enabled

Source: https://unsloth.ai/docs/models/qwen3.5.md

Start llama-server with the Qwen3.5-9B GGUF model, enabling thinking mode via chat-template-kwargs.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-9B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --alias "unsloth/Qwen3.5-9B-GGUF" \
    --port 8001 \
    --chat-template-kwargs '{"enable_thinking":true}'
```

--------------------------------

### Loader example for MoE (bf16 LoRA) with FastModel

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune.md

Use this snippet to load a Qwen3.5 MoE model (e.g., unsloth/Qwen3.5-35B-A3B) with bf16 LoRA. After loading, attach LoRA adapters and train similarly to the SFT example above.

```python
import os
import torch
from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained(
    model_name = "unsloth/Qwen3.5-35B-A3B",
    max_seq_length = 2048,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 27B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)
```

--------------------------------

### Download Qwen3.5-122B-A10B GGUF model with hf download

Source: https://unsloth.ai/docs/models/qwen3.5.md

Downloads the model using huggingface_hub and hf_transfer. Includes mmproj-F16 and UD-Q4_K_XL files; use '*UD-Q2_K_XL*' for Dynamic 2bit. Requires pip install huggingface_hub hf_transfer.

```bash
hf download unsloth/Qwen3.5-122B-A10B-GGUF \
    --local-dir unsloth/Qwen3.5-122B-A10B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Fetch Qwen3.5 YAML for LM Studio Thinking Toggle

Source: https://unsloth.ai/docs/models/qwen3.5.md

Run this command in Terminal/PowerShell after installing LM Studio to enable the '💡Thinking' and 'Non-thinking' toggle for your GGUF model. Change '4b' to the desired quant size.

```bash
lms get unsloth/qwen3.5-4b
```

--------------------------------

### Download Qwen3.5-397B-A17B GGUF model via Hugging Face

Source: https://unsloth.ai/docs/models/qwen3.5.md

Downloads the model files using hf download. Requires pip install huggingface_hub hf_transfer. Use --include "*UD-Q2_K_XL*" for dynamic 2-bit quantization.

```bash
hf download unsloth/Qwen3.5-397B-A17B-GGUF \
    --local-dir unsloth/Qwen3.5-397B-A17B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Minimal SFT recipe for Qwen3.5 with Unsloth

Source: https://unsloth.ai/docs/models/qwen3.5/fine-tune.md

Use this snippet to fine-tune Qwen3.5 text-only models with Unsloth Core. It loads a dataset with a 'text' column, configures a 16-bit LoRA, and trains with SFTTrainer. Ensure vision dependencies (torchvision, pillow) are installed if needed and keep Transformers up-to-date. If OOM, reduce batch size or max_seq_length and keep gradient checkpointing enabled.

```python
from unsloth import FastLanguageModel
import torch
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

max_seq_length = 2048  # start small; scale up after it works

# Example dataset (replace with yours). Needs a "text" column.
url = "https://huggingface.co/datasets/laion/OIG/resolve/main/unified_chip2.jsonl"
dataset = load_dataset("json", data_files={"train": url}, split="train")

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "Qwen/Qwen3.5-27B",
    max_seq_length = max_seq_length,
    load_in_4bit = False,     # MoE QLoRA not recommended, dense 27B is fine
    load_in_16bit = True,     # bf16/16-bit LoRA
    full_finetuning = False,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = [
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    # "unsloth" checkpointing is intended for very long context + lower VRAM
    use_gradient_checkpointing = "unsloth",
    random_state = 3407,
    max_seq_length = max_seq_length,
)

trainer = SFTTrainer(
    model = model,
    train_dataset = dataset,
    tokenizer = tokenizer,
    args = SFTConfig(
        max_seq_length = max_seq_length,
        per_device_train_batch_size = 1,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = 100,
        logging_steps = 1,
        output_dir = "outputs_qwen35",
        optim = "adamw_8bit",
        seed = 3407,
        dataset_num_proc = 1,
    ),
)

trainer.train()
```

--------------------------------

### Run Qwen3.5-27B in thinking mode for general tasks

Source: https://unsloth.ai/docs/models/qwen3.5.md

Uses llama-cli with UD-Q4_K_XL quantization, temp 1.0, top-p 0.95, top-k 20, min-p 0.00.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-35B-A3B in thinking mode for general tasks

Source: https://unsloth.ai/docs/models/qwen3.5.md

Uses llama-cli with the UD-Q4_K_XL quantized model. Sets temperature to 1.0 for general tasks. Use export LLAMA_CACHE to specify a save location.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-35B-A3B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-35B-A3B-GGUF:UD-Q4_K_XL \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Download Qwen3.5-27B GGUF model via Hugging Face

Source: https://unsloth.ai/docs/models/qwen3.5.md

Downloads the model with mmproj and UD-Q4_K_XL files. Use --include "*UD-Q2_K_XL*" for Dynamic 2bit. Requires huggingface_hub and hf_transfer.

```bash
hf download unsloth/Qwen3.5-27B-GGUF \
    --local-dir unsloth/Qwen3.5-27B-GGUF \
    --include "*mmproj-F16*" \
    --include "*UD-Q4_K_XL*" # Use "*UD-Q2_K_XL*" for Dynamic 2bit
```

--------------------------------

### Use OpenAI completion library with llama-server

Source: https://unsloth.ai/docs/models/qwen3.5

Uses OpenAI's Python library to interact with the llama-server. Requires pip install openai.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-no-key-required")
response = client.chat.completions.create(
    model="qwen3.5-397b-a17b",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)
```

--------------------------------

### Instruct (non-thinking) mode settings for general tasks

Source: https://unsloth.ai/docs/models/qwen3.5.md

Recommended sampling parameters for Qwen3.5 instruct (non-thinking) mode when handling general tasks. Use temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, and repetition_penalty=1.0.

```bash
temperature=0.7, top_p=0.8, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
```

--------------------------------

### Query Qwen3.5-9B with OpenAI Python client

Source: https://unsloth.ai/docs/models/qwen3.5.md

Python client to query the llama-server endpoint and print both the content and reasoning_content from the response. Requires the server from the previous example to be running.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-9B-GGUF",
    messages = [{"role": "user", "content": "What is 2+2?"},],
)
print(completion.choices[0].message.content)
print(completion.choices[0].message.reasoning_content)

```

--------------------------------

### Run Qwen3.5-397B-A17B with llama.cpp (custom threads and GPU layers)

Source: https://unsloth.ai/docs/models/qwen3.5

Runs the model with custom CPU threads and GPU offloading layers. Adjust --threads and --n-gpu-layers if GPU runs out of memory. Remove for CPU-only inference.

```bash
llama-cli -hf unsloth/Qwen3.5-397B-A17B-GGUF:Q4_K_M --threads 32 --n-gpu-layers 2
```

--------------------------------

### Run Qwen3.5-27B in conversation mode

Source: https://unsloth.ai/docs/models/qwen3.5.md

Runs llama-cli with the downloaded model and mmproj file, using temp 1.0, top-p 0.95, min-p 0.00, top-k 20.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-27B-GGUF/Qwen3.5-27B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-27B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

--------------------------------

### Run Qwen3.5-397B-A17B with local GGUF files and mmproj

Source: https://unsloth.ai/docs/models/qwen3.5.md

Runs llama-cli with a local GGUF file and mmproj. Adjust --threads for CPU threads and --n-gpu-layers for GPU offloading; remove if CPU-only.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-397B-A17B-GGUF/UD-Q4_K_XL/Qwen3.5-397B-A17B-UD-Q4_K_XL-00001-of-00006.gguf \
    --mmproj unsloth/Qwen3.5-397B-A17B-GGUF/mmproj-F16.gguf \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### Run Qwen3.5-9B in conversation mode with llama-cli

Source: https://unsloth.ai/docs/models/qwen3.5.md

Run the model in conversation mode with multimodal support using the downloaded GGUF files.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-9B-GGUF/Qwen3.5-9B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-9B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```

--------------------------------

### Query Qwen3.5 via OpenAI Python Client

Source: https://unsloth.ai/docs/models/qwen3.5.md

After installing the openai package, this script connects to the llama-server endpoint and sends a chat completion request. Requires the server from the previous snippet to be running on port 8001.

```python
from openai import OpenAI
import json
openai_client = OpenAI(
    base_url = "http://127.0.0.1:8001/v1",
    api_key = "sk-no-key-required",
)
completion = openai_client.chat.completions.create(
    model = "unsloth/Qwen3.5-397B-A17B",
    messages = [{"role": "user", "content": "Create a Snake game."},],
)
print(completion.choices[0].message.content)
```

--------------------------------

### Run Qwen3.5-397B-A17B in non-thinking mode with llama-server

Source: https://unsloth.ai/docs/models/qwen3.5.md

Starts llama-server with the UD-Q4_K_XL quantization and disables thinking via --chat-template-kwargs '{"enable_thinking":false}'. Use export LLAMA_CACHE to set a custom cache location.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-397B-A17B-GGUF"
./llama.cpp/llama-server \
    -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-Q4_K_XL \
    --temp 0.7 \
    --top-p 0.8 \
    --top-k 20 \
    --min-p 0.00 \
    --chat-template-kwargs '{"enable_thinking":false}'
```

--------------------------------

### Run Qwen3.5-27B in thinking mode for precise coding tasks

Source: https://unsloth.ai/docs/models/qwen3.5.md

Uses llama-cli with UD-Q4_K_XL quantization, temp 0.6, top-p 0.95, top-k 20, min-p 0.00. Set LLAMA_CACHE to force save location.

```bash
export LLAMA_CACHE="unsloth/Qwen3.5-27B-GGUF"
./llama.cpp/llama-cli \
    -hf unsloth/Qwen3.5-27B-GGUF:UD-Q4_K_XL \
    --temp 0.6 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00
```

--------------------------------

### HTTP GET request to query Qwen3.5 documentation

Source: https://unsloth.ai/docs/models/qwen3.5.md

Use this request to query the documentation when the answer is not on the page. The ask parameter is the specific question, and goal describes the user's ultimate objective. The response contains a direct answer and relevant excerpts.

```http
GET https://unsloth.ai/docs/models/qwen3.5.md?ask=<question>&goal=<user_goal>
```

--------------------------------

### Run Qwen3.5-35B-A3B in conversation mode with multimodal support

Source: https://unsloth.ai/docs/models/qwen3.5.md

Runs llama-cli with the downloaded GGUF model and mmproj file for multimodal conversation. Uses temperature 1.0, top-p 0.95, min-p 0.00, and top-k 20.

```bash
./llama.cpp/llama-cli \
    --model unsloth/Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf \
    --mmproj unsloth/Qwen3.5-35B-A3B-GGUF/mmproj-F16.gguf \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.00 \
    --top-k 20
```
Tokens:
Raw