LLaMA Factory

https://llamafactory.readthedocs.io
Admin
LLaMA Factory is a simple and efficient platform for training and fine-tuning large language models...
Tokens:50,795
Snippets:582
Trust Score:8.3
Update:4 months ago
Show doc for...
Tokens:
Raw
### Start API Service and Call Example

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/inference.html

Example of starting an API service and a Python script to call it for inference.

```bash
API_PORT=8000 CUDA_VISIBLE_DEVICES=0 llamafactory-cli api examples/inference/qwen3_lora_sft.yaml

```

```python
# api_call_example.py
from openai import OpenAI
client = OpenAI(api_key="0",base_url="http://0.0.0.0:8000/v1")
messages = [{"role": "user", "content": "Who are you?"}]
result = client.chat.completions.create(messages=messages, model="Qwen/Qwen3-4B-Instruct-2507")
print(result.choices[0].message)

```

--------------------------------

### Start Docker Container for NPU Training

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_training.rst.txt

Launches a Docker container with necessary NPU device mappings and shared memory for training. Ensure to adjust the device mappings according to your specific NPU setup.

```bash
docker run -itd \
    --net=host \
    --device=/dev/davinci0 \
    --device=/dev/davinci1 \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    --shm-size=1200g \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --name llama_factory_npu \
    hiyouga/llamafactory:latest-npu-a2 \
    /bin/bash
```

--------------------------------

### Example NPU Training Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_training.rst.txt

A sample YAML configuration for LLaMA-Factory training optimized for lower memory usage and to ensure the training process can start. Adjust parameters based on your specific hardware and task requirements.

```yaml
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
cutoff_len: 4096
gradient_checkpointing: true
```

--------------------------------

### API Service Startup and Client Call

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/inference.rst.txt

Example of starting the LLaMA-Factory API service and making a client call to it using OpenAI's library. Ensure the API port and device are correctly set.

```python
# api_call_example.py
from openai import OpenAI
client = OpenAI(api_key="0",base_url="http://0.0.0.0:8000/v1")
messages = [{"role": "user", "content": "Who are you?"}]
result = client.chat.completions.create(messages=messages, model="Qwen/Qwen3-4B-Instruct-2507")
print(result.choices[0].message)
```

--------------------------------

### Launch LLaMA Factory WebUI

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/webui.rst.txt

Use this command to start the WebUI after installation. The WebUI allows for zero-code fine-tuning of LLMs.

```bash
llamafactory-cli webui
```

--------------------------------

### Install CANN Toolkit

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Install the CANN Toolkit development suite, which is used for training, inference, and development. The installation path defaults to /usr/local/Ascend for root users and ${HOME}/Ascend for regular users.

```bash
./Ascend-cann-toolkit_<version>_linux-aarch64.run --install

```

--------------------------------

### Install CANN Toolkit

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the CANN Toolkit, which is essential for training, inference, and development on Ascend NPUs. Ensure sufficient disk space before installation.

```shell
./Ascend-cann-toolkit_<version>_linux-aarch64.run --install
```

--------------------------------

### SFT Configuration Example (qwen3_lora_sft.yaml)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/sft.rst.txt

An example YAML configuration file for SFT training with LoRA. It specifies model, LoRA parameters, dataset, and training settings. Adjust 'model_name_or_path' and 'dataset' to match your requirements.

```yaml
### examples/train_lora/qwen3_lora_sft.yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
trust_remote_code: true

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: identity,alpaca_en_demo
template: qwen3_nothink
cutoff_len: 2048
max_samples: 1000
preprocessing_num_workers: 16
dataloader_num_workers: 4

output_dir: saves/qwen3-4b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

# eval_dataset: alpaca_en_demo
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500
```

--------------------------------

### Build and Start Container with Docker Compose (Ascend-A2)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Builds the Docker image and starts the container for Ascend-A2 using Docker Compose.

```bash
# Ascend-A2
docker-compose up -d

```

--------------------------------

### Launching Training with Accelerate Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/distributed.rst.txt

Command to start training using Accelerate with a specified multi-node configuration file.

```bash
accelerate launch \
--config_file accelerate_multiNode_config.yaml \
train.py llm_config.yaml

```

--------------------------------

### Install CANN Ops Package

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the CANN ops package after the CANN Toolkit. Use the --install flag for standard installation or --devel for static library installation.

```shell
./Ascend-cann-<chip_type>-ops_<version>_linux-aarch64.run --install
```

--------------------------------

### Install CANN Ops Package

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Install the CANN ops package after the CANN Toolkit has been installed. Use the --devel flag instead of --install if you need to install static libraries.

```bash
./Ascend-cann-<chip_type>-ops_<version>_linux-aarch64.run --install

```

--------------------------------

### Start NativeDDP Training with torchrun (Multi-Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Start multi-node NativeDDP training using torchrun. Configure master port, number of processes per node, number of nodes, node rank, and master address.

```bash
torchrun --master_port 29500 --nproc_per_node=8 --nnodes=2 --node_rank=0  \
--master_addr=192.168.0.1  train.py
torchrun --master_port 29500 --nproc_per_node=8 --nnodes=2 --node_rank=1  \
--master_addr=192.168.0.1  train.py
```

--------------------------------

### Install bitsandbytes for QLoRA on Windows

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Install a specific version of the bitsandbytes library compatible with Windows and your CUDA version to enable QLoRA.

```bash
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.2.post2-py3-none-win_amd64.whl
```

--------------------------------

### Install HDK Firmware

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the Ascend HDK firmware using the .run package. The --full flag ensures a complete installation. A success message indicates proper installation.

```shell
./Ascend-hdk-<chip_type>-npu-firmware_<version>.run --full
```

--------------------------------

### Launch Training with DeepSpeed Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/distributed.rst.txt

Use the accelerate launch command with a DeepSpeed configuration file to start the training process.

```bash
accelerate launch \
--config_file deepspeed_config.yaml \
train.py llm_config.yaml
```

--------------------------------

### Pre-training Configuration Example

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/trainers.rst.txt

Configuration for pre-training a large language model. Ensure the dataset format matches the pre-training dataset requirements.

```yaml
### model
model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct

### method
stage: pt
do_train: true
finetuning_type: lora
lora_target: all

### dataset
dataset: c4_demo
cutoff_len: 1024
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/llama3-8b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
```

--------------------------------

### Download and Install CUDA 12.2

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/installation.html

Download the CUDA 12.2 installer and execute it. It is recommended to skip the driver installation if unsure about compatibility.

```bash
wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run
sudo sh cuda_12.2.0_535.54.03_linux.run

```

--------------------------------

### Example accelerate Configuration for Multi-Node Multi-GPU

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

This is an example configuration file for multi-node multi-GPU training using accelerate. It specifies the number of machines, processes, and main process details.

```yaml
# accelerate_multiNode_config.yaml
compute_environment: LOCAL_MACHINE
debug: true
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_process_ip: '192.168.0.1'
main_process_port: 29500
main_training_function: main
mixed_precision: fp16
num_machines: 2
num_processes: 16
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Alpaca Format Example (Single Turn)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/data_preparation.html

Example of a single-turn instruction tuning dataset in Alpaca format. The 'instruction' and 'input' fields are combined for the final prompt.

```json
{
  "instruction": "计算这些物品的总费用。 ",
  "input": "输入：汽车 - $3000，衣服 - $100，书 - $20。",
  "output": "汽车、衣服和书的总费用为 $3000 + $100 + $20 = $3120。"
}
```

--------------------------------

### Build and Start with Docker Compose (A2)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Builds the Docker image and starts the container using Docker Compose for Ascend A2 NPUs. Navigate to the docker/docker-npu directory first.

```shell
cd docker/docker-npu
# Ascend-A2
docker-compose up -d
```

--------------------------------

### Start LLaMA Factory Training

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_training.html

Initiates the training process using the LLaMA Factory CLI with a specified configuration file.

```bash
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml

```

--------------------------------

### Download and Install CUDA 12.2

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Download the CUDA 12.2 installer and execute it. It is recommended to deselect the driver installation if you are unsure about compatibility.

```bash
wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run
sudo sh cuda_12.2.0_535.54.03_linux.run
```

--------------------------------

### Start Container from Custom Docker Build

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Launches a Docker container using a locally built LLaMA Factory image, with NPU devices and host volumes mapped. This is the same command as starting from a pre-built image, but uses the custom-built DOCKER_IMAGE.

```bash
CONTAINER_NAME=llama_factory_npu
DOCKER_IMAGE=llamafactory:latest
docker run -itd \
    --cap-add=SYS_PTRACE \
    --net=host \
    --device=/dev/davinci0 \
    --device=/dev/davinci1 \
    --device=/dev/davinci2 \
    --device=/dev/davinci3 \
    --device=/dev/davinci4 \
    --device=/dev/davinci5 \
    --device=/dev/davinci6 \
    --device=/dev/davinci7 \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    --shm-size=1200g \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /data:/data \
    --name "$CONTAINER_NAME" \
    "$DOCKER_IMAGE" \
    /bin/bash
```

--------------------------------

### Start Full Parameter Fine-Tuning (Full) Job

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_training.html

Command to initiate a full parameter fine-tuning job using a specified configuration file.

```bash
llamafactory-cli train examples/train_full/qwen3_full_sft.yaml

```

--------------------------------

### Install NNAL Neural Network Acceleration Library

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the NNAL library, which includes ATB and SiP acceleration libraries. This is an optional step and should be done after installing the CANN Toolkit.

```shell
./Ascend-cann-nnal_<version>_linux-aarch64.run --install
```

--------------------------------

### Install LLaMA-Factory and Dependencies

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Clone the LLaMA-Factory repository and install the package along with its core dependencies and metrics requirements.

```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
pip install -r requirements/metrics.txt
```

--------------------------------

### Build and Start with Docker Compose (A3)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Builds and starts the Docker container using Docker Compose for Ascend A3 NPUs. This command uses a specific profile for A3 devices.

```shell
cd docker/docker-npu
# Ascend-A3
docker-compose --profile a3 up -d llamafactory-a3
```

--------------------------------

### Start NativeDDP Training with torchrun (Single Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Use torchrun to start NativeDDP training on a single node with multiple GPUs. Configure nnodes, nproc_per_node, and other training parameters as needed.

```bash
torchrun  --standalone --nnodes=1 --nproc-per-node=8  src/train.py \
--stage sft \
--model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct  \
--do_train \
--dataset alpaca_en_demo \
--template llama3 \
--finetuning_type lora \
--output_dir  saves/llama3-8b/lora/ \
--overwrite_cache \
--per_device_train_batch_size 1 \
--gradient_accumulation_steps 8 \
--lr_scheduler_type cosine \
--logging_steps 100 \
--save_steps 500 \
--learning_rate 1e-4 \
--num_train_epochs 2.0 \
--plot_loss \
--bf16
```

--------------------------------

### Alpaca Format Example (JSON)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/data_preparation.rst.txt

An example of a single-turn instruction tuning data entry in Alpaca format. It includes instruction, input, and output fields.

```json
{
    "instruction": "计算这些物品的总费用。 ",
    "input": "输入：汽车 - $3000，衣服 - $100，书 - $20。",
    "output": "汽车、衣服和书的总费用为 $3000 + $100 + $20 = $3120。"
  }
```

--------------------------------

### Start Ray Head Node

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Command to start the Ray head node for multi-node distributed training.

```bash
ray start --head --port=6379

```

--------------------------------

### Start NativeDDP Training with llamafactory-cli (Multi-Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Start multi-node NativeDDP training using llamafactory-cli. Set FORCE_TORCHRUN to 1 and provide node information (NNODES, NODE_RANK, MASTER_ADDR, MASTER_PORT). Run this command on each node with the appropriate NODE_RANK.

```bash
FORCE_TORCHRUN=1 NNODES=2 NODE_RANK=0 MASTER_ADDR=192.168.0.1 MASTER_PORT=29500 \
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml

FORCE_TORCHRUN=1 NNODES=2 NODE_RANK=1 MASTER_ADDR=192.168.0.1 MASTER_PORT=29500 \
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

--------------------------------

### Alpaca Format Example (Text Input)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/data_preparation.rst.txt

Demonstrates how instruction and input are combined to form the final human input for instruction tuning.

```text
计算这些物品的总费用。
输入：汽车 - $3000，衣服 - $100，书 - $20。
```

--------------------------------

### Start Pre-training (PT) Job

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_training.html

Command to initiate a pre-training job using a specified configuration file.

```bash
llamafactory-cli train examples/train_lora/qwen3_lora_pretrain.yaml

```

--------------------------------

### Start NativeDDP Training with llamafactory-cli (Single GPU)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Use llamafactory-cli to start NativeDDP training. Ensure FORCE_TORCHRUN is set to 1. This command defaults to using all available GPUs if CUDA_VISIBLE_DEVICES is not specified.

```bash
FORCE_TORCHRUN=1 llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

--------------------------------

### Start NativeDDP Training with llamafactory-cli (Specific GPUs)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Use llamafactory-cli to start NativeDDP training on specific GPUs. Set FORCE_TORCHRUN to 1 and specify the desired GPUs using CUDA_VISIBLE_DEVICES.

```bash
FORCE_TORCHRUN=1 CUDA_VISIBLE_DEVICES=0,1 llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

--------------------------------

### Install LLaMA Factory and Transformers

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/best_practice/gpt-oss.html

Clone the LLaMA Factory repository, install the library and its dependencies, including a specific version of transformers.

```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
pip install -r requirements/metrics.txt --no-build-isolation
pip install "transformers==4.55.0"

```

--------------------------------

### Example accelerate Configuration for Single Node Multi-GPU

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

This is an example configuration file for single-node multi-GPU training using accelerate. It specifies distributed type, mixed precision, and GPU IDs.

```yaml
# accelerate_singleNode_config.yaml
compute_environment: LOCAL_MACHINE
debug: true
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Alpaca Format Example (Text Output)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/data_preparation.rst.txt

Shows the expected model response for the given instruction and input in instruction tuning.

```text
汽车、衣服和书的总费用为 $3000 + $100 + $20 = $3120。
```

--------------------------------

### Start NPU Docker Container

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Launches a Docker container with NPU devices attached. Ensure DOCKER_IMAGE and device paths are correctly set.

```bash
CONTAINER_NAME=llama_factory_npu
DOCKER_IMAGE=hiyouga/llamafactory:latest-npu-a2

docker run -itd \
    --cap-add=SYS_PTRACE \
    --net=host \
    --device=/dev/davinci0 \
    --device=/dev/davinci1 \
    --device=/dev/davinci2 \
    --device=/dev/davinci3 \
    --device=/dev/davinci4 \
    --device=/dev/davinci5 \
    --device=/dev/davinci6 \
    --device=/dev/davinci7 \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    --shm-size=1200g \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /data:/data \
    --name "$CONTAINER_NAME" \
    "$DOCKER_IMAGE" \
    /bin/bash

```

--------------------------------

### Alpaca Format Example (Multi-turn)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/data_preparation.html

Example of a multi-turn instruction tuning dataset in Alpaca format, including conversation history. The 'history' field contains pairs of previous instructions and responses.

```json
[
  {
    "instruction": "今天的天气怎么样？",
    "input": "",
    "output": "今天的天气不错，是晴天。",
    "history": [
      [
        "今天会下雨吗？",
        "今天不会下雨，是个好天气。"
      ],
      [
        "今天适合出去玩吗？",
        "非常适合，空气质量很好。"
      ]
    ]
  }
]
```

--------------------------------

### BAdam Configuration Example

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/tuning_algorithms.html

Example configuration for using the BAdam optimizer. Ensure `finetuning_type` is set to `full` and `pure_bf16` to `True` when using BAdam. Note that `badam_mode = layer` requires DeepSpeed ZeRO3 for single or multi-GPU training, while `badam_mode = ratio` is single-GPU only.

```yaml
stage: sft
do_train: true
finetuning_type: full
use_badam: true
badam_mode: layer
badam_switch_mode: ascending
badam_switch_interval: 50
badam_verbose: 2
pure_bf16: true
```

--------------------------------

### Train with NativeDDP using llamafactory-cli (Single-Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/distributed.rst.txt

Use llamafactory-cli to start NativeDDP training. Specify CUDA_VISIBLE_DEVICES to select GPUs.

```bash
FORCE_TORCHRUN=1 llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

```bash
FORCE_TORCHRUN=1 CUDA_VISIBLE_DEVICES=0,1 llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

--------------------------------

### Example SFT Configuration File (qwen3_lora_sft.yaml)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/sft.html

A sample configuration file for SFT training using LoRA with the Qwen3 model. This file specifies model, dataset, training, and output parameters. Adjust these settings according to your specific requirements.

```yaml
model_name_or_path: Qwen/Qwen3-4B-Instruct-2507
trust_remote_code: true

stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8
lora_target: all

dataset: identity,alpaca_en_demo
template: qwen3_nothink
cutoff_len: 2048
max_samples: 1000
preprocessing_num_workers: 16
data_loader_num_workers: 4

output_dir: saves/qwen3-4b/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none

per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

# eval_dataset: alpaca_en_demo
# val_size: 0.1
# per_device_eval_batch_size: 1
# eval_strategy: steps
# eval_steps: 500

```

--------------------------------

### Configure Accelerate for Single-Node Multi-GPU Training

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/distributed.rst.txt

Run 'accelerate config' to generate a configuration file for single-node multi-GPU training. The example shows a typical setup with fp16 mixed precision.

```bash
accelerate config
```

```yaml
# accelerate_singleNode_config.yaml
compute_environment: LOCAL_MACHINE
debug: true
distributed_type: MULTI_GPU
downcast_bf16: 'no'
enable_cpu_affinity: false
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

--------------------------------

### Launch FSDP+QLoRA Fine-tuning with Bash Script

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/distributed.rst.txt

Execute a bash script to initiate FSDP and QLoRA fine-tuning, after modifying the provided configuration files.

```bash
bash examples/extras/fsdp_qlora/train.sh
```

--------------------------------

### Launch accelerate Training (Multi-Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Launch accelerate training for a multi-node setup using a pre-generated configuration file. Specify the config file and the training script with its arguments.

```bash
accelerate launch \
--config_file accelerate_multiNode_config.yaml \
train.py llm_config.yaml
```

--------------------------------

### Multimodal Model Inference Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/inference.rst.txt

Configuration example for multimodal model inference. Includes model path, template, and backend settings.

```yaml
model_name_or_path: Qwen/Qwen3-VL-4B-Instruct
template: qwen3_vl
infer_backend: huggingface #choices： [huggingface, vllm]
trust_remote_code: true
```

--------------------------------

### Launch accelerate Training (Single Node)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Launch accelerate training using a pre-generated configuration file. Specify the config file and the training script with its arguments.

```bash
accelerate launch \
--config_file accelerate_singleNode_config.yaml \
src/train.py training_config.yaml
```

--------------------------------

### Install torch-npu via requirements

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Install the torch-npu plugin along with other dependencies by running pip install with the requirements/npu.txt file. This is the recommended method during LLaMA-Factory installation.

```bash
pip install -r requirements/npu.txt

```

--------------------------------

### Configure BAdam Optimizer

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/tuning_algorithms.rst.txt

Example configuration for enabling and setting up the BAdam optimizer. Ensure to follow the specified constraints for optimal performance and compatibility.

```yaml
### model
...
### method
stage: sft
do_train: true
finetuning_type: full
use_badam: true
badam_mode: layer
badam_switch_mode: ascending
badam_switch_interval: 50
badam_verbose: 2
pure_bf16: true
...
```

--------------------------------

### Install HDK Driver

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the Ascend HDK driver using the .run package. The --full and --install-for-all flags ensure a complete installation. A success message indicates proper installation.

```shell
./Ascend-hdk-<chip_type>-npu-driver_<version>_linux-<arch>.run --full --install-for-all
```

--------------------------------

### Resolve Environment Conflicts during Installation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/getting_started/installation.html

If environment conflicts arise during the LLaMA-Factory installation, use this command to install without dependencies.

```bash
pip install --no-deps -e .

```

--------------------------------

### Uninstall Existing CUDA (if necessary)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Remove previous CUDA installations before installing a new version. This ensures a clean installation.

```bash
sudo /usr/local/cuda-12.1/bin/cuda-uninstaller
```

```bash
sudo rm -r /usr/local/cuda-12.1/
sudo apt clean && sudo apt autoclean
```

--------------------------------

### Start SFT Training with YAML Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/sft.rst.txt

Initiates the SFT training process using a specified YAML configuration file. Ensure the YAML file is correctly configured for your model and dataset.

```bash
llamafactory-cli train examples/train_lora/qwen3_lora_sft.yaml
```

--------------------------------

### Install torch-npu manually

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Manually install the torch-npu plugin by providing the specific wheel file name. Ensure the installed torch-npu and torch versions are compatible.

```bash
pip install torch_npu-version-cp311-cp311-manylinux_2_17_aarch64.whl

```

--------------------------------

### Launch FSDP Training with Ray and Accelerate

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Command to launch FSDP training with Ray enabled using Accelerate. Requires setting USE_RAY=1 and configuring fsdp_config.yaml with num_processes=1.

```bash
USE_RAY=1 accelerate launch \
--config_file fsdp_config.yaml \
src/train.py training_config.yaml

```

--------------------------------

### Install torch-npu via requirements

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Install the torch-npu plugin along with other LLaMA-Factory dependencies by using the provided requirements file. This is the recommended method for installing torch-npu.

```bash
pip install -r requirements/npu.txt
```

--------------------------------

### Grant Execute Permissions for CANN Toolkit

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Before installing the CANN Toolkit, grant execute permissions to the .run installation file. This is necessary to run the installation script.

```shell
chmod +x Ascend-cann-toolkit_<version>_linux-aarch64.run
```

--------------------------------

### Launch FSDP+QLoRA Fine-tuning with accelerate

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Shell script to launch FSDP+QLoRA fine-tuning. Ensure accelerate is installed and FSDP configuration files are correctly modified.

```bash
examples/extras/fsdp_qlora/train.sh
```

--------------------------------

### Install torch-npu manually

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Manually install the torch-npu plugin by providing the specific wheel file. Ensure the Python version and torch version are compatible with the installed torch-npu version.

```bash
pip install torch_npu-version-cp311-cp311-manylinux_2_17_aarch64.whl
```

--------------------------------

### Grant Execute Permissions for HDK Installation Packages

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_installation.rst.txt

Before installing the HDK driver and firmware, grant execute permissions to the respective .run files. This is a prerequisite for running the installation commands.

```shell
chmod +x Ascend-hdk-<chip_type>-npu-driver_<version>_linux-<arch>.run
chmod +x Ascend-hdk-<chip_type>-npu-firmware_<version>.run
```

--------------------------------

### Verify LLaMA-Factory Installation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Confirm that LLaMA-Factory has been installed successfully by checking its version using the command-line interface.

```bash
llamafactory-cli version
```

--------------------------------

### Resolve Environment Conflicts

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

If environment conflicts arise during installation, try installing the package without its dependencies.

```bash
pip install --no-deps -e .
```

--------------------------------

### Verify CUDA Installation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Check if the CUDA toolkit has been installed correctly by verifying the NVCC compiler version.

```bash
nvcc -V
```

--------------------------------

### Example _get_mm_inputs Implementation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/model_support.rst.txt

Illustrates the _get_mm_inputs method, showing how to process visual inputs and return model-compatible tensors for different multimodal architectures like LLaVA, Qwen2-VL, and MLlama.

```python
# https://github.com/hiyouga/LLaMA-Factory/blob/da971c37640de20f97b4d774e77e6f8d5c00b40a/src/llamafactory/data/mm_plugin.py#L264
def _get_mm_inputs(
    self,
    images: list["ImageInput"],
    videos: list["VideoInput"],
    audios: list["AudioInput"],
    processor: "MMProcessor",
    imglens: Optional[list[int]] = None,
) -> dict[str, "torch.Tensor"]:
    r"""Process visual inputs.

    Returns: (llava and paligemma)
        pixel_values: tensor with shape (B, C, H, W)

    Returns: (qwen2-vl)
        pixel_values: tensor with shape (num_patches, patch_dim)
        image_grid_thw: tensor with shape (num_images, 3), where the three numbers are time, width, height
        where num_patches == torch.prod(image_grid_thw)

    Returns: (mllama)
        pixel_values: tensor with shape
                      (batch_size, max_num_images, max_image_tiles, channels, tile_height, tile_width)
                      For example, (2, 1, 4, 3, 560, 560).
        aspect_ratio_ids: tensor with shape (batch_size, max_num_images). For example, (2, 1).
        aspect_ratio_mask: tensor with shape (batch_size, max_num_images, max_image_tiles). For example, (2, 1, 4).
        num_tiles: List[List[int]] with shape (batch_size, num_images_in_batch). For example, (2, 1).

    """
    pass
```

--------------------------------

### Install LLaMA-Factory and Dependencies

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/advanced/best_practice/gpt-oss.rst.txt

Installs the LLaMA-Factory library, its dependencies, and the transformers library, which is required for GPT-OSS models.

```bash
git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .
pip install -r requirements/metrics.txt --no-build-isolation
pip install "transformers==4.55.0"
```

--------------------------------

### Start Ray Worker Node

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Command to start a Ray worker node and connect it to the head node.

```bash
ray start --address='master node IP:6379'

```

--------------------------------

### Start Direct Preference Optimization (DPO) Training Job

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_training.html

Command to initiate a DPO training job using a specified configuration file.

```bash
llamafactory-cli train examples/train_lora/qwen3_lora_dpo.yaml

```

--------------------------------

### Verify HDK Installation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Check the status of the NPU driver after installation and system reboot by running the 'npu-smi info' command.

```bash
npu-smi info

```

--------------------------------

### FSDP2 Configuration Example

Source: https://llamafactory.readthedocs.io/zh-cn/latest/advanced/distributed.html

Configuration file for FSDP2 distributed training. Specifies environment, distributed type, mixed precision, and FSDP-specific settings.

```yaml
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_cpu_ram_efficient_loading: true
  fsdp_offload_params: false
  fsdp_reshard_after_forward: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_version: 2
machine_rank: 0
main_training_function: main
mixed_precision: bf16  # or fp16
num_machines: 1  # the number of nodes
num_processes: 2  # the number of GPUs in all nodes
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

```

--------------------------------

### Check GCC Installation

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/installation.rst.txt

Ensure that the GCC compiler is installed on your Linux system. This is often required for building software from source.

```bash
gcc --version
```

--------------------------------

### Start NPU Docker Container (Built Locally)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/multibackend/npu/npu_installation.html

Launches a Docker container using a locally built Llama Factory image with NPU devices attached.

```bash
CONTAINER_NAME=llama_factory_npu
DOCKER_IMAGE=llamafactory:latest
docker run -itd \
    --cap-add=SYS_PTRACE \
    --net=host \
    --device=/dev/davinci0 \
    --device=/dev/davinci1 \
    --device=/dev/davinci2 \
    --device=/dev/davinci3 \
    --device=/dev/davinci4 \
    --device=/dev/davinci5 \
    --device=/dev/davinci6 \
    --device=/dev/davinci7 \
    --device=/dev/davinci_manager \
    --device=/dev/devmm_svm \
    --device=/dev/hisi_hdc \
    --shm-size=1200g \
    -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /data:/data \
    --name "$CONTAINER_NAME" \
    "$DOCKER_IMAGE" \
    /bin/bash

```

--------------------------------

### Direct Preference Optimization (DPO) Training Configuration

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/multibackend/npu/npu_training.rst.txt

This command initiates DPO training. It requires a specific configuration file tailored for DPO.

```bash
llamafactory-cli train examples/train_lora/qwen3_lora_dpo.yaml
```

--------------------------------

### Alpaca Multi-turn Conversation Example (JSON)

Source: https://llamafactory.readthedocs.io/zh-cn/latest/_sources/getting_started/data_preparation.rst.txt

An example of a multi-turn conversation dataset in Alpaca format, demonstrating the use of the 'history' field.

```json
[
    {
      "instruction": "今天的天气怎么样？",
      "input": "",
      "output": "今天的天气不错，是晴天。",
      "history": [
        [
          "今天会下雨吗？", 
          "今天不会下雨，是个好天气。"
        ],
        [
          "今天适合出去玩吗？", 
          "非常适合，空气质量很好。"
        ]
      ]
    }
  ]
```
Tokens:
Raw