### Pre-DPO Installation and Environment Setup (Bash)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This bash script outlines the steps for setting up the Pre-DPO development environment. It includes creating a Conda environment, cloning the repository, installing PyTorch with CUDA support, installing the Pre-DPO framework and its optional dependencies (like bitsandbytes, vLLM, galore), and verifying the installation by checking the `llamafactory-cli` version and GPU availability.

```bash
# Create conda environment
conda create -n predpo python=3.10
conda activate predpo

# Clone repository
git clone https://github.com/DtYXs/Pre-DPO.git
cd Pre-DPO

# Install PyTorch with CUDA support
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124

# Install Pre-DPO framework
pip install -e ".[torch,metrics]"

# Install DeepSpeed for distributed training
pip install deepspeed==0.15.4

# Optional: Install additional dependencies
pip install -e ".[bitsandbytes]"  # For quantization
pip install -e ".[vllm]"          # For vLLM inference backend
pip install -e ".[galore]"        # For GaLore optimizer
pip install -e ".[dev]"           # For development tools

# Verify installation
llamafactory-cli version

# Check GPU availability
python -c "import torch; print(f'CUDA available: {torch.cuda.is_available()}')"
python -c "import torch; print(f'GPU count: {torch.cuda.device_count()}')"
```

--------------------------------

### Pre-DPO Training with Guiding Reference Model

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Conducts Pre-DPO training, which enhances DPO by incorporating a guiding reference model. This script specifies the SFT model path, the reference model path (a previously trained DPO model), the binarized preference dataset, and training hyperparameters like preference beta, batch size, learning rate, and GPU configuration.

```bash
bash scripts/train_predpo.sh \
    --sft_model_path meta-llama/Llama-3.2-3B-Instruct \
    --ref_model_path experiments/llama3.2-3b-dpo \
    --dataset llama3.2-3b-ultrafeedback-armorm-binarized \
    --template llama3 \
    --pref_beta 0.1 \
    --bsz 1 \
    --gradient_accumulation_steps 16 \
    --lr 6.0e-7 \
    --gpus 0,1,2,3 \
    --output_dir experiments/llama3.2-3b-predpo
```

--------------------------------

### Start OpenAI-Compatible API Server (Python)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Initiate an API server compatible with OpenAI's format using the `run_api` function from `llamafactory.api.app` or by running `src/api.py`. Access API documentation at `http://localhost:8000/docs`.

```python
# Start the API server
from llamafactory.api.app import run_api
run_api()

# Or via command line
python src/api.py
# Visit http://localhost:8000/docs for API documentation
```

--------------------------------

### Launch API Server with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Start an API server for model inference using the `llamafactory-cli api` command. Configure model path, template, and inference backend. Environment variables can be set for host, port, API key, and model name.

```bash
# Start API server for model inference
llamafactory-cli api \
    --model_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --infer_backend huggingface

# Environment variables for API configuration
export API_HOST=0.0.0.0
export API_PORT=8000
export API_KEY=your_secret_key
export API_MODEL_NAME=llama3.2-predpo
llamafactory-cli api --model_name_or_path experiments/llama3.2-3b-predpo
```

--------------------------------

### Display Version Information with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Check the installed version of the llamafactory-cli tool using the `llamafactory-cli version` command.

```bash
# Show version information
llamafactory-cli version
```

--------------------------------

### Install Pre-DPO Environment and Dependencies

Source: https://github.com/dtyxs/pre-dpo/blob/main/README.md

Sets up a Conda environment for Pre-DPO, clones the repository, and installs PyTorch with CUDA support, other dependencies, and DeepSpeed.

```shell
conda create -n predpo python=3.10 && conda activate predpo

git clone https://github.com/DtYXs/Pre-DPO.git
cd Pre-DPO
pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu124
pip install -e ".[torch,metrics]"
pip install deepspeed==0.15.4
```

--------------------------------

### Make API Requests with curl (OpenAI-Compatible)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Interact with the deployed OpenAI-compatible API server using `curl` commands. Examples include sending chat completion requests (both standard and streaming) and listing available models. Requires setting the `Authorization` header with your API key.

```bash
# Make API requests with curl
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_secret_key" \
  -d '{
    "model": "llama3.2-predpo",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    "temperature": 0.7,
    "max_tokens": 500,
    "stream": false
  }'

# Streaming response
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your_secret_key" \
  -d '{
    "model": "llama3.2-predpo",
    "messages": [
      {"role": "user", "content": "Write a short story about AI."}
    ],
    "stream": true
  }'

# List available models
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer your_secret_key"

# Response format:
# {
#   "id": "chatcmpl-123",
#   "object": "chat.completion",
#   "created": 1677652288,
#   "model": "llama3.2-predpo",
#   "choices": [{"index": 0,
#     "message": {"role": "assistant", "content": "Quantum computing harnesses..."},
#     "finish_reason": "stop" }],
#   "usage": {
#     "prompt_tokens": 20,
#     "completion_tokens": 150,
#     "total_tokens": 170
#   }
# }
```

--------------------------------

### DeepSpeed Configuration Command Line Arguments

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This section shows command-line arguments for utilizing different DeepSpeed ZeRO stages during training. It specifies paths to JSON configuration files for ZeRO-0, ZeRO-2 (with and without CPU offload), and ZeRO-3 (with and without CPU offload). It also includes an example of training with a custom DeepSpeed configuration using `llamafactory-cli`.

```bash
# Use different ZeRO stages based on GPU memory
# ZeRO-0: No optimization (baseline)
--deepspeed scripts/deepspeed_config/ds_z0_config.json

# ZeRO-2: Optimizer state + gradients partitioning (recommended for most cases)
--deepspeed scripts/deepspeed_config/ds_z2_config.json

# ZeRO-2 with CPU offload: For limited GPU memory
--deepspeed scripts/deepspeed_config/ds_z2_offload_config.json

# ZeRO-3: Full parameter partitioning (largest models)
--deepspeed scripts/deepspeed_config/ds_z3_config.json

# ZeRO-3 with CPU offload: Maximum memory efficiency
--deepspeed scripts/deepspeed_config/ds_z3_offload_config.json

# Training with custom DeepSpeed config
llamafactory-cli train \
    --model_name_or_path meta-llama/Llama-3.2-3B \
    --stage sft \
    --deepspeed scripts/deepspeed_config/ds_z2_config.json \
    --dataset ultrachat_200k \
    --output_dir experiments/custom-training
```

--------------------------------

### OpenAI-Compatible API Server

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Deploy trained models as API endpoints compatible with OpenAI's chat completion format. You can start the server using the Python script or the command line interface.

```APIDOC
## Start OpenAI-Compatible API Server

### Description
Deploys trained models as API endpoints compatible with OpenAI's chat completion format.

### Method
GET

### Endpoint
/

### Usage

**Python:**
```python
from llamafactory.api.app import run_api
run_api()
```

**Command Line:**
```bash
python src/api.py
```

**Environment Variables:**
* `API_HOST`: Host address for the API server (default: `0.0.0.0`).
* `API_PORT`: Port for the API server (default: `8000`).
* `API_KEY`: Secret key for API authentication.
* `API_MODEL_NAME`: The name of the model to serve.

### API Endpoints

#### Chat Completions

##### POST /v1/chat/completions

###### Description
Generates chat completions for a given conversation.

###### Method
POST

###### Endpoint
`/v1/chat/completions`

###### Parameters

*   **Path Parameters**
    *   None

*   **Query Parameters**
    *   None

*   **Request Body**
    *   `model` (string) - Required - The ID of the model to use for generation.
    *   `messages` (array of objects) - Required - A list of message objects representing the conversation.
        *   `role` (string) - Required - The role of the message sender (`system`, `user`, or `assistant`).
        *   `content` (string) - Required - The content of the message.
    *   `temperature` (number) - Optional - Controls randomness. Lower values make the output more focused and deterministic.
    *   `top_p` (number) - Optional - Controls diversity via nucleus sampling.
    *   `max_tokens` (integer) - Optional - The maximum number of tokens to generate.
    *   `stream` (boolean) - Optional - Whether to stream back partial progress.

###### Request Example (Non-streaming)
```json
{
  "model": "llama3.2-predpo",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."}, 
    {"role": "user", "content": "Explain quantum computing in simple terms."}
  ],
  "temperature": 0.7,
  "max_tokens": 500,
  "stream": false
}
```

###### Request Example (Streaming)
```json
{
  "model": "llama3.2-predpo",
  "messages": [
    {"role": "user", "content": "Write a short story about AI."}
  ],
  "stream": true
}
```

###### Success Response (200)
*   `id` (string) - The ID of the chat completion.
*   `object` (string) - The type of object returned, usually `chat.completion`.
*   `created` (integer) - Unix timestamp of creation.
*   `model` (string) - The model used for the completion.
*   `choices` (array) - A list of completion choices.
    *   `index` (integer) - Index of the choice.
    *   `message` (object) - The message object.
        *   `role` (string) - The role of the assistant.
        *   `content` (string) - The generated content.
    *   `finish_reason` (string) - The reason the model stopped generating tokens.
*   `usage` (object) - Usage statistics for the request.
    *   `prompt_tokens` (integer) - Number of tokens in the prompt.
    *   `completion_tokens` (integer) - Number of tokens in the completion.
    *   `total_tokens` (integer) - Total tokens used.

###### Response Example
```json
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "llama3.2-predpo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Quantum computing harnesses..."
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 20,
    "completion_tokens": 150,
    "total_tokens": 170
  }
}
```

#### List Available Models

##### GET /v1/models

###### Description
Retrieves a list of available models.

###### Method
GET

###### Endpoint
`/v1/models`

###### Parameters

*   **Path Parameters**
    *   None

*   **Query Parameters**
    *   None

###### Request Example
```bash
curl http://localhost:8000/v1/models \
  -H "Authorization: Bearer your_secret_key"
```

###### Success Response (200)
*   `data` (array) - A list of model objects.
    *   `id` (string) - The ID of the model.
    *   `object` (string) - The type of object, usually `model`.
    *   `created` (integer) - Unix timestamp of creation.
    *   `owned_by` (string) - The owner of the model.

###### Response Example
```json
{
  "data": [
    {
      "id": "llama3.2-predpo",
      "object": "model",
      "created": 1677652288,
      "owned_by": "organization"
    }
  ],
  "object": "list"
}
```
```

--------------------------------

### Alpaca Format Column Mapping Example (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Provides an example of how to map dataset columns to the Alpaca format within `dataset_info.json`. This configuration is essential for correctly interpreting the 'instruction', 'input', 'output', 'system', and 'history' fields during training.

```json
"dataset_name": {
  "file_name": "data.json",
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "response": "output",
    "system": "system",
    "history": "history"
  }
}
```

--------------------------------

### Advanced DPO Training with Custom Hyperparameters

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Performs advanced DPO training with custom hyperparameters, starting from a Llama-3.2-3B-Instruct model and using the UltraFeedback dataset. Key parameters include the SFT model path, dataset, preference beta, batch size, learning rate, warmup ratio, and epochs.

```bash
bash scripts/train_dpo.sh \
    --sft_model_path meta-llama/Llama-3.2-3B-Instruct \
    --dataset ultrafeedback \
    --template llama3 \
    --pref_beta 0.05 \
    --bsz 2 \
    --gradient_accumulation_steps 8 \
    --lr 1e-6 \
    --warmup_ratio 0.1 \
    --epoch 1.0 \
    --output_dir experiments/custom-dpo
```

--------------------------------

### Pre-training Dataset Format (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This format is used for pre-training models, where only the 'text' column is utilized for learning. The example shows a simple JSON array of objects, each containing a 'text' field.

```json
[
  {"text": "document"},
  {"text": "document"}
]
```

--------------------------------

### Get Reward Scores from Chat Model

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This snippet illustrates how to obtain reward scores for a list of texts using the `get_scores` method of a `chat_model`. This is typically used for reward models. The input is a list of strings, and the output is a list of scores.

```python
# Get reward scores (for reward models)
batch_texts = [
    "This is a helpful response.",
    "This is not a helpful response."
]
scores = chat_model.get_scores(batch_texts)
print(f"Scores: {scores}")  # [0.85, 0.23]
```

--------------------------------

### Train Qwen 2.5 7B with SFT

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Fine-tunes a Qwen 2.5 7B model using the UltraChat dataset. This bash script configures the SFT training process, including base model path, dataset, template, GPU usage, batch size, and learning rate, saving the results to a designated output directory.

```bash
bash scripts/train_sft.sh \
    --base_model_path Qwen/Qwen2.5-7B \
    --dataset ultrachat_200k \
    --template qwen \
    --gpus 0,1,2,3,4,5,6,7 \
    --bsz 4 \
    --lr 2.0e-6 \
    --output_dir experiments/qwen2.5-7b-sft
```

--------------------------------

### Train with Standard DPO on UltraFeedback

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Applies Direct Preference Optimization (DPO) to align a Supervised Fine-Tuned (SFT) model with human preferences using the UltraFeedback dataset. This script configures DPO training parameters such as SFT model path, dataset, preference beta, batch size, gradient accumulation steps, learning rate, and GPU allocation.

```bash
bash scripts/train_dpo.sh \
    --sft_model_path experiments/llama3.2-3b-sft \
    --dataset ultrafeedback \
    --template llama3 \
    --pref_beta 0.01 \
    --bsz 1 \
    --gradient_accumulation_steps 16 \
    --lr 6.0e-7 \
    --gpus 0,1,2,3 \
    --output_dir experiments/llama3.2-3b-dpo
```

--------------------------------

### Train LLaMA 3.2 3B with SFT

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Fine-tunes a LLaMA 3.2 3B model using the UltraChat dataset with supervised instruction-following data. This script sets up the training environment, specifying model paths, dataset, template, GPU allocation, batch size, learning rate, and number of epochs. The output is saved to a specified directory.

```bash
bash scripts/train_sft.sh \
    --base_model_path meta-llama/Llama-3.2-3B \
    --dataset ultrachat_200k \
    --template llama3 \
    --gpus 0,1,2,3 \
    --bsz 4 \
    --gradient_accumulation_steps 1 \
    --lr 2.0e-6 \
    --epoch 3.0 \
    --cutoff_len 4096 \
    --output_dir experiments/llama3.2-3b-sft
```

--------------------------------

### Launch Web UI Chat with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Launch a web-based chat interface for model interaction using the `llamafactory-cli webchat` command. Specify the model path and template.

```bash
# Launch web UI for model interaction
llamafactory-cli webchat \
    --model_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3
```

--------------------------------

### Configure Datasets with dataset_info.json

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Define custom dataset configurations for training by specifying Hugging Face hub URLs, formatting types, column mappings, and tag configurations in a JSON file. Supports standard conversational datasets and ranking-based preference datasets.

```json
{
  "ultrachat_200k": {
    "hf_hub_url": "HuggingFaceH4/ultrachat_200k",
    "formatting": "sharegpt",
    "columns": {
      "messages": "messages"
    },
    "tags": {
      "role_tag": "role",
      "content_tag": "content",
      "user_tag": "user",
      "assistant_tag": "assistant"
    }
  },
  "ultrafeedback": {
    "hf_hub_url": "llamafactory/ultrafeedback_binarized",
    "ranking": true,
    "columns": {
      "prompt": "instruction",
      "chosen": "chosen",
      "rejected": "rejected"
    }
  },
  "custom_preference_dataset": {
    "hf_hub_url": "username/custom-dataset",
    "ranking": true,
    "columns": {
      "prompt": "question",
      "chosen": "best_response",
      "rejected": "worst_response"
    }
  }
}
```

--------------------------------

### Initialize ChatModel in Python

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Instantiate the `ChatModel` class in Python to programmatically interact with trained models. Configuration includes model path, template, and inference backend. This allows for both synchronous and asynchronous inference.

```python
from llamafactory.chat import ChatModel

# Initialize chat model
chat_model = ChatModel({
    "model_name_or_path": "experiments/llama3.2-3b-predpo",
    "template": "llama3",
    "infer_backend": "huggingface"
})
```

--------------------------------

### Pre-DPO with SimPO Loss (Pre-DPO-SimPO Variant)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Trains a model using the Pre-DPO-SimPO variant, combining Pre-DPO with SimPO loss. This script requires paths to the SFT and reference models, a binarized preference dataset, and tuning of parameters such as preference beta, learning rate, gradient accumulation steps, and GPU allocation.

```bash
bash scripts/train_predpo.sh \
    --sft_model_path experiments/qwen2.5-7b-sft \
    --ref_model_path experiments/qwen2.5-7b-dpo \
    --dataset qwen2.5-7b-ultrafeedback-armorm-binarized \
    --template qwen \
    --pref_beta 0.05 \
    --lr 1e-6 \
    --gradient_accumulation_steps 32 \
    --gpus 0,1,2,3,4,5,6,7 \
    --output_dir experiments/qwen2.5-7b-predpo
```

--------------------------------

### SimPO Training with Gamma Parameter

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Trains a model using Simple Preference Optimization (SimPO) with a specified gamma parameter for reward shaping. This script uses an SFT model and a binarized preference dataset, configuring SimPO-specific hyperparameters like simpo_gamma and pref_beta, along with standard training settings.

```bash
bash scripts/train_simpo.sh \
    --sft_model_path meta-llama/Llama-3.2-3B-Instruct \
    --dataset llama3.2-3b-ultrafeedback-armorm-binarized \
    --template llama3 \
    --simpo_gamma 1.0 \
    --pref_beta 2.0 \
    --bsz 1 \
    --gradient_accumulation_steps 16 \
    --lr 6.0e-7 \
    --gpus 0,1,2,3 \
    --output_dir experiments/llama3.2-3b-simpo
```

--------------------------------

### Train Supervised Fine-Tuning (SFT) Model

Source: https://github.com/dtyxs/pre-dpo/blob/main/README.md

A script to perform Supervised Fine-Tuning (SFT) on a base model. Requires specifying the model name, dataset, output directory, and template.

```shell
bash scripts/train_sft.sh --model_name_or_path <MODEL_NAME_OR_PATH> --dataset <DATASET_NAME> --output_dir <OUTPUT_DIR> --template <TEMPLATE>
```

--------------------------------

### SimPO with Adjusted Reward Shaping

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Performs SimPO training with adjusted reward shaping parameters. This script configures the SFT model path, dataset, and SimPO hyperparameters, including simpo_gamma and pref_beta, to tailor the reward shaping for the optimization process.

```bash
bash scripts/train_simpo.sh \
    --sft_model_path experiments/qwen2.5-7b-sft \
    --dataset ultrafeedback \
    --template qwen \
    --simpo_gamma 0.5 \
    --pref_beta 1.5 \
    --output_dir experiments/qwen2.5-7b-simpo
```

--------------------------------

### Alpaca Format for Supervised Fine-Tuning (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Illustrates the structure of a supervised fine-tuning dataset in Alpaca format. It specifies required fields like 'instruction' and 'output', and optional fields such as 'input', 'system', and 'history' for customized training.

```json
[
  {
    "instruction": "human instruction (required)",
    "input": "human input (optional)",
    "output": "model response (required)",
    "system": "system prompt (optional)",
    "history": [
      ["human instruction in the first round (optional)", "model response in the first round (optional)"],
      ["human instruction in the second round (optional)", "model response in the second round (optional)"]
    ]
  }
]
```

--------------------------------

### Train Pre-DPO Model

Source: https://github.com/dtyxs/pre-dpo/blob/main/README.md

The main script for training Pre-DPO models. It utilizes an SFT model, a reference model, and a dataset, along with DPO-specific hyperparameters like beta, batch size, gradient accumulation steps, and learning rate.

```shell
bash scripts/train_predpo.sh --sft_model_path <SFT_MODEL_PATH> --ref_model_path <REF_MODEL_PATH> --dataset <DATASET_NAME> --output_dir <OUTPUT_DIR> --template <TEMPLATE> --pref_beta <BETA_IN_PREDPO> --bsz <BATCH_SIZE> --gradient_accumulation_steps <GRADIENT_ACCUMULATION_STEPS> --lr <LEARNING_RATE>
```

--------------------------------

### Train SimPO Model

Source: https://github.com/dtyxs/pre-dpo/blob/main/README.md

A script to train a model using SimPO (Simplified Preference Optimization). It requires the SFT model path, dataset, and hyperparameters including beta, gamma, batch size, gradient accumulation steps, and learning rate.

```shell
bash scripts/train_simpo.sh --sft_model_path <SFT_MODEL_PATH> --dataset <DATASET_NAME> --output_dir <OUTPUT_DIR> --template <TEMPLATE> --pref_beta <BETA_IN_SIMPO> --simpo_gamma <GAMMA_IN_SIMPO> --bsz <BATCH_SIZE> --gradient_accumulation_steps <GRADIENT_ACCUMULATION_STEPS> --lr <LEARNING_RATE>
```

--------------------------------

### Interactive Chat in Terminal with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Engage in interactive chat directly from the terminal using the `llamafactory-cli chat` command. Specify the model path and template for the chat session.

```bash
# Interactive chat in terminal
llamafactory-cli chat \
    --model_name_or_path meta-llama/Llama-3.2-3B-Instruct \
    --template llama3
```

--------------------------------

### Evaluate Trained Model with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Evaluate a trained model using the `llamafactory-cli eval` command. Specify the model path, template, and the evaluation task, such as 'mmlu'.

```bash
# Evaluate trained model
llamafactory-cli eval \
    --model_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --task mmlu
```

--------------------------------

### Multi-turn Conversation with Chat Model

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This snippet demonstrates how to conduct multi-turn conversations using a chat model. It appends user and assistant messages to a list and streams the assistant's response. It requires a `chat_model` object with a `stream_chat` method.

```python
messages = []
while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    messages.append({"role": "user", "content": user_input})

    response_text = ""
    print("Assistant: ", end="", flush=True)
    for chunk in chat_model.stream_chat(messages):
        print(chunk, end="", flush=True)
        response_text += chunk
    print()

    messages.append({"role": "assistant", "content": response_text})
```

--------------------------------

### Dataset Description for Pre-training (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This JSON snippet defines the dataset description for a pre-training dataset. It specifies the file name and maps the 'prompt' column to the 'text' field in the dataset.

```json
"dataset_name": {
  "file_name": "data.json",
  "columns": {
    "prompt": "text"
  }
}
```

--------------------------------

### Model Export and Merging with LlamaFactory CLI (Bash)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This section provides bash commands for exporting and merging trained models using the `llamafactory-cli`. It covers exporting the full model, exporting a quantized model (INT4), pushing a model to the HuggingFace Hub, and exporting with a specific torch dtype (float16).

```bash
# Export full model after training
llamafactory-cli export \
    --model_name_or_path meta-llama/Llama-3.2-3B \
    --adapter_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --export_dir models/llama3.2-3b-predpo-merged \
    --export_size 2 \
    --export_device cpu

# Export quantized model
llamafactory-cli export \
    --model_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --export_dir models/llama3.2-3b-predpo-int4 \
    --export_quantization_bit 4 \
    --export_quantization_type q4_k_m

# Push to HuggingFace Hub
llamafactory-cli export \
    --model_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --export_dir models/temp \
    --export_hub_model_id username/llama3.2-3b-predpo

# Export with specific torch dtype
llamafactory-cli export \
    --model_name_or_path experiments/qwen2.5-7b-predpo \
    --template qwen \
    --export_dir models/qwen2.5-7b-predpo-fp16 \
    --export_size 2 \
    --export_device cuda \
    --torch_dtype float16
```

--------------------------------

### Asynchronous Chat and Streaming with Chat Model

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This snippet shows asynchronous usage patterns for a chat model, including making an asynchronous chat request and streaming responses asynchronously. It relies on Python's `asyncio` library and requires a `chat_model` object with `achat` and `astream_chat` methods.

```python
import asyncio

async def async_chat():
    responses = await chat_model.achat(messages, temperature=0.8)
    return responses[0].response_text

async def async_stream():
    async for chunk in chat_model.astream_chat(messages):
        print(chunk, end="", flush=True)

# Run async functions
asyncio.run(async_chat())
asyncio.run(async_stream())
```

--------------------------------

### Train Direct Preference Optimization (DPO) Model

Source: https://github.com/dtyxs/pre-dpo/blob/main/README.md

A script for Direct Preference Optimization (DPO). It takes the SFT model path, dataset, and various training hyperparameters like beta, batch size, gradient accumulation steps, and learning rate.

```shell
bash scripts/train_dpo.sh --sft_model_path <SFT_MODEL_PATH> --dataset <DATASET_NAME> --output_dir <OUTPUT_DIR> --template <TEMPLATE> --pref_beta <BETA_IN_DPO> --bsz <BATCH_SIZE> --gradient_accumulation_steps <GRADIENT_ACCUMULATION_STEPS> --lr <LEARNING_RATE>
```

--------------------------------

### Dataset Description for ShareGPT SFT (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This JSON snippet configures the dataset description for a ShareGPT formatted dataset used for Supervised Fine-Tuning (SFT). It specifies the file name, formatting type, and maps 'messages', 'system', and 'tools' columns.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "system": "system",
    "tools": "tools"
  }
}
```

--------------------------------

### Dataset Description for Preference Data (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This JSON snippet configures the dataset description for a preference dataset. It includes the file name, sets 'ranking' to true, and maps 'prompt', 'query', 'chosen', and 'rejected' columns.

```json
"dataset_name": {
  "file_name": "data.json",
  "ranking": true,
  "columns": {
    "prompt": "instruction",
    "query": "input",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}
```

--------------------------------

### OpenAI Format Configuration (dataset_info.json)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Configuration for OpenAI format datasets within `dataset_info.json`. Uses 'sharegpt' formatting and maps message columns, along with specific tags for roles and content.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "messages"
  },
  "tags": {
    "role_tag": "role",
    "content_tag": "content",
    "user_tag": "user",
    "assistant_tag": "assistant",
    "system_tag": "system"
  }
}
```

--------------------------------

### DeepSpeed ZeRO Optimization Configuration (JSON)

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

This JSON configuration defines DeepSpeed's ZeRO optimization stages for distributed training. It includes settings for batch size, gradient accumulation, FP16/BF16 precision, and specific ZeRO stage configurations (e.g., stage 2 with various partitioning and offloading options).

```json
{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "zero_allow_untested_optimizer": true,
  "fp16": {
    "enabled": false
  },
  "bf16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": 5e8,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e8,
    "overlap_comm": false,
    "contiguous_gradients": true
  }
}
```

--------------------------------

### Python ChatModel API

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Programmatically interact with trained models using the ChatModel class for synchronous and asynchronous inference.

```APIDOC
## Python ChatModel API

### Description
Provides a Python interface to interact with LlamaFactory models for chat-based tasks, supporting both synchronous and streaming responses.

### Usage

**Initialization:**
```python
from llamafactory.chat import ChatModel

chat_model = ChatModel({
    "model_name_or_path": "path/to/your/model",
    "template": "your_template_name",
    "infer_backend": "huggingface" # or "vllm", "llama.cpp"
})
```

**Synchronous Chat:**
```python
messages = [
    {"role": "user", "content": "What is machine learning?"}
]
responses = chat_model.chat(
    messages,
    temperature=0.7,
    top_p=0.9,
    max_tokens=200
)

# Access the response text
print(responses[0].response_text)
```

**Streaming Chat:**
```python
messages = [
    {"role": "system", "content": "You are a coding assistant."} 
    {"role": "user", "content": "Write a Python function to calculate fibonacci numbers."} 
]

for chunk in chat_model.stream_chat(messages):
    print(chunk, end="", flush=True)
print()
```

### Parameters

**`ChatModel` Initialization:**
*   `model_name_or_path` (str): Path or Hugging Face identifier for the model.
*   `template` (str): The template to use for formatting prompts.
*   `infer_backend` (str): The inference backend to use (`huggingface`, `vllm`, `llama.cpp`).
*   Other arguments passed will be forwarded to the underlying model loader.

**`chat` and `stream_chat` methods:**
*   `messages` (list[dict]): A list of message dictionaries, where each dictionary has `role` (`user`, `assistant`, `system`) and `content` (str).
*   `temperature` (float, optional): Controls randomness. Defaults to `0.7`.
*   `top_p` (float, optional): Nucleus sampling parameter. Defaults to `0.9`.
*   `max_tokens` (int, optional): Maximum number of tokens to generate. Defaults to `None`.
*   `stop` (list[str], optional): A list of strings to stop generation when encountered.
*   `stream` (bool, optional): If `True`, the method will yield chunks of text. This is implicitly handled by `stream_chat`.

### Response Object (`responses[0]` in synchronous chat)
*   `response_text` (str): The full response text from the model.
```

--------------------------------

### Export Merged Model with llamafactory-cli

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Export a merged model by combining LoRA adapters with the base model using the `llamafactory-cli export` command. Requires specifying base model, adapter path, template, and export directory.

```bash
# Export merged model (LoRA adapters + base model)
llamafactory-cli export \
    --model_name_or_path meta-llama/Llama-3.2-3B \
    --adapter_name_or_path experiments/llama3.2-3b-predpo \
    --template llama3 \
    --export_dir models/llama3.2-3b-predpo-merged
```

--------------------------------

### Multimodal Video Dataset Configuration (dataset_info.json)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Configuration for multimodal video datasets within `dataset_info.json`. Specifies file name, formatting as 'sharegpt', and column mappings for messages and videos.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "videos": "videos"
  }
}
```

--------------------------------

### Dataset Description for ShareGPT KTO (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This JSON snippet configures the dataset description for a ShareGPT formatted KTO dataset. It specifies the file name, formatting type, and maps 'messages' and 'kto_tag' columns.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "kto_tag": "kto_tag"
  }
}
```

--------------------------------

### ShareGPT Format for Supervised Fine-Tuning (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

The ShareGPT format supports multi-turn conversations with roles like 'human', 'gpt', 'observation', and 'function'. The 'conversations' column is a list of message objects, and optional 'system' and 'tools' fields can be included.

```json
[
  {
    "conversations": [
      {
        "from": "human",
        "value": "human instruction"
      },
      {
        "from": "function_call",
        "value": "tool arguments"
      },
      {
        "from": "observation",
        "value": "tool result"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "system": "system prompt (optional)",
    "tools": "tool description (optional)"
  }
]
```

--------------------------------

### Multimodal Video Dataset Structure (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Defines the JSON structure for multimodal video datasets. Requires a 'videos' column with paths matching '<video>' tokens in conversations. The number of videos must correspond to the number of '<video>' tokens.

```json
[
  {
    "conversations": [
      {
        "from": "human",
        "value": "<video>human instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "videos": [
      "video path (required)"
    ]
  }
]
```

--------------------------------

### Dataset Information Schema (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Defines the structure for dataset configuration in `dataset_info.json`. It includes parameters for specifying dataset sources (Hugging Face Hub, Model Scope Hub), file locations, formatting, and detailed column mappings for various data fields.

```json
{
  "dataset_name": {
    "hf_hub_url": "the name of the dataset repository on the Hugging Face hub. (if specified, ignore script_url and file_name)",
    "ms_hub_url": "the name of the dataset repository on the Model Scope hub. (if specified, ignore script_url and file_name)",
    "script_url": "the name of the directory containing a dataset loading script. (if specified, ignore file_name)",
    "file_name": "the name of the dataset folder or dataset file in this directory. (required if above are not specified)",
    "formatting": "the format of the dataset. (optional, default: alpaca, can be chosen from {alpaca, sharegpt})",
    "ranking": "whether the dataset is a preference dataset or not. (default: False)",
    "subset": "the name of the subset. (optional, default: None)",
    "split": "the name of dataset split to be used. (optional, default: train)",
    "folder": "the name of the folder of the dataset repository on the Hugging Face hub. (optional, default: None)",
    "num_samples": "the number of samples in the dataset to be used. (optional, default: None)",
    "columns (optional)": {
      "prompt": "the column name in the dataset containing the prompts. (default: instruction)",
      "query": "the column name in the dataset containing the queries. (default: input)",
      "response": "the column name in the dataset containing the responses. (default: output)",
      "history": "the column name in the dataset containing the histories. (default: None)",
      "messages": "the column name in the dataset containing the messages. (default: conversations)",
      "system": "the column name in the dataset containing the system prompts. (default: None)",
      "tools": "the column name in the dataset containing the tool description. (default: None)",
      "images": "the column name in the dataset containing the image inputs. (default: None)",
      "videos": "the column name in the dataset containing the videos inputs. (default: None)",
      "chosen": "the column name in the dataset containing the chosen answers. (default: None)",
      "rejected": "the column name in the dataset containing the rejected answers. (default: None)",
      "kto_tag": "the column name in the dataset containing the kto tags. (default: None)"
    },
    "tags (optional, used for the sharegpt format)": {
      "role_tag": "the key in the message represents the identity. (default: from)",
      "content_tag": "the key in the message represents the content. (default: value)",
      "user_tag": "the value of the role_tag represents the user. (default: human)",
      "assistant_tag": "the value of the role_tag represents the assistant. (default: gpt)",
      "observation_tag": "the value of the role_tag represents the tool results. (default: observation)",
      "function_tag": "the value of the role_tag represents the function call. (default: function_call)",
      "system_tag": "the value of the role_tag represents the system prompt. (default: system, can override system column)"
    }
  }
}
```

--------------------------------

### Preference Dataset Format (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Preference datasets are designed for reward modeling and DPO, ORPO, and SimPO training. They require 'chosen' (better response) and 'rejected' (worse response) columns, along with an optional 'input' and a required 'instruction'.

```json
[
  {
    "instruction": "human instruction (required)",
    "input": "human input (optional)",
    "chosen": "chosen answer (required)",
    "rejected": "rejected answer (required)"
  }
]
```

--------------------------------

### Perform Synchronous Chat with ChatModel API

Source: https://context7.com/dtyxs/pre-dpo/llms.txt

Execute synchronous chat requests using the `chat` method of the `ChatModel` instance. Pass a list of message dictionaries, and optionally control generation parameters like temperature, top_p, and max_tokens. The response text is accessed via `response_text`.

```python
# Synchronous chat
messages = [
    {"role": "user", "content": "What is machine learning?"}
]
responses = chat_model.chat(messages, temperature=0.7, top_p=0.9, max_tokens=200)
print(responses[0].response_text)
```

--------------------------------

### Dataset Description for ShareGPT Preference Data (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

This JSON snippet configures the dataset description for a ShareGPT formatted preference dataset. It includes the file name, formatting type, sets 'ranking' to true, and maps 'messages', 'chosen', and 'rejected' columns.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "ranking": true,
  "columns": {
    "messages": "conversations",
    "chosen": "chosen",
    "rejected": "rejected"
  }
}
```

--------------------------------

### Multimodal Image Dataset Configuration (dataset_info.json)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

Configuration for multimodal image datasets within `dataset_info.json`. Specifies file name, formatting as 'sharegpt', and column mappings for messages and images.

```json
"dataset_name": {
  "file_name": "data.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations",
    "images": "images"
  }
}
```

--------------------------------

### ShareGPT Format for KTO (JSON)

Source: https://github.com/dtyxs/pre-dpo/blob/main/data/README.md

The KTO ( কিনা ট্যাগিং অবজেক্টিভ) dataset format in ShareGPT requires an additional 'kto_tag' column for human feedback (boolean true/false). The 'conversations' column contains the dialogue history.

```json
[
  {
    "conversations": [
      {
        "from": "human",
        "value": "human instruction"
      },
      {
        "from": "gpt",
        "value": "model response"
      }
    ],
    "kto_tag": "human feedback [true/false] (required)"
  }
]
```