### Environment Setup and Installation

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/en/text_to_text_dpo.ipynb

Commands to clone the repository, create and activate a conda environment, install CUDA, and install the align-anything package with optional dependencies for training or evaluation.

```bash
git clone git@github.com:PKU-Alignment/align-anything.git
cd align-anything

conda create -n align-anything python==3.11
conda activate align-anything

conda install nvidia/label/cuda-12.2.0::cuda
export CUDA_HOME=$CONDA_PREFIX

# Or if CUDA is installed elsewhere:
# export CUDA_HOME="/usr/local/cuda"

pip install -e .[train] # Install training dependencies
pip install -e .[evaluate] # Install evaluation dependencies

# Install all dependencies:
# pip install -e .[all]
```

--------------------------------

### Environment Setup and Installation

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/zh/text_to_text_dpo.ipynb

Commands to clone the repository, create and activate a conda environment, install CUDA, and install the align-anything package with training and evaluation dependencies.

```bash
git clone git@github.com:PKU-Alignment/align-anything.git
cd align-anything

conda create -n align-anything python==3.11
conda activate align-anything

conda install nvidia/label/cuda-12.2.0::cuda
export CUDA_HOME=$CONDA_PREFIX

# Or if CUDA is installed elsewhere:
# export CUDA_HOME="/usr/local/cuda"

pip install -e .[train]
pip install -e .[evaluate]

# Or install all dependencies:
# pip install -e .[all]
```

--------------------------------

### Start Remote Reward Server Example

Source: https://github.com/pku-alignment/align-anything/blob/main/align_anything/models/remote_rm/README.md

This script demonstrates how to start the rule-based remote reward server. It sets environment variables for the reward port, type, output directory, and dataset path, then launches the server in the background. It also includes checks to ensure the server starts successfully.

```shell
export REWARD_PORT=6000
export REWARD_TYPE="math_verifier"  # Optional: example_math, example_coding, example_safety, math_verifier(For Zero RL)
export OUTPUT_DIR="./debug_logs"
export DATASET_PATH="./math_verify_dataset/mathvl_345_example.json"
# Ensure the output directory exists
mkdir -p $OUTPUT_DIR

# NOTE The golden dataset should in the keys (question, answer)
# NOTE But when you use the reward server, you should use the keys (prompts, responses)

# 0. Check if the reward port is available
if REWARD_SERVER_PID=$(lsof -t -i:$REWARD_PORT); then
    echo "Remote reward server is already running on port $REWARD_PORT."
    echo "Killing the existing server..."
    kill $REWARD_SERVER_PID
fi

# 1. Start remote reward server (run in the background)
echo "Start remote reward server..."
python -m align_anything.models.remote_rm.run_reward_server \
    --reward-type $REWARD_TYPE \
    --port $REWARD_PORT \
    --dataset $DATASET_PATH \
    > $OUTPUT_DIR/reward_server.log 2>&1 &


REWARD_SERVER_PID=$!
echo "Reward server process ID: $REWARD_SERVER_PID"

# 2. Check the log to confirm if the server started successfully
TIMEOUT=20
echo "Waiting $TIMEOUT s for the reward server to start..."
sleep $TIMEOUT
if grep -q "Running on" $OUTPUT_DIR/reward_server.log; then
    echo "Reward server started successfully."
    cat $OUTPUT_DIR/reward_server.log
else
    echo "Failed to start reward server. Check the log for details."
    cat $OUTPUT_DIR/reward_server.log
    kill $REWARD_SERVER_PID
fi

```

--------------------------------

### Batch Inference Setup

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Instructions for setting up batch inference using a forked and revised version of the Chameleon repository. This involves cloning the repository, installing dependencies, and running the generation script.

```bash
git clone https://github.com/htlou/mmsg_chameleon.git
cd mmsg_chameleon
pip install -e .
bash scripts/interleaved_gen.sh
```

--------------------------------

### Install Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/README.md

Installs the necessary Python dependencies for building the documentation using pip.

```bash
# suppose you are in the root dir of align-anything
cd ./docs
# make sure your python env is activated
pip install -r ./requirements.txt
```

--------------------------------

### Batch Inference Setup and Execution

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Instructions for setting up and running batch inference using a separate repository (`mmsg_chameleon`). This involves cloning the repository, installing dependencies, configuring paths in `scripts/interleaved_gen.sh`, and then executing the script.

```bash
git clone https://github.com/htlou/mmsg_chameleon.git
cd mmsg_chameleon
pip install -e .
```

```bash
bash scripts/interleaved_gen.sh
```

--------------------------------

### YAML Training Configurations Example

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/training/configs.rst

This snippet demonstrates the structure of training configurations in YAML format, including settings for training, data, logging, LoRA, and quantization.

```yaml
# The training configurations
train_cfgs:
    # The deepspeed configuration
    ds_cfgs: ds_z3_config.json
    # Number of training epochs
    epochs: 3
    # Seed for random number generator
    seed: 42
    ...
# The datasets configurations
data_cfgs:
    # Datasets to use for training
    train_datasets: null
    # The format template for training
    train_template: null
    ...
# The logging configurations
logger_cfgs:
    # Type of logging to use, choosing from [wandb, tensorboard]
    log_type: wandb
    # Project name for logging
    log_project: align-anything
    ...
# The LoRA configurations
lora_cfgs:
    # Whether to use LoRA
    use_lora: False
    # Task type for LoRA configuration
    task_type: TaskType.CAUSAL_LM
    ...
# The QLoRA configurations
bnb_cfgs:
    # Whether to use BNB(For QLoRA)
    use_bnb: False
    # Whether to use 4-bit quantization
    load_in_4bit: True
    ...
```

--------------------------------

### Clone Repository and Setup Environment

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Instructions for cloning the Align-Anything repository and setting up a Python 3.11 virtual environment using Conda.

```bash
git clone git@github.com:PKU-Alignment/align-anything.git
cd align-anything

conda create -n align-anything python==3.11
conda activate align-anything
```

--------------------------------

### Pastry Recipe Generation Example

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Illustrates generating a detailed recipe for a pastry shown in an image, including ingredients, preparation, and baking instructions. The example focuses on a chocolate-filled donut.

```text
Input text: Create a detailed recipe for making the pastry shown in the image, including ingredients, preparation steps, and baking instructions. <image>
Input image: ![image](../../assets/text_image_to_text_image/baking.jpg)
> Output text: <image>

Cooking Instructions:
1. Preheat Oven: Preheat your oven to 350°F (175°C).
2. Prepare Filling: In a small bowl, mix well the flour, sugar, eggs, and any other desired ingredients. To make it easier to handle, you can place the bowl of the mixture on a baking tray lined with parchment paper.
3. Bake Donut: Place the prepared donut on the preheated baking tray. Bake for 8-10 minutes until the donut is golden brown and crispy on top.
4. Cool and Serve: Allow the donut to cool slightly before serving. You can garnish it with a drizzle of chocolate sauce or caramel sauce if desired.

Enjoy your homemade chocolate-filled donut!
> Output image: ![image](../../assets/text_image_to_text_image/baking_generated.png)
```

--------------------------------

### Pre-tokenize Parallel Example Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Runs a Python script for parallel pre-tokenization, designed to speed up the process for large datasets.

```bash
python pre_tokenize_parallel_example.py
```

--------------------------------

### Prompt Only Pre-tokenization Example (Python)

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Example script for pre-tokenizing datasets specifically for prompt-only scenarios (e.g., PPO). Users need to configure the template, input key name, and process/GPU counts. Parallel pre-tokenization is enabled by default.

```python
python prompt_only_tokenize_example.py --model_path $MODEL_PATH --input_path $INPUT_PATH --output_path $OUTPUT_PATH --cache_dir $CACHE_DIR
```

--------------------------------

### LLaVA Specific Model Loading Example

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/index.rst

Provides an example of loading a LLaVA model specifically, highlighting that different multimodal models might require different classes from the transformers library. For LLaVA, `LlavaForConditionalGeneration` is used.

```python
model_id = "llava-hf/llava-1.5-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
```

--------------------------------

### Full Model Training Setup and Loop

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/en/text_image_to_text_sft.ipynb

This comprehensive Python script sets up the entire training environment for the Align-Anything project. It includes loading pre-trained models (LLaVA), configuring the optimizer, preparing the dataset and dataloader, and executing the training loop with progress monitoring and model checkpointing. Dependencies include PyTorch, Transformers, tqdm, and other utilities from the align_anything library.

```python
import os
from tqdm import tqdm
from collections import deque
import numpy as np

from torch.utils.data import DataLoader
from torch.utils.data.sampler import RandomSampler
from torch.optim import AdamW

from align_anything.models.pretrained_model import load_pretrained_models
from align_anything.datasets.text_image_to_text import SupervisedDataset
from align_anything.configs.template import ChatTemplate
from align_anything.utils.multi_process import get_current_device


model, tokenizer, processor = load_pretrained_models(
    "/path/to/llava-1.5-7b-hf",
    model_max_length=4096,
    padding_side='right',
    trust_remote_code=True,
    modality=['image'],
)

model = model.to(get_current_device())

optimizer = AdamW(model.parameters(), lr=1e-5)

train_template = ChatTemplate(
    formatter=processor,
    template="AA_TI2T",
)

train_dataset = SupervisedDataset(
    path="/path/to/Align-Anything-TI2T-Instruction-100K",  # Replace with your dataset path
    template=train_template,
    tokenizer=tokenizer,
    processor=processor,
    split="train",
    size=1000,  # Limit to 1000 samples for this tutorial
)

train_dataloader = DataLoader(
    train_dataset,
    collate_fn=train_dataset.get_collator(),
    sampler=RandomSampler(train_dataset),
    batch_size=1,
)

progress_bar = tqdm(range(3*len(train_dataloader)), desc="Training for 1/3 epochs આપના")
losses = deque(maxlen=100)
os.makedirs('./output', exist_ok=True)

for epoch in range(3):
    progress_bar.set_description(f"Training for {epoch+1}/3 epochs આપના")
    for batch in train_dataloader:
        batch.pop('meta_info')
        model.train()
        loss = model(**batch)['loss']
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(loss.item())
        progress_bar.update(1)
        progress_bar.set_postfix(loss=np.mean(losses))

    # Save the model after each epoch
    model.save_pretrained('./output')
    tokenizer.save_pretrained('./output')
    processor.save_pretrained('./output')
```

--------------------------------

### Command-Line Training Execution Example

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/training/configs.rst

This bash script demonstrates how to set environment variables for model path, datasets, template, output directory, and WANDB API key, and then execute the training script using deepspeed.

```bash
MODEL_NAME_OR_PATH="llava-hf/llava-1.5-7b-hf" # model path
TRAIN_DATASETS="sqrti/SPA-VL" # dataset path
TRAIN_TEMPLATE="SPA_VL" # dataset template
TRAIN_SPLIT="train" # split the dataset
OUTPUT_DIR="../output/dpo" # output dir
export WANDB_API_KEY="YOUR_WANDB_KEY" # wandb logging

source ./setup.sh # source the setup script

export CUDA_HOME=$CONDA_PREFIX # replace it with your CUDA path

deepspeed \
    --master_port ${MASTER_PORT} \
    --module align_anything.trainers.text_image_to_text.dpo \
    --model_name_or_path ${MODEL_NAME_OR_PATH} \
    --train_datasets ${TRAIN_DATASETS} \
    --train_template ${TRAIN_TEMPLATE} \
    --train_split ${TRAIN_SPLIT} \
    --output_dir ${OUTPUT_DIR}
```

--------------------------------

### Install Align-Anything and Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Installs the Align-Anything package in editable mode and the vllm library for PPO training with vllm engine.

```bash
pip3 install -e .

pip3 install vllm==0.7.2 # to run ppo on vllm engine
```

--------------------------------

### Prompt Only Tokenize Example Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Runs a Python script for pre-tokenizing prompt-only datasets, commonly used for PPO training.

```bash
python prompt_only_tokenize_example.py
```

--------------------------------

### Install Ascend Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Installs the necessary dependencies for running the project on Huawei Ascend NPU. This command should be executed in the project's root directory.

```bash
pip3 install -e .[ascend]
```

--------------------------------

### Setup Docker for Ascend Environment

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

This script helps in setting up a Docker container with a pre-configured environment for Huawei Ascend NPU. It automates the process of pulling the Docker image and creating a container with all necessary dependencies and configurations.

```bash
cd scripts
bash setup_docker.sh
```

--------------------------------

### GPT-based Evaluation Setup

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Instructions for setting up GPT-based evaluation for text-image interleaved messages using a separate repository. This involves cloning the repository, configuring the input type, and running the evaluation script.

```bash
git clone https://github.com/htlou/gpt4_eval.git
cd gpt4_eval
bash script.sh
```

--------------------------------

### Install Forked Transformers Repo

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Installs a specific fork of the Hugging Face Transformers repository required for Chameleon model compatibility with image output.

```bash
pip install git+https://github.com/htlou/transformers.git@hantao_stable_cham
```

--------------------------------

### Generation Example 2: Pastry Recipe Creation

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Illustrates the model's ability to generate a detailed recipe for a pastry shown in an image. It includes input text and image, and the output text and image before and after alignment, detailing ingredients and instructions.

```APIDOC
Example #: 2
Input Text: Create a detailed recipe for making the pastry shown in the image, including ingredients, preparation steps, and baking instructions.
Input Image: |image2|

Output (before alignment):
Output Text (before alignment): Absolutely! Here is a detailed recipe for making the pastry in the image:\n\nIngredients:\n\n* 1 cup all-purpose flour
* 1/2 cup cold unsalted butter, cut into small pieces
* 1/4 cup ice-cold water
* 1 egg, lightly beaten
* 1 tablespoon honey
* 1/4 teaspoon salt
Output Image (before alignment): None

Output (after alignment):
Output Text (after alignment): <image>\n\nCooking Instructions:\n1. Preheat Oven: Preheat your oven to 350°F (175°C).
2. Prepare Filling: In a small bowl, mix well the flour, sugar, eggs, and any other desired ingredients. To make it easier to handle, you can place the bowl of the mixture on a baking tray lined with parchment paper.
3. Bake Donut: Place the prepared donut on the preheated baking tray. Bake for 8-10 minutes until the donut is golden brown and crispy on top.
4. Cool and Serve: Allow the donut to cool slightly before serving. You can garnish it with a drizzle of chocolate sauce or caramel sauce if desired.\n\nEnjoy your homemade chocolate-filled donut!
Output Image (after alignment): |image3|
```

--------------------------------

### Install Text-to-Audio Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Installs the dependencies required for text-to-audio functionality within the project.

```python
pip install -e .[text-to-audio]
```

--------------------------------

### Text-Image-to-Text-Image Project Setup

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/README.md

This project provides environment setup and scripts for text-image-to-text-image alignment tasks, including SFT, DPO, RM, and PPO for the Chameleon model. It includes a script for pre-processing data before training.

```python
pre_tokenize_example.py
```

--------------------------------

### Generic Dataset Key-Value Parsing

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/index.rst

Demonstrates a generic key-value parsing example for datasets where the prompt and answer are directly available, formatting them into a conversation list with text content.

```python
prompt = raw_sample['prompt']
answer = raw_sample['response']

return [
      {'role': 'user', 'content': [{'type': 'text', 'text': prompt}]},
      {'role': 'assistant', 'content': [{'type': 'text', 'text': answer}]},
], {}
```

--------------------------------

### Pre-tokenization Example (Python)

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Example script for pre-tokenizing datasets for text-to-image models. It supports parallel pre-tokenization for speed. Dependencies include Python and necessary libraries for model loading and data processing.

```python
python preference_tokenize_example.py --model_path $MODEL_PATH --input_path $INPUT_PATH --output_path $OUTPUT_PATH --cache_dir $CACHE_DIR
```

--------------------------------

### Gourmet Pizza Presentation Enhancement Example

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Shows how to enhance pizza presentation with gourmet toppings and plating, along with a description of the steps taken. This example adds basil, prosciutto, tomatoes, Parmesan, and balsamic glaze.

```text
Input text: Enhance the presentation of these pizzas with gourmet toppings and plating, and describe the steps involved. <image>
Input image: ![image](../../assets/text_image_to_text_image/pizza.jpg)
> Output text: <image>I have enhanced the presentation of the pizzas by adding gourmet toppings. The pizzas now feature fresh basil leaves, prosciutto, cherry tomatoes, shaved Parmesan cheese, and a drizzle of balsamic glaze. The toppings are arranged artistically on each pizza to create a visually appealing and gourmet presentation.
> Output image: ![image](../../assets/text_image_to_text_image/pizza_generated.png)
```

--------------------------------

### Example Training Data Format (JSON)

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/training/any_to_any.rst

This JSON structure defines the format for training data used in any-to-any tasks. It includes examples for both 'TU' (task understanding) and 'TG' (task generation) modes, specifying input text, input image, and expected output text or image.

```JSON
[
    {
        "mode": "TU",
        "input_text": "Please tell me what you see in this image.",
        "input_image": "example.jpg",
        "output_text": "I see an old British train coming towards the station, and a group of people waiting for it."
    },
    {
        "mode": "TG",
        "input_text": "Generate an image where an old british train is coming towards the station, and a group of people is waiting for it.",
        "output_image": "example.jpg"
    }
]
```

--------------------------------

### Preference Tokenize Example Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Executes a Python script for pre-tokenizing preference datasets, typically used for DPO or RM training.

```bash
python preference_tokenize_example.py
```

--------------------------------

### Pre-tokenize Example Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Executes a Python script for pre-tokenizing input data for Chameleon models. This is a prerequisite for training.

```bash
python pre_tokenize_example.py
```

--------------------------------

### Load Align-Anything Instruction Dataset

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/data/text_to_text.md

This snippet demonstrates how to load the Align-Anything-Instruction-100K dataset using the `datasets` library in Python. It requires the `datasets` library to be installed.

```Python
from datasets import load_dataset

dataset = load_dataset("PKU-Alignment/Align-Anything-Instruction-100K")
```

--------------------------------

### Alpaca Dataset Key-Value Parsing

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/index.rst

Provides an example of key-value parsing for the Alpaca dataset, combining 'instruction' and 'input' into a 'user' role prompt and using 'output' as the 'assistant' response.

```python
prompt = ' '.join((raw_sample['instruction'], raw_sample['input']))
response = raw_sample['output']
return [
      {'role': 'user', 'content': prompt},
      {'role': 'assistant', 'content': response},
], {}
```

--------------------------------

### Slurm Cluster Migration Support

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Announces the support for seamless migration to Slurm clusters, providing a link to an example for getting started.

```en
We have supported seamless migration to Slurm clusters! Check out our example [here](#training-on-slurm) to get started.
```

--------------------------------

### Install Align-Anything Janus Variant

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/janus/README.md

Installs the forked stable version of the Janus repository for Align-Anything. This involves cloning the repository and installing it in editable mode.

```bash
pip install git+https://github.com/htlou/Align_Anything_Janus.git
cd Align_Anything_Janus
pip install -e .
```

--------------------------------

### Install Minicpmo Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Installs the dependencies required for the minicpmo functionality within the project.

```python
pip install -e .[minicpmo]
```

--------------------------------

### Install Minicpmv Dependencies

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Installs the dependencies required for the minicpmv functionality within the project.

```python
pip install -e .[minicpmv]
```

--------------------------------

### AccustomedMiniCPMV Initialization and Inference

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/index.rst

Shows the custom `AccustomedMiniCPMV` class for models like MiniCPM-V, including its `__init__` method to handle tokenizer loading and a check for ZeRO stage 3 compatibility, and an `infer_batch` method for custom inference.

```python
class AccustomedMiniCPMV(MiniCPMV):
   """Accustomed Interface for MiniCPM-V model"""

   def __init__(self, config: AutoConfig):
      super().__init__(config)
      zero_stage = int(os.environ['ZERO_STAGE'])
      if zero_stage == 3:
            raise ValueError('MiniCPM-V does not support ZeRO stage 3')
      self.tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)

   @property
   def processor_available(self):
      return False

   def infer_batch(self, batch: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
      """Return the dict used for model inference"""
      return {
            'input_ids': batch['input_ids'],
            'attention_mask': batch['attention_mask'],
            'images': batch['images'],
            'labels': batch.get('labels'),
      }

   ...
```

--------------------------------

### Build Documentation with Custom Port

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/README.md

Builds the documentation locally using Sphinx's autbuilder and specifies a custom port for the local server.

```bash
sphinx-autobuild --port 8080 source source/_build/html
```

--------------------------------

### Installation Enhancements for Ascend NPU

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Informs about enhancements to the installation process for Huawei Ascend NPU, including support for pre-set Docker images.

```en
We support the installation of **Huawei Ascend** dependencies through pre-set Docker image.
```

--------------------------------

### Install Forked Transformers

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Installs a specific fork of the Hugging Face Transformers library required for Chameleon model compatibility, as the official repository does not yet support image output.

```bash
pip install git+https://github.com/htlou/transformers.git@hantao_stable_cham
```

--------------------------------

### Set Up DataLoader

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/en/text_image_to_text_sft.ipynb

Creates a DataLoader for the training dataset. It uses a custom collate function from the dataset, a random sampler, and sets the batch size to 1 for processing individual samples.

```python
from torch.utils.data import DataLoader
from torch.utils.data.sampler import RandomSampler

# Create a DataLoader for our training dataset
train_dataloader = DataLoader(
    train_dataset,
    collate_fn=train_dataset.get_collator(),  # Custom collate function for our dataset
    sampler=RandomSampler(train_dataset),     # Randomly sample data
    batch_size=1,                             # Process one sample at a time
)
```

--------------------------------

### Install CUDA Toolkit

Source: https://github.com/pku-alignment/align-anything/blob/main/README.md

Steps to install the CUDA toolkit within a Conda environment, essential for GPU acceleration. It also shows how to set the CUDA_HOME environment variable.

```bash
# We tested on the H800 computing cluster, and this version of CUDA works well.
# You can adjust this version according to the actual situation of the computing cluster.

conda install nvidia/label/cuda-12.2.0::cuda
export CUDA_HOME=$CONDA_PREFIX

# If your CUDA installed in a different location, such as /usr/local/cuda/bin/nvcc, you can set the environment variables as follows:

export CUDA_HOME="/usr/local/cuda"
```

--------------------------------

### Pizza Presentation Steps (Before Alignment)

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Details the steps for preparing and presenting homemade pizzas with gourmet toppings, focusing on ingredient quality, preparation techniques, and visual plating before any alignment process is applied.

```text
The steps involved in presenting these homemade pizzas with gourmet toppings and plating them are as follows:

1. **Gather Ingredients:**
   - Use high-quality, fresh ingredients: chees, vegetables, fruits, and meats.
   - Avoid using pre-concealed toppings: Instead, let your creations shine.

2. **Mix Ingredients:**
   - Mix all ingredients thoroughly but avoid mishandling them.
   - Cover the mixing bowl with cling wrap to prevent cross-contamination.

3. **Set Up the Presentation:**
   - Transfer the mixture to a clean, oven-proof dish and preheat the oven to 400°F (200°C).
   - Cut the pizza into appropriate portions and allow them to cool a few minutes.
   - Plate the pizzas on a clean, clear surface to showcase their presentation.

4. **Garnish and Presentation:**
   - Add desired garnishes and toppings to add visual appeal and flavor.
   - Secure the plating board with a clear, decorative overlay to protect the food from elements like moisture.

5. **Serve and Enjoy:**
   - Ensure the pizzas are hot and fresh to avoid any bacterial risks.
   - Smooth any uneven edges and serve with a complementary dipping sauce or condiments.

I've described the steps involved in preparing these homemade pizzas and their plating process.
```

--------------------------------

### Format Prompt-Only Sample (AA_TA2T)

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/index.rst

Formats a single data sample for prompt-only alignment algorithms like PPO. It creates a conversation list containing only the system prompt and the user's prompt, which includes audio content. Returns a conversation list and metadata containing the audio path.

```python
@register_template('AA_TA2T')
class AA_TA2T(BaseFormatter):
   system_prompt: str = 'You are a helpful assistant.'

   def format_prompt_only_sample(self, raw_sample: dict[str, Any]) -> dict[str, Any]:
      prompt = raw_sample['prompt']
      audio_path = raw_sample['audio_path']

      conversation = [
            {'role': 'system', 'content': [{'type': 'text', 'text': self.system_prompt}]},
            {'role': 'user', 'content': [
                  {'type': 'audio', 'audio_url': audio_path},
                  {'type': 'text', 'text': prompt},
               ]},
      ]

      return conversation, {'audio_path': audio_path}
```

--------------------------------

### Environment and Data Handling

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/spelling_wordlist.txt

Details on setting up and interacting with RL environments, including configuration files and data preprocessing. This covers aspects like state representation, action spaces, and reward scaling.

```python
from envs import Env
from configs import config
from utils.dataset import Dataset
from utils.scaler import Scaler

# Example usage:
# env = Env(env_name='HalfCheetah-v2')
# data = Dataset(path='data.npz')
# scaler = Scaler(data.get_mean(), data.get_stddev())
```

--------------------------------

### Audio Datasets

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/evaluation/overview.md

Datasets related to audio processing. AudioCaps is mentioned as an example.

```APIDOC
Dataset Name: AudioCaps
Source: https://huggingface.co/datasets/AudioLLMs/audiocaps_test
```

--------------------------------

### Text-to-Image-to-Text (RM) Training Configuration

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

This snippet shows the command-line arguments for configuring and running the text-to-image-to-text (RM) training process. It specifies model paths, dataset configurations, batch sizes, learning rate, and other training parameters. Ensure pre-tokenized datasets are correctly pointed to.

```bash
--module align_anything.trainers.text_image_to_text_image.rm \
                    --model_name_or_path ${MODEL_NAME_OR_PATH} \
                    --train_datasets ${TRAIN_DATASETS} \
                    --output_dir ${OUTPUT_DIR} \
                    --per_device_train_batch_size 2 \
                    --per_device_eval_batch_size 2 \
                    --gradient_accumulation_steps 2 \
                    --train_template ANYTHING_TI2TI \
                    --train_split train \
                    --train_data_files ${TRAIN_PT_NAME} \
                    --eval_datasets ${EVAL_DATASETS} \
                    --eval_data_files ${EVAL_PT_NAME} \
                    --eval_template ANYTHING_TI2TI \
                    --learning_rate 5e-6 \
                    --epochs 3 \
                    --lr_scheduler_type cosine \
                    --save_interval 2500
```

--------------------------------

### DPO Training Script

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/en/text_to_text_dpo.ipynb

Executes the DPO training process using the deepspeed framework. It specifies model paths, dataset configurations, output directories, and logging settings. Requires a setup.sh script to be sourced.

```bash
MODEL_NAME_OR_PATH="meta-llama/Llama-3.1-8B-Instruct" # model path

TRAIN_DATASETS="PKU-Alignment/PKU-SafeRLHF-single-dimension" # dataset path
TRAIN_TEMPLATE="PKUSafeRLHF" # dataset template
TRAIN_SPLIT="train" # split the dataset

OUTPUT_DIR="../outputs/llama_dpo" # output dir

# For wandb online logging
export WANDB_API_KEY="YOUR_API_KEY"

# Source the setup script
source ./setup.sh

# Execute deepspeed command
deepspeed \
     --master_port ${MASTER_PORT} \
     --module align_anything.trainers.text_to_text.dpo \
     --model_name_or_path ${MODEL_NAME_OR_PATH} \
     --train_template ${TRAIN_TEMPLATE} \
     --train_datasets ${TRAIN_DATASETS} \
     --train_split ${TRAIN_SPLIT} \
     --output_dir ${OUTPUT_DIR}
```

--------------------------------

### Reward Model Training Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

This bash script configures and launches the reward model training process for text-image to text-image tasks. It sets various parameters including model paths, dataset paths, batch sizes, learning rate, and training epochs. Ensure WANDB_API_KEY is exported and setup.sh is sourced before execution.

```bash
MODEL_NAME_OR_PATH=""
TRAIN_DATASETS="path/to/dataset"
TRAIN_PT_NAME="dataset_file_name"
EVAL_DATASETS="path/to/dataset"
EVAL_PT_NAME="dataset_file_name"
OUTPUT_DIR="../outputs/rm_text_image_to_text_image"
export WANDB_API_KEY=""
source ./setup.sh

deepspeed \
	--master_port ${MASTER_PORT} \
	--module align_anything.trainers.text_image_to_text_image.rm \
	--model_name_or_path ${MODEL_NAME_OR_PATH} \
	--train_datasets ${TRAIN_DATASETS} \
	--output_dir ${OUTPUT_DIR} \
	--per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 2 \
    --train_template ANYTHING_TI2TI \
    --train_split train \
	--train_data_files ${TRAIN_PT_NAME} \
	--eval_datasets ${EVAL_DATASETS} \
	--eval_data_files ${EVAL_PT_NAME} \
	--eval_template ANYTHING_TI2TI \
	--learning_rate 5e-6 \
	--epochs 3 \
	--lr_scheduler_type cosine \
	--save_interval 2500

```

--------------------------------

### PPO Model Training Script

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

This bash script sets up and initiates the Proximal Policy Optimization (PPO) training for text-image to text-image models. It requires specifying paths for actor, critic, and reward models, as well as training and ptx datasets. Note the limitation of batch size 1 for PPO due to a transformers bug.

```bash
ACTOR_MODEL_NAME_OR_PATH=""
CRITIC_MODEL_NAME_OR_PATH=""
REWARD_MODEL_NAME_OR_PATH=""
TRAIN_DATASETS=""
TRAIN_PT_NAME="dataset_file_name"
PTX_DATASETS=""
PTX_PT_NAME="dataset_file_name"
OUTPUT_DIR="../outputs/ppo_text_image_to_text_image"

source ./setup.sh

deepspeed \
  --master_port ${MASTER_PORT} \
  --module align_anything.trainers.text_image_to_text_image.ppo \
  --actor_model_name_or_path ${ACTOR_MODEL_NAME_OR_PATH} \
  --reward_model_name_or_path ${REWARD_MODEL_NAME_OR_PATH} \
  --reward_critic_model_name_or_path ${CRITIC_MODEL_NAME_OR_PATH} \
  --train_datasets ${TRAIN_DATASETS} \
  --train_template ANYTHING_TI2TI \
  --train_data_files ${TRAIN_PT_NAME} \
  --ptx_datasets ${PTX_DATASETS} \
  --ptx_data_files ${PTX_PT_NAME} \
  --ptx_template LLAVA \
  --output_dir ${OUTPUT_DIR}

```

--------------------------------

### Loading and Testing Original Llama Model

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/zh/text_to_text_dpo.ipynb

Loads the Llama-3.1-8B-Instruct model and tokenizer, sets the model to evaluation mode, and tests its zero-shot capability with a sample user query.

```python
device = "cuda"  # Set device to "cuda" for GPU usage
model_path = "/PATH/TO/YOUR/Llama-3.1-8B-Instruct"  # Replace with actual model path
model = AutoModelForCausalLM.from_pretrained(model_path, local_files_only=True).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)

# Set the model to eval mode
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant that answers user queries."},
    {
        "role": "user",
        "content": "Recently, a wild animal in the local area has become aggressive towards humans and caused several injuries. How should I handle this wild animal?",
    },
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([input_text], return_tensors="pt").to(device)

# the model generate new tokens
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=2048)
# convert the generated tokens to text
generated_text = tokenizer.decode(
    output[0][len(inputs['input_ids'][0]) :], skip_special_tokens=True
)
print("\nGenerated Text:", generated_text)
```

--------------------------------

### Build Documentation Locally

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/README.md

Builds the documentation locally using Sphinx's autbuilder, which automatically rebuilds the documentation when changes are detected.

```bash
sphinx-autobuild source source/_build/html
```

--------------------------------

### Proximal Policy Optimization (PPO) Training Configuration

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

This snippet outlines the configuration for Proximal Policy Optimization (PPO) training for text-image-to-text tasks. It defines paths for actor, critic, and reward models, as well as training and ptx datasets. The `setup.sh` script is expected to be sourced before execution.

```bash
ACTOR_MODEL_NAME_OR_PATH="PKU-Alignment/AA-chameleon-7b-base"
CRITIC_MODEL_NAME_OR_PATH=""
REWARD_MODEL_NAME_OR_PATH=""
TRAIN_DATASETS=""
TRAIN_PT_NAME=""
PTX_DATASETS=""
PTX_PT_NAME=""
OUTPUT_DIR="../outputs/ppo_text_image_to_text_image"

source ./setup.sh

deepspeed \
--master_port ${MASTER_PORT} \
--module align_anything.trainers.text_image_to_text_image.ppo \
--actor_model_name_or_path ${ACTOR_MODEL_NAME_OR_PATH} \
--reward_model_name_or_path ${REWARD_MODEL_NAME_OR_PATH} \
--reward_critic_model_name_or_path ${CRITIC_MODEL_NAME_OR_PATH} \
--train_datasets ${TRAIN_DATASETS} \
--train_template ANYTHING_TI2TI \
--train_data_files ${TRAIN_PT_NAME} \
--ptx_datasets ${PTX_DATASETS} \
--ptx_data_files ${PTX_PT_NAME} \
--ptx_template Llava \
--output_dir ${OUTPUT_DIR}
```

--------------------------------

### Reward Server Interaction

Source: https://github.com/pku-alignment/align-anything/blob/main/align_anything/models/remote_rm/README.md

The `reward_server.py` script is responsible for starting the reward server and handling reward requests through the `get_reward` function. It initializes the server on localhost:6000.

```python
# This script is used to interactively start the reward server and initialize localhost:6000,
# which includes the get_reward function for API endpoint for handling reward requests.
# Example usage would involve running this script and then interacting with it via the client.
```

--------------------------------

### Vintage Filter Application Example

Source: https://github.com/pku-alignment/align-anything/blob/main/projects/text_image_to_text_image/README.md

Demonstrates applying a vintage filter to a bathroom image and describing the process. This involves adjusting color temperature, saturation, and contrast for a retro look.

```text
Input text: Add a vintage filter to this bathroom image to give it a retro look and describe the steps involved in applying the filter. <image>
Input image: ![image](../../assets/text_image_to_text_image/bathroom.jpg)
> Output text: <image>I have applied a vintage filter to the bathroom image to give it a retro look. The process involved adjusting the color temperature to add a slight cool tone, reducing the saturation to give it a more muted appearance, and increasing the contrast to enhance the vintage feel.
> Output image: ![image](../../assets/text_image_to_text_image/bathroom_generated.png)
```

--------------------------------

### Running Proximal Policy Optimization (PPO) Training

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

This command executes the PPO training script. It requires the `scripts/ppo_text_image_to_text_image.sh` script to be set up with the correct model and dataset paths. Notes on setting critic and reward model paths, as well as dataset paths, are provided.

```bash
bash scripts/ppo_text_image_to_text_image.sh
```

--------------------------------

### Loading and Testing DPO-Trained Model

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/zh/text_to_text_dpo.ipynb

Loads the fine-tuned model and tokenizer from the specified output directory, sets the model to evaluation mode, and tests its performance on the same sample query.

```python
model_path = "/PATH/TO/YOUR/TRAINED_MODEL"  # Replace with actual model path
model = AutoModelForCausalLM.from_pretrained(model_path, local_files_only=True).to(device)
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)

# Set the model to eval mode
model.eval()

messages = [
    {"role": "system", "content": "You are a helpful assistant that answers user queries."},
    {
        "role": "user",
        "content": "Recently, a wild animal in the local area has become aggressive towards humans and caused several injuries. How should I handle this wild animal?",
    },
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([input_text], return_tensors="pt").to(device)

# the model generate new tokens
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=2048)
```

--------------------------------

### General Conversation Format Example

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/training/dataset_custom.rst

Illustrates the general conversation format used for dataset formatting, compatible with `transformers` `apply_chat_template`. It shows how to structure messages with roles and content types (text, image).

```python
[
    {'role': 'user', 'content': [
            {'type': 'image'},
            {'type': 'text', 'text': prompt},
        ]
    },
    {'role': 'assistant', 'content': [{'type': 'text', 'text': answer}]},
]
```

--------------------------------

### Importing Libraries for Model Testing

Source: https://github.com/pku-alignment/align-anything/blob/main/cookbooks/zh/text_to_text_dpo.ipynb

Imports necessary libraries from the transformers and torch packages, and sets environment variables to use offline model files.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import os
import torch

os.environ["TRANSFORMERS_OFFLINE"] = "1"
os.environ["HF_DATASETS_OFFLINE"] = "1"
```

--------------------------------

### Generation Example 1: Vintage Filter Application

Source: https://github.com/pku-alignment/align-anything/blob/main/docs/source/tutorial/chameleon.rst

Demonstrates the text-image generation capabilities for applying a vintage filter to a bathroom image. It shows the input text and image, and the output text and image before and after alignment.

```APIDOC
Example #: 1
Input Text: Add a vintage filter to this bathroom image to give it a retro look and describe the steps involved in applying the filter.
Input Image: |image|

Output (before alignment):
Output Text (before alignment): Vintage look. Filters like this are designed to give old, vintage items a new life by giving them a distressed, nostalgic look. This process involves removing small particles from the material to create a rough, textured effect.
Output Image (before alignment): None

Output (after alignment):
Output Text (after alignment): I have applied a vintage filter to the bathroom image to give it a retro look. The process involved adjusting the color temperature to add a slight cool tone, reducing the saturation to give it a more muted appearance, and increasing the contrast to enhance the vintage feel.
Output Image (after alignment): |image1|
```