LLM-Blender

https://github.com/yuchenlin/llm-blender
Admin
LLM-Blender is an innovative ensembling framework that achieves consistently superior performance by...
Tokens:11,039
Snippets:117
Trust Score:8
License:Apache-2.0
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install LLM-Blender with Example Dependencies

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Install the LLM-Blender library with the necessary dependencies for running examples. This command ensures all required packages are available.

```bash
pip install -e .[example]
```

--------------------------------

### Install Training Dependencies

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Install the project with training dependencies. Ensure you have the correct environment activated.

```bash
pip install -e .[train]
```

--------------------------------

### Install LLM-Blender

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Install the LLM-Blender package using pip. For development, clone the repository and install in editable mode.

```bash
pip install llm-blender
# pip install git+https://github.com/yuchenlin/LLM-Blender.git
```

```bash
git clone https://github.com/yuchenlin/LLM-Blender.git
cd LLM-Blender
pip install -e .
```

--------------------------------

### RLHF Tuning with PairRM using Default Reference

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

This example demonstrates using LLM-Blender's rank_with_ref method to obtain scalar rewards for RLHF tuning. By default, it compares candidates against the longest generation as a reference.

```python
rewards = blender.rank_with_ref(inputs, candidates_texts, return_scores=True, batch_size=2, mode="longest")
print("Rewards for input 1:", rewards[0]) # rewards of candidates for input 1

```

--------------------------------

### Load PairRM with Hugging Face Transformers for DPO

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Load the PairRM model directly using `from_pretrained` from Hugging Face transformers. This allows using PairRM for DPO without installing the `llm-blender` library. Ensure CUDA is available and set the device map.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from llm_blender.pair_ranker.pairrm import DebertaV2PairRM # or copy the DebertaV2PairRM definition here, https://github.com/yuchenlin/LLM-Blender/blob/main/llm_blender/pair_ranker/pairrm.py
from transformers import AutoTokenizer
from typing import List
pairrm = DebertaV2PairRM.from_pretrained("llm-blender/PairRM-hf", device_map="cuda:0").eval()
tokenizer = AutoTokenizer.from_pretrained('llm-blender/PairRM-hf')
source_prefix = "<|source|>")
cand1_prefix = "<|candidate1|>")
cand2_prefix = "<|candidate2|>")
inputs = ["hello!", "I love you!"]
candidates_A = ["hi!", "I hate you!"]
candidates_B = ["f**k off!", "I love you, too!"]
def tokenize_pair(sources:List[str], candidate1s:List[str], candidate2s:List[str], source_max_length=1224, candidate_max_length=412):
    ids = []
    assert len(sources) == len(candidate1s) == len(candidate2s)
    max_length = source_max_length + 2 * candidate_max_length
    for i in range(len(sources)):
        source_ids = tokenizer.encode(source_prefix + sources[i], max_length=source_max_length, truncation=True)
        candidate_max_length = (max_length - len(source_ids)) // 2
        candidate1_ids = tokenizer.encode(cand1_prefix + candidate1s[i], max_length=candidate_max_length, truncation=True)
        candidate2_ids = tokenizer.encode(cand2_prefix + candidate2s[i], max_length=candidate_max_length, truncation=True)
        ids.append(source_ids + candidate1_ids + candidate2_ids)
    encodings = tokenizer.pad({"input_ids": ids}, return_tensors="pt", padding="max_length", max_length=max_length)
    return encodings

encodings = tokenize_pair(inputs, candidates_A, candidates_B)
encodings = {k:v.to(pairrm.device) for k,v in encodings.items()}
outputs = pairrm(**encodings)
logits = outputs.logits.tolist()
comparison_results = outputs.logits > 0
print(logits)
# [1.9003021717071533, -1.2547134160995483]
print(comparison_results)
# tensor([ True, False], device='cuda:0'), which means whether candidate A is better than candidate B for each input

```

--------------------------------

### Rank and Fuse in One Step

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Performs both ranking and fusion in a single operation using `rank_and_fuse`. This is an efficient way to get fused generations directly from candidate texts.

```python
# # Or do rank and fuser together
fuse_generations, ranks = blender.rank_and_fuse(inputs, candidates_texts, instructions=insts, return_scores=False, batch_size=2, top_k=3)
```

--------------------------------

### Rank Candidates with LLM-Blender

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Ranks candidate texts based on pairwise comparisons using LLM-Blender. Set `return_scores=False` to get only the ranks.

```python
ranks = blender.rank(inputs, candidates_texts, instructions=insts, return_scores=False, batch_size=2)
```

--------------------------------

### Compare Two Candidates with LLM-Blender

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Directly compares two sets of candidate texts (A and B) for given inputs using LLM-Blender. Set `return_logits=False` to get comparison outcomes instead of raw logits.

```python
candidates_A = [x['candidates'][0]['text'] for x in few_examples]
candidates_B = [x['candidates'][1]['text'] for x in few_examples]
comparison_results = blender.compare(
    inputs, candidates_A, candidates_B, instructions=insts, 
    batch_size=2, return_logits=False)
print("Comparison results for inputs:", comparison_results) # comparison results for input 1
```

--------------------------------

### Load Ranker and Rank Outputs

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Initialize the Blender and load a pre-trained ranker checkpoint. Then, use the rank function to rank candidate texts for given inputs.

```python
import llm_blender
blender = llm_blender.Blender()
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint
```

```python
inputs = ["hello, how are you!", "I love you!"]
candidates_texts = [["get out!", "hi! I am fine, thanks!", "bye!"], 
                    ["I love you too!", "I hate you!", "Thanks! You're a good guy!"]]
ranks = blender.rank(inputs, candidates_texts, return_scores=False, batch_size=1)
# ranks is a list of ranks where ranks[i][j] represents the ranks of candidate-j for input-i
"""
ranks -->
array([[3, 1, 2], # it means "hi! I am fine, thanks!" ranks the 1st, "bye" ranks the 2nd, and "get out!" ranks the 3rd. 
       [1, 3, 2]], # it means "I love you too"! ranks the the 1st, and "I hate you!" ranks the 3rd.
       dtype=int32)

"""
```

--------------------------------

### Load and Use PairRM Ranker

Source: https://github.com/yuchenlin/llm-blender/blob/main/pairrm_to_hf.ipynb

Demonstrates loading the 'llm-blender/PairRM' ranker and performing pairwise comparisons between candidate responses for given inputs. It shows how to obtain logits and determine comparison results.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import llm_blender
blender = llm_blender.Blender()
# Load Ranker
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint
inputs = ["hello!", "I love you!"]
candidates_A = ["hi!", "I hate you!"]
candidates_B = ["f**k off!", "I love you, too!"]
logits = blender.compare(inputs, candidates_A, candidates_B, return_logits=True, mode="[A,B]")
comparison_results = logits > 0
print(logits)
# [1.9003021717071533, -1.2547134160995483]
print(comparison_results)
# tensor([ True, False], device='cuda:0'), which means whether candidate A is better than candidate B for each input
```

--------------------------------

### Load LLM-Blender Models (Ranker and Fuser)

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Initialize the LLM-Blender and load pre-trained checkpoints for the ranker and optionally the fuser. Set the CUDA visible devices to '0' for GPU acceleration.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
import llm_blender
blender = llm_blender.Blender()
# Load Ranker
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint
# blender.loadranker("OpenAssistant/reward-model-deberta-v3-large-v2") # load ranker checkpoint
# Load Fuser
# blender.loadfuser("llm-blender/gen_fuser_3b") # load fuser checkpoint if you want to use pre-trained fuser; or you can use ranker only
```

--------------------------------

### Load and Prepare MixInstruct Dataset

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Load the MixInstruct dataset using the `datasets` library and prepare it for use. This involves taking a subset of the data, parsing JSON strings, and filtering out entries with missing comparison results.

```python
import datasets
import json
from llm_blender.gpt_eval.cor_eval import COR_MAPS
from llm_blender.gpt_eval.utils import get_ranks_from_chatgpt_cmps
mixinstruct_test = datasets.load_dataset("llm-blender/mix-instruct", split="test", streaming=True)
few_examples = list(mixinstruct_test.take(8))
# remove cmp_results with none cmp results
for ex in few_examples:
    ex['cmp_results'] = json.loads(ex['cmp_results'])
few_examples = [x for x in few_examples if x['cmp_results']]
insts = [x['instruction'] for x in few_examples]
inputs = [x['input'] for x in few_examples]
candidates_texts = [[cand['text'] for cand in x['candidates']] for x in few_examples]
print("Example:")
print("Instruction 1:\n", insts[0])
print("Input 1:\n", inputs[0])
print("Candidate 1 for input 1:\n")
print(candidates_texts[0][0])
```

--------------------------------

### Load Hugging Face Formatted PairRM Model

Source: https://github.com/yuchenlin/llm-blender/blob/main/pairrm_to_hf.ipynb

This snippet demonstrates how to load a Hugging Face formatted PairRM model using `from_pretrained` and prepare inputs for inference. It includes tokenization, padding, and generating comparison logits.

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
from llm_blender.pair_ranker.pairrm import DebertaV2PairRM
from transformers import AutoTokenizer
from typing import List
pairrm = DebertaV2PairRM.from_pretrained("llm-blender/PairRM-hf", device_map="cuda:0").eval()
tokenizer = AutoTokenizer.from_pretrained('llm-blender/PairRM-hf')
source_prefix = "<|source|>")
cand1_prefix = "<|candidate1|>")
cand2_prefix = "<|candidate2|>")
inputs = ["hello!", "I love you!"]
candidates_A = ["hi!", "I hate you!"]
candidates_B = ["f**k off!", "I love you, too!"]
def tokenize_pair(sources:List[str], candidate1s:List[str], candidate2s:List[str], source_max_length=1224, candidate_max_length=412):
    ids = []
    assert len(sources) == len(candidate1s) == len(candidate2s)
    max_length = source_max_length + 2 * candidate_max_length
    for i in range(len(sources)):
        source_ids = tokenizer.encode(source_prefix + sources[i], max_length=source_max_length, truncation=True)
        candidate_max_length = (max_length - len(source_ids)) // 2
        candidate1_ids = tokenizer.encode(cand1_prefix + candidate1s[i], max_length=candidate_max_length, truncation=True)
        candidate2_ids = tokenizer.encode(cand2_prefix + candidate2s[i], max_length=candidate_max_length, truncation=True)
        ids.append(source_ids + candidate1_ids + candidate2_ids)
    encodings = tokenizer.pad({"input_ids": ids}, return_tensors="pt", padding="max_length", max_length=max_length)
    return encodings

encodings = tokenize_pair(inputs, candidates_A, candidates_B)
encodings = {k:v.to(pairrm.device) for k,v in encodings.items()}
outputs = pairrm(**encodings)
logits = outputs.logits.tolist()
comparison_results = outputs.logits > 0
print(logits)
# [1.9003021717071533, -1.2547134160995483]
print(comparison_results)
# tensor([ True, False], device='cuda:0'), which means whether candidate A is better than candidate B for each input
```

--------------------------------

### Set Dataset for Training

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Define the dataset to be used for training the ranker model.

```bash
dataset="<your dataset>"
```

--------------------------------

### Set Metrics for Training

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Specify the metrics to be used for training the ranker signal. Common metrics like ROUGE and BLEU are supported.

```bash
using_metrics="rouge1,rouge2,rougeLsum,bleu"
```

--------------------------------

### Control Training or Inference Mode

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Set the `do_inference` flag to `False` for training and `True` for inference. Inference mode can be further specified as 'bubble' or 'full'.

```bash
do_inference=False # training
do_inference=True # inference
```

--------------------------------

### Fuse Top-Ranked Candidates

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Load a fuser checkpoint and use get_topk_candidates_from_ranks to select top candidates, then fuse them using the fuse function.

```python
blender.loadfuser("llm-blender/gen_fuser_3b") # load fuser checkpoint if you want to use pre-trained fuser; or you can use ranker only
from llm_blender.blender.blender_utils import get_topk_candidates_from_ranks
topk_candidates = get_topk_candidates_from_ranks(ranks, candidates_texts, top_k=3)
fuse_generations = blender.fuse(inputs, topk_candidates, batch_size=2)
# fuse_generations are the fused generations from our fine-tuned checkpoint
```

```python
# You can also do the rank and fusion with a single function

fuse_generations, ranks = blender.rank_and_fuse(inputs, candidates_texts, return_scores=False, batch_size=2, top_k=3)
```

--------------------------------

### Filter Candidates for Training

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Configure parameters for filtering candidate responses used during training, including the candidate model, decoding method, and number of candidates.

```bash
candidate_model="flan-t5-xxl" # or "alpaca-native"
candidate_decoding_method="top_p_sampling"
n_candidates=15
```

--------------------------------

### Set Ranker Type

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Select the type of ranker model to train. Options include 'Pairranker', 'Summaranker', or 'SimCLS'.

```bash
ranker="Pairranker" # "PairRanker" or "Summaranker" or "SimCLS"
```

--------------------------------

### RLHF Tuning with PairRM using Custom References

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

This snippet shows how to use LLM-Blender's rank_with_ref method for RLHF tuning when you want to specify custom reference texts for comparison, overriding the default mode.

```python
ref_candidates = [_c[0] for _c in candidates_texts] # use the first candidate as the reference, same as mode="first"
rewards = blender.rank_with_ref(inputs, candidates_texts, return_scores=True, batch_size=2, ref_candidates=ref_candidates)

```

--------------------------------

### Fuse Generations using Top-Ranked Candidates

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Generates fused outputs by selecting the top-k ranked candidates and then using LLM-Blender's fuse function. Requires `get_topk_candidates_from_ranks` utility.

```python
from llm_blender.blender.blender_utils import get_topk_candidates_from_ranks
topk_candidates = get_topk_candidates_from_ranks(ranks, candidates_texts, top_k=3)
fuse_generations = blender.fuse(inputs, topk_candidates, instructions=insts, batch_size=2)
print("fuse_generations for input 1:", fuse_generations[0])
```

--------------------------------

### Transform Original PairRM Model to Hugging Face Format

Source: https://github.com/yuchenlin/llm-blender/blob/main/pairrm_to_hf.ipynb

This snippet configures a DebertaV2 model for the PairRM task, adds special tokens, and loads an original PairRM model checkpoint. It then saves the model in a format compatible with Hugging Face.

```python
from llm_blender.pair_ranker.pairrm import DebertaV2PairRM
from transformers import DebertaV2Config, AutoTokenizer
config = DebertaV2Config.from_pretrained('microsoft/deberta-v3-large')
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v3-large')
source_prefix = "<|source|>")
cand1_prefix = "<|candidate1|>")
cand2_prefix = "<|candidate2|>")
cand_prefix = "<|candidate|>")
tokenizer.add_tokens([source_prefix, cand1_prefix, cand2_prefix, cand_prefix])

config.n_tasks = 1
config.source_prefix_id = 128001
config.cand1_prefix_id = 128002
config.cand2_prefix_id = 128003
config.cand_prefix_id = 128004
config.drop_out = 0.05
pairrm = DebertaV2PairRM(config)
pairrm.pretrained_model.resize_token_embeddings(len(tokenizer))
```

```python
!git clone https://huggingface.co/llm-blender/PairRM
import safetensors
import logging

load_result = safetensors.torch.load_model(pairrm, "./PairRM/model.safetensors") # path of original pairrm model
missing_keys, unexpected_keys = load_result
if missing_keys:
    print(f"Missing keys: {missing_keys}")
if unexpected_keys:
    print(f"Unexpected keys: {unexpected_keys}")
if not missing_keys and not unexpected_keys:
    print(f"Successfully loaded checkpoint from './PairRM/model.safetensors'")
```

```python
from transformers import Trainer, TrainingArguments
trainer = Trainer(
    model=pairrm,
    args=TrainingArguments(
        output_dir="./hf_PairRM",
        overwrite_output_dir=True,
    ),
    tokenizer=tokenizer,
)
trainer.save_model("./hf_PairRM/final_checkpoint")
```

--------------------------------

### Set Ranker Backbone Type

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Choose the backbone architecture for the ranker. Options include 'deberta' or 'roberta'.

```bash
backbone_type="deberta" # "deberta" or "roberta"
```

--------------------------------

### Generate Scalar Rewards with PairRM

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Use `blender.rank_with_ref` to compare candidates against a reference and obtain scalar rewards. The `mode` parameter controls how the reference is selected (e.g., 'longest', 'shortest', 'first').

```python
import llm_blender
blender = llm_blender.Blender()
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint

inputs = ["hello, how are you!", "I love you!"]
candidates_texts = [["get out!", "hi! I am fine, thanks!", "bye!"], 
    ["I love you too!", "I hate you!", "Thanks! You're a good guy!"]]
rewards = blender.rank_with_ref(inputs, candidates_texts, return_scores=True, batch_size=2, mode="longest")
print("Rewards for input 1:", rewards[0]) # rewards of candidates for input 1

```

--------------------------------

### Best-of-N Decoding Enhancement with Zephyr-7B

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

This snippet shows how to use LLM-Blender's best_of_n_generate function for decoding enhancement with a causal language model. It requires initializing the tokenizer and model first.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceH4/zephyr-7b-beta", device_map="auto")

system_message = {
    "role": "system",
    "content": "You are a friendly chatbot who always responds in the style of a pirate",
}
messages = [
    [
        system_message,
        {"role": "user", "content": _inst + "\n" + _input},
    ]
    for _inst, _input in zip(insts, inputs)
]
prompts = [tokenizer.apply_chat_template(m, tokenize=False, add_generation_prompt=True) for m in messages]
outputs = blender.best_of_n_generate(model, tokenizer, prompts, n=10)
print("### Prompt:")
print(prompts[0])
print("### best-of-n generations:")
print(outputs[0])

```

--------------------------------

### Best-of-N Sampling with LLM-Blender

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Enhance LLM response quality using best-of-N sampling by re-ranking generated outputs with a loaded ranker.

```python
import llm_blender
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
model = AutoModelForCausalLM.from_pretrained("HuggingFaceH4/zephyr-7b-beta", device_map="auto")
system_message = {"role": "system", "content": "You are a friendly chatbot."}

inputs = ["can you tell me a joke about OpenAI?"]
messages = [[system_message, {"role": "user", "content": _input}] for _input in inputs]
prompts = [tokenizer.apply_chat_template(m, tokenize=False, add_generation_prompt=True) for m in messages]

# standard sampling generation 
input_ids = tokenizer(prompts[0], return_tensors="pt").input_ids
sampled_outputs = model.generate(input_ids, do_sample=True, top_k=50, top_p=0.95, num_return_sequences=1)
print(tokenizer.decode(sampled_outputs[0][len(input_ids[0]):], skip_special_tokens=False))
# --> `Sure`
```

```python
# using our PairRM for best-of-n sampling
blender = llm_blender.Blender()
blender.loadranker("llm-blender/PairRM") # load ranker checkpoint
outputs = blender.best_of_n_generate(model, tokenizer, prompts, n=10)

print("### Prompt:")
print(prompts[0])
print("### best-of-n generations:")
print(outputs[0])
# --> 
"""
Sure, here's a joke about OpenAI:

Why did OpenAI decide to hire a mime as their new AI researcher?

Because they wanted someone who could communicate complex ideas without making a sound!

(Note: This is a joke, not a reflection of OpenAI's actual hiring practices.)
"""
```

--------------------------------

### Inference on Dataset A with Ranker Trained on Dataset B

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Perform inference on one dataset (A) using a ranker model that was trained on a different dataset (B). Ensure `do_inference` is set to `True`.

```bash
dataset=<A>
checkpoint_trained_dataset=<B>
do_inference=True
```

--------------------------------

### Limit Dataset Size for Training/Evaluation

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Control the maximum number of data points used for training, evaluation, and prediction. Set to -1 for no limit.

```bash
max_train_data_size=-1 # -1 means no limit
max_eval_data_size=-1 # -1 means no limit
max_predict_data_size=-1 # -1 means no limit
```

--------------------------------

### Compare Two Candidates

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Use the compare function to directly evaluate which of two candidate texts is better for a given input.

```python
inputs = ["hello!", "I love you!"]
candidates_A = ["hi!", "I hate you!"]
candidates_B = ["f**k off!", "I love you, too!"]
comparison_results = blender.compare(inputs, candidates_A, candidates_B)
# comparison_results is a list of bool, where comparison_results[i] denotes whether candidates_A[i] is better than candidates_B[i] for inputs[i]
# comparison_results[0]--> True
```

--------------------------------

### Print Ranks for Input

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Prints the calculated ranks for the candidates associated with a specific input. This is useful for inspecting the ranking results.

```python
print("Ranks for input 1:", ranks[0]) # ranks of candidates for input 1
# Ranks for input 1: [ 1 11  4  9 12  5  2  8  6  3 10  7]
```

--------------------------------

### Evaluate Ranking Correlation with ChatGPT

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Calculates and prints the correlation between LLM-Blender's ranks and ChatGPT's ranks using various correlation functions. Requires `numpy` and pre-defined `COR_MAPS`.

```python
import numpy as np
llm_ranks_map, gpt_cmp_results = get_ranks_from_chatgpt_cmps(few_examples)
gpt_ranks = np.array(list(llm_ranks_map.values())).T
print("Correlation with ChatGPT")
print("------------------------")
for cor_name, cor_func in COR_MAPS.items():
    print(cor_name, cor_func(ranks, gpt_ranks))
```

--------------------------------

### Set Torchrun Command

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Specify the path to your torchrun executable. This is typically found in your activated conda environment.

```bash
TORCHRUN_CMD=<you torchrun cmd path>
```

--------------------------------

### Tokenize Conversational Pairs

Source: https://github.com/yuchenlin/llm-blender/blob/main/pairrm_to_hf.ipynb

A utility function to prepare multi-turn conversations for pairwise comparison. It validates conversation structure, ensures user and assistant roles are correct, and formats inputs and candidates for the tokenizer.

```python
from typing import List

def tokenize_conv_pair(convAs: List[str], convBs: List[str]):
    """Compare two conversations by takeing USER turns as inputs and ASSISTANT turns as candidates
        Multi-turn conversations comparison is also supportted.
        a conversation format is:
        ```python
        [
            {
                "content": "hello",
                "role": "USER"
            },
            {
                "content": "hi",
                "role": "ASSISTANT"
            },
            ...
        ]
        """

    for c in convAs + convBs:
        assert len(c) % 2 == 0, "Each conversation must have even number of turns"
        assert all([c[i]['role'] == 'USER' for i in range(0, len(c), 2)]), "Each even turn must be USER"
        assert all([c[i]['role'] == 'ASSISTANT' for i in range(1, len(c), 2)]), "Each odd turn must be ASSISTANT"
    # check conversations correctness
    assert len(convAs) == len(convBs), "Number of conversations must be the same"
    for c_a, c_b in zip(convAs, convBs):
        assert len(c_a) == len(c_b), "Number of turns in each conversation must be the same"
        assert all([c_a[i]['content'] == c_b[i]['content'] for i in range(0, len(c_a), 2)]), "USER turns must be the same"
    
    instructions = ["Finish the following coversation in each i-th turn by filling in <Response i> with your response."] * len(convAs)
    inputs = [
        "\n".join([
            "USER: " + x[i]['content'] +
            f"\nAssistant: <Response {i//2+1}>" for i in range(0, len(x), 2)
        ]) for x in convAs
    ]
    cand1_texts = [
        "\n".join([
            f"<Response {i//2+1}>: " + x[i]['content'] for i in range(1, len(x), 2)
        ]) for x in convAs
    ]
    cand2_texts = [
        "\n".join([
            f"<Response {i//2+1}>: " + x[i]['content'] for i in range(1, len(x), 2)
        ]) for x in convBs
    ]
    inputs = [inst + inp for inst, inp in zip(instructions, inputs)]
    encodings = tokenize_pair(inputs, cand1_texts, cand2_texts)
    return encodings
```

--------------------------------

### Set Ranker Backbone Name

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Specify the pre-trained model name for the chosen backbone. Options are 'microsoft/deberta-v3-large' or 'roberta-large'.

```bash
backbone_name="microsoft/deberta-v3-large" # "microsoft/deberta-v3-large" or "roberta-large"
```

--------------------------------

### Specify Reference Candidates for Reward Calculation

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

Override the default reference selection by providing a list of explicit reference candidates using the `ref_candidates` parameter in `blender.rank_with_ref`.

```python
ref_candidates = [_c[0] for _c in candidates_texts] # use the first candidate as the reference, same as mode="first"
rewards = blender.rank_with_ref(inputs, candidates_texts, return_scores=True, batch_size=2, ref_candidates=ref_candidates) 

```

--------------------------------

### LLM-Blender Citation (BibTeX)

Source: https://github.com/yuchenlin/llm-blender/blob/main/README.md

BibTeX entry for the LLM-Blender paper, useful for academic citations.

```bibtex
@inproceedings{llm-blender-2023,
    title = "LLM-Blender: Ensembling Large Language Models with Pairwise Comparison and Generative Fusion",
    author = "Jiang, Dongfu and Ren, Xiang and Lin, Bill Yuchen",
    booktitle = "Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023)",
    year = "2023"
}
```

--------------------------------

### Evaluate Fused Generation Quality

Source: https://github.com/yuchenlin/llm-blender/blob/main/blender_usage.ipynb

Evaluates the quality of fused generations using specified metrics (e.g., 'bartscore') against target outputs. It also calculates scores for individual LLMs.

```python
from llm_blender.common.evaluation import overall_eval
metrics = ['bartscore']
targets = [x['output'] for x in few_examples]
scores = overall_eval(fuse_generations, targets, metrics)

print("Fusion Scores")
for key, value in scores.items():
    print("  ", key+":", np.mean(value))

print("LLM Scores")
llms = [x['model'] for x in few_examples[0]['candidates']]
llm_scores_map = {llm: {metric: [] for metric in metrics} for llm in llms}
for ex in few_examples:
    for cand in ex['candidates']:
        for metric in metrics:
            llm_scores_map[cand['model']][metric].append(cand['scores'][metric])
for i, (llm, scores_map) in enumerate(llm_scores_map.items()):
    print(f"{i} {llm}")
    for metric, llm_scores in llm_scores_map[llm].items():
        print("  ", metric+":", "{:.4f}".format(np.mean(llm_scores)))
```
Tokens:
Raw