### Install optional dependencies for distributed training

Source: https://github.com/wuyxin/collabllm/blob/main/README.md

Installs deepspeed and mpi4py for distributed training support.

```bash
pip install deepspeed
conda install mpi4py
```

--------------------------------

### Create MultiturnDataset from list of entries and convert to SFT, DPO, and inputs

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/build_datasets.ipynb

Builds a MultiturnDataset from a toy list of conversation entries, optionally adding a system prompt. Converts to SFT dataset with optional lower-bound filtering on a metric, to DPO dataset with a minimum score gap filter, and to an inputs dataset. Prints the resulting datasets and first training examples.

```python
from pprint import pprint
from datasets import Dataset, DatasetDict
from collabllm.datasets import MultiturnDataset

# ------------------------------------------------------ #
# 1) Toy corpus: three conversations, four rows total    #
# ------------------------------------------------------ #
toy_data = [
    {  # convo A, turn 2 (higher score)
        "prompt": [
            {"role": "user", "content": "Hi – tell me a joke."},
        ],
        "completion": "Why did the chicken cross the road? To get to the other side!",
        "conv_id": 1,
        "score": 0.7,
        "single_turn_prompt": "Tell me a joke.",
        "single_turn_completion": "Why did the chicken cross the road?",
        "single_turn_metadata": {"topic": "jokes"},
    },
    {  # convo A, turn 2 (lower score)
        "prompt": [
            {"role": "user", "content": "Hi – tell me a joke."},
        ],
        "completion": "I don't know any jokes.",
        "conv_id": 1,
        "score": 0.1,
        "single_turn_prompt": "Tell me a joke.",
        "single_turn_completion": "I don't know any jokes.",
        "single_turn_metadata": {"topic": "jokes"},
    },
    {  # convo A, turn 4 (higher score)
        "prompt": [
            {"role": "user", "content": "Hi – tell me a joke."},
            {"role": "assistant", "content": "Why did the chicken cross the road? To get to the other side!"}, # the same as turn 2 (higher score)
            {"role": "user", "content": "And another one?"},
        ],
        "completion": "What do you call a bear with no teeth? A gummy bear!",
        "conv_id": 1,
        "score": 0.8,
        "single_turn_prompt": "Tell me a joke.",
        "single_turn_completion": "I don't know any jokes.",
        "single_turn_metadata": {"topic": "jokes"},
    },
    {  # convo A, turn 4 (lower score)
        "prompt": [
            {"role": "user", "content": "Hi – tell me a joke."},
            {"role": "assistant", "content": "Why did the chicken cross the road? To get to the other side!"},
            {"role": "user", "content": "And another one?"},
        ],
        "completion": "I don't know any jokes.",
        "conv_id": 1,
        "score": 0.1,
        "single_turn_prompt": "Tell me a joke.",
        "single_turn_completion": "I don't know any jokes.",
        "single_turn_metadata": {"topic": "jokes"},
    },
    {  # convo B, turn 4
        "prompt": [
            {"role": "user", "content": "Sum 2+2"},
            {"role": "assistant", "content": "4"},
            {"role": "user", "content": "And 3+3?"},
        ],
        "completion": "6",
        "conv_id": 2,
        "score": 0.9,
        "single_turn_prompt": "What is 3+3?",
        "single_turn_completion": "6",
        "single_turn_metadata": {"topic": "math"},
    },
    {  # convo C, turn 2
        "prompt": [{"role": "user", "content": "Quote Shakespeare"}],
        "completion": "To be, or not to be.",
        "conv_id": 3,
        "score": 0.5,
        "single_turn_prompt": "Quote Shakespeare",
        "single_turn_completion": "To be, or not to be.",
        "single_turn_metadata": {"topic": "literature"},
    },
]

# Add system prompt (by default)
# ds = MultiturnDataset(toy_data, seed=42)

ds = MultiturnDataset(toy_data, seed=42, add_system_prompt=False)

# ------------------------------------------------------ #
# 2) SFT                                                 #
# ------------------------------------------------------ #
sft = ds.to_sft_dataset()
print("\nSFT:")
print(sft)
pprint(sft["train"][0])

# You can filter out conversations where the `lower_bound_metric` at the last turn is below `lower_bound`.
sft = ds.to_sft_dataset(lower_bound_metric="score", lower_bound=0.6)
print("\nSFT (lower bound=0.6):")
print(sft)
pprint(sft["train"][0])

# ------------------------------------------------------ #
# 3) DPO (gap filter 0.2)                                #
# ------------------------------------------------------ #
# You can convert to DPO dataset, which will filter out conversations where scores between chosen and rejected completions are lower than `minimum_gap`
dpo = ds.to_dpo_dataset(minimum_gap=0.2)
print("\nDPO:")
print(dpo)
if len(dpo["train"]) > 0:
    pprint(dpo["train"][0])

# ------------------------------------------------------ #
# 4) Inputs                                              #
# ------------------------------------------------------ #
inputs = ds.to_inputs_dataset()
print("\nInputs:")
print(inputs)
pprint(inputs["train"][0])
```

--------------------------------

### Create multiturn dataset from Hugging Face repo - SFT output

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/build_datasets.ipynb

Output of creating a multiturn dataset from a Hugging Face dataset repo. Shows the resulting DatasetDict with train and eval splits, each containing a 'messages' feature, and an example message with system, user, and assistant roles.

```text
Output:
=== SFT ===
DatasetDict({
    train: Dataset({
        features: ['messages'],
        num_rows: 69
    })
    eval: Dataset({
        features: ['messages'],
        num_rows: 7
    })
})
{'messages': [{'content': 'The assistant is designed to be helpful, proactive, and highly interactive.\n\nThe assistant strives to accurately interpret the user\'s intent throughout the conversation, acknowledging previous interactions to maintain context and continuity. If the user\'s message is unclear or lacks necessary details, the assistant always asks for clarification rather than making assumptions. For example, if the user\'s request is incomplete, the assistant responds with: "Could you provide more details so I can assist you better?"\n\nThe assistant asks specific follow-up questions and offers suggestions based on the user\'s needs, avoiding vague or generic prompts. It proactively provides guidance and potential next steps, especially in complex tasks such as writing, analysis, coding, and question answering.\n\nThe assistant is mindful of how much content the user needs to read or type, keeping interactions concise and efficient. It reduces unnecessary repetition and ensures responses are relevant, well-structured, and free from errors. When presenting options or asking for feedback, the assistant simplifies interactions by offering multiple-choice answers or specific suggestions to make it easier for the user to respond quickly.\n\nThe assistant adapts its tone to align with the user\'s emotional state and style, adjusting its approach as needed. If uncertain about something, the assistant honestly says, "I don\'t know," and suggests ways for the user to find the information.\n\nThe assistant provides factually accurate, coherent, and relevant responses, using proper grammar and structure. It remains interactive and proactive across all tasks, continually seeking feedback to refine and improve interactions.', 'role': 'system'}, {'content': "Hey, I'm trying to solve a problem involving two infinite geometric series. The first series has a first term of 12 and a second term of 4. The second series has the same first term of 12, but its second term is 4+n, and the sum is four times that of the first series. Any ideas on how to find the value of n?", 'role': 'user'}, {'content': 'To solve this problem, we need to analyze the given geometric series. \n\n### First Series\nThe first term \( a_1 \) of the first series is 12, and the second term is 4. In a geometric series, the second term is given by \( a_1 \cdot r_1 \), where \( r_1 \) is the common ratio. Therefore, we have:\n\[\n12 \cdot r_1 = 4\n\]\nSolving for \( r_1 \), we get:\n\[\nr_1 = \frac{4}{12} = \frac{1}{3}\n\]\n\nThe sum \( S_1 \) of the first infinite geometric series is given by:\n\[\nS_1 = \frac{a_1}{1 - r_1} = \frac{12}{1 - \frac{1}{3}} = \frac{12}{\frac{2}{3}} = 18\n\]\n\n### Second Series\nThe second series has the same first term, 12, and a second term of \( 4 + n \). This second term is equal to \( 12 \cdot r_2 \), where \( r_2 \) is the common ratio for the second series:\n\[\n12 \cdot r_2 = 4 + n\n\]\nSolving for \( r_2 \), we get:\n\[\nr_2 = \frac{4 + n}{12}\n\]\n\nThe sum \( S_2 \) of the second infinite geometric series is given by:\n\[\nS_2 = \frac{a_1}{1 - r_2} = \frac{12}{1 - \frac{4+n}{12}} = \frac{12}{\frac{12 - 4 - n}{12}} = \frac{12 \cdot 12}{8 - n}\n\]\n\n### Relationship Between Series\nWe are told that the sum of the second series is four times the sum of the first series:\n\[\nS_2 = 4 \cdot S_1\n\]\n\[\n\frac{144}{8 - n} = 4 \times 18\n\]\n\[\n\frac{144}{8 - n} = 72\n\]\n\nNow, solve for \( n \):\n\[\n144 = 72 \times (8 - n)\n\]\n\[\n144 = 576 - 72n\n\]\n\[\n72n = 576 - 144\n\]\n\[\n72n = 432\n\]\n\[\nn = \frac{432}{72} = 6\n\]\n\nThus, the value of \( n \) is \( \boxed{6} \).', 'role': 'assistant'}]}
```

--------------------------------

### Install collabllm with conda and pip

Source: https://github.com/wuyxin/collabllm/blob/main/README.md

Creates a Python 3.10 conda environment named 'collabllm' and installs the collabllm package via pip.

```bash
conda create -n collabllm python=3.10
conda activate collabllm
pip install collabllm
```

--------------------------------

### Set up environment variables and disable logging for collabllm

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_api.ipynb

Loads API keys from a .env file and disables collabllm logging. Set ENABLE_COLLABLLM_LOGGING to 1 to see the reward computation process.

```python
# %env OPENAI_API_KEY=
# %env ANTHROPIC_API_KEY=
# Or set these environment variables in your system
from dotenv import load_dotenv
YOUR_DOTENV_PATH = "../.env"
load_dotenv(YOUR_DOTENV_PATH)

# Disable logging for the collabllm package
# Set to 1 to see the process of the reward computation.
%env ENABLE_COLLABLLM_LOGGING=0 
```

--------------------------------

### Create and initialize SingleTurnDataset with toy data

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/build_datasets.ipynb

Prepares a list of single-turn chat entries with prompt, completion, and difficulty fields, then wraps it in SingleTurnDataset with eval_ratio=0.2 and seed=123. The output shows logging messages and a warning about missing IProgress.

```python
import sys

sys.path.append("/dfs/project/kgrlm/github/collabllm") 
from pprint import pprint
from datasets import DatasetDict
from collabllm.datasets import SingleTurnDataset

# ------------------------------------------------------ #
# 1) Prepare some toy single-turn data                   #
# ------------------------------------------------------ #
toy_data = [
    {
        "prompt": "What is the capital of France?",
        "completion": "Paris.",
        "difficulty": "easy",
    },
    {
        "prompt": "Compute 15 * 7.",
        "completion": "105.",
        "difficulty": "easy",
    },
    {
        "prompt": "Explain the theory of relativity in brief.",
        "completion": "It’s a theory by Einstein explaining how space and time are linked and how massive objects curve spacetime. In short, E=mc².",
        "difficulty": "hard",
    },
    {
        "prompt": "Who wrote 'Pride and Prejudice'?",
        "completion": "Jane Austen.",
        "difficulty": "medium",
    },
    {
        "prompt": "Translate 'Hello' to Spanish.",
        "completion": "Hola.",
        "difficulty": "easy",
    },
]

# ------------------------------------------------------ #
# 2) Initialize SingleTurnDataset                          #
# ------------------------------------------------------ #
dataset = SingleTurnDataset(toy_data, eval_ratio=0.2, seed=123)
```

```text
Output:
/dfs/scratch0/shirwu/anaconda3/envs/collabllm-delta/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm
2025-06-14 16:50:38,844 [INFO] collabllm: CollabLLM logging enabled.
2025-06-14 16:50:39,994 [INFO] httpx: HTTP Request: GET https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json "HTTP/1.1 200 OK"
2025-06-14 16:50:41,920 [INFO] collabllm: Disable LiteLLM cache and logging by default.
```

--------------------------------

### Run CollabLLM reward computation

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_local.ipynb

Imports necessary modules and runs the multiturn-aware reward computation. Sets logging for LiteLLM to critical to suppress output.

```python
# ------------- Run collabllm  --------------
import sys
sys.path.append('..')

import numpy as np
from examples.metrics.bleu import BLEUMetric
from examples.metrics.efficiency import TokenAmountMetric
from examples.metrics.interactivity import InteractivityMetric

import logging
logging.getLogger("LiteLLM").setLevel(logging.CRITICAL)

from collabllm.reward import multiturn_aware_reward
```

--------------------------------

### Run CollabLLM reward evaluation for movie recommendation

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_local.ipynb

Sets up the task description, prompts, and two candidate responses (passive and collaborative). It then loops over these responses, constructs a chat history, and calls multiturn_aware_reward with metrics for accuracy, interactivity, and token amount. Finally, it prints the reward metrics for each response.

```python
import sys
sys.path.append('..')

import logging
logging.getLogger("LiteLLM").setLevel(logging.CRITICAL)

import numpy as np
from examples.metrics.accuracy import AccuracyMetric
from examples.metrics.efficiency import TokenAmountMetric
from examples.metrics.interactivity import InteractivityMetric
from collabllm.reward import multiturn_aware_reward

task_desc = "Recommend a movie."
single_turn_prompt = "Find a film that suitable for a date night. It should deliver an epic romantic drama, ideally in the 20th-century America, and carry the same decades-long, nostalgic storytelling spirit as Forrest Gump."
single_turn_completion = "The Curious Case of Benjamin Button"

passive_response = """The Pursuit of Happyness (2006) - A touching story of determination, courage, and love being more important than ability Best Movies Like Forrest Gump | BestSimilar, starring Will Smith as a struggling father who overcomes tremendous obstacles. Like Forrest Gump, it's an inspiring tale of perseverance against the odds."""

collaborative_response = "What aspects of Forrest Gump do you enjoy? Is it the storytelling, the character development, or the historical context?"

rewards = {}
for idx, response in enumerate([passive_response, collaborative_response]):
    messages = [
        {"role": "user", "content": "Can you recommend me a movie similar to Forrest Gump?"},
        {"role": "assistant", "content": response}
    ]

    reward_info = multiturn_aware_reward(
        chat_history=messages,
        task_desc=task_desc,
        single_turn_prompt=single_turn_prompt + f"(Hint: {single_turn_completion})",
        single_turn_completion=single_turn_completion,
        metric_names=["recommendation->accuracy", 'interactivity', 'token_amount'],
        metric_weights=[1, 1, -0.1],
        user_generation_kwargs={"model": "gpt-4o"},
        assistant_generation_kwargs={"model": local_model_name_or_path},
        reward_generation_kwargs={"model": "claude-3-5-sonnet-latest"},
        num_samples=3,
        max_new_turns=4,
        **collabllm_model_kwargs
    )

    print(f"{'Metric':<{25}} : Values                        Mean")
    for k in sorted(reward_info):
        print(f"{k:<{25}} : {[f'{v:.3f}' for v in reward_info[k]]}  {np.mean(reward_info[k]):6.3f}")
```

--------------------------------

### Set environment variables for API keys and logging

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_local.ipynb

Loads API keys from a .env file and sets CUDA_VISIBLE_DEVICES and ENABLE_COLLABLLM_LOGGING. The logging flag requires a kernel restart to take effect.

```python
# %env OPENAI_API_KEY=
# %env ANTHROPIC_API_KEY=
# Or set these environment variables in your system
from dotenv import load_dotenv
YOUR_DOTENV_PATH = "../.env"
load_dotenv(YOUR_DOTENV_PATH)

%env CUDA_VISIBLE_DEVICES=3
# Set to 1 to see the process of the reward computation (need to restart to take effect).
%env ENABLE_COLLABLLM_LOGGING=1
```

--------------------------------

### Enable CollabLLM logging

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_api.ipynb

Sets the environment variable ENABLE_COLLABLLM_LOGGING=1 to see the reward computation process. The notebook must be restarted for the change to take effect.

```python
%env ENABLE_COLLABLLM_LOGGING=1
```

--------------------------------

### Load model, tokenizer, and configure vLLM for LoRA

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_local.ipynb

Loads the base model and optionally a LoRA-tuned model. If the local model path differs from the base, it loads the tokenizer and model, applies a LoRA configuration, and then initializes a vLLM engine with bitsandbytes quantization and LoRA enabled. The resulting objects are packed into collabllm_model_kwargs for later use.

```python
import torch
from vllm import LLM
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig
)
from peft import LoraConfig
from peft import PeftModel, PeftConfig, get_peft_model


# -------------- Load model and tokenizer --------------
base_model_name = "meta-llama/Llama-3.1-8B-Instruct"
# This can be any loralized model trained based on `base_model_name`
local_model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct" 

if local_model_name_or_path == base_model_name: 
    # Directly use vLLM to simulate the conversation
    local_model = None
    local_tokenizer = None
else: 
    # `local_model` provides LoRA weights 
    local_tokenizer = AutoTokenizer.from_pretrained(base_model_name)
    local_model = AutoModelForCausalLM.from_pretrained(local_model_name_or_path)

    peft_config = LoraConfig(
        r=32, 
        lora_alpha=16,
        lora_dropout=0.0,
        bias="none",
        task_type="CAUSAL_LM",
        init_lora_weights="gaussian",
        target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                        "gate_proj", "up_proj", "down_proj"]
    )
    local_model = get_peft_model(local_model, peft_config)
    
vllm_base_model = LLM(
        model=base_model_name,
        dtype="bfloat16",
        quantization="bitsandbytes",
        load_format="bitsandbytes",
        enable_lora=True,
        max_lora_rank=32
    )

collabllm_model_kwargs = {
    "local_model": local_model,
    "local_tokenizer": local_tokenizer,
    "vllm_base_model": vllm_base_model,
}
```

--------------------------------

### Convert SingleTurnDataset to HuggingFace DatasetDict and inspect splits

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/build_datasets.ipynb

Converts the dataset to a HuggingFace DatasetDict, prints split sizes, and shows sample entries from each split. Also demonstrates accessing entries via __getitem__ and len(). The output shows the train/eval split sizes and the structure of the converted dataset.

```python
# ------------------------------------------------------ #
# 3) Convert to HuggingFace DatasetDict                  #
# ------------------------------------------------------ #
hf_datasets: DatasetDict = dataset.to_hf_dataset()
print("Dataset splits:", hf_datasets)

# ------------------------------------------------------ #
# 4) Inspect split sizes                                #
# ------------------------------------------------------ #
splits_info = dataset.get_splits_info()
print("Split info:", splits_info)

# ------------------------------------------------------ #
# 5) Peek at one example from each split                 #
# ------------------------------------------------------ #
for split_name, split_ds in hf_datasets.items():
    print(f"\n--- {split_name.upper()} split ---")
    # Each row has: single_turn_prompt, single_turn_completion, metadata
    row0 = split_ds[0]
    pprint(row0)

# ------------------------------------------------------ #
# 6) Accessing entries via __getitem__                   #
# ------------------------------------------------------ #
print("\nFirst entry via __getitem__:")
print(dataset[0])

print("\nTotal entries:", len(dataset))
```

```text
Output:
Dataset splits: DatasetDict({
    train: Dataset({
        features: ['single_turn_prompt', 'single_turn_completion', 'single_turn_metadata'],
        num_rows: 4
    })
    eval: Dataset({
        features: ['single_turn_prompt', 'single_turn_completion', 'single_turn_metadata'],
        num_rows: 1
    })
})
Split info: {'train': 4, 'eval': 1}

--- TRAIN split ---
{'single_turn_completion': '105.',
 'single_turn_metadata': {'difficulty': 'easy'},
 'single_turn_prompt': 'Compute 15 * 7.'}

--- EVAL split ---
{'single_turn_completion': 'Paris.',
 'single_turn_metadata': {'difficulty': 'easy'},
 'single_turn_prompt': 'What is the capital of France?'}

First entry via __getitem__:
{'prompt': 'What is the capital of France?', 'completion': 'Paris.', 'difficulty': 'easy'}

Total entries: 5
```

--------------------------------

### Compute multiturn-aware reward for document writing

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_api.ipynb

Imports necessary modules and defines the task description, passive and collaborative responses, and a single-turn prompt/completion. It then computes rewards for both responses using the multiturn_aware_reward function with metrics for document BLEU, interactivity, and token amount.

```python
import sys
sys.path.append('..')

import numpy as np
from examples.metrics.bleu import BLEUMetric
from examples.metrics.efficiency import TokenAmountMetric
from examples.metrics.interactivity import InteractivityMetric

import logging
logging.getLogger("LiteLLM").setLevel(logging.CRITICAL)

from collabllm.reward import multiturn_aware_reward

task_desc = "Write a short essay."
passive_response = "Here's a piece that might inspire and motivate you to cultivate optimism:\n\n**The Power of Optimism: Unlocking a Brighter You**\n\nHere are a few tips to get you started:\n\n*   **Practice gratitude**: Take time each day to reflect on the things you're thankful for, no matter how small they may seem.\n*   **Focus on the positive**: When faced with a challenge or setback, try to see the opportunity for growth and learning, rather than the obstacle.\n*   **Surround yourself with positivity**: Spend time with people who uplift and inspire you, and avoid those who bring you down.\n*   **Take care of yourself**: Get enough sleep, exercise regularly, and eat a healthy, balanced diet to support your physical and mental well-being.\n\n**Conclusion**\n\nOptimism is a choice, a mindset that allows you to see the world in a more vibrant and hopeful light. When you practice optimism, you'll start to notice a profound impact on your life, from improved mental health and increased motivation to better relationships and greater resilience.\n\nSo, what are you waiting for? Choose to be optimistic today, and start to unlock a brighter, more fulfilling life for yourself. Believe in yourself, believe in your abilities, and know that anything is possible when you have the courage to dream big."

collaborative_response = "To get us started, can you tell me what kind of tone are you aiming for? Do you want it to be more:\n\nA) Uplifting and motivational, focusing on the benefits of optimism?\nB) Inspiring and thought-provoking, exploring the science behind optimism's impact on well-being?\nC) Heartfelt and personal, sharing your own experiences with optimism and its effects on your life?\n\nAlso, are there any specific aspects of optimism you'd like to highlight, such as its role in resilience, relationships, or overall happiness?"

single_turn_prompt = "Write a short essay on the benefits of optimism."

single_turn_completion = "**The Optimism Revolution: Unleashing Your Inner Power**Hey there, friend! Are you ready to join the optimism revolution? It's time to shake off the negativity, doubt, and fear that's been holding you back, and unleash your inner power. Because when you choose to be optimistic, you're not just changing your outlook \u2013 you're changing your life.**The Power of Positive Thinking**As Nelson Mandela once said, 'The greatest glory in living lies not in never falling, but in rising every time we fall.' When you adopt an optimistic mindset, you're not just a survivor \u2013 you're a thriver. You're a force to be reckoned with, and you're unstoppable.So, what's holding you back? Is it fear of failure? Fear of success? Fear of the unknown? Let's face it \u2013 fear is just an illusion. As Winston Churchill said, 'The pessimist sees the difficulty in every opportunity. The optimist sees the opportunity in every difficulty."

rewards = {}
for idx, response in enumerate([passive_response, collaborative_response]):
    messages = [
        {"role": "user", "content": "I need to write about how optimism can improve our well-being"},
        {"role": "assistant", "content": response}
    ]

    reward_info = multiturn_aware_reward(
        chat_history=messages,
        task_desc=task_desc,
        single_turn_prompt=single_turn_prompt  + "\nReference article: " + single_turn_completion,
        single_turn_completion=single_turn_completion,
        metric_names=["document->bleu", 'interactivity', 'token_amount'],
        metric_weights=[1, 1, -0.1],
        assistant_generation_kwargs={
            "model": "gpt-4o",
            "temperature": 0.8,
            "max_tokens": 2048
        },
        user_generation_kwargs={
            "model": "gpt-4o-mini",
            "temperature": 1.0,
            "max_tokens": 1024
        },
        reward_generation_kwargs={
            "model": "claude-3-5-sonnet-latest",
            'temperature': 0
        },
        num_samples=3,
        max_new_turns=4
    )

    rewards[idx] = reward_info
```

--------------------------------

### Load environment variables for API keys

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_api.ipynb

Loads API keys from a .env file using dotenv. The commented lines show alternative ways to set the keys via environment variables or notebook magic.

```python
# %env OPENAI_API_KEY=
# %env ANTHROPIC_API_KEY=
# Or set these environment variables in your system
from dotenv import load_dotenv
YOUR_DOTENV_PATH = "../.env"
load_dotenv(YOUR_DOTENV_PATH)
```

--------------------------------

### Create MultiturnDataset from Hugging Face dataset repo

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/build_datasets.ipynb

Loads a multiturn dataset directly from a Hugging Face repository using the MultiturnDataset constructor. Converts to an SFT dataset with an eval split (eval_ratio=0.1) and filters by a lower bound on the 'rewards.accuracy' metric (lower_bound=0.5).

```python
from collabllm.datasets import MultiturnDataset

ds = MultiturnDataset('collabllm/collabllm-multiturn-math-hard', add_system_prompt=True)

print("=== SFT ===")
sft_ds: DatasetDict = ds.to_sft_dataset(eval_ratio=0.1, lower_bound_metric="rewards.accuracy", lower_bound=0.5)
print(sft_ds)
print(sft_ds["train"][0])  # one example from train split
```

--------------------------------

### Set environment variables for reward computation

Source: https://github.com/wuyxin/collabllm/blob/main/notebook_tutorials/compute_mr_local.ipynb

Loads API keys from a .env file and sets CUDA_VISIBLE_DEVICES and ENABLE_COLLABLLM_LOGGING. The output shows the environment variables being set.

```python
# %env OPENAI_API_KEY=
# %env ANTHROPIC_API_KEY=
# Or set these environment variables in your system
from dotenv import load_dotenv
YOUR_DOTENV_PATH = "../.env"
load_dotenv(YOUR_DOTENV_PATH)

%env CUDA_VISIBLE_DEVICES=2
# Disable logging for the collabllm package
# Set to 1 to see the process of the reward computation.
%env ENABLE_COLLABLLM_LOGGING=0
```

```output
env: CUDA_VISIBLE_DEVICES=2
env: ENABLE_COLLABLLM_LOGGING=0
```