### FewShotPrompt Configuration Example

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Defines the configuration for the FewShotPrompt step, which processes inputs using in-context examples and an instruction with an LLM.

```python
FewShotPrompt(
	name = 'The name of the step.',
	inputs = {
		'input_examples': 'The in-context example inputs to include in the prompt.',
		'output_examples': 'The in-context example outputs to include in the prompt.',
		'inputs': 'The inputs to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'input_label': "The label to use for inputs. (optional, defaults to 'Input:')",
		'output_label': "The label to use for outputs. (optional, defaults to 'Output:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'instruction': 'An instruction to include in the prompt. (optional)',
		'sep': "The separator to use between instructions and in-context examples. (optional, defaults to '\n')",
		'min_in_context_examples': 'The minimum number of in-context examples to include. (optional)',
		'max_in_context_examples': 'The maximum number of in-context examples to include. (optional)',
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'inputs': 'The inputs processed with the LLM.',

```

--------------------------------

### Install DataDreamer

Source: https://datadreamer.dev/docs/latest/index.html

Install the DataDreamer library using pip.

```bash
pip3 install datadreamer.dev
```

--------------------------------

### ProcessWithPrompt Configuration Example

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Defines the configuration for the ProcessWithPrompt step, used for processing inputs with an LLM based on a given instruction.

```python
ProcessWithPrompt(
	name = 'The name of the step.',
	inputs = {
		'inputs': 'The inputs to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'An instruction that describes how to process the input.',
		'input_label': "The label to use for inputs. (optional, defaults to 'Input:')",
		'instruction_label': "The label to use for the instruction. (optional, defaults to 'Instruction:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'sep': "The separator to use between instructions and the input. (optional, defaults to '\n\n')",
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'inputs': 'The inputs processed with the LLM.',
		'prompts': 'The prompts processed with the LLM.',
		'generations': 'The generations by the LLM.'
	},
)


```

--------------------------------

### Setup Project Git Hooks

Source: https://datadreamer.dev/docs/latest/pages/contributing.html

Run this command after cloning the repository to configure local Git hooks for development.

```bash
cd DataDreamer
git config --local core.hooksPath ./scripts/.githooks/
./scripts/.githooks/post-checkout

```

--------------------------------

### RAGPrompt Configuration Example

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Defines the configuration for the RAGPrompt step, specifying inputs, arguments, and outputs for retrieval-augmented generation.

```python
RAGPrompt(
	name = 'The name of the step.',
	inputs = {
		'prompts': 'The prompts to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'retriever': 'The retriever to use to retrieve in-context texts.',
		'k': 'The number of texts to retrieve.',
		'retrieved_text_label': "The label to use for retrieved texts. (optional, defaults to 'Document:')",
		'prompt_label': "The label to use for the prompt. (optional, defaults to 'Question:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'sep': "The separator to use between in-context retrieved texts and each prompt. (optional, defaults to '\n')",
		'min_in_context_retrieved_texts': 'The minimum number of in-context retrieved texts to include. (optional)',
		'max_in_context_retrieved_texts': 'The maximum number of in-context retrieved texts to include. (optional)',
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'prompts': 'The prompts processed with the LLM.',
		'generations': 'The generations by the LLM.'
	},
)


```

--------------------------------

### FewShotPrompt

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Processes a set of inputs using in-context examples and an instruction with a `LLM`.

```APIDOC
## FewShotPrompt

### Description
Processes a set of inputs using in-context examples and an instruction with a `LLM`.

### Inputs
- `input_examples`: The in-context example inputs to include in the prompt.
- `output_examples`: The in-context example outputs to include in the prompt.
- `inputs`: The inputs to process with the LLM.

### Arguments
- `llm`: The LLM to use.
- `input_label`: The label to use for inputs. (optional, defaults to 'Input:')
- `output_label`: The label to use for outputs. (optional, defaults to 'Output:')
- `max_new_tokens`: The maximum number of tokens to generate. (optional)
- `instruction`: An instruction to include in the prompt. (optional)
- `sep`: The separator to use between instructions and in-context examples. (optional, defaults to '\n')
- `min_in_context_examples`: The minimum number of in-context examples to include. (optional)
- `max_in_context_examples`: The maximum number of in-context examples to include. (optional)
- `post_process`: A function to post-process the generations. (optional)
- `lazy`: Whether to run lazily or not. (optional, defaults to False)
- `**kwargs`: Any other arguments you want to pass to the .run() method of the LLM. (optional)

### Outputs
- `inputs`: The inputs processed with the LLM.
```

--------------------------------

### FewShotPromptWithRetrieval

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Processes inputs using in-context examples and an instruction with an LLM. It retrieves the k most relevant in-context examples for each input using an Embedder.

```APIDOC
## FewShotPromptWithRetrieval

### Description
Processes a set of inputs using in-context examples and an instruction with a `LLM`. The `k` most relevant in-context examples are retrieved for each input using an `Embedder`.

### Parameters
#### Inputs
- **input_examples** (dict) - The in-context example inputs to retrieve from to include in the prompt.
- **output_examples** (dict) - The in-context example outputs to retrieve from to include in the prompt.
- **inputs** (dict) - The inputs to process with the LLM.

#### Arguments
- **llm** (object) - The LLM to use.
- **embedder** (object) - The embedder to use to retrieve in-context examples.
- **k** (int) - The number of in-context examples to retrieve.
- **input_label** (str, optional) - The label to use for inputs. Defaults to 'Input:'.
- **output_label** (str, optional) - The label to use for outputs. Defaults to 'Output:'.
- **max_new_tokens** (int, optional) - The maximum number of tokens to generate.
- **instruction** (str, optional) - An instruction to include in the prompt.
- **sep** (str, optional) - The separator to use between instructions and in-context examples. Defaults to '\n'.
- **min_in_context_examples** (int, optional) - The minimum number of in-context examples to include.
- **max_in_context_examples** (int, optional) - The maximum number of in-context examples to include.
- **post_process** (function, optional) - A function to post-process the generations.
- **lazy** (bool, optional) - Whether to run lazily or not. Defaults to False.
- **kwargs** - Any other arguments you want to pass to the .run() method of the LLM. (optional)

### Outputs
- **inputs** (object) - The inputs processed with the LLM.
- **prompts** (object) - The prompts processed with the LLM.
- **generations** (object) - The generations by the LLM.
```

--------------------------------

### Utility Methods

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Utility methods for steps, including getting output paths and data serialization.

```APIDOC
## Step Utility Methods

### `get_run_output_folder_path()`
Get the run output folder path that can be used by the step for writing persistent data.

Return type:
`str`

### `pickle(_value_, _*args_, _**kwargs_)`
Pickle a value so it can be stored in a row produced by this step.

Parameters:
- **value** (`Any`) – The value to pickle.
- **args** (`Any`) – The arguments to pass to `dumps()`.
- **kwargs** (`Any`) – The keyword arguments to pass to `dumps()`.

Return type:
`bytes`

### `unpickle(_value_)`
Unpickle a value that was stored in a row produced by this step with `pickle()`.

Parameters:
- **value** (`bytes`) – The value to unpickle.

Return type:
`Any`
```

--------------------------------

### End-to-End DataDreamer Workflow Example

Source: https://datadreamer.dev/docs/latest/index.html

This example demonstrates a complete DataDreamer workflow: generating synthetic abstracts, converting them to tweets, training a model to perform this conversion, and publishing the dataset and model. It utilizes OpenAI for generation and Hugging Face's T5 model for fine-tuning with LoRA.

```python
from datadreamer import DataDreamer
from datadreamer.llms import OpenAI
from datadreamer.steps import DataFromPrompt, ProcessWithPrompt
from datadreamer.trainers import TrainHFFineTune
from peft import LoraConfig

with DataDreamer("./output"):
   # Load GPT-4
   gpt_4 = OpenAI(model_name="gpt-4")

   # Generate synthetic arXiv-style research paper abstracts with GPT-4
   arxiv_dataset = DataFromPrompt(
      "Generate Research Paper Abstracts",
      args={
         "llm": gpt_4,
         "n": 1000,
         "temperature": 1.2,
         "instruction": (
            "Generate an arXiv abstract of an NLP research paper."
            " Return just the abstract, no titles."
         ),
      },
      outputs={"generations": "abstracts"},
   )

   # Ask GPT-4 to convert the abstracts to tweets
   abstracts_and_tweets = ProcessWithPrompt(
      "Generate Tweets from Abstracts",
      inputs={"inputs": arxiv_dataset.output["abstracts"]},
      args={
         "llm": gpt_4,
         "instruction": (
            "Given the abstract, write a tweet to summarize the work."
         ),
         "top_p": 1.0,
      },
      outputs={"inputs": "abstracts", "generations": "tweets"},
   )

   # Create training data splits
   splits = abstracts_and_tweets.splits(train_size=0.90, validation_size=0.10)

   # Train a model to convert research paper abstracts to tweets
   # with the synthetic dataset
   trainer = TrainHFFineTune(
      "Train an Abstract => Tweet Model",
      model_name="google/t5-v1_1-base",
      peft_config=LoraConfig(),
   )
   trainer.train(
      train_input=splits["train"].output["abstracts"],
      train_output=splits["train"].output["tweets"],
      validation_input=splits["validation"].output["abstracts"],
      validation_output=splits["validation"].output["tweets"],
      epochs=30,
      batch_size=8,
   )

   # Publish and share the synthetic dataset
   abstracts_and_tweets.publish_to_hf_hub(
      "datadreamer-dev/abstracts_and_tweets",
      train_size=0.90,
      validation_size=0.10,
   )

   # Publish and share the trained model
   trainer.publish_to_hf_hub("datadreamer-dev/abstracts_to_tweet_model")

```

--------------------------------

### FewShotPromptWithRetrieval Constructor

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Initializes a FewShotPromptWithRetrieval step. Use this to process inputs using in-context examples retrieved by an embedder.

```python
FewShotPromptWithRetrieval(
	name = 'The name of the step.',
	inputs = {
		'input_examples': 'The in-context example inputs to retrieve from to include in the prompt.',
		'output_examples': 'The in-context example outputs to retrieve from to include in the prompt.',
		'inputs': 'The inputs to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'embedder': 'The embedder to use to retrieve in-context examples.',
		'k': 'The number of in-context examples to retrieve.',
		'input_label': "The label to use for inputs. (optional, defaults to 'Input:')",
		'output_label': "The label to use for outputs. (optional, defaults to 'Output:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'instruction': 'An instruction to include in the prompt. (optional)',
		'sep': "The separator to use between instructions and in-context examples. (optional, defaults to '\n')",
		'min_in_context_examples': 'The minimum number of in-context examples to include. (optional)',
		'max_in_context_examples': 'The maximum number of in-context examples to include. (optional)',
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'inputs': 'The inputs processed with the LLM.',
		'prompts': 'The prompts processed with the LLM.',
		'generations': 'The generations by the LLM.'
	},
)
```

--------------------------------

### Skip Dependency Installation

Source: https://datadreamer.dev/docs/latest/pages/contributing.html

Set the environment variable `PROJECT_SKIP_INSTALL_REQS` to 1 to skip dependency installation on subsequent runs for faster execution.

```bash
export PROJECT_SKIP_INSTALL_REQS=1
./scripts/run.sh -k "TestFewShotPrompt"

```

--------------------------------

### Basic Step Structure in DataDreamer

Source: https://datadreamer.dev/docs/latest/pages/advanced_usage/creating_a_new_datadreamer_.../step.html

Define a new custom step by subclassing the `Step` class and implementing the `setup()` and `run()` methods. This provides a foundation for custom data processing logic.

```python
from datadreamer.steps import Step

class MyNewStep(Step):
    def setup(self):
        # Register inputs, arguments, outputs, and data card information here

    def run(self):
        # Implement your custom data processing / transformation logic here

```

--------------------------------

### DataDreamer Session Management (Start/Stop)

Source: https://datadreamer.dev/docs/latest/datadreamer.html

Provides an alternative method for managing DataDreamer sessions using explicit `start()` and `stop()` calls. This is useful in interactive environments like notebooks where context managers might be less convenient.

```APIDOC
## DataDreamer Session with Start/Stop Methods

### Description
Manually start and stop a DataDreamer session using `dd.start()` and `dd.stop()`. While the context manager (`with` block) is preferred, this method is useful for interactive environments.

### Usage
```python
from datadreamer import DataDreamer

dd = DataDreamer('./output/')
dd.start()
# ... run steps or trainers here ...
dd.stop()
```
```

--------------------------------

### Translate Test Set with Few-shot Examples

Source: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/bootstrapping_machine_translation.html

Translates the English test set to Tamil using the final bootstrapped synthetic few-shot examples generated in the previous rounds.

```python
from datadreamer.steps import HFHubDataSource, FewShotPrompt

# Load the test set of English sentences
english_test_dataset = HFHubDataSource(
    "Get FLORES-101 English Sentences (Test Set)",
    "gsarti/flores_101",
    config_name="eng",
    split="devtest",
).select_columns(["sentence"])

# Finally translate the test set with the final bootstrapped synthetic few-shot examples
english_test_to_tamil = FewShotPrompt(
    "Few-shot Translate from English To Tamil (Test Set)",
    inputs={
        "input_examples": best_translation_pairs.output["english"],
        "output_examples": best_translation_pairs.output["tamil"],
        "inputs": english_test_dataset.output["sentence"],
    },
    args={

```

--------------------------------

### take

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Takes the first 'n' rows from the step's output. This is useful for previewing or limiting the dataset size.

```APIDOC
## take(_n_ , _name =None_, _lazy =True_, _progress_interval =DEFAULT_, _force =False_, _writer_batch_size =1000_, _save_num_proc =DEFAULT_, _save_num_shards =DEFAULT_, _background =False_)

### Description
Take the first `n` rows from the step’s output. See `take()` for more details.

### Parameters
#### Path Parameters
- **n** (`int`) – The number of rows to take.

#### Query Parameters
- **name** (`Optional`[`str`], default: `None`) – The name of the operation.
- **lazy** (`bool`, default: `True`) – Whether to run the operation lazily.
- **progress_interval** (`UnionType`[`None`, `int`, `Default`], default: `DEFAULT`) – How often to log progress in seconds.
- **force** (`bool`, default: `False`) – Whether to force run the step (ignore saved results).
- **writer_batch_size** (`Optional`[`int`], default: `1000`) – The batch size to use if saving to disk.
- **save_num_proc** (`UnionType`[`None`, `int`, `Default`], default: `DEFAULT`) – The number of processes to use if saving to disk.
- **save_num_shards** (`UnionType`[`None`, `int`, `Default`], default: `DEFAULT`) – The number of shards on disk to save the dataset into.
- **background** (`bool`, default: `False`) – Whether to run the operation in the background.

### Return type
`Step`

### Returns
A new step with the operation applied.
```

--------------------------------

### Run DataDreamer Session with Start and Stop Methods

Source: https://datadreamer.dev/docs/latest/datadreamer.html

An alternative to the context manager for starting and stopping DataDreamer sessions manually. This approach is useful in interactive environments like Jupyter notebooks.

```python
from datadreamer import DataDreamer

dd = DataDreamer('./output/')
dd.start()
# ... run steps or trainers here ...
dd.stop()

```

--------------------------------

### Initialize DataDreamer and Load LLM

Source: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/bootstrapping_machine_translation.html

Sets up the DataDreamer environment and loads the OpenAI GPT-4 model for translation tasks.

```python
from datadreamer import DataDreamer
from datadreamer.llms import OpenAI

with DataDreamer("./output"):
    # Load GPT-4
    gpt_4 = OpenAI(model_name="gpt-4")
```

--------------------------------

### DataDreamer.__version__

Source: https://datadreamer.dev/docs/latest/datadreamer.html

Accesses the installed version of the DataDreamer library.

```APIDOC
## `datadreamer.__version__`

### Description
Provides the version string of the installed DataDreamer library.

### Type
*   `str`
```

--------------------------------

### Run All Project Tests

Source: https://datadreamer.dev/docs/latest/pages/contributing.html

Execute all test cases located in the `src/tests` folder to verify your changes.

```bash
./scripts/run.sh

```

--------------------------------

### Define and Configure a Prompt Step

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

The Prompt step is used to run a series of prompts against an LLM. It allows specifying inputs, arguments like the LLM to use, and post-processing functions. Outputs include the processed prompts and the LLM generations.

```python
Prompt(
	name='my_llm_prompts',
	inputs={'prompts': ['What is the capital of France?', 'Who wrote Hamlet?']},
	args={
		'llm': my_llm_instance,
		'post_process': my_post_processing_function
	}
)
```

--------------------------------

### Step Class Initialization

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Parameters for initializing a base Step object.

```APIDOC
## `_datadreamer.steps.Step` Constructor

### Description
Base class for all steps in Datadreamer.

### Parameters
- **name** (`str`) – The name of the step.
- **inputs** (`Optional[dict[str, OutputDatasetColumn | OutputIterableDatasetColumn]]`, default: `None`) – The inputs to the step.
- **args** (`Optional[dict[str, Any]]`, default: `None`) – The arguments to the step.
- **outputs** (`Optional[dict[str, str]]`, default: `None`) – The name mapping to rename outputs produced by the step.
- **progress_interval** (`Optional[int]`, default: `60`) – How often to log progress in seconds.
- **force** (`bool`, default: `False`) – Whether to force run the step (ignore saved results).
- **verbose** (`Optional[bool]`, default: `None`) – Whether or not to print verbose logs.
- **log_level** (`Optional[int]`, default: `None`) – The logging level to use (`DEBUG`, `INFO`, etc.).
- **save_num_proc** (`Optional[int]`, default: `None`) – The number of processes to use if saving to disk.
- **save_num_shards** (`Optional[int]`, default: `None`) – The number of shards on disk to save the dataset into.
- **background** (`bool`, default: `False`) – Whether to run the operation in the background.
```

--------------------------------

### Retrieve Step Configuration

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Set up the Retrieve step to fetch results for a set of queries using a specified Retriever. The 'retriever' argument is required.

```python
Retrieve(
	name = 'The name of the step.',
	inputs = {
		'queries': 'The queries to retrieve results for.'
	},
	args = {
		'retriever': 'The Retriever to use.',
		'k': 'How many results to retrieve. (optional, defaults to 5)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the Retriever. (optional)'
	},
	outputs = {
		'queries': 'The queries used to retrieve results.',
		'results': 'The results from the Retriever.'
	},
)
```

--------------------------------

### DataFromPrompt Constructor

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Initializes a DataFromPrompt step. Use this to generate a specified number of data rows using an LLM and an instruction.

```python
DataFromPrompt(
	name = 'The name of the step.',
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'The instruction to use to generate data.',
		'n': 'The number of rows to generate from the prompt.',
		'temperature': 'The temperature to use when generating data. (optional, defaults to 1.0)',
		'top_p': 'The top_p to use when generating data. (optional, defaults to 1.0)',
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'prompts': 'The prompts processed with the LLM.',
		'generations': 'The generations by the LLM.'
	},
)
```

--------------------------------

### Getting Run Output Folder Path

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Retrieve the path to the run output folder, which can be used by the step for writing persistent data.

```python
get_run_output_folder_path()
```

--------------------------------

### Few-shot Translation for Subsequent Rounds

Source: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/bootstrapping_machine_translation.html

Utilizes previously generated best translation pairs as few-shot examples to improve translation quality in subsequent bootstrapping rounds.

```python
from datadreamer.steps import FewShotPrompt

# On subsequent rounds, use the best synthetic translation pairs from the previous round
# as few-shot examples to translate more English sentences to create even better synthetic pairs
english_to_tamil = FewShotPrompt(
    f"Round #{r+1}: Few-shot Translate from English To Tamil",
    inputs={
        "input_examples": best_translation_pairs.output["english"],
        "output_examples": best_translation_pairs.output["tamil"],
        "inputs": sentences_for_round.output["sentence"],
    },
    args={
        "llm": gpt_4,
        "input_label": "English:",
        "output_label": "Tamil:",
        "instruction": "Translate the sentence to Tamil.",
        "max_new_tokens": 1000,
    },
    outputs={"inputs": "english", "generations": "tamil"},
).select_columns(["english", "tamil"])
```

--------------------------------

### DataDreamer Session Management

Source: https://datadreamer.dev/docs/latest/datadreamer.html

Demonstrates how to initiate and manage DataDreamer sessions using a context manager for automatic handling of session setup and teardown. This is the recommended approach for running workflows.

```APIDOC
## DataDreamer Session with Context Manager

### Description
Use a `with` statement to manage a DataDreamer session. This ensures that resources are properly initialized and cleaned up automatically, organizing, caching, and saving results to the specified output folder.

### Usage
```python
from datadreamer import DataDreamer

with DataDreamer('./output/'):
    # ... run steps or trainers here ...
```

### In-Memory Sessions
To run DataDreamer without saving any data to disk, pass `':memory:'` as the `output_folder_path`.

```python
from datadreamer import DataDreamer

with DataDreamer(':memory:'):
    # ... run steps or trainers here ...
```
```

--------------------------------

### DataFromAttributedPrompt Step Configuration

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Defines the configuration for the DataFromAttributedPrompt step, specifying arguments for LLM, instruction, attributes, and generation parameters.

```python
DataFromAttributedPrompt(
	name = 'The name of the step.',
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'The attributed instruction to use to generate data.',
		'attributes': 'The attributes to use in the instruction.',
		'n': 'The number of rows to generate from the prompt.',
		'temperature': 'The temperature to use when generating data. (optional, defaults to 1.0)',
		'top_p': 'The top_p to use when generating data. (optional, defaults to 1.0)',
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'attributes': 'The attributes used to generate the data.',
		'prompts': 'The prompts processed with the LLM.',
		'generations': 'The generations by the LLM.'
	},
)
```

--------------------------------

### Monitor GPU Memory Usage

Source: https://datadreamer.dev/docs/latest/pages/advanced_usage/parallelization/training_models_on_multiple_gpus.html

Set the `verbose` parameter to `True` in the `Trainer` constructor to monitor GPU memory usage during training. This logs memory usage at the start and end of training, and after each epoch.

```python
from datadreamer.trainer import Trainer

# Example: Enable verbose logging for GPU memory usage
trainer = Trainer(device='cuda', verbose=True)

```

--------------------------------

### RankWithPrompt Step Configuration

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Configuration for the RankWithPrompt step, used to produce scores for inputs via an LLM instruction. It supports sorting and filtering based on scores.

```python
RankWithPrompt(
	name = 'The name of the step.',
	inputs = {
		'inputs': 'The inputs to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'An instruction that describes how to process the input.',
		'sort': 'Whether or not to sort the inputs by the score produced by the instruction. (optional, defaults to True)',
		'reverse': 'Whether or not to reverse the sort direction. (optional, defaults to True)',
		'score_threshold': 'A score threshold. If specified, filter out input rows that scored below the threshold. (optional)',
		'input_label': "The label to use for inputs. (optional, defaults to 'Input:')",
		'instruction_label': "The label to use for the instruction. (optional, defaults to 'Instruction:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'sep': "The separator to use between instructions and the input. (optional, defaults to '\n\n')",
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
)
```

--------------------------------

### Configure Trainer for Multi-GPU Training

Source: https://datadreamer.dev/docs/latest/pages/advanced_usage/parallelization/training_models_on_multiple_gpus.html

Pass a list of GPU devices to the `device` parameter of the `Trainer` constructor to enable multi-GPU training. This is the primary method for utilizing multiple GPUs.

```python
from datadreamer.trainer import Trainer

# Example: Train on GPUs 0 and 1
trainer = Trainer(device=['cuda:0', 'cuda:1'])

# Example: Train on all available GPUs
trainer = Trainer(device='cuda')

```

--------------------------------

### Run DataDreamer Session with Context Manager

Source: https://datadreamer.dev/docs/latest/datadreamer.html

Use a context manager to run DataDreamer workflows. This automatically handles session setup and teardown, organizing, caching, and saving results to the specified output folder.

```python
from datadreamer import DataDreamer

with DataDreamer('./output/'):
    # ... run steps or trainers here ...

```

--------------------------------

### JudgePairsWithPrompt Step Configuration

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Configures the JudgePairsWithPrompt step for LLM-based judgment of input pairs. Use this to evaluate and compare two sets of inputs based on a given instruction.

```python
JudgePairsWithPrompt(
	name = 'The name of the step.',
	inputs = {
		'a': 'A set of inputs to judge between.',
		'b': 'A set of inputs to judge between.'
	},
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'An instruction that describes how to judge the input. (optional, defaults to "Which input is better? Respond with \'Input A\' or \'Input B\'.")',
		'judgement_func': "A function to get the judgement from the generation. (optional, defaults to judging by parsing 'a_label' / 'b_label' ('Input A' / 'Input B') from the generation)",
		'randomize_order': "Whether to randomly swap the order of 'a' and 'b' in the prompt to mitigate position bias. (optional, defaults to True)",
		'a_label': "The label to use for inputs 'a'. (optional, defaults to 'Input A:')",
		'b_label': "The label to use for inputs 'b'. (optional, defaults to 'Input B:')",
		'instruction_label': "The label to use for the instruction. (optional, defaults to 'Instruction:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'sep': "The separator to use between instructions and the input. (optional, defaults to '\n')",
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'a': 'A set of inputs judged with the LLM.',
		'b': 'A set of inputs judged with the LLM.',
		'judge_prompts': 'The prompts processed with the LLM to judge inputs.',
		'judge_generations': 'The judgement generations by the LLM.',
		'judgements': 'The judgements by the LLM.'
	},
)
```

--------------------------------

### Format Project Code

Source: https://datadreamer.dev/docs/latest/pages/contributing.html

Execute the `format.sh` script to check for and auto-format code according to project style guidelines.

```bash
./scripts/format.sh

```

--------------------------------

### Viewing Step Output

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Method to view the first N rows of a step's output.

```APIDOC
## `head(_n=5_, _shuffle=False_, _seed=None_, _buffer_size=1000_)`

### Description
Return the first `n` rows of the step’s output as a pandas `DataFrame` for easy viewing.

### Parameters
- **n** (`int`, default: `5`) – The number of rows to return.
- **shuffle** (`bool`, default: `False`) – Whether to shuffle the rows before returning.
- **seed** (`Optional[int]`, default: `None`) – The seed for shuffling.
- **buffer_size** (`int`, default: `1000`) – The buffer size for shuffling.
```

--------------------------------

### Step Registration Methods

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Methods for registering inputs, arguments, outputs, and data cards for a step.

```APIDOC
## Step Registration

### `register_input(_input_column_name_, _required=True_, _help=None_)`
Registers an input for the step.

Parameters:
- **input_column_name** (`str`) – The name of the input column.
- **required** (`bool`, default: `True`) – Whether the input is required.
- **help** (`Optional[str]`, default: `None`) – The help string for the input.

### `register_arg(_arg_name_, _required=True_, _default=None_, _help=None_, _default_help=None_)`
Registers an argument for the step.

Parameters:
- **arg_name** (`str`) – The name of the argument.
- **required** (`bool`, default: `True`) – Whether the argument is required.
- **default** (`Optional[Any]`, default: `None`) – The default value of the argument.
- **help** (`Optional[str]`, default: `None`) – The help string for the argument.
- **default_help** (`Optional[str]`, default: `None`) – The help string for the default value of the argument.

### `register_output(_output_column_name_, _help=None_)`
Registers an output for the step.

Parameters:
- **output_column_name** (`str`) – The name of the output column.
- **help** (`Optional[str]`, default: `None`) – The help string for the output.

### `register_data_card(_data_card_type_, _data_card_)`
Registers a data card for the step.

Parameters:
- **data_card_type** (`str`) – The type of the data card.
- **data_card** (`Any`) – The data card.
```

--------------------------------

### FilterWithPrompt Step Configuration

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Configuration for the FilterWithPrompt step, which filters inputs using an LLM and a provided filter function. It accepts arguments for LLM, instruction, and filtering logic.

```python
FilterWithPrompt(
	name = 'The name of the step.',
	inputs = {
		'inputs': 'The inputs to process with the LLM.'
	},
	args = {
		'llm': 'The LLM to use.',
		'instruction': 'An instruction that describes how to process the input.',
		'filter_func': "A function to filter the generations with. (optional, defaults to filtering by parsing 'Yes' / 'No' from the generation)",
		'input_label': "The label to use for inputs. (optional, defaults to 'Input:')",
		'instruction_label': "The label to use for the instruction. (optional, defaults to 'Instruction:')",
		'max_new_tokens': 'The maximum number of tokens to generate. (optional)',
		'sep': "The separator to use between instructions and the input. (optional, defaults to '\n\n')",
		'post_process': 'A function to post-process the generations. (optional)',
		'lazy': 'Whether to run lazily or not. (optional, defaults to False)',
		'**kwargs': 'Any other arguments you want to pass to the .run() method of the LLM. (optional)'
	},
	outputs = {
		'*columns': "All of the columns of the step producing the 'inputs'."
	},
)
```

--------------------------------

### TrainHFFineTune.export_to_disk

Source: https://datadreamer.dev/docs/latest/datadreamer.trainers.html

Exports the trained model to a specified directory on disk.

```APIDOC
## TrainHFFineTune.export_to_disk

### Description
Exports the trained model to disk.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **path** (str) - The path to export the model to.
* **adapter_only** (bool, default: `False`) - Whether to export only the adapter.

### Return Type
`PreTrainedModel`

### Returns
The exported model.
```

--------------------------------

### TrainHFFineTune.export_to_disk

Source: https://datadreamer.dev/docs/latest/datadreamer.trainers.html

Exports the trained model to a specified directory on disk.

```APIDOC
## export_to_disk

### Description
Exports the trained model to a specified directory on disk.

### Method
`export_to_disk(_path_ , _adapter_only =False_)`

### Parameters
* `_path` (str) - Required - The path to export the model to.
* `_adapter_only` (bool, default: `False`) - Whether to export only the adapter.

### Return Type
`PreTrainedModel`

### Returns
The exported model.
```

--------------------------------

### Instruction-Tuning with DataDreamer and LoRA

Source: https://datadreamer.dev/docs/latest/pages/get_started/quick_tour/instruction_tuning.html

This snippet demonstrates how to instruction-tune a base LLM using DataDreamer. It fetches the Alpaca dataset, formats instructions, creates data splits, defines a chat prompt template, and configures a LoRA fine-tuning trainer.

```python
from datadreamer import DataDreamer
from datadreamer.steps import HFHubDataSource
from datadreamer.trainers import TrainHFFineTune
from peft import LoraConfig

with DataDreamer("./output"):
    # Get the Alpaca instruction-tuning dataset (cleaned version)
    instruction_tuning_dataset = HFHubDataSource(
        "Get Alpaca Instruction-Tuning Dataset", "yahma/alpaca-cleaned", split="train"
    )

    # Keep only 1000 examples as a quick demo
    instruction_tuning_dataset = instruction_tuning_dataset.take(1000)

    # Some examples taken in an "input", we'll format those into the instruction
    instruction_tuning_dataset.map(
        lambda row: {
            "instruction": (
                row["instruction"]
                if len(row["input"]) == 0
                else f"Input: {row['input']}\n\n{row['instruction']}"
            ),
            "output": row["output"],
        },
        lazy=False,
    )

    # Create training data splits
    splits = instruction_tuning_dataset.splits(train_size=0.90, validation_size=0.10)

    # Define what the prompt template should be when instruction-tuning
    chat_prompt_template = "### Instruction:\n{{prompt}}\n\n### Response:\n"

    # Instruction-tune the base TinyLlama model to make it follow instructions
    trainer = TrainHFFineTune(
        "Instruction-Tune TinyLlama",
        model_name="TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
        chat_prompt_template=chat_prompt_template,
        peft_config=LoraConfig(),
        device=["cuda:0", "cuda:1"],
        dtype="bfloat16",
    )
    trainer.train(
        train_input=splits["train"].output["instruction"],
        train_output=splits["train"].output["output"],
        validation_input=splits["validation"].output["instruction"],
        validation_output=splits["validation"].output["output"],
        epochs=3,
        batch_size=1,
        gradient_accumulation_steps=32,
    )

```

--------------------------------

### Viewing Step Output Head

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Return the first `n` rows of the step's output as a pandas DataFrame for easy viewing. Options for shuffling and seeding are available.

```python
head(_n =5_, _shuffle =False_, _seed =None_, _buffer_size =1000_)
```

--------------------------------

### ProcessWithPrompt

Source: https://datadreamer.dev/docs/latest/datadreamer.steps.html

Processes a set of inputs using an instruction with a `LLM`.

```APIDOC
## ProcessWithPrompt

### Description
Processes a set of inputs using an instruction with a `LLM`.

### Inputs
- `inputs`: The inputs to process with the LLM.

### Arguments
- `llm`: The LLM to use.
- `instruction`: An instruction that describes how to process the input.
- `input_label`: The label to use for inputs. (optional, defaults to 'Input:')
- `instruction_label`: The label to use for the instruction. (optional, defaults to 'Instruction:')
- `max_new_tokens`: The maximum number of tokens to generate. (optional)
- `sep`: The separator to use between instructions and the input. (optional, defaults to '\n\n')
- `post_process`: A function to post-process the generations. (optional)
- `lazy`: Whether to run lazily or not. (optional, defaults to False)
- `**kwargs`: Any other arguments you want to pass to the .run() method of the LLM. (optional)

### Outputs
- `inputs`: The inputs processed with the LLM.
- `prompts`: The prompts processed with the LLM.
- `generations`: The generations by the LLM.
```