### Instantiate DeepSeekProverAutoFormalization with Examples

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deepseek_prover.md

This example demonstrates how to instantiate the `DeepSeekProverAutoFormalization` task with a list of few-shot examples. The `dedent` function from `textwrap` is used to format the multi-line example strings cleanly. This setup is beneficial for improving the model's performance in generating accurate Lean 4 formalizations.

```python
from textwrap import dedent

examples = [
    dedent("""
    ## Statement in natural language:
    For real numbers k and x:
    If x is equal to (13 - √131) / 4, and
    If the equation 2x² - 13x + k = 0 is satisfied,
    Then k must be equal to 19/4.
    ## Formalized:
    theorem mathd_algebra_116 (k x : ℝ) (h₀ : x = (13 - Real.sqrt 131) / 4)
        (h₁ : 2 * x ^ 2 - 13 * x + k = 0) : k = 19 / 4 :="""),
    dedent("""
    ## Statement in natural language:
    The greatest common divisor (GCD) of 20 factorial (20!) and 200,000 is equal to 40,000.
    ## Formalized:
    theorem mathd_algebra_116 (k x : ℝ) (h₀ : x = (13 - Real.sqrt 131) / 4)
        (h₁ : 2 * x ^ 2 - 13 * x + k = 0) : k = 19 / 4 :="""),
    dedent("""
    ## Statement in natural language:
    Given two integers x and y:
    If y is positive (greater than 0),
    And y is less than x,
    And the equation x + y + xy = 80 is true,
    Then x must be equal to 26.
    ## Formalized:
    theorem mathd_algebra_116 (k x : ℝ) (h₀ : x = (13 - Real.sqrt 131) / 4)
        (h₁ : 2 * x ^ 2 - 13 * x + k = 0) : k = 19 / 4 :="""),
]

auto_formalization = DeepSeekProverAutoFormalization(
    name="auto_formalization",
    input_batch_size=8,
    llm=llm,
    examples=examples
)

```

--------------------------------

### Evolved Instructions Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deita.md

Example output demonstrating the progression of instruction complexity generated by the EvolInstruct pipeline, starting from a simple question and evolving into more detailed and specific queries.

```bash
( 1, 'How many fish are there in a dozen fish?')
( 2, 'How many rainbow trout are there in a dozen rainbow trout?')
( 3, 'What is the average weight in pounds of a dozen rainbow trout caught in a specific river in Alaska during the month of May?')
```

--------------------------------

### Running the Image Generation Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/examples/image_generation.md

Execute the image generation pipeline using the provided Python script. Make sure to install the 'distilabel[vision]' package first.

```bash
python examples/image_generation.py

```

--------------------------------

### Install Transformers, Torch, and SetFit

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/generate_textcat_dataset.ipynb

Install specific versions of transformers, torch, and setfit libraries. Ensure compatibility with the distilabel setup.

```python
!pip install "transformers~=4.40" "torch~=2.0" "setfit~=1.0"
```

--------------------------------

### Define and Run Pipeline with Load Groups

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/load_groups_and_execution_stages.md

This example demonstrates how to define a pipeline with multiple TextGeneration steps using vLLM and then execute them using load groups to manage sequential execution on a single GPU. It shows how to get load stages and run the pipeline with specified load groups.

```python
from datasets import load_dataset

from distilabel.llms import vLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import StepResources
from distilabel.steps.tasks import TextGeneration

dataset = load_dataset(
    "distilabel-internal-testing/instruction-dataset-mini", split="test"
).rename_column("prompt", "instruction")

with Pipeline() as pipeline:
    text_generation_0 = TextGeneration(
        llm=vLLM(
            model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
            extra_kwargs={"max_model_len": 1024},
        ),
        resources=StepResources(gpus=1),
    )

    text_generation_1 = TextGeneration(
        llm=vLLM(
            model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
            extra_kwargs={"max_model_len": 1024},
        ),
        resources=StepResources(gpus=1),
    )

if __name__ == "__main__":
    load_stages = pipeline.get_load_stages(load_groups=[[text_generation_1.name]])

    for i, steps_stage in enumerate(load_stages[0]):
        print(f"Stage {i}: {steps_stage}")

    # Output:
    # Stage 0: ['text_generation_0']
    # Stage 1: ['text_generation_1']

    distiset = pipeline.run(dataset=dataset, load_groups=[[text_generation_0.name]])
```

--------------------------------

### Create Social Network with FinePersonas Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example shows how to leverage FinePersonas to create a synthetic social network and fine-tune adapters for Multi-LoRA. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.payload import PayloadTask
from pydantic import BaseModel

class UserProfile(BaseModel):
    name: str
    bio: str
    interests: list[str]

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the payload task for user profiles
task = PayloadTask(
    "Generate a user profile for a social network.",
    model=UserProfile,
    llm=llm,
)

# Generate a synthetic social network
dataset = task.generate(num_examples=50)
dataset.save("social_network.jsonl")

# Fine-tune adapters for Multi-LoRA (example placeholder)
# This part would involve additional libraries and specific configurations
print("Fine-tuning adapters for Multi-LoRA would go here.")
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Install the dependencies required for building and serving the project documentation locally using mkdocs.

```bash
uv pip install -e ".[docs]"
```

--------------------------------

### Benchmarking with Distilabel Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example demonstrates how to reproduce the Arena Hard benchmark using Distilabel. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.preference import PreferenceTask

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the preference task
task = PreferenceTask(
    "What is the best response to the user?",
    "Which of the two responses is better?",
    llm=llm,
)

# Generate a preference dataset for benchmarking
dataset = task.generate(num_examples=1000)
dataset.save("arena_hard_benchmark.jsonl")
```

--------------------------------

### Define Pipeline with GlobalStep and Get Load Stages

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/load_groups_and_execution_stages.md

This example defines a Distilabel pipeline including a GlobalStep and then retrieves and prints the defined load stages. Use this to visualize how your pipeline is segmented for execution.

```python
from typing import TYPE_CHECKING

from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromDicts, StepInput, step

if TYPE_CHECKING:
    from distilabel.typing import StepOutput


@step(inputs=["instruction"], outputs=["instruction2"])
def DummyStep(inputs: StepInput) -> "StepOutput":
    for input in inputs:
        input["instruction2"] = "miau"
    yield inputs


@step(inputs=["instruction"], outputs=["instruction2"], step_type="global")
def GlobalDummyStep(inputs: StepInput) -> "StepOutput":
    for input in inputs:
        input["instruction2"] = "miau"
    yield inputs


with Pipeline() as pipeline:
    generator = LoadDataFromDicts(data=[{"instruction": "Hi"}] * 50)
    dummy_step_0 = DummyStep()
    global_dummy_step = GlobalDummyStep()
    dummy_step_1 = DummyStep()

    generator >> dummy_step_0 >> global_dummy_step >> dummy_step_1

if __name__ == "__main__":
    load_stages = pipeline.get_load_stages()

    for i, steps_stage in enumerate(load_stages[0]):
        print(f"Stage {i}: {steps_stage}")
```

--------------------------------

### Run Distilabel Exam Generation Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/examples/exam_questions.md

Execute the Python script to run the Distilabel pipeline for generating exam questions. Ensure the 'wikipedia' library is installed.

```python
python examples/exam_questions.py
```

--------------------------------

### Structured Generation with Outlines Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example shows how to generate RPG characters following a pydantic.BaseModel with outlines in Distilabel. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.payload import PayloadTask
from pydantic import BaseModel

class RPGCharacter(BaseModel):
    name: str
    class_name: str
    level: int

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the payload task with outlines
task = PayloadTask(
    "Generate an RPG character.",
    model=RPGCharacter,
    llm=llm,
)

# Generate an RPG character
character = task.generate_one()
print(character.model_dump())

# Generate multiple RPG characters
dataset = task.generate(num_examples=5)
dataset.save("rpg_characters.jsonl")
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Start the mkdocs development server to preview the documentation locally. Changes are typically reloaded automatically.

```bash
mkdocs serve
```

--------------------------------

### Load and Prepare Few-Shot Examples

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/apigen.md

Loads a subset of a dataset from Hugging Face and shuffles it to be used as few-shot examples. The dataset is then converted to a list of dictionaries for further processing.

```python
ds_og = (
    load_dataset("Salesforce/xlam-function-calling-60k", split="train")
    .shuffle(seed=42)
    .select(range(500))
    .to_list()
)
```

--------------------------------

### Install Test Dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Install the necessary dependencies to run unit and integration tests for the distilabel project.

```bash
uv pip install ".[tests]"
```

--------------------------------

### Install Transformers and Torch

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/clean_existing_dataset.ipynb

Installs specific versions of the transformers and torch libraries required for the tutorial.

```python
!pip install "transformers~=4.0" "torch~=2.0"
```

--------------------------------

### Navigate to Distilabel Directory

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Change the current directory to the distilabel folder to begin the setup process.

```bash
cd distilabel
```

--------------------------------

### Install Distilabel with Outlines Support

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/structured_generation.md

Install Distilabel with the necessary dependencies for Outlines integration to enable structured output generation.

```bash
pip install distilabel[outlines]
```

--------------------------------

### Install Distilabel with HF Inference Endpoints Extra

Source: https://github.com/argilla-io/distilabel/blob/main/README.md

To run the example using Hugging Face Inference Endpoints, install distilabel with the 'hf-inference-endpoints' extra. This command also upgrades the package.

```sh
pip install "distilabel[hf-inference-endpoints]" --upgrade
```

--------------------------------

### Install Graphviz for Visualization

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/examples/mistralai_with_instructor.md

Install the graphviz library using pip to enable visualization of generated knowledge graphs.

```console
pip install graphviz
```

--------------------------------

### Distilabel Text Generation Pipeline Example

Source: https://github.com/argilla-io/distilabel/blob/main/README.md

This Python script demonstrates how to set up a text generation pipeline using Distilabel with Hugging Face Inference Endpoints. It loads a dataset, runs the pipeline, and pushes the results to the Hugging Face Hub. Ensure you have installed the necessary extras.

```python
from datasets import load_dataset

from distilabel.models import InferenceEndpointsLLM
from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration

with Pipeline() as pipeline:
    TextGeneration(
        llm=InferenceEndpointsLLM(
            model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
            generation_kwargs={"temperature": 0.7, "max_new_tokens": 512},
        ),
    )

if __name__ == "__main__":
    dataset = load_dataset("distilabel-internal-testing/instructions", split="test")
    distiset = pipeline.run(dataset=dataset)
    distiset.push_to_hub(repo_id="distilabel-example")
```

--------------------------------

### Text Generation with Images Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example demonstrates how to ask questions about images using Distilabel. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.multimodal import MultimodalTask

# Define the LLM to use
llm = OpenAILLM(model="gpt-4-vision-preview")

# Define the multimodal task
task = MultimodalTask(
    "What is in this image?",
    llm=llm,
)

# Load image data (example placeholder)
image_path = "path/to/your/image.jpg"

# Ask a question about the image
response = task.generate_one(image=image_path)
print(f"Response to image question: {response}")

# Generate multiple responses for different images or questions
# This would typically involve a dataset of images and corresponding questions
dataset = task.generate(num_examples=10, images=["path/to/image1.jpg", "path/to/image2.jpg"])
dataset.save("image_qa_responses.jsonl")
```

--------------------------------

### Image Generation Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example shows how to generate synthetic images using Distilabel. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.image_generation import ImageGenerationTask

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the image generation task
task = ImageGenerationTask(
    "Generate an image of a futuristic city.",
    llm=llm,
)

# Generate a synthetic image
image_data = task.generate_one()
# The image_data would typically be a URL or base64 encoded string
print(f"Generated image data: {image_data}")

# Generate multiple synthetic images
dataset = task.generate(num_examples=5)
dataset.save("generated_images.jsonl")
```

--------------------------------

### Install distilabel with Instructor Dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/structured_generation.md

Install the distilabel library with the necessary dependencies for Instructor integration. This command ensures all required packages are available for structured output generation.

```bash
pip install distilabel[instructor]
```

--------------------------------

### Structured Generation with Instructor Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example demonstrates answering instructions with knowledge graphs defined as pydantic.BaseModel objects using instructor in Distilabel. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.payload import PayloadTask
from pydantic import BaseModel

class KnowledgeGraph(BaseModel):
    subject: str
    predicate: str
    object: str

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the payload task with instructor
task = PayloadTask(
    "Extract a knowledge graph triple from the following text.",
    model=KnowledgeGraph,
    llm=llm,
)

# Generate a knowledge graph triple
triple = task.generate_one()
print(triple.model_dump())

# Generate multiple knowledge graph triples
dataset = task.generate(num_examples=10)
dataset.save("knowledge_graph_triples.jsonl")
```

--------------------------------

### Example Pipeline with OpenAILLM Offline Batch Generation

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/offline_batch_generation.md

This example demonstrates setting up a Distilabel pipeline that utilizes OpenAILLM with offline batch generation enabled. Ensure pipeline caching is enabled for this feature to work correctly.

```python
from distilabel.models import OpenAILLM
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromHub
from distilabel.steps.tasks import TextGeneration

with Pipeline() as pipeline:
    load_data = LoadDataFromHub(output_mappings={"prompt": "instruction"})

    text_generation = TextGeneration(
        llm=OpenAILLM(
            model="gpt-3.5-turbo",
            use_offline_batch_generation=True,  # (1)
        )
    )

    load_data >> text_generation


if __name__ == "__main__":
    distiset = pipeline.run(
        parameters={
            load_data.name: {
                "repo_id": "distilabel-internal-testing/instruction-dataset",
                "split": "test",
                "batch_size": 500,
            },
        }
    )
```

--------------------------------

### Install Distilabel with Extras

Source: https://context7.com/argilla-io/distilabel/llms.txt

Install Distilabel with specific LLM provider extras or features like structured generation and distributed execution. Multiple extras can be installed at once.

```bash
pip install distilabel --upgrade
```

```bash
pip install "distilabel[hf-inference-endpoints]" --upgrade
```

```bash
pip install "distilabel[openai]" --upgrade
```

```bash
pip install "distilabel[vllm]" --upgrade
```

```bash
pip install "distilabel[anthropic]" --upgrade
```

```bash
pip install "distilabel[outlines]" --upgrade
```

```bash
pip install "distilabel[instructor]" --upgrade
```

```bash
pip install "distilabel[ray]" --upgrade
```

```bash
pip install "distilabel[openai,vllm,ray]" --upgrade
```

--------------------------------

### Install Development Dependencies and Pre-commit Hooks

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Install development dependencies, including tools for linting and formatting like ruff, and set up pre-commit hooks to ensure code quality.

```bash
uv pip install -e ".[dev]"
pre-commit install
```

--------------------------------

### Create Exam Questions and Answers Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This example demonstrates how to generate questions and answers for an exam using a raw Wikipedia page and structured generation. It is part of the Examples section.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.payload import PayloadTask
from pydantic import BaseModel

class QuestionAnswer(BaseModel):
    question: str
    answer: str

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the payload task for question-answer generation
task = PayloadTask(
    "Generate a question and answer pair from the following text.",
    model=QuestionAnswer,
    llm=llm,
)

# Load raw text from a Wikipedia page (example placeholder)
with open("wikipedia_page.txt", "r") as f:
    wikipedia_text = f.read()

# Generate questions and answers
dataset = task.generate(num_examples=20, prompt_prefix=wikipedia_text)
dataset.save("exam_questions_answers.jsonl")
```

--------------------------------

### Install distilabel with Ray support

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/scaling_with_ray.md

Install the necessary package to enable distilabel's integration with Ray for distributed pipeline execution. It's recommended to use a virtual environment.

```bash
pip install distilabel[ray]
```

--------------------------------

### Install distilabel with vLLM support

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/prometheus.md

Install the distilabel library with vLLM support for enhanced performance. This is a prerequisite for using Prometheus 2 within distilabel.

```bash
pip install "distilabel[vllm]>=1.1.0"
```

--------------------------------

### Dataset Structure Example (JSON)

Source: https://github.com/argilla-io/distilabel/blob/main/src/distilabel/utils/card/distilabel_template.md

Example JSON structure for a dataset record within a specific configuration. This illustrates the data format.

```json
{
    "text": "A simple example sentence.",
    "label": "positive",
    "metadata": {
        "source": "example.com",
        "timestamp": "2023-01-01T12:00:00Z"
    }
}
```

--------------------------------

### Install distilabel and dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deita.md

Installs the distilabel library with support for OpenAI and Hugging Face Transformers, along with other necessary packages like pynvml and argilla.

```bash
pip install "distilabel[openai,hf-transformers]>=1.0.0"
pip install pynvml huggingface_hub argilla
```

--------------------------------

### Install Distilabel from Source

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/getting_started/installation.md

Install the latest unreleased version of Distilabel directly from its GitHub repository. This is useful for accessing the newest features and bug fixes before they are officially released.

```sh
pip install "distilabel @ git+https://github.com/argilla-io/distilabel.git@develop" --upgrade
```

--------------------------------

### Install Distilabel with Argilla, OpenAI, and vLLM

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/ultrafeedback.md

Install the distilabel library with support for Argilla, OpenAI, and vLLM. Ensure you are using version 1.0.0 or higher.

```bash
pip install "distilabel[argilla,openai,vllm]>=1.0.0"
```

--------------------------------

### Text Generation Task Setup

Source: https://context7.com/argilla-io/distilabel/llms.txt

Instantiate and use the TextGeneration task for generating multiple responses to a single instruction. Ensure the LLM is loaded before processing.

```python
from distilabel.models import InferenceEndpointsLLM
from distilabel.steps.tasks import TextGeneration

task = TextGeneration(
    name="text-generation",
    llm=InferenceEndpointsLLM(
        model_id="meta-llama/Meta-Llama-3-70B-Instruct",
        tokenizer_id="meta-llama/Meta-Llama-3-70B-Instruct",
    ),
    num_generations=3,  # Generate multiple responses
    group_generations=True,  # Group into single output row
)
task.load()
```

--------------------------------

### Initialize and Configure Pipeline Steps

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/math_shepherd.md

This snippet demonstrates initializing various Distilabel steps, including data loading, LLM configurations, and specialized tasks like MathShepherdGenerator and MathShepherdCompleter. It also shows how to set up parallel and sequential execution of these steps.

```python
from datasets import load_dataset

from distilabel.steps.tasks import MathShepherdCompleter, MathShepherdGenerator, FormatPRM
from distilabel.models import InferenceEndpointsLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import CombineOutputs, ExpandColumns

ds_name = "openai/gsm8k"

ds = load_dataset(ds_name, "main", split="test").rename_column("question", "instruction").select(range(3))  # (1)

with Pipeline(name="Math-Shepherd") as pipe:
    model_id_70B = "meta-llama/Meta-Llama-3.1-70B-Instruct"
    model_id_8B = "meta-llama/Meta-Llama-3.1-8B-Instruct"

    llm_70B = InferenceEndpointsLLM(
        model_id=model_id_70B,
        tokenizer_id=model_id_70B,
        generation_kwargs={"max_new_tokens": 1024, "temperature": 0.6},
    )
    llm_8B = InferenceEndpointsLLM(
        model_id=model_id_8B,
        tokenizer_id=model_id_8B,
        generation_kwargs={"max_new_tokens": 2048, "temperature": 0.6},
    )  # (2)

    generator_golden = MathShepherdGenerator(
        name="golden_generator",
        llm=llm_70B,
    )  # (3)
    generator = MathShepherdGenerator(
        name="generator",
        llm=llm_8B,
        use_default_structured_output=True,  # (9)
        M=5
    )  #… (4)
    completer = MathShepherdCompleter(
        name="completer",
        llm=llm_8B,
        use_default_structured_output=True,
        N=4
    )  # (5)

    combine = CombineOutputs()

    expand = ExpandColumns(
        name="expand_columns",
        columns=["solutions"],
        split_statistics=True,
    )  #… (6)
    formatter = FormatPRM(name="format_prm")  # (7)

    [generator_golden, generator] >> combine >> completer >> expand >> formatter # (8)
```

--------------------------------

### Define and run a distributed pipeline with Ray

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/scaling_with_ray.md

This example demonstrates how to define a distilabel pipeline with multiple steps and configure it to run on a Ray cluster. It includes setting up resources like replicas and GPUs for specific steps and pushing the resulting dataset to the Hugging Face Hub.

```python
from distilabel.models import vLLM
from distilabel.pipeline import Pipeline
from distilabel.steps import LoadDataFromHub
from distilabel.steps.tasks import TextGeneration

with Pipeline(name="text-generation-ray-pipeline") as pipeline:
    load_data_from_hub = LoadDataFromHub(output_mappings={"prompt": "instruction"})

    text_generation = TextGeneration(
        llm=vLLM(
            model="meta-llama/Meta-Llama-3-8B-Instruct",
            tokenizer="meta-llama/Meta-Llama-3-8B-Instruct",
        )
    )

    load_data_from_hub >> text_generation

if __name__ == "__main__":
    distiset = pipeline.run(
        parameters={
            load_data_from_hub.name: {
                "repo_id": "HuggingFaceH4/instruction-dataset",
                "split": "test",
            },
            text_generation.name: {
                "llm": {
                    "generation_kwargs": {
                        "temperature": 0.7,
                        "max_new_tokens": 4096,
                    }
                },
                "resources": {"replicas": 2, "gpus": 1}, # (1)
            },
        }
    )

    distiset.push_to_hub(
        "<YOUR_HF_USERNAME_OR_ORGANIZATION>/text-generation-distilabel-ray" # (2)
    )
```

--------------------------------

### Generate Preference Dataset Tutorial

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This tutorial covers synthetic data generation for ORPO and DPO. It is part of the end-to-end tutorials.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.preference import PreferenceTask

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the preference task
task = PreferenceTask(
    "What is the best response to the user?",
    "Which of the two responses is better?",
    llm=llm,
)

# Generate a preference dataset
dataset = task.generate(num_examples=100)
dataset.save("preference_dataset.jsonl")
```

--------------------------------

### Clean Existing Preference Dataset Tutorial

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/index.md

This tutorial demonstrates how to provide AI feedback to clean an existing dataset. It is part of the end-to-end tutorials.

```python
from distilabel.llms import OpenAILLM
from distilabel.tasks.preference import PreferenceTask

# Define the LLM to use
llm = OpenAILLM(model="gpt-4")

# Define the preference task
task = PreferenceTask(
    "What is the best response to the user?",
    "Which of the two responses is better?",
    llm=llm,
)

# Load an existing dataset
dataset = load_dataset("preference_dataset.jsonl")

# Clean the dataset
cleaned_dataset = task.clean(dataset)
cleaned_dataset.save("cleaned_preference_dataset.jsonl")
```

--------------------------------

### Use InstructionResponsePipeline

Source: https://context7.com/argilla-io/distilabel/llms.txt

Instantiate and run the pre-built InstructionResponsePipeline for supervised fine-tuning data generation. Customize the pipeline with a system prompt.

```python
from distilabel.pipeline import InstructionResponsePipeline

# Use the generic pipeline with default settings
pipeline = InstructionResponsePipeline()
dataset = pipeline.run()

# Customize with a system prompt
pipeline = InstructionResponsePipeline(
    system_prompt="You are a helpful coding assistant that provides clear explanations."
)
dataset = pipeline.run()
```

--------------------------------

### Install Distilabel with HF Inference Endpoints

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/getting_started/quickstart.md

Install the latest release of Distilabel, including the `hf-inference-endpoints` extra, using pip. This command upgrades the package if it's already installed.

```sh
pip install distilabel[hf-inference-endpoints] --upgrade
```

--------------------------------

### Install Sentence Transformers

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/GenerateSentencePair.ipynb

Installs the sentence-transformers library, specifying a version compatible with the project.

```python
!pip install "sentence-transformers~=3.0"
```

--------------------------------

### Install fsspec implementation for cloud storage

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/fs_to_pass_data.md

To use specific file systems like Google Cloud Storage, install the corresponding fsspec implementation package. For GCS, use `pip install gcsfs`.

```bash
pip install gcsfs
```

--------------------------------

### Run Pipeline with Artifact Saving and Push to Hub

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/saving_step_generated_artifacts.md

This example demonstrates how to set up a Distilabel pipeline that includes the `CountTextCharacters` step. It then runs the pipeline with a dataset loaded from Hugging Face Hub and prepares the resulting `Distiset` for potential upload.

```python
from typing import TYPE_CHECKING, List

import matplotlib.pyplot as plt
from datasets import load_dataset
from distilabel.pipeline import Pipeline
from distilabel.steps import GlobalStep, StepInput, StepOutput

if TYPE_CHECKING:
    from distilabel.steps import StepOutput


class CountTextCharacters(GlobalStep):
    @property
    def inputs(self) -> List[str]:
        return ["text"]

    @property
    def outputs(self) -> List[str]:
        return ["text_character_count"]

    def process(self, inputs: StepInput) -> "StepOutput":  # type: ignore
        character_counts = []

        for input in inputs:
            text_character_count = len(input["text"])
            input["text_character_count"] = text_character_count
            character_counts.append(text_character_count)

        # Generate plot with the distribution of text character counts
        plt.figure(figsize=(10, 6))
        plt.hist(character_counts, bins=30, edgecolor="black")
        plt.title("Distribution of Text Character Counts")
        plt.xlabel("Character Count")
        plt.ylabel("Frequency")

        # Save the plot as an artifact of the step
        self.save_artifact(
            name="text_character_count_distribution",
            write_function=lambda path: plt.savefig(path / "figure.png"),
            metadata={"type": "image", "library": "matplotlib"},
        )

        plt.close()

        yield inputs


with Pipeline() as pipeline:
    count_text_characters = CountTextCharacters()

if __name__ == "__main__":
    distiset = pipeline.run(
        dataset=load_dataset(
            "HuggingFaceH4/instruction-dataset", split="test"
        ).rename_column("prompt", "text"),
    )
```

--------------------------------

### Install Distilabel

Source: https://github.com/argilla-io/distilabel/blob/main/README.md

Install the distilabel package using pip. This command upgrades the package to the latest version.

```sh
pip install distilabel --upgrade
```

--------------------------------

### Install distilabel

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/clair.md

Install distilabel with version 1.4.0 or higher. Additional dependencies may be required based on the LLM provider.

```bash
pip install "distilabel>=1.4.0"
```

--------------------------------

### Initialize Argilla Client

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/generate_textcat_dataset.ipynb

Instantiate the Argilla client with API URL and key. Uncomment the last line and set your HF_TOKEN if your space is private.

```python
client = rg.Argilla(
    api_url="https://[your-owner-name]-[your_space_name].hf.space",
    api_key="[your-api-key]",
    # headers={"Authorization": f"Bearer {HF_TOKEN}"}
)
```

--------------------------------

### Install Distilabel with Extra Dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Install the distilabel package with specific extra dependencies, such as 'vllm' and 'outlines', for specialized development.

```bash
uv pip install -e ".[vllm,outlines]"
```

--------------------------------

### Use DummyLLM with Task.print

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/basic/task/index.md

Instantiate a task with a `DummyLLM` and call `load()` and `print()` to test prompt formatting without a real LLM. This is useful for development and testing.

```python
uf = UltraFeedback(llm=DummyLLM())
uf.load()
uf.print()
```

--------------------------------

### Start Ray Workers on SLURM

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/math_shepherd.md

This script configures and starts Ray worker nodes on a SLURM cluster. It iterates through available nodes, starts a Ray instance on each, and waits for them to connect. Ensure the head node IP and SLURM environment variables are correctly set.

```bash
worker_num=$((SLURM_JOB_NUM_NODES - 1))

    # Start from 1 (0 is head node)
    for ((i = 1; i <= worker_num; i++)); do
        node_i=${nodes_array[$i]}
        worker_tmp_dir="/tmp/ray_tmp_${SLURM_JOB_ID}_worker_$i"
        echo "Starting WORKER $i at $node_i"
        srun --nodes=1 --ntasks=1 -w "$node_i" \
            ray start --address "$ip_head" \
            --temp-dir="$worker_tmp_dir" \
            --block &
        sleep 5
    done

    # Give some time to the Ray cluster to gather info
    sleep 60
```

--------------------------------

### Distilabel Pipeline Run CLI Help

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/cli/index.md

Displays the available options and usage for the `distilabel pipeline run` command. Use this to understand the various parameters for configuring pipeline execution.

```bash
distilabel pipeline run --help
```

--------------------------------

### Initialize LLM with Instructor for Structured Output

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/how_to_guides/advanced/structured_generation.md

Initialize an LLM (e.g., InferenceEndpointsLLM) and configure it to use Instructor for structured output by providing the Pydantic schema. Ensure the model supports structured outputs.

```python
from distilabel.models import InferenceEndpointsLLM
from pydantic import BaseModel

class User(BaseModel):
    name: str
    last_name: str
    id: int

llm = InferenceEndpointsLLM(
    model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
    tokenizer_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
    structured_output={"schema": User}
)
llm.load()
```

--------------------------------

### Preference Dataset Structure Example

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/generate_preference_dataset.ipynb

This is an example of the resulting preference dataset structure after processing with `FormatTextGenerationDPO`. It includes 'chosen' and 'rejected' fields for model comparison.

```python
Result:
[{'instruction': "What's the capital of Spain?",
  'generations': ['Madrid', 'Barcelona'],
  'generation_models': ['Meta-Llama-3-8B-Instruct',
   'Mixtral-8x7B-Instruct-v0.1'],
  'ratings': [5, 1],
  'prompt': "What's the capital of Spain?",
  'prompt_id': '26174c953df26b3049484e4721102dca6b25d2de9e3aa22aa84f25ed1c798512',
  'chosen': [{'role': 'user', 'content': "What's the capital of Spain?"},
   {'role': 'assistant', 'content': 'Madrid'}],
  'chosen_model': 'Meta-Llama-3-8B-Instruct',
  'chosen_rating': 5,
  'rejected': [{'role': 'user', 'content': "What's the capital of Spain?"},
   {'role': 'assistant', 'content': 'Barcelona'}],
  'rejected_model': 'Mixtral-8x7B-Instruct-v0.1',
  'rejected_rating': 1}]
```

--------------------------------

### Create and Run a Basic Text Generation Pipeline

Source: https://context7.com/argilla-io/distilabel/llms.txt

Define a text generation pipeline using a context manager, specifying the LLM and generation parameters. Load a dataset and run the pipeline, then push the results to Hugging Face Hub.

```python
from datasets import load_dataset
from distilabel.models import InferenceEndpointsLLM
from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration

# Create pipeline using context manager
with Pipeline(name="text-generation-pipeline", description="Generate responses") as pipeline:
    TextGeneration(
        llm=InferenceEndpointsLLM(
            model_id="meta-llama/Meta-Llama-3.1-8B-Instruct",
            generation_kwargs={"temperature": 0.7, "max_new_tokens": 512},
        ),
    )

if __name__ == "__main__":
    # Load dataset and run pipeline
    dataset = load_dataset("distilabel-internal-testing/instructions", split="test")
    distiset = pipeline.run(dataset=dataset)

    # Push results to Hugging Face Hub
    distiset.push_to_hub(repo_id="my-org/generated-dataset")
```

--------------------------------

### Install distilabel with HF Inference Endpoints

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deepseek_prover.md

Install distilabel with the necessary dependencies for using Hugging Face inference endpoints. This is a prerequisite for using the InferenceEndpointsLLM.

```bash
pip install "distilabel[hf-inference-endpoints]"
```

--------------------------------

### Full Pipeline Script

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deepseek_prover.md

This script demonstrates how to assemble Distilabel building blocks to create a complete pipeline for generating mathematical proofs. It utilizes the DeepSeekProverSolver and can be run with or without a dry run.

```python
--8<-- "examples/deepseek_prover.py"
```

--------------------------------

### Initialize Argilla Client and Define Dataset Settings

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/GenerateSentencePair.ipynb

Sets up the Argilla client with API credentials and defines the schema for datasets, including text fields, label questions, metadata properties, and vector fields. Handles potential conflicts if datasets already exist.

```python
import argilla as rg
from argilla._exceptions import ConflictError

api_key = "ohh so secret"
api_url = "https://[your-owner-name]-[your-space-name].hf.space"

client = rg.Argilla(api_url=api_url, api_key=api_key)

settings = rg.Settings(
    fields=[
        rg.TextField("anchor")
    ],
    questions=[
        rg.TextQuestion("positive"),
        rg.TextQuestion("negative"),
        rg.LabelQuestion(
            name="is_positive_relevant",
            title="Is the positive query relevant?",
            labels=["yes", "no"],
        ),
        rg.LabelQuestion(
            name="is_negative_irrelevant",
            title="Is the negative query irrelevant?",
            labels=["yes", "no"],
        )
    ],
    metadata=[
        rg.TermsMetadataProperty("filename"),
        rg.FloatMetadataProperty("similarity-positive-negative"),
        rg.FloatMetadataProperty("similarity-anchor-positive"),
        rg.FloatMetadataProperty("similarity-anchor-negative"),
    ],
    vectors=[
        rg.VectorField("anchor-vector", dimensions=model.get_sentence_embedding_dimension())
    ]
)
rg_datasets = []
for dataset_name in ["generate_retrieval_pairs", "generate_reranking_pairs"]:
    ds = rg.Dataset(
        name=dataset_name,
        settings=settings
    )
    try:
        ds.create()
    except ConflictError:
        ds = client.datasets(dataset_name)
    rg_datasets.append(ds)
```

--------------------------------

### Define Initial Instruction and Response

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deita.md

Sets up the initial instruction and response pair for the quality evaluation process. Ensure these are correctly formatted strings.

```python
instruction_0 = "Give three tips for staying healthy."
reponse_0 = "1. Eat a balanced and nutritious diet: Make sure your meals are inclusive of a variety of fruits and vegetables, lean protein, whole grains, and healthy fats. This helps to provide your body with the essential nutrients to function at its best and can help prevent chronic diseases. 2. Engage in regular physical activity: Exercise is crucial for maintaining strong bones, muscles, and cardiovascular health. Aim for at least 150 minutes of moderate aerobic exercise or 75 minutes of vigorous exercise each week. 3. Get enough sleep: Getting enough quality sleep is crucial for physical and mental well-being. It helps to regulate mood, improve cognitive function, and supports healthy growth and immune function. Aim for 7-9 hours of sleep each night."
```

--------------------------------

### Structured Generation with Instructor

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/examples/mistralai_with_instructor.md

This script utilizes MistralLLM and Instructor for structured output generation, creating knowledge graphs from complex topics. Ensure necessary libraries are installed.

```python
--8<-- "examples/structured_generation_with_instructor.py"
```

--------------------------------

### Install Distilabel with HF Inference Endpoints

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/tutorials/clean_existing_dataset.ipynb

Installs the distilabel SDK along with the necessary dependencies for using Hugging Face's serverless Inference API.

```python
!pip install "distilabel[hf-inference-endpoints]"
```

--------------------------------

### Install Distilabel with Extra Dependencies

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/instruction_backtranslation.md

Install distilabel with necessary dependencies for Hugging Face inference endpoints and OpenAI integration. Ensure you have version 1.0.0 or higher.

```bash
pip install "distilabel[hf-inference-endpoints,openai]>=
1.0.0"
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Create a virtual environment using uv with Python 3.11 and then activate it. This isolates project dependencies.

```bash
uv venv .venv --python 3.11
source .venv/bin/activate
```

--------------------------------

### Install Distilabel Locally

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/community/developer_documentation.md

Install the distilabel package in editable mode from the local source using uv pip. This allows for direct code changes to be reflected.

```bash
uv pip install -e .
```

--------------------------------

### Initialize and Load EvolInstruct Pipeline

Source: https://github.com/argilla-io/distilabel/blob/main/docs/sections/pipeline_samples/papers/deita.md

Initializes the EvolInstruct pipeline with specified parameters, including the LLM, number of evolutions, and whether to store intermediate steps and generate answers. The pipeline is then loaded.

```python
evol_instruction_complexity = EvolInstruct(
    name="evol_instruction_complexity",
    llm=OpenAILLM(model="gpt-3.5-turbo"),
    num_evolutions=5,
    store_evolutions=True,
    generate_answers=True,
    include_original_instruction=True,
    pipeline=pipeline,
)

evol_instruction_complexity.load()
```