### Install Pre-commit Hooks

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Install pre-commit hooks to ensure code style consistency before committing.

```bash
pre-commit install
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Installs all optional dependencies for DataTrove using uv. Ensure you have created a virtual environment first.

```bash
uv sync --all-extras
```

--------------------------------

### Python Example for Optional Dependencies

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Demonstrates how to declare optional dependencies for a pipeline step using `_requires_dependencies`. This prevents import errors if the dependency is not installed.

```python
from pipeline.base import PipelineStep

class StepWithOptionalDeps(PipelineStep):
    def _requires_dependencies(self):
        return {"my_optional_lib": "my_optional_lib"}

    def run(self, docs):
        # Code that uses my_optional_lib can go here
        # It will only be executed if the dependency is met
        return docs
```

--------------------------------

### Minimal End-to-End Inference Example

Source: https://github.com/huggingface/datatrove/blob/main/README.md

This example demonstrates a basic setup for running inference with Datatrove, including document preparation, inference configuration, and pipeline execution. It utilizes local checkpoints and S3 for output.

```python
from datatrove.data import Document
from datatrove.executor.local import LocalPipelineExecutor
from datatrove.pipeline.inference.run_inference import InferenceConfig, InferenceRunner
from datatrove.pipeline.writers import JsonlWriter

async def simple_rollout(doc: Document, generate):
    payload = {"messages": [{"role": "user", "content": [{"type": "text", "text": doc.text}]}], "max_tokens": 2048}
    return await generate(payload)

documents = [Document(text="What's the weather in Tokyo?", id=str(i)) for i in range(1005)]
config = InferenceConfig(server_type="vllm", model_name_or_path="google/gemma-3-27b-it", rollouts_per_document=1, max_concurrent_generations=500)

LocalPipelineExecutor(
    pipeline=[
        documents,
        InferenceRunner(
            rollout_fn=simple_rollout,
            config=config,
            skip_bad_requests=True,
            records_per_chunk=500,
            checkpoints_local_dir="/fsx/.../translate-checkpoints",
            output_writer=JsonlWriter("s3://.../final_output_data", output_filename="${rank}_chunk_${chunk_index}.jsonl"),
        ),
    ],
    logging_dir="/fsx/.../inference_logs",
    tasks=1,
).run()
```

--------------------------------

### Install DataTrove with all dependencies

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs all available dependencies for DataTrove. This is a convenient option for users who need the full suite of features.

```bash
uv sync --extra all
```

--------------------------------

### Basic DataTrove Installation

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs the core DataTrove package. Requires Python 3.10+.

```bash
uv sync
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Clone the DataTrove repository and install development dependencies using uv.

```bash
git clone git@github.com:huggingface/datatrove.git && cd datatrove
uv sync --extra dev
```

--------------------------------

### RayPipelineExecutor Example

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Launches a pipeline on a Ray cluster using Ray tasks. Configure tasks, workers, and logging directory.

```python
import ray
from datatrove.executor import RayPipelineExecutor
ray.init()
executor = RayPipelineExecutor(
    pipeline=[
        ...
    ],
    logging_dir="logs/",
    tasks=500,
    workers=100,  # omit to run all at once
)
executor.run()
```

--------------------------------

### Bash Example for Running Pytest

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Shows how to run Pytest tests, including options for verbose output, specific files, and rerunning failed tests. Useful for local development and CI.

```bash
pytest -sv ./tests/
```

```bash
pytest -sv ./tests/pipeline/test_filters.py
```

--------------------------------

### Example of a Simple Single-Request Rollout Function

Source: https://github.com/huggingface/datatrove/blob/main/README.md

A basic async callable that receives a Document and a generate callback. Suitable for straightforward generation tasks.

```python
async def my_rollout(doc, generate, **kwargs):
    # Your generation logic here
    pass
```

--------------------------------

### Install DataTrove with Inference Dependencies

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Installs the datatrove package along with necessary inference dependencies. Ensure you are logged into your Hugging Face account with write access.

```sh
uv sync --extra inference
```

--------------------------------

### Install DataTrove with CLI tools

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for the command-line interface, providing tools to manage and interact with DataTrove from the terminal.

```bash
uv sync --extra cli
```

--------------------------------

### Install DataTrove with decontamination support

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for decontamination tasks using lighteval. This is relevant for evaluating and mitigating data contamination in models.

```bash
uv sync --extra decont
```

--------------------------------

### Install DataTrove with multilingual processing

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for multilingual text processing. This is essential for working with datasets in multiple languages.

```bash
uv sync --extra multilingual
```

--------------------------------

### Speculative Decoding Configuration Example

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Example YAML configuration for sweeping speculative decoding methods. Includes None, N-gram, and Suffix decoding options.

```yaml
experiments:
  - name: "gemma-spec-sweep"
    args:
      model-name-or-path: "google/gemma-3-1b-it"
      speculative-config:
        - None  # No speculative decoding
        - '{"method": "ngram", "num_speculative_tokens": 5}'  # N-gram
        - '{"method": "suffix", "num_speculative_tokens": 32}'  # Suffix
        # Draft model (not yet supported as of vLLM 0.15.0):
        # - '{"model": "facebook/opt-125m", "num_speculative_tokens": 5}'
```

--------------------------------

### Install DataTrove with processing dependencies

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies necessary for text extraction, filtering, and tokenization. This is useful for data cleaning and preparation pipelines.

```bash
uv sync --extra processing
```

--------------------------------

### Inference Progress Monitoring Setup

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Configure progress monitoring and dataset card generation for long-running inference jobs. The monitor runs in parallel with inference, while the generator runs after completion.

```python
from datatrove.pipeline.inference import InferenceDatasetCardParams, InferenceProgressMonitor, InferenceDatasetCardGenerator

params = InferenceDatasetCardParams(
    output_repo_id="your-username/output-dataset",
    input_dataset_name="simplescaling/s1K-1.1",
    input_dataset_split="train",
    model_name="Qwen/Qwen3-0.6B",
    # ... other params
)

# Monitor pipeline (runs in parallel with inference on Slurm)
monitor_pipeline = [InferenceProgressMonitor(params=params, update_interval=3600)]

# Final card generation (runs after inference completes)
datacard_pipeline = [InferenceDatasetCardGenerator(params=params)]
```

--------------------------------

### Install DataTrove with IO dependencies

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies required for reading various file formats like warc, arc, wet, arrow, parquet, and optimized-parquet. Use this when working with diverse data sources.

```bash
uv sync --extra io
```

--------------------------------

### JSON Example: Prompt with Template

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Example of a dataset row from WikiText when using `--prompt-template`. The `[[DOCUMENT]]` variable in the template is replaced with the text from the specified column.

```json
{"text": "The sky appears blue during the day due to Rayleigh scattering..."}
```

--------------------------------

### Install DataTrove with inference pipelines

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies required for LLM inference pipelines. Use this when deploying or experimenting with large language models.

```bash
uv sync --extra inference
```

--------------------------------

### Install DataTrove with S3 support

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for S3 integration, enabling DataTrove to interact with S3 buckets for data storage and retrieval.

```bash
uv sync --extra s3
```

--------------------------------

### Example of Writing Raw Data to HuggingFace Buckets

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Use this pattern for storing raw or intermediate data before cleaning and publishing.

```python
hf://buckets/<org>/<bucket>/...
```

--------------------------------

### SlurmPipelineExecutor Example

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Launches a pipeline on a Slurm cluster using job arrays. Define tasks, time, partition, and dependencies between executors.

```python
from datatrove.executor import SlurmPipelineExecutor
executor1 = SlurmPipelineExecutor(
    pipeline=[
        ...
    ],
    job_name="my_cool_job1",
    logging_dir="logs/job1",
    tasks=500,
    workers=100,  # omit to run all at once
    time="10:00:00",  # 10 hours
    partition="hopper-cpu"
)
executor2 = SlurmPipelineExecutor(
    pipeline=[
        ...
    ],
    job_name="my_cool_job2",
    logging_dir="logs/job2",
    tasks=1,
    time="5:00:00",  # 5 hours
    partition="hopper-cpu",
    depends=executor1  # this pipeline will only be launched after executor1 successfully completes
)
# executor1.run()
executor2.run() # this will actually launch executor1, as it is a dependency, so no need to launch it explicitly
```

--------------------------------

### Define a DataTrove Pipeline

Source: https://github.com/huggingface/datatrove/blob/main/README.md

This example shows how to define a pipeline using CSVReader, SamplerFilter, and JsonlWriter. Ensure the specified input and output paths are correctly configured.

```python
from datatrove.pipeline.readers import CSVReader
from datatrove.pipeline.filters import SamplerFilter
from datatrove.pipeline.writers import JsonlWriter

pipeline = [
    CSVReader(
        data_folder="/my/input/path"
    ),
    SamplerFilter(rate=0.5),
    JsonlWriter(
        output_folder="/my/output/path"
    )
]
```

--------------------------------

### Python Example for Custom Writer

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Illustrates creating a custom data writer by inheriting from `DiskWriter`. Implement the `_write` method to handle document writing, utilizing output filename templates for sharding.

```python
from pipeline.writers.disk_base import DiskWriter

class MyCustomWriter(DiskWriter):
    def _write(self, document, output_file):
        # Write the document to the output file
        # Use self.output_filename template (e.g., '${rank}_${id}.txt')
        pass
```

--------------------------------

### Install DataTrove with Ray for distributed compute

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for Ray, a distributed compute engine, allowing DataTrove to leverage distributed computing for larger tasks.

```bash
uv sync --extra ray
```

--------------------------------

### JSON Example: Multi-turn Chat Messages

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Example of a dataset row for multi-turn chat messages. The `--prompt-column` is set to 'messages', and a system message from `--system-prompt` is automatically prepended.

```json
{
  "messages": [
    {"role": "user", "content": "Hi"},
    {"role": "assistant", "content": "Hello! How can I help?"},
    {"role": "user", "content": "What color is the sky?"}
  ]
}
```

--------------------------------

### Python Example for Pipeline Block

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Illustrates the basic structure for adding a new pipeline block by inheriting from a base class and implementing required methods. Ensure to use provided utilities for metrics and dependency checks.

```python
from pipeline.base import PipelineStep

class MyCustomStep(PipelineStep):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        # Initialize custom attributes here

    def run(self, docs):
        # Process documents here
        # Use self.stat_update() and self.track_time() for metrics
        return docs
```

--------------------------------

### Create Virtual Environment

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Creates a Python virtual environment using uv, specifying the Python version. This is a prerequisite for installing dependencies.

```bash
uv venv --python 3.12
```

--------------------------------

### JSON Example: Single-turn Prompt

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Example of a dataset row when using `--prompt-column` with a plain string. The request sent to the model includes an optional system message and the prompt content as a user message.

```json
{"question": "What color is the sky?"}
```

--------------------------------

### Install DataTrove with specific IO and S3 dependencies

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Installs dependencies for both IO operations and S3 support, allowing DataTrove to read various file formats and interact with S3.

```bash
uv sync --extra io --extra s3
```

--------------------------------

### LocalPipelineExecutor Example

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Instantiates and runs a pipeline using the `LocalPipelineExecutor`. This executor runs tasks on a local machine, with options for total tasks and concurrent workers.

```python
from datatrove.executor import LocalPipelineExecutor
executor = LocalPipelineExecutor(
    pipeline=[
        ...
    ],
    logging_dir="logs/",
    tasks=10,
    workers=5
)
executor.run()
```

--------------------------------

### JSON Example: Single-turn Chat Message

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Example of a dataset row for a single-turn chat message. The `--prompt-column` is set to 'messages', and a system message from `--system-prompt` is automatically prepended.

```json
{"messages": [{"role": "user", "content": "What color is the sky?"}]}
```

--------------------------------

### Python Example for Custom Formatter

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Provides a template for creating a custom text formatter by inheriting from `BaseFormatter`. Implement the `format` method to transform document content as needed.

```python
from pipeline.formatters.base import BaseFormatter

class MyCustomFormatter(BaseFormatter):
    def format(self, doc):
        # Format the document content
        return doc
```

--------------------------------

### Configure Reader with Glob Pattern

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Use a glob pattern to match specific files within the data folder. This example matches .warc.gz files in subdirectories.

```python
glob_pattern="*/warc/*.warc.gz"
```

--------------------------------

### Python Example for Custom Reader

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Demonstrates how to create a custom data reader by inheriting from `BaseDiskReader`. Implement the `read_file` method to yield `Document` objects, respecting specified keys and adapters.

```python
from pipeline.readers.base import BaseDiskReader
from documents import Document

class MyCustomReader(BaseDiskReader):
    def read_file(self, filepath, file_doc_idx):
        # Read file content and yield Document objects
        # Example: yield Document(text='...', id='...')
        pass
```

--------------------------------

### Python Example for Custom Stats Collector

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Shows how to implement a custom statistics collector by inheriting from `BaseStats`. Implement the `extract_stats` method to gather and return statistics from documents.

```python
from pipeline.stats.base import BaseStats

class MyCustomStats(BaseStats):
    def extract_stats(self, doc):
        # Extract statistics from the document
        return { "my_stat": 1 }
```

--------------------------------

### Python Example for Custom Filter

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Shows how to implement a custom filter by inheriting from `BaseFilter`. The `filter` method should return a boolean to indicate whether to keep the document. Supports batch processing and exclusion writers.

```python
from pipeline.filters.base_filter import BaseFilter

class MyCustomFilter(BaseFilter):
    def filter(self, doc):
        # Return True to keep the document, False to discard
        return True

    def batch_filter(self, batch):
        # Optional: Implement batched filtering for efficiency
        return [True] * len(batch)
```

--------------------------------

### Specify Input Folder with FSSpec Options Dictionary

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Provide a string path and a dictionary of options to initialize the fsspec filesystem.

```python
input_folder=("s3://mybucket/myinputdata", {"client_kwargs": {"endpoint_url": endpoint_uri}})
```

--------------------------------

### Submit Benchmark Experiments

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Launch benchmark jobs using a specified configuration file. The script automatically skips completed jobs and can be configured to re-run specific failure types.

```bash
python examples/inference/benchmark/launch_experiments.py \
    --config examples/inference/benchmark/sample_benchmark_config.yaml
```

--------------------------------

### Build and Run Rust MinHash

Source: https://github.com/huggingface/datatrove/blob/main/src/datatrove/tools/fast_mh3/README.md

Build the Rust MinHash implementation in release mode and view its help information.

```bash
cargo build --release
./target/release/s3 --help
```

--------------------------------

### Benchmark Configuration YAML

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Define models, parameters, and experiment sweeps for benchmarking. Use `script` to specify the inference script and `fixed_args` for common parameters. `experiments` allows sweeping over parameters like `model-name-or-path`, `tp`, and `speculative-config`.

```yaml
script: "examples/inference/generate_data.py"
continue_on_failure: true

fixed_args:
  qos: "high"
  time: "1:00:00"
  model-max-context: 2048
  max-tokens: 1024
  input-dataset-name: "simplescaling/s1K-1.1"
  input-dataset-split: "train"
  prompt-column: "question"
  output-dataset-name: "s1K-1.1-benchmark"
  output-dir: "data"

experiments:
  - name: "Qwen3-4B"
    args:
      model-name-or-path: "Qwen/Qwen3-4B-Thinking-2507"
      tp: [1, 2, 4]  # Sweep over TP configurations
      speculative-config: [None]
```

--------------------------------

### Specify Input Folder with FSSpec Filesystem Instance

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Provide a string path along with an initialized fsspec filesystem object.

```python
from s3fs import S3FileSystem
input_folder=("s3://mybucket/myinputdata", S3FileSystem(client_kwargs={"endpoint_url": endpoint_uri}))
```

--------------------------------

### Run Quality Checks with make

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Lints changed files using Ruff for code quality and formatting. For full repository checks, use 'make quality-full'.

```bash
make quality
```

--------------------------------

### Run Code Style Checks

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Execute local code style checks. Use 'make quality' and 'make style' for fast checks on changed files, or 'make quality-full' and 'make style-full' for comprehensive checks across the entire repository.

```bash
# Fast local loop (changed Python files only)
make quality
make style

# Full repository checks (same scope as CI)
make quality-full
make style-full
```

--------------------------------

### Analyze Benchmark Results

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Process benchmark run outputs to parse throughput metrics and generate summary CSV files. Specify the root directory where results are stored.

```bash
python examples/inference/benchmark/analyze_results.py \
    --root data
```

--------------------------------

### Dry Run Benchmark Experiments

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Preview SLURM commands without submitting jobs. Use the `--dry-run` flag to see the commands that would be executed.

```bash
python examples/inference/benchmark/launch_experiments.py \
    --config examples/inference/benchmark/sample_benchmark_config.yaml \
    --dry-run
```

--------------------------------

### Read from and Write to Hugging Face Hub Buckets

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Demonstrates reading data from an HF Hub bucket using `ParquetReader` and writing processed data back to a bucket using `HuggingFaceBucketWriter`. The writer can be configured to overwrite existing data.

```python
from datatrove.pipeline.readers import ParquetReader
from datatrove.pipeline.writers import HuggingFaceBucketWriter

# Reading: any reader with an hf://buckets/... path works.
reader = ParquetReader(data_folder="hf://buckets/myorg/my-bucket/raw/")

# Writing: HuggingFaceBucketWriter stages files locally, then pushes via Xet.
# Set overwrite=True to replace existing files at the prefix (default: append).
writer = HuggingFaceBucketWriter(
    bucket="myorg/my-bucket",
    prefix="v1/filtered",
    private=True,
    cleanup=True,
    overwrite=True,
)
```

--------------------------------

### Load and Access MetricStatsDict

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Demonstrates how to load a MetricStatsDict from a JSON file and access specific statistics like total length or mean for a given domain.

```python
from datatrove.pipeline.stats.summary_stats import MetricStatsDict
import json
stats = MetricStatsDict.from_dict(json.load(open("fqdn/length/metric.json")))

# E.g for total length of nytimes.com docs
stats["nytimes.com"].total

# Or for mean of cnn.com docs
stats["cnn.com"].mean
```

--------------------------------

### Run MinHash S3 Command

Source: https://github.com/huggingface/datatrove/blob/main/src/datatrove/tools/fast_mh3/README.md

Execute the Rust MinHash deduplication process directly from the command line, targeting S3 for input and output. The `--total-files` and `--downloads` parameters should be set according to your data and processing needs.

```bash
./target/release/s3 --input-folder s3://some-bucket/minhash/buckets/ --output-folder s3://some-bucket/minhash/remove_ids/ --total-files 700 --downloads 20
```

--------------------------------

### Run Tests with make

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Executes the test suite for DataTrove. Ensure all tests pass before pushing new changes.

```bash
make test
```

--------------------------------

### Specify Input Folder as String

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Pass a single string for the input folder path. Supports local paths and Hugging Face storage URIs.

```python
input_folder="/home/user/mydir"
```

```python
input_folder="s3://mybucket/myinputdata"
```

```python
input_folder="hf://buckets/myorg/my-bucket/raw/"
```

```python
input_folder="hf://datasets/allenai/c4/en/"
```

--------------------------------

### Re-run Specific Failed Benchmark Experiments

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/benchmark/README.md

Re-run benchmark experiments that failed due to specific reasons like timeouts or server errors, while skipping others like OOM errors. Use `--skip-failure-reasons none` to re-run all failed experiments.

```bash
# Re-run timeouts and server failures, but still skip OOM
python examples/inference/benchmark/launch_experiments.py \
    --config examples/inference/benchmark/sample_benchmark_config.yaml \
    --skip-failure-reasons OOM

# Re-run all previously failed experiments
python examples/inference/benchmark/launch_experiments.py \
    --config examples/inference/benchmark/sample_benchmark_config.yaml \
    --skip-failure-reasons none
```

--------------------------------

### Run MinHash Local Command

Source: https://github.com/huggingface/datatrove/blob/main/src/datatrove/tools/fast_mh3/README.md

Execute the Rust MinHash deduplication process using the local file system for input and output. Ensure the paths are correct and the `--total-files` and `--downloads` parameters are appropriately configured.

```bash
./target/release/local --input-folder /fsx/some-path/minhash/buckets/ --output-folder /fsx/some-path/minhash/remove_ids/ --total-files 700 --downloads 20
```

--------------------------------

### Python Configuration for MinHash S3

Source: https://github.com/huggingface/datatrove/blob/main/src/datatrove/tools/fast_mh3/README.md

Configure and run the MinHash deduplication process using Python with S3 storage. Ensure the SlurmPipelineExecutor and MinhashDedupCluster are correctly imported and configured.

```python
BASE_PATH = "s3://some-bucket/minhash"
s3 = SlurmPipelineExecutor(
    job_name=f"mh3",
    pipeline=[
        MinhashDedupCluster(
            input_folder=f"{BASE_PATH}/buckets",
            output_folder=f"{BASE_PATH}/remove_ids",
            save_cluster_size=True
        ),
    ],
    tasks=1,
    cpus_per_task=2,
    mem_per_cpu_gb=450,
    logging_dir=f"logs/clusters",
    partition="hopper-cpu",
    time="100:00:00"
).run()
```

--------------------------------

### Run Tests

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Execute the test suite for the DataTrove project using pytest.

```bash
pytest -sv ./tests/
```

--------------------------------

### Generate Synthetic Data Locally

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

Use this command for development or small-scale synthetic data generation on a single GPU. It specifies input dataset, model, output details, and execution mode.

```sh
python examples/inference/generate_data.py \
    --input-dataset-name simplescaling/s1K-1.1 \
    --input-dataset-split train \
    --prompt-column question \
    --model-name-or-path Qwen/Qwen3-4B-Thinking-2507 \
    --output-dataset-name s1K-1.1-synthetic \
    --output-dir data \
    --tasks 1 \
    --examples-per-chunk 50 \
    --local-execution
```

--------------------------------

### Creating a Custom Block Class

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Define a custom block by inheriting from PipelineStep or its subclasses. This allows for more complex logic, parameterization, and data handling, including reading from and writing to data folders.

```python
from datatrove.pipeline.base import PipelineStep
from datatrove.data import DocumentsPipeline
from datatrove.io import DataFolderLike, get_datafolder


class UppercaserBlock(PipelineStep):
    def __init__(self, some_folder: DataFolderLike, some_param: int = 5):
        super().__init__()
        # you can take whatever parameters you need and save them here
        self.some_param = some_param
        # to load datafolders use get_datafolder()
        self.some_folder = get_datafolder(some_folder)

    def run(self, data: DocumentsPipeline, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
        # you could also load data from the `some_folder`:
        for filepath in self.some_folder.get_shard(rank, world_size): # it also accepts a glob pattern, among other things
            with self.some_folder.open(filepath, "rt") as f:
                # do something
                ...
                yield doc

        #
        # OR process data from previous blocks (`data`)
        #

        for doc in data:
            with self.track_time():
                # you can wrap the main processing code in `track_time` to know how much each document took to process
                nr_uppercase_letters = sum(map(lambda c: c.isupper(), doc.text))
                # you can also keep track of stats per document using stat_update
                self.stat_update("og_upper_letters", value=nr_uppercase_letters)
                doc.text = doc.text.upper()
            # make sure you keep the yield outside the track_time block, or it will affect the time calculation
            yield doc

        #
        # OR save data to disk
        #

        with self.some_folder.open("myoutput", "wt") as f:
            for doc in data:
                f.write(doc...)

```

```python
pipeline = [
    ...,
    UppercaserBlock("somepath"),
    ...
]
```

--------------------------------

### Datatrove Logging Directory Structure

Source: https://github.com/huggingface/datatrove/blob/main/README.md

This structure is created when a pipeline is run with a specified logging directory. It includes executor configurations, task-specific logs, and aggregated statistics.

```text
└── mylogspath/exp1
    │── executor.json ⟵ json dump of the executor options and pipeline steps
    │── launch_script.slurm ⟵ the slurm config created and used to launch this job (if running on slurm)
    │── executor.pik ⟵ the slurm config created and used to launch this job (if running on slurm)
    │── ranks_to_run.json ⟵ list of tasks that are being run
    │── logs/
    │   └──[task_00000.log, task_00001.log, task_00002.log, ...] ⟵ individual logging files for each task
    │── completions/
    │   └──[00004, 00007, 00204, ...] ⟵ empty files marking a task as completed. Using when relaunching/resuming a job (only unfinished tasks will be run)
    │── stats/
    │   └──[00000.json, 00001.json, 00002.json, ...] ⟵ individual stats for each task (number of samples processed, filtered, removed, etc)
    └── stats.json ⟵ global stats from all tasks
```

--------------------------------

### Write to Hugging Face Hub Bucket

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Utilize `HuggingFaceBucketWriter` for raw or intermediate data storage on the Hugging Face Hub. It supports overwriting existing files and cleaning up local staged files after upload.

```python
HuggingFaceBucketWriter(
    bucket="myorg/my-bucket",
    prefix="v1/raw",                # path inside the bucket
    private=True,
    overwrite=True,                  # delete existing files at prefix first (default: False = append)
)
```

--------------------------------

### Write to Hugging Face Hub Dataset

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Employ `HuggingFaceDatasetWriter` to publish processed data as a final, ready-to-share dataset on the Hugging Face Hub.

```python
HuggingFaceDatasetWriter(
    dataset="myorg/my-dataset",
    private=True,
)
```

--------------------------------

### Configure JsonlWriter with Dynamic Filename Template

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Use this writer to save data in JSON Lines format. The `output_filename` supports template variables like `${language}` and `${rank}` for dynamic folder and file structuring.

```python
JsonlWriter(
    f"{MAIN_OUTPUT_PATH}/non_english/",
    output_filename="${language}/" + DUMP + "/${rank}.jsonl.gz",  # folder structure: language/dump/file
)
```

--------------------------------

### Auto-fix Code Style with make

Source: https://github.com/huggingface/datatrove/blob/main/AGENTS.md

Automatically fixes code style issues in changed files using Ruff. For the entire repository, use 'make style-full'.

```bash
make style
```

--------------------------------

### Defining a Custom Function for Simple Processing

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Define a custom function with a specific signature to perform simple processing within the pipeline. Imports can be moved inside the function body to avoid pickling issues.

```python
from datatrove.data import DocumentsPipeline

def uppercase_everything(data: DocumentsPipeline, rank: int = 0, world_size: int = 1) -> DocumentsPipeline:
    """
        `data` is a generator of Document. You must also return a generator of Document (yield)
        You can optionally use `rank` and `world_size` for sharding
    """
    for document in data:
        document.text = document.text.upper()
        yield document

pipeline = [
    ...,
    uppercase_everything,
    ...
]
```

--------------------------------

### Passing Simple Data as a Pipeline Block

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Use an iterable of Document objects directly as a pipeline block for small workloads or testing. Note that this iterable will not be sharded across multiple tasks.

```python
from datatrove.data import Document
from datatrove.pipeline.filters import SamplerFilter
from datatrove.pipeline.writers import JsonlWriter

pipeline = [
    [
        Document(text="some data", id="0"),
        Document(text="some more data", id="1"),
        Document(text="even more data", id="2"),
    ],
    SamplerFilter(rate=0.5),
    JsonlWriter(
        output_folder="/my/output/path"
    )
]
```

--------------------------------

### Enable Log File Colorization

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Set the DATATROVE_COLORIZE_LOG_FILES environment variable to '1' to enable ANSI colors in log messages saved to files (e.g., logs/task_XXXXX.log).

```bash
export DATATROVE_COLORIZE_LOG_FILES="1"
```

--------------------------------

### Enable Console Log Colorization

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Set the DATATROVE_COLORIZE_LOGS environment variable to '1' to enable ANSI colors in console log messages. This is useful for improving readability in interactive sessions.

```bash
export DATATROVE_COLORIZE_LOGS="1"
```

--------------------------------

### Generate Synthetic Data on Slurm Cluster

Source: https://github.com/huggingface/datatrove/blob/main/examples/inference/README.md

This command is for large-scale production workloads, distributing processing across multiple nodes using Slurm. It configures the number of concurrent workers and Slurm tasks.

```sh
python examples/inference/generate_data.py \
    --input-dataset-name simplescaling/s1K-1.1 \
    --input-dataset-split train \
    --prompt-column question \
    --model-name-or-path Qwen/Qwen3-4B-Thinking-2507 \
    --output-dataset-name s1K-1.1-dataforge \
    --output-dir data \
    --workers 10 \
    --tasks 20 \
    --examples-per-chunk 50
```

--------------------------------

### DataTrove Citation

Source: https://github.com/huggingface/datatrove/blob/main/README.md

BibTeX entry for citing the DataTrove project in academic work.

```bibtex
@misc{penedo2024datatrove,
  author = {Penedo, Guilherme and Kydlíček, Hynek and Cappelli, Alessandro and Sasko, Mario and Wolf, Thomas},
  title = {DataTrove: large scale data processing},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://github.com/huggingface/datatrove}
}
```

--------------------------------

### Disable Log File Colorization

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Set the DATATROVE_COLORIZE_LOG_FILES environment variable to '0' to disable ANSI colors in log messages saved to files. This is the default behavior.

```bash
export DATATROVE_COLORIZE_LOG_FILES="0"
```

--------------------------------

### Disable Console Log Colorization

Source: https://github.com/huggingface/datatrove/blob/main/README.md

Set the DATATROVE_COLORIZE_LOGS environment variable to '0' to disable ANSI colors in console log messages. This is the default behavior for log files.

```bash
export DATATROVE_COLORIZE_LOGS="0"
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.