### Install Data Prep Toolkit with Ray

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/tokenization/tokenization-ray.ipynb

Install the data prep toolkit with Ray support. This example is for testing with a cloned repository.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
# When testing with cloned repo, no need to use pypi wheels
#%uv pip install 'data-prep-toolkit[ray]'
## When testing with wheel from pypi
#%uv pip install 'data-prep-toolkit-transforms[tokenization]'
```

--------------------------------

### Set up Conda Environment and Install Dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/pdf-processing-1/README.md

This snippet shows how to create and activate a Conda environment for the project, navigate to the example directory, and install the required Python packages using uv pip.

```bash
conda create -n data-prep-kit -y python=3.11
conda activate data-prep-kit

# install the following in 'data-prep-kit' environment
cd examples/pdf-processing-1
uv pip install -r requirements.txt

# start jupyter and run the notebooks with this jupyter
jupyter lab
```

--------------------------------

### Install Data Prep Toolkit and Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/profiler/profiler-python.ipynb

Installs the Data Prep Toolkit and its profiler transforms using uv pip. This example is for reference; users and application developers should use the appropriate tag for the latest version from PyPI.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
%uv pip install data-prep-toolkit
%uv pip install data-prep-toolkit-transforms[profiler]
```

--------------------------------

### Install Data Prep Toolkit and Dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/docling2parquet/docling2parquet-ray.ipynb

Installs the necessary data prep toolkit and its Ray and docling2parquet components, along with pandas and ipywidgets for environment setup.

```python
%%capture
!uv pip install 'data-prep-toolkit[ray]'
%uv pip install 'data-prep-toolkit-transforms[docling2parquet]'
!uv pip install pandas
!uv pip install -U ipywidgets
```

--------------------------------

### Setup Environment for a Specific Transform (fdedup)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md

Navigates to the 'fdedup' transform directory, creates a virtual environment, activates it, installs Jupyter Lab, and launches Jupyter Lab. This prepares the environment to run the fdedup transform notebooks.

```bash
cd data-prep-kit/transforms/universal/fdedup
make venv
source venv/bin/activate 
uv pip install jupyterlab
jupyter lab
```

--------------------------------

### Install Data Prep Toolkit (Complete)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md

Installs the complete Data Prep Toolkit library, including all transforms and the Ray runtime. Use this for a full-featured installation.

```shell
uv pip install 'data-prep-toolkit-transforms[ray,all]'
```

--------------------------------

### Navigate and Install Dependencies (Conda)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/rag-pdf-1/setup-python-dev-env.md

Change directory to the RAG PDF example folder and install all required Python packages listed in 'requirements.txt'.

```bash
cd examples/rag-pdf-1
pip  install  -r requirements.txt
```

--------------------------------

### Install data-prep-toolkit and dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/notebooks/fine tuning/code/sample-notebook.ipynb

Installs the data-prep-toolkit with Ray and code transforms, along with the datasets and pandas libraries. Captures installation logs.

```python
%%capture logpip --no-stderr
%uv pip install 'data-prep-toolkit-transforms[ray,code]==1.1.1'
%uv pip install datasets
%uv pip install pandas
```

--------------------------------

### Run Kubernetes Setup Script

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/kfp/doc/deployment_on_MacOS.md

Execute the setup script to install necessary Kubernetes components, such as Kubeflow pipelines. This process may take time, and users should wait until all pods reach the 'Ready' state.

```bash
make setup
```

--------------------------------

### Install Web2Parquet from Git Fork

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/web2parquet/README.md

Sets up a virtual environment and installs dependencies from requirements.txt when working from a forked git repository.

```bash
cd transforms/universal/web2parquet
make venv
source venv/bin/activate
uv pip install -r requirements.txt
```

--------------------------------

### Install Data Prep Toolkit and Requirements

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/images/faces/Faces.ipynb

Installs the data-prep-toolkit and requirements.txt using uv pip. This is a reference for setting up the environment.

```python
%%capture
## This is here as a reference only and provide the same results as `make venv` above
#%uv pip install data-prep-toolkit
#%uv pip install -r requirements.txt
```

--------------------------------

### Install Dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/agentic/Planning_DPK_agent.ipynb

Installs the necessary packages from the requirements.txt file. Use this to set up the environment for the agent.

```python
%uv pip install -qq -r requirements.txt
```

--------------------------------

### Install data-prep-toolkit-transforms with Ray and all extras

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/README-list.md

Installs the data-prep-toolkit-transforms library with Ray and all other extras using uv. This is useful for distributed processing with Ray.

```bash
python -m uv pip install data-prep-toolkit-transforms[ray, all]
```

--------------------------------

### Install Web2Parquet Transform Prerequisites

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/web2parquet/README.md

Installs necessary Python packages including the data-prep-connector and data-prep-toolkit.

```bash
uv pip install data-prep-connector
uv pip install data-prep-toolkit>=0.2.2.dev2
uv pip install 'data-prep-toolkit-transforms[web2parquet]>=0.2.2.dev3'
```

--------------------------------

### Install data-prep-toolkit-transforms with all extras

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/README-list.md

Installs the data-prep-toolkit-transforms library with all available extras using uv. This command ensures all optional dependencies are included.

```bash
python -m uv pip install data-prep-toolkit-transforms[all]
```

--------------------------------

### Install Development Tools and Clone Repository

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/repo.md

Installs Python development tools uv, pre-commit, and twine using uv, clones the repository, navigates into the directory, and installs pre-commit hooks.

```shell
pip install uv
uv pip install pre-commit
uv pip install twine
...
git clone git@github.com:IBM/data-prep-kit.git
cd data-prep-kit
pre-commit install
```

--------------------------------

### Setup Input and Output Directories

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/pdf-processing-1/pdf_processing_1_python.ipynb

Configures input and output directories, creating them if they don't exist. It also clears the output directory before starting. Handles differences between Colab and local environments for the input directory.

```python
import os, sys
import shutil

if RUNNING_IN_COLAB:
    input_dir = "input"
    shutil.os.makedirs(input_dir, exist_ok=True)
else:
    input_dir = "../data-files/pdf-processing-1/"

output_dir = "output"

output_docling2pq_dir = os.path.join (output_dir, 'dpk_01_docling2pq_out')
output_docid_dir = os.path.join (output_dir, 'dpk_02_docid_out')
output_exact_dedupe_dir = os.path.join (output_dir, 'dpk_03_exact_dedupe_out')
output_fuzzy_dedupe_dir = os.path.join (output_dir, 'dpk_04_fuzzy_dedupe_out')
output_doc_quality_dir = os.path.join (output_dir, 'dpk_05_doc_quality_out')
output_final_dir = os.path.join (output_dir, 'dpk_output_final')

## clear output folder
shutil.rmtree(output_dir, ignore_errors=True)
shutil.os.makedirs(output_dir, exist_ok=True)
print ("✅ Cleared output directory")
```

--------------------------------

### Local Development Setup

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/yara/README.md

Steps to set up and run the YARA transform locally. This includes creating a virtual environment, installing dependencies, fetching rules, and activating the environment.

```bash
make venv
source venv/bin/activate
python -m dpk_yara.local_python
```

--------------------------------

### Install DPK Connector from GitHub

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/data-connector-lib/doc/overview.md

Install the Data Prep Connector directly from its GitHub repository using uv pip.

```sh
uv pip install git+https://github.com/data-prep-kit/data-prep-kit.git@dev#subdirectory=data-connector-lib
```

--------------------------------

### Install dpk_transform_chain and data-process-lib

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/chain/dpk_transform_chain/README.md

Install the necessary packages by navigating to the transforms directory and using uv pip. Ensure you are in a virtual environment.

```bash
cd transforms

# Optional: create virtual environment
python -m venv venv
source venv/bin/activate


uv pip install ".[all]"

cd ../data-process-lib
uv pip install .
```

--------------------------------

### Install Data Prep Toolkit

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/tokenization/tokenization.ipynb

Installs the data-prep-toolkit package. Use the appropriate tag for the latest from PyPI or when testing with a cloned repository.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
# When testing with cloned repo, no need to use pypi wheels
#%uv pip install 'data-prep-toolkit'
## When testing with wheel from pypi
#%uv pip install 'data-prep-toolkit-transforms[tokenization]'
```

--------------------------------

### Run the Elasticsearch Setup Script

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/similarity/tools/load_elasticsearch/README.md

Execute the main Python script to perform the data setup and loading into Elasticsearch.

```bash
python set_up_elasticsearch.py
```

--------------------------------

### Install Data Prep Toolkit

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/ededup/ededup-ray.ipynb

Install the data-prep-toolkit and its ededup transforms. Use the appropriate release tag for the latest version from PyPI.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
#%uv pip install data-prep-toolkit[ray]
#%uv pip install data-prep-toolkit-transforms[ededup]
```

--------------------------------

### Clone Repository and Install Pre-commit Hooks

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/kfp/kfp_support_lib/README.md

Clone the data-prep-kit repository and navigate to the kfp_support_lib directory to install pre-commit hooks for development.

```shell
git clone git@github.com:IBM/data-prep-kit.git
cd kfp/kfp_support_lib
pre-commit install
```

--------------------------------

### Install data-prep-toolkit-transforms[similarity]

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/similarity/similarity.ipynb

Install the similarity transform package from PyPI. Ensure you use the correct tag for the latest version.

```python
%uv pip install data-prep-toolkit-transforms[similarity]
```

--------------------------------

### Install Data Prep Toolkit

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/images/nsfw/nsfw.ipynb

Reference for installing the data-prep-toolkit and its requirements. This is typically done within a Jupyter environment using %%capture.

```python
%%capture
## This is here as a reference only. 
#%uv pip install data-prep-toolkit
#%uv pip install -r requirements.txt
```

--------------------------------

### Install DPK Connector from PyPI

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/data-connector-lib/doc/overview.md

Install the Data Prep Kit Connector using uv pip from PyPI.

```sh
uv pip install data-prep-connector
```

--------------------------------

### Install huggingface_hub

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/ADVANCED.md

Install the huggingface_hub library using uv pip. This is a prerequisite for downloading files from HuggingFace.

```bash
uv pip install --upgrade huggingface_hub
```

--------------------------------

### Set up Local Environment for Jupyter Notebook

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/web2parquet/README.md

Installs Python virtual environment, activates it, and installs Jupyter Lab.

```bash
python -v venv venv
source venv/bin/activate
pip install uv
uv pip install jupyter lab
```

--------------------------------

### Install Kubeflow Pipelines (KFP)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/scripts/k8s-setup/README.md

Sets the KFP version and installs KFP using a script. Waits for all KFP pods to be ready.

```shell
# Set required KFP version. You can reference to the latest supported version in the [requirements.env](./requirements.env) file.
# Currently, we support 1.8.5 for KFPv1 and 2.2.0 for KFP v2
export PIPELINE_VERSION=1.8.5
cd $REPOROOT/scripts/k8s-setup/tools/ && ./install_kubeflow.sh deploy && cd -
kubectl wait --for=condition=ready --all pod -n kubeflow --timeout=300s
```

--------------------------------

### Clone Repository and Navigate

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/agentic/README.md

Clone the data-prep-kit repository and navigate to the agentic examples directory. This is the initial step for setting up the project.

```bash
git clone git@github.com:IBM/data-prep-kit.git
cd examples/agentic
```

--------------------------------

### Install Data Prep Toolkit Packages

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/web2parquet/web2parquet.ipynb

Installs the core data prep toolkit, transforms, and connector packages using uv pip.

```python
%%capture
!uv pip install data-prep-toolkit
!uv pip install data-prep-toolkit-transforms
!uv pip install data-prep-connector
```

--------------------------------

### Setting up Virtual Environment and Installing Dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/tokenization/tokenization2arrow-ray.ipynb

This snippet outlines the steps to create a virtual environment, activate it, and install necessary packages like jupyterlab and ipywidgets for the tokenization transform. It's intended for developers working from a git clone of the repository.

```bash
cd tokenization2arrow
make venv 
source venv/bin/activate 
uv pip install jupyterlab
uv pip install -U ipywidgets
./venv/bin/jupyter lab
```

--------------------------------

### Install Dependencies and Run Jupyter Lab

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/code_quality/code_quality-ray.ipynb

Installs necessary packages using 'uv pip' and starts Jupyter Lab. Ensure to adapt the 'uv pip install' command to your specific release level or pre-configure your virtual environment.

```bash
make venv 
source venv/bin/activate 
uv pip install jupyterlab
virtualenv/bin/jupyter lab
```

--------------------------------

### Install DPK and Helper Applications

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/recipes/DPK-sequence/pdf_processing_python.ipynb

Installs the Data Prep Toolkit with language support, tqdm for progress bars, and humanfriendly for friendly output formatting. Use this command in your Python environment.

```python
%%capture cap --no-stderr
!uv pip install 'data-prep-toolkit-transforms[language]' tqdm humanfriendly

```

--------------------------------

### Install pandas using uv pip

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/code_quality/code_quality.ipynb

Installs the pandas library using uv pip. This is a reference example; users should use the correct tag for the latest PyPI release.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
#%uv pip install 'data-prep-toolkit-transforms[code_quality]'
! uv pip install pandas
```

--------------------------------

### Install Pandas using uv pip

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/code_quality/code_quality-ray.ipynb

Installs the pandas library using 'uv pip'. This is a reference example; users should use the appropriate tag for the latest version from PyPI.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
#%uv pip install 'data-prep-toolkit-transforms[ray,code_quality]'
!uv pip install pandas

```

--------------------------------

### Install Bloom Transform Package

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/bloom/bloom_python.ipynb

Install the 'data-prep-toolkit-transforms' package with Bloom support from PyPI using 'uv pip'. This command is for users and application developers to get the latest version.

```python
%uv pip install 'data-prep-toolkit-transforms[bloom]'
```

--------------------------------

### Install Data Prep Toolkit and Datasets

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/notebooks/fine tuning/language/fine-tune-language.ipynb

Installs the necessary data-prep-toolkit and datasets libraries. Requires at least 8 CPUs; consider using TPUs on Google Colab for sufficient resources.

```python
%%capture
!uv pip install 'data-prep-toolkit-transforms[language]==1.1.1.dev0'
import pyarrow.parquet as pq
import pandas as pd
```

--------------------------------

### Running DocQualityTransform Sample

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/doc_quality/README.md

Example of how to run the dpk_doc_quality/transform.py sample using the 'run-cli-sample' make target.

```shell
make run-cli-sample
...
ls output
```

--------------------------------

### Setup Jupyter Lab for Local Experimentation

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md

Installs Jupyter Lab, ipykernel, and ipywidgets, then installs the 'dataprepkit' kernel for use in Jupyter environments. This enables local experimentation with transform notebooks.

```bash
uv pip install jupyterlab ipykernel ipywidgets
python -m ipykernel install --user --name=data-prep-kit --display-name "dataprepkit"
```

--------------------------------

### Install Header Cleanser and Pandas

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/header_cleanser/header_cleanser.ipynb

Install the Header Cleanser transform and Pandas using 'uv pip'. This example is for reference; users should use the correct tag for the latest PyPI version.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
%uv pip install 'data-prep-toolkit-transforms[header_cleanser]'
! uv pip install pandas
```

--------------------------------

### Setup Utility Path

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/rag-pdf-1/rag_1_dpk_process_python.ipynb

Appends the '../utils' directory to the system path to allow importing utility functions.

```python
import sys
sys.path.append('../utils')
```

--------------------------------

### Activate Conda Environment

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/rag-pdf-1/setup-python-dev-env.md

Activate the newly created conda environment to start using its installed packages and Python interpreter.

```bash
conda activate data-prep-kit-rag
```

--------------------------------

### Install KFP on Existing Cluster

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/kfp/doc/setup.md

Set this environment variable to 1 to indicate that you are deploying to an external Kubernetes cluster. This is used in conjunction with the setup command.

```bash
export EXTERNAL_CLUSTER=1
make -C scripts/k8s-setup setup
```

--------------------------------

### Local Python Environment Setup

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/recipes/PII/Run_your_first_PII_redactor_transform.ipynb

Commands to set up a local Python environment for running the notebook. Includes creating a virtual environment, activating it, installing JupyterLab, and launching the notebook.

```bash
python -m venv venv
source venv/bin/activate
uv pip install jupyterlab
jupyter lab Run_your_first_PII_redactor_transform.ipynb
```

--------------------------------

### PIIRedactor Transform Output Log

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/pii_redactor/piiredactor.ipynb

Example log output from the PIIRedactor transform execution. It shows pipeline information, data access details, and the start time of the PII redactor orchestrator.

```text
17:17:13 INFO - pipeline id pipeline_id
17:17:13 INFO - code location None
17:17:13 INFO - data factory data_ is using local data access: input_folder - ray/test-data/input output_folder - output
17:17:13 INFO - data factory data_ max_files -1, n_sample -1
17:17:13 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']
17:17:13 INFO - orchestrator pii_redactor started at 2025-01-16 17:17:13
17:17:13 INFO - Number of files is 1, source profile {'max_file_size': 0.0023164749145507812, 'min_file_size': 0.0023164749145507812, 'total_file_size': 0.0023164749145507812}
17:17:13 INFO - Loading model from flair/ner-english-large
```

--------------------------------

### Running Transform Tests with Makefile

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/data-processing-lib/doc/transform-standalone-testing.md

This example demonstrates how to run all defined tests, including setting up the virtual environment, using the project's Makefile.

```shell
$ make test
...

```

--------------------------------

### Load Example Task

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/agentic/Planning_DPK_agent.ipynb

Imports and prints an example task string. This can be used to enhance planning results by providing a sample task structure.

```python
# This is an example of a plan for a simple task. It is possed to the prompt to enhance the planning results.
from llm_utils.dpk.examples import *
print(example_task)
```

--------------------------------

### Setup Input/Output Directories and Download Sample Data

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/recipes/DPK-sequence/pdf_processing_python.ipynb

Sets up the necessary input and output directories for the PDF processing workflow. It downloads sample PDF files into the input directory and clears the output directory before creating it. This code is essential for preparing the environment before running DPK transformations.

```python
%%capture cap --no-stderr
import os, sys
import urllib.request
import shutil
import pandas as pd
import glob

shutil.os.makedirs("tmp/input", exist_ok=True)
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/earth.pdf", "tmp/input/earth.pdf")
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/earth-copy.pdf", "tmp/input/earth-copy.pdf")
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/earth2.pdf", "tmp/input/earth2.pdf")
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/mars.pdf", "tmp/input/mars.pdf")
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/spam.pdf", "tmp/input/spam.pdf")
urllib.request.urlretrieve("https://raw.githubusercontent.com/data-prep-kit/data-prep-kit/dev/examples/data-files/pdf-processing-1/lorem-ipsum.pdf", "tmp/input/lorem-ipsum.pdf")

input_dir = "tmp/input"
output_dir = "output"


output_docling2pq_dir = os.path.join (output_dir, '01_docling2pq_out')
output_docid_dir = os.path.join (output_dir, '02_docid_out')
output_exact_dedupe_dir = os.path.join (output_dir, '03_exact_dedupe_out')
output_fuzzy_dedupe_dir = os.path.join (output_dir, '04_fuzzy_dedupe_out')
output_doc_quality_dir = os.path.join (output_dir, '05_doc_quality_out')
output_final_dir = os.path.join (output_dir, 'output_final')

## clear output folder
shutil.rmtree(output_dir, ignore_errors=True)
shutil.os.makedirs(output_dir, exist_ok=True)
print ("✅ Cleared output directory")

```

--------------------------------

### Install Data Prep Toolkit Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/yara/yara-pipeline-demo.ipynb

Installs the data-prep-toolkit-transforms package with folder2parquet and yara capabilities. This is a reference for PyPI installation; use `%uv pip install` for non-developer usage.

```python
%%capture
# Reference only — install from pypi for non-developer usage:
# %uv pip install 'data-prep-toolkit-transforms[folder2parquet,yara]'
! uv pip install pandas
```

--------------------------------

### Setup Output Directories and Clear Previous Runs

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/agentic/dpk_as_tools.ipynb

Initializes the base output path and cleans up any previous output to ensure a fresh run. It also sets up the input folder.

```python
import shutil
import os
cwd = os.getcwd()

output_base_path = f"{cwd}/output"

input_folder = f"{cwd}/test-data/input/"
output_folder =  f"{output_base_path}/final_1/"

shutil.rmtree(output_base_path, ignore_errors=True)
print (f"✅ Cleared {output_base_path} directory")
```

--------------------------------

### Install Dependencies

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/yara/yara-pipeline-demo.ipynb

Installs necessary Python packages for the pipeline. Use `pip install` for non-developer usage.

```bash
make venv
source venv/bin/activate
uv pip install jupyterlab
venv/bin/jupyter lab
```

--------------------------------

### Install Specific Transform (lang_id)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md

Installs only the 'lang_id' transform, which is used for identifying the language of content. This is useful for minimal installations.

```shell
uv pip install 'data-prep-toolkit-transforms[lang_id]'
```

--------------------------------

### Install Minio Server

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/data-processing-lib/doc/transform-s3-testing.md

Installs the Minio server on macOS using Homebrew. Ensure Homebrew is installed before running this command.

```shell
brew install minio/stable/minio
```

--------------------------------

### Install Specific Transform (docling2parquet)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/run-transform-cli.md

Installs only the specified transform, 'docling2parquet', to reduce installation size. This command can be adapted for any other available transform.

```bash
uv pip install 'data-prep-toolkit-transforms[docling2parquet]'
```

--------------------------------

### Install Data Prep Kit Transforms (Reference)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/doc_id/doc_id_jsonl.ipynb

Reference installation command for the data-prep-kit-transforms with the doc_id extra. Users should refer to PyPI for the latest version tags.

```python
# Users and application developers must use the right tag for the latest from pypi
# uv pip install data-prep-kit-transforms[doc_id]
```

--------------------------------

### Install Data Prep Toolkit Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/transforms-dev1-testing.ipynb

Installs the data-prep-toolkit-transforms package with language support. Ensure you are in a virtual environment (venv) where the package will be installed.

```python
%%capture
%uv pip install 'data-prep-toolkit-transforms[language]==1.1.1'
```

--------------------------------

### Install data-prep-toolkit-transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/notebooks/GneissWeb/GneissWeb.ipynb

Install the data-prep-toolkit-transforms package with gneissweb extras using uv pip. This command ensures all necessary dependencies for GneissWeb are installed.

```bash
!uv pip install --no-cache "data-prep-toolkit-transforms[gneissweb]==1.0.1.dev2]"
```

--------------------------------

### Install data-prep-toolkit and HTML2Parquet

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/html2parquet/html2parquet.ipynb

Install the necessary packages for HTML to Parquet transformation. This includes the core toolkit, the HTML2Parquet transform, and pandas for data manipulation.

```bash
!uv pip install data-prep-toolkit
!uv pip install 'data-prep-toolkit-transforms[html2parquet]'
!uv pip install pandas
```

--------------------------------

### Install Gneissweb Classification Transform

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/gneissweb_classification/gneissweb_classification.ipynb

Installs the data-prep-toolkit-transforms with the gneissweb_classification extra and pandas. This is a reference for users and application developers to install the correct PyPI tag.

```python
%uv pip install 'data-prep-toolkit-transforms[gneissweb_classification]'
%uv pip install pandas
```

--------------------------------

### Install Data Prep Toolkit and License Select Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/code/license_select/license_select.ipynb

Install the data-prep-toolkit and its license_select transforms using uv pip. This is typically done within a virtual environment.

```python
%%capture
## This is here as a reference only
# Users and application developers must use the right tag for the latest from pypi
%uv pip install data-prep-toolkit
%uv pip install data-prep-toolkit-transforms[license_select]
```

--------------------------------

### Install Requirements with UV (Virtual Env)

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/examples/rag-pdf-1/setup-python-dev-env.md

Install project dependencies using 'uv pip' after activating the virtual environment. Ensure 'uv' is installed.

```bash
uv pip install -r requirements.txt
```

--------------------------------

### Install data-prep-kit Toolkit and Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/recipes/PII/Run_your_first_PII_redactor_transform.ipynb

Installs the data-prep-toolkit and specific transforms, including pii_redactor, using uv pip. Also installs pandas for data manipulation.

```python
#%%capture cap --no-stderr
#%uv pip install "data-prep-toolkit @ git+https://github.com/touma-I/data-prep-kit-pkg@numpy-dependencies#subdirectory=data-processing-lib"
#%uv pip install "data-prep-toolkit-transforms[docling2parquet,pii_redactor] @ git+https://github.com/touma-I/data-prep-kit-pkg@numpy-dependencies#subdirectory=transforms"
!uv pip install "data-prep-toolkit-transforms[docling2parquet,pii_redactor]==1.1.6"
!uv pip install pandas 
import pandas as pd
import os
```

--------------------------------

### Configure and Run Resize Transform

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/resize/resize-ray.ipynb

Sets up runtime parameters for the Resize transform and invokes it. This example specifies input and output folders, enables local execution, and sets a maximum number of rows per table.

```python
Resize(input_folder= "test-data/input",
        output_folder= "output",
        run_locally= True,
        resize_max_rows_per_table= 125).transform()
```

--------------------------------

### Install GCC/G++ for fasttext on Linux

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/quick-start.md

Installs the necessary gcc and g++ compilers on Linux systems, required for compiling and installing the fasttext library used by some transforms.

```shell
conda install gcc_linux-64
conda install gxx_linux-64
```

--------------------------------

### Install Data Prep Toolkit with Ray

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/language/doc_chunk/doc_chunk-ray.ipynb

Install the data-prep-toolkit with the Ray extra and the doc_chunk transforms. This is typically done within a virtual environment.

```bash
make venv
source venv/bin/activate && uv pip install jupyterlab
```

--------------------------------

### Install data-prep-toolkit-transforms for Gneissweb

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/recipes/GneissWeb/GneissWeb.ipynb

Installs the specific version of the data-prep-toolkit-transforms package with Gneissweb dependencies using uv pip. Use --no-cache to ensure a fresh installation.

```python
!uv pip install --no-cache "data-prep-toolkit-transforms[gneissweb]==1.1.8"
```

--------------------------------

### Install Data Prep Toolkit and Profiler Transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/transforms/universal/profiler/profiler-ray.ipynb

Install the data-prep-toolkit with Ray support and the profiler transforms. Ensure you use the correct tags for the latest versions from PyPI.

```python
%uv pip install 'data-prep-toolkit[ray]'
%uv pip install 'data-prep-toolkit-transforms[profiler]'
```

--------------------------------

### Install Data Prep Kit with all transforms

Source: https://github.com/data-prep-kit/data-prep-kit/blob/dev/README.md

Installs the latest version of the Data Prep Kit along with all available transforms using uv and pip. Ensure you have Python 3.10+ installed.

```bash
pip install uv
uv pip install 'data-prep-toolkit-transforms[all]'
```