### Start Label Studio with Pip and Create Test Project

Source: https://github.com/datajuicer/data-juicer/blob/main/tools/humanops/README.md

Example of starting Label Studio using a local pip installation on a specific port and automatically creating a test project. After execution, check the generated JSON file for connection details.

```bash
python label_studio_service.py --use-pip --port 7070 --create-test-project
```

--------------------------------

### Install and Run Data-Juicer CLI

Source: https://github.com/datajuicer/data-juicer/blob/main/README_ZH.md

Install the Data-Juicer Python package and run a sample processing pipeline using the command-line interface. This is a quick way to get started with a predefined configuration.

```bash
uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml
```

--------------------------------

### Vision Processing Setup

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with generic and vision capabilities for computer vision tasks.

```bash
uv pip install "py-data-juicer[generic,vision]"
```

--------------------------------

### Install Data-Juicer from Source

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer from its source repository, enabling the use of the latest features. Includes installing specific domains like vision.

```bash
# Clone repository
git clone https://github.com/datajuicer/data-juicer.git
cd data-juicer
uv pip install -e .
```

```bash
# You can install specific domain as well
uv pip install -e ".[vision]"
```

--------------------------------

### Install EasyAnimate

Source: https://github.com/datajuicer/data-juicer/blob/main/thirdparty/models/README.md

Run this script to install the EasyAnimate library. Ensure you have the necessary permissions and dependencies.

```shell
bash ./setup_easyanimate.sh
```

--------------------------------

### Install Package with All Dependencies

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/DeveloperGuide.md

Install the Data-Juicer package including all optional dependencies using uv.

```bash
uv pip install -e ".[all]"
```

--------------------------------

### Get Partition Recommendations

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/JobManagement.md

Instantiate PartitionSizeOptimizer with a configuration object to get partition recommendations for a given dataset and pipeline.

```python
from data_juicer.core.executor.partition_size_optimizer import PartitionSizeOptimizer

optimizer = PartitionSizeOptimizer(cfg)
recommendations = optimizer.get_partition_recommendations(dataset, pipeline)
```

--------------------------------

### Install uv Package Installer

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/DeveloperGuide.md

Instructions for installing uv, a fast Python package installer and resolver. It can be installed using curl or pip.

```bash
# Using curl
curl -LsSf https://astral.sh/uv/install.sh | sh
```

```bash
# Or using pip
pip install uv
```

--------------------------------

### Example 5: Hash Comments at Start

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/clean_copyright_mapper.md

Demonstrates the removal of lines starting with '#' at the beginning of the text.

```text
#if start with
that will be cleaned 
#evenly
```

--------------------------------

### Full Processing Pipeline Setup

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with generic, NLP, vision, and distributed computing capabilities for a comprehensive processing pipeline.

```bash
uv pip install "py-data-juicer[generic,nlp,vision,distributed]"
```

--------------------------------

### Install Core Dependencies

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/agent/minimal_configs/README.md

Installs the core dependencies for the project in an editable environment. This is a prerequisite for running `dj-process` correctly.

```bash
uv pip install -e .
```

--------------------------------

### Complete Data-Juicer Installation

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs all features of Data-Juicer, excluding the sandbox environment, using the '[all]' tag.

```bash
# Install all features (except sandbox)
uv pip install "py-data-juicer[all]"
```

--------------------------------

### Text Processing Setup

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with generic and NLP capabilities for text processing tasks.

```bash
uv pip install "py-data-juicer[generic,nlp]"
```

--------------------------------

### Install Data-Juicer with Development & Testing Tools

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with tools necessary for development and testing.

```bash
# Development & Testing
uv pip install "py-data-juicer[dev]"
```

--------------------------------

### Install Project with AI Services Extra

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/agent/minimal_configs/README.md

Installs the project in an editable environment, including the 'ai_services' extra dependencies. This is recommended for full functionality.

```bash
uv pip install -e ".[ai_services]"
```

--------------------------------

### Install and Configure Pre-commit

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/DeveloperGuide.md

Install the pre-commit tool and configure it for the Data-Juicer project to automatically check and format code before committing.

```shell
# ===========install pre-commit tool===========
uv pip install pre-commit

cd <path_to_data_juicer>
# install pre-commit script for data_juicer
pre-commit install


# ===========check all files===========
git add .
pre-commit run --all-files

# commit after all checking are passed
git commit -m "xxxx"
```

--------------------------------

### Install Data-Juicer with Audio Processing Features

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with specific support for Audio Processing tasks.

```bash
# Audio Processing
uv pip install "py-data-juicer[audio]"
```

--------------------------------

### Basic Data-Juicer Installation

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs the core Data-Juicer package from PyPI using 'uv'. This provides essential data processing capabilities.

```bash
uv pip install py-data-juicer
```

--------------------------------

### Example Operator List for DJ_OPS_LIST_PATH

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/DJ_service.md

This is an example text file content that can be used with the DJ_OPS_LIST_PATH environment variable to specify which Data-Juicer operators the MCP server should return.

```text
text_length_filter
flagged_words_filter
image_nsfw_filter
text_pair_similarity_filter
```

--------------------------------

### Example 4: URL

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/clean_copyright_mapper.md

This example shows a URL, which is not affected by the copyright cleaning.

```text
http://www.nasosnsncc.com
```

--------------------------------

### Start the API Service

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/DJ_service.md

Run this command to start the Data-Juicer API service using uvicorn.

```bash
uvicorn service:app
```

--------------------------------

### Install Data-Juicer with AI Services & APIs Features

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with components for AI Services and APIs.

```bash
# AI Services & APIs
uv pip install "py-data-juicer[ai_services]"
```

--------------------------------

### Install Data-Juicer with Computer Vision Features

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with specific support for Computer Vision tasks.

```bash
# Computer Vision
uv pip install "py-data-juicer[vision]"
```

--------------------------------

### Install Data-Juicer with Generic ML/DL Capabilities

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with generic Machine Learning and Deep Learning capabilities, including support for PyTorch, Transformers, and VLLM.

```bash
# Generic ML/DL capabilities
uv pip install "py-data-juicer[generic]"
```

--------------------------------

### Silent Audio Data URI Example

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/BadDataExhibition.md

This example demonstrates how to create a data URI for a silent audio file, specifically a 0-second WAVE file. This can be useful for testing or as a placeholder.

```text
data:audio/wav;base64,UklGRjIAAABXQVZFZm10IBIAAAABAAEAQB8AAEAfAAABAAgAAABmYWN0BAAAAAAAAABkYXRhAAAAAA==
```

```text
data:audio/x-wav;base64,UklGRooWAABXQVZFZm10IBAAAAABAAEAIlYAAESsAAACABAAZGF0YWYWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
```

--------------------------------

### Run Standard Demo Process

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/agent/minimal_configs/README.md

Executes a simple data processing pipeline using a standard demo configuration. This is recommended to verify the environment setup before testing agent-specific configurations.

```bash
dj-process --config demos/process_simple/process.yaml
```

--------------------------------

### Example 3: Single-line Comments at Start

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/clean_copyright_mapper.md

Illustrates the removal of lines starting with '//' at the beginning of the text.

```text
//if start with
//that will be cleaned 
 evenly
```

--------------------------------

### Example 6: Double Dash Comments at Start

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/clean_copyright_mapper.md

Illustrates the removal of lines starting with '--' at the beginning of the text.

```text
--if start with
--that will be cleaned 
#evenly
```

--------------------------------

### Run Label Studio Service Utility

Source: https://github.com/datajuicer/data-juicer/blob/main/tools/humanops/README.md

Basic command to run the utility script. Use --use-pip to run with a local pip installation instead of Docker. Use --create-test-project to automatically set up a test project and sample task.

```bash
python label_studio_service.py [--port PORT] [--create-test-project] [--use-pip]
```

--------------------------------

### Install Megatron-LM and HELM

Source: https://github.com/datajuicer/data-juicer/blob/main/thirdparty/LLM_ecosystems/README.md

Execute these scripts sequentially within the Docker container on evaluation machines to install both Megatron-LM and HELM. This setup is required for comprehensive evaluation.

```shell
./setup_megatron.sh
./setup_helm.sh
```

--------------------------------

### Example 1: Multiline and Single-line Comments

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/clean_copyright_mapper.md

Demonstrates cleaning of multiline comments containing 'copyright' and single-line comments starting with '//'.

```text
这是一段 /* 多行注释
注释内容copyright
*/ 的文本。另外还有一些 // 单行注释。
```

--------------------------------

### Configure Data-Juicer Tools

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/api_service/react_data_filter_process.ipynb

Sets up the Data-Juicer service toolkit by importing and calling setup_service_toolkit. This function is used to enable Data-Juicer operators for the agent.

```python
from utils import setup_service_toolkit

service_toolkit = setup_service_toolkit('wrapped_filters')
```

--------------------------------

### Initialize VideoResizeResolutionMapper

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/video_resize_resolution_mapper.md

Instantiate the VideoResizeResolutionMapper with default parameters. This will resize videos according to the default constraints.

```python
VideoResizeResolutionMapper()
```

--------------------------------

### Manual Minimal Dependency Installation

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Manually installs minimal dependencies for Data-Juicer from source, tailored to a specific configuration file. This command is intended for source installations.

```shell
# only for installation from source
python tools/dj_install.py --config path_to_your_data-juicer_config_file
```

```shell
# use command line tool
dj-install --config path_to_your_data-juicer_config_file
```

--------------------------------

### Check Data Juicer Installation Version

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Run this Python code to verify the installed version of the data-juicer library. Ensure FFmpeg is installed and accessible via the $PATH for video-related operators.

```python
import data_juicer as dj
print(dj.__version__)
```

--------------------------------

### Run Data-Juicer Demos

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/README.md

Navigate to a demo's subdirectory and execute `app.py` using Streamlit to launch the demo.

```shell
cd <subdir_of_demos>
streamlit run app.py
```

--------------------------------

### Install Data-Juicer with Distributed Computing Features

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/Installation.md

Installs Data-Juicer with components for Distributed Computing.

```bash
# Distributed Computing
uv pip install "py-data-juicer[distributed]"
```

--------------------------------

### Install Data Juicer with AI Services

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/agent/QUICKSTART_BAD_CASE.md

Install the Data Juicer package with AI services enabled, which is required for full pipeline runs involving LLMs. For smoke tests only, a minimal installation is sufficient.

```bash
cd /path/to/data-juicer
uv venv && source .venv/bin/activate
uv pip install -e ".[ai_services]"   # 全量菜谱需要 LLM；仅 smoke 可只装 -e .
```

--------------------------------

### Analyze Data (Source Install)

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/QuickStart.md

Run the data analysis script when DataJuicer is installed from source.

```shell
# only for installation from source
python tools/analyze_data.py --config demos/analyze_simple/analyzer.yaml
```

--------------------------------

### Initialize VideoTaggingFromFramesMapper

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/video_tagging_from_frames_mapper.md

Instantiate the VideoTaggingFromFramesMapper operator. No specific parameters are shown in this basic example, implying default configurations are used.

```python
VideoTaggingFromFramesMapper()
```

--------------------------------

### Run Full Recipe (Multi-LLM)

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/agent/QUICKSTART_BAD_CASE.md

Execute the full pipeline, which involves multiple LLMs and can be resource-intensive and time-consuming. This is recommended for comprehensive analysis.

```bash
bash demos/agent/scripts/run_bad_case_pipeline.sh full
```

--------------------------------

### Initialize Tracer in Default Mode (Python)

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/Tracing.md

Instantiate the Tracer class with specified working directory, operators to trace, sample display count, and keys for trace output.

```python
from data_juicer.core.tracer import Tracer

tracer = Tracer(
    work_dir="./outputs",
    op_list_to_trace=["clean_email_mapper", "words_num_filter"],
    show_num=10,
    trace_keys=["sample_id"]
)
```

--------------------------------

### Run Data Processing (Source Install)

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/tutorial/QuickStart.md

Execute the data processing script when DataJuicer is installed from source.

```shell
# only for installation from source
python tools/process_data.py --config demos/process_simple/process.yaml
```

--------------------------------

### Command Line Cache Configuration

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/Cache_ZH.md

Examples of configuring cache settings using the command line interface.

```APIDOC
## Command Line

```bash
# Enable cache (default)
dj-process --config config.yaml --use_cache true

# Disable cache
dj-process --config config.yaml --use_cache false

# Enable cache compression
dj-process --config config.yaml --cache_compress zstd

# Custom cache directory
dj-process --config config.yaml --ds_cache_dir /fast-storage/dj-cache
```
```

--------------------------------

### Cache vs. Checkpoint Configuration

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/Cache_ZH.md

Configuration examples showing the relationship and mutual exclusivity between cache and checkpoint modes.

```APIDOC
## Cache and Checkpoint

Cache and checkpoint are mutually exclusive—enabling checkpoint automatically disables cache:

| Feature | Cache | Checkpoint |
|---|---|---|
| **Purpose** | Speed up repeated runs with the same configuration | Fault recovery and resuming from breakpoints |
| **Granularity** | Result of each operator | Full dataset snapshot |
| **Storage Location** | HuggingFace cache directory | Working directory |
| **Recovery Method** | Automatic (hash-based) | Manual (configuration-based) |
| **Compression** | Supported (`cache_compress`) | Not applicable |
| **Scenario** | Iterative development, parameter tuning | Long-running production tasks |

```yaml
# Cache mode (default)
use_cache: true
use_checkpoint: false

# Checkpoint mode (cache automatically disabled)
use_cache: true           # Will be overridden to false
use_checkpoint: true
```
```

--------------------------------

### Initialize VideoSplitByKeyFrameMapper

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/operators/mapper/video_split_by_key_frame_mapper.md

Initializes the VideoSplitByKeyFrameMapper with specific configurations for keeping original samples and saving split videos to a temporary directory.

```python
VideoSplitByKeyFrameMapper(keep_original_sample=False, save_dir='tempfile.TemporaryDirectory().name')
```

--------------------------------

### Install Data-Juicer with Distributed Requirements

Source: https://github.com/datajuicer/data-juicer/blob/main/docs/Distributed.md

Install Data-Juicer and its distributed dependencies, including Ray, using pip.

```shell
uv pip install -v -e .  # Install the minimal requirements of Data-Juicer
uv pip install -v -e ".[dist]"  # Include dependencies on Ray and other distributed libraries
```

--------------------------------

### Configure Data-Juicer Tools

Source: https://github.com/datajuicer/data-juicer/blob/main/demos/api_service/react_data_mapper_process.ipynb

Sets up the Data-Juicer service toolkit by loading predefined mapper functions. This allows the agent to access Data-Juicer operations.

```python
from utils import setup_service_toolkit

service_toolkit = setup_service_toolkit('wrapped_mappers')
```