DocETL

https://ucbepic.github.io/docetl/
Admin
DocETL is a low-code, declarative YAML interface tool for creating and executing LLM-powered data...
Tokens:353,683
Snippets:2,059
Trust Score:8.3
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install DocETL with pip

Source: https://ucbepic.github.io/docetl/installation

Install DocETL using pip. This is the standard installation method.

```bash
pip install docetl
```

--------------------------------

### Install dependencies with uv (all extras)

Source: https://ucbepic.github.io/docetl/installation

Install project dependencies and DocETL with all extras using uv.

```bash
uv sync --all-extras
```

--------------------------------

### Install dependencies with uv (parsing extra only)

Source: https://ucbepic.github.io/docetl/installation

Install project dependencies and DocETL with only the parsing extra using uv.

```bash
uv sync --extra parsing
```

--------------------------------

### Start development server with make run-ui-dev

Source: https://ucbepic.github.io/docetl/playground

Starts the development server. After running, navigate to http://localhost:3000/playground to access DocWrangler.

```bash
make run-ui-dev

```

--------------------------------

### Install dependencies with make install and make install-ui

Source: https://ucbepic.github.io/docetl/playground

Installs the Python package and UI dependencies for local development.

```bash
make install      # Install Python package
make install-ui   # Install UI dependencies

```

--------------------------------

### Install uv

Source: https://ucbepic.github.io/docetl/installation

Install the uv package manager if it is not already installed.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

--------------------------------

### Install DocETL with parsing extra

Source: https://ucbepic.github.io/docetl/installation

Install DocETL with the optional 'parsing' extra to enable parsing tools.

```bash
pip install docetl[parsing]
```

--------------------------------

### Example Calibration Output

Source: https://ucbepic.github.io/docetl/operators/map

Example of how the original prompt is augmented with reference anchors during calibration. This output is illustrative and shows the type of guidance appended to the prompt for all documents.

```text
# Original prompt gets augmented with something like:
# 
# For reference, consider 'Server completely down for 500+ users' → critical as your baseline for critical issues.
# Documents similar to 'Login button not working for one user' → low priority.
# For reference, consider 'Payment processing delays affecting checkout' → high as your standard for high priority issues.

```

--------------------------------

### Install DocETL with TUI extra

Source: https://ucbepic.github.io/docetl/execution/interactive-progress

Install the optional 'tui' extra to enable the interactive dashboard. This pulls in Textual.

```bash
pip install "docetl[tui]"
```

--------------------------------

### Install DocETL and configure model credentials

Source: https://ucbepic.github.io/docetl/examples/tool-equipped-research-agents

Install Docetl and set model credentials. For Azure OpenAI, set the AZURE_API_BASE, AZURE_API_KEY, and AZURE_API_VERSION environment variables. The model stays on each operation; for Azure OpenAI use the LiteLLM deployment name, for example model="azure/gpt-4o-mini".

```bash
pip install docetl
export OPENAI_API_KEY="..."
# or Azure OpenAI:
export AZURE_API_BASE="https://<resource>.openai.azure.com/"
export AZURE_API_KEY="..."
export AZURE_API_VERSION="2024-12-01-preview"
```

--------------------------------

### Verify DocETL installation

Source: https://ucbepic.github.io/docetl/installation

Check that DocETL is installed correctly by running the version command.

```bash
docetl version
```

--------------------------------

### Install DocETL and Claude Code skill via pip

Source: https://ucbepic.github.io/docetl/quickstart-claude-code

Installs DocETL via pip and then installs the Claude Code skill, copying it to ~/.claude/skills/docetl/. Run `claude` in any directory after installation.

```bash
pip install docetl
docetl install-skill

```

--------------------------------

### Example JSON file (reviews.json)

Source: https://ucbepic.github.io/docetl/concepts/datasets

A JSON file is a list of objects; each object is one row.

```json
// reviews.json
[
  {"id": 1, "product": "headphones", "review": "Battery died after a week."},
  {"id": 2, "product": "keyboard", "review": "Keys feel great, very quiet."}
]

```

--------------------------------

### experiment_summary.json example

Source: https://ucbepic.github.io/docetl/optimization/moar/results

Example content of experiment_summary.json, the high-level summary file generated after running `docetl build pipeline.yaml`. Contains optimizer name, input pipeline, model, iteration count, save directory, dataset, timestamps, duration, and search metrics.

```json
{
  "optimizer": "moar",
  "input_pipeline": "pipeline.yaml",
  "rewrite_agent_model": "gpt-5.1",
  "max_iterations": 40,
  "save_dir": "results/moar_optimization",
  "dataset": "transcripts",
  "start_time": "2024-01-15T10:30:00",
  "end_time": "2024-01-15T11:15:00",
  "duration_seconds": 2700,
  "num_best_nodes": 5,
  "total_nodes_explored": 120,
  "total_search_cost": 15.50
}
```

--------------------------------

### Excel Parsing Output Example

Source: https://ucbepic.github.io/docetl/examples/custom-parsing

Example output from the top_products_report function for an Excel file, showing a report of top-selling products.

```text
Top Products Report - January 2023

1. Widget A - 1500 units sold
2. Gadget B - 1200 units sold
3. Gizmo C - 950 units sold
4. Doohickey D - 800 units sold
5. Thingamajig E - 650 units sold
   ...

Total Revenue: $245,000
Best Selling Category: Electronics

```

--------------------------------

### YAML prompt example for reduce operation

Source: https://ucbepic.github.io/docetl/concepts/operators

Example of a YAML prompt that references 'inputs' in a reduce operation, iterating over a list of inputs to summarize insights and supporting actions.

```yaml
prompt: |
  Summarize the user behavior insights for the country: {{ inputs[0].country }}

  Insights and supporting actions:
  {% for item in inputs %}
  - Insight: {{ item.insight }}
  Supporting actions:
  {% for action in item.supporting_actions %}
  - {{ action }}
  {% endfor %}
  {% endfor %}

```

--------------------------------

### Python prompt example for reduce operation

Source: https://ucbepic.github.io/docetl/concepts/operators

Example of a Python prompt string that references 'inputs' in a reduce operation, iterating over a list of inputs to summarize insights and supporting actions.

```python
prompt="""Summarize the user behavior insights for the country: {{ inputs[0].country }}

Insights and supporting actions:
{% for item in inputs %}
- Insight: {{ item.insight }}
Supporting actions:
{% for action in item.supporting_actions %}
- {{ action }}
{% endfor %}
{% endfor %}"""

```

--------------------------------

### Example output of collect()

Source: https://ucbepic.github.io/docetl

Example output of the Python pipeline's collect() method, showing the top_themes list with name, description, and num_mentions fields.

```json
[
    {"name": "onboarding friction", "description": "confusion during first-week setup", "num_mentions": 12},
    {"name": "pricing transparency", "description": "unclear billing and plan limits", "num_mentions": 8},
    ...
]
```

--------------------------------

### pareto_frontier.json example

Source: https://ucbepic.github.io/docetl/optimization/moar/results

Example content of pareto_frontier.json, listing Pareto-optimal solutions with node_id, yaml_path, cost, and accuracy. Use this file to choose a solution based on cost or accuracy priorities.

```json
[
  {
    "node_id": 5,
    "yaml_path": "results/moar_optimization/pipeline_5.yaml",
    "cost": 0.05,
    "accuracy": 0.92
  },
  {
    "node_id": 12,
    "yaml_path": "results/moar_optimization/pipeline_12.yaml",
    "cost": 0.08,
    "accuracy": 0.95
  }
]
```

--------------------------------

### EquijoinOperation usage example

Source: https://ucbepic.github.io/docetl/api-reference/operations

Shows how to configure and run an EquijoinOperation with blocking keys, limits, a comparison prompt, a blocking threshold, blocking conditions, and a comparison limit. The execute method returns a tuple of joined results and total cost, which are printed to the console.

```python
from docetl.operations import EquijoinOperation

config = {
    "blocking_keys": {
        "left": ["id"],
        "right": ["user_id"]
    },
    "limits": {
        "left": 1,
        "right": 1
    },
    "comparison_prompt": "Compare {{left}} and {{right}} and determine if they match.",
    "blocking_threshold": 0.8,
    "blocking_conditions": ["left['id'] == right['user_id']"],
    "limit_comparisons": 1000
}
equijoin_op = EquijoinOperation(config)
left_data = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
right_data = [{"user_id": 1, "age": 30}, {"user_id": 2, "age": 25}]
results, cost = equijoin_op.execute(left_data, right_data)
print(f"Joined results: {results}")
print(f"Total cost: {cost}")
```

--------------------------------

### Sample dataset

Source: https://ucbepic.github.io/docetl/examples/split-gather

Example dataset containing a single document with a pdf_url field.

```json
[
  {
    "pdf_url": "https://storage.courtlistener.com/recap/gov.uscourts.dcd.258148/gov.uscourts.dcd.258148.252.0.pdf"
  }
]

```

--------------------------------

### EquijoinOperation.execute

Source: https://ucbepic.github.io/docetl/api-reference/operations

Documentation for the EquijoinOperation.execute method, including configuration parameters, usage example, and behavior.

```APIDOC
## EquijoinOperation.execute

### Description
Executes an equijoin operation on two datasets, performing blocking, comparison, and aggregation to produce joined results and the total cost.

### Method
execute

### Signature
```python
def execute(self, left_data: list[dict], right_data: list[dict]) -> tuple[list[dict], float]
```

### Parameters
- **left_data** (list[dict]) - Required - The left dataset to join.
- **right_data** (list[dict]) - Required - The right dataset to join.

### Returns
- **tuple[list[dict], float]** - A tuple containing the joined results and the total cost of the operation.

### Usage Example
```python
from docetl.operations import EquijoinOperation

config = {
    "blocking_keys": {
        "left": ["id"],
        "right": ["user_id"]
    },
    "limits": {
        "left": 1,
        "right": 1
    },
    "comparison_prompt": "Compare {{left}} and {{right}} and determine if they match.",
    "blocking_threshold": 0.8,
    "blocking_conditions": ["left['id'] == right['user_id']"],
    "limit_comparisons": 1000
}
equijoin_op = EquijoinOperation(config)
left_data = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
right_data = [{"user_id": 1, "age": 30}, {"user_id": 2, "age": 25}]
results, cost = equijoin_op.execute(left_data, right_data)
print(f"Joined results: {results}")
print(f"Total cost: {cost}")
```

### Configuration Parameters
- **blocking_keys** (dict) - Optional - Keys to use for blocking, with "left" and "right" lists.
- **limits** (dict) - Optional - Limits for left and right datasets.
- **comparison_prompt** (str) - Required - Prompt template for comparing pairs.
- **blocking_threshold** (float) - Optional - Threshold for embedding-based blocking.
- **blocking_conditions** (list) - Optional - Conditions for initial blocking.
- **limit_comparisons** (int) - Optional - Maximum number of comparisons.
- **blocking_target_recall** (float) - Optional - Target recall for auto-computed threshold (default 0.95).
- **comparison_model** (str) - Optional - Model to use for comparisons.
- **timeout** (int) - Optional - Timeout in seconds for comparisons (default 120).
- **max_retries_per_timeout** (int) - Optional - Max retries per timeout (default 2).

### Behavior
1. Initial blocking based on specified conditions (if any).
2. Embedding-based blocking (if threshold is provided).
3. LLM-based comparison for blocked pairs.
4. Result aggregation and validation.

### Statistics
Calculates and logs statistics such as comparisons saved by blocking and join selectivity.
```

--------------------------------

### Example CSV file (tickets.csv)

Source: https://ucbepic.github.io/docetl/concepts/datasets

Each row of the table is one row of the dataset; column names become keys.

```csv
ticket_id,customer,message
101,acme,"Cannot log in since the update"
102,globex,"Invoice total looks wrong"

```

--------------------------------

### UnnestOperation examples: unnesting a list and a dictionary

Source: https://ucbepic.github.io/docetl/api-reference/operations

Two examples from the UnnestOperation docstring: unnesting a list of values and unnesting a dictionary with expand_fields. When unnesting dictionaries, the original nested dictionary is preserved and the specified fields are expanded into the parent dictionary.

```python
# Unnesting a list
unnest_op = UnnestOperation({"unnest_key": "colors"})
input_data = [
    {"id": 1, "colors": ["red", "blue"]},
    {"id": 2, "colors": ["green"]}
]
result, _ = unnest_op.execute(input_data)
# Result will be:
# [
#     {"id": 1, "colors": "red"},
#     {"id": 1, "colors": "blue"},
#     {"id": 2, "colors": "green"}
# ]
```

```python
# Unnesting a dictionary
unnest_op = UnnestOperation({"unnest_key": "details", "expand_fields": ["color", "size"]})
input_data = [
    {"id": 1, "details": {"color": "red", "size": "large", "stock": 5}},
    {"id": 2, "details": {"color": "blue", "size": "medium", "stock": 3}}
]
result, _ = unnest_op.execute(input_data)
# Result will be:
# [
#     {"id": 1, "details": {"color": "red", "size": "large", "stock": 5}, "color": "red", "size": "large"},
#     {"id": 2, "details": {"color": "blue", "size": "medium", "stock": 3}, "color": "blue", "size": "medium"}
# ]
```

--------------------------------

### UnnestOperation

Source: https://ucbepic.github.io/docetl/api-reference/operations

Documentation for the UnnestOperation class, including its configuration schema and usage examples.

```APIDOC
## UnnestOperation

### Description
A class that represents an operation to unnest a list-like or dictionary value in a dictionary into multiple dictionaries. This operation takes a list of dictionaries and a specified key, and creates new dictionaries based on the value type: for list-like values, it creates a new dictionary for each element in the list, copying all other key-value pairs; for dictionary values, it expands specified fields from the nested dictionary into the parent dictionary.

### Usage
```python
from docetl.operations import UnnestOperation

# Unnesting a list
config_list = {"unnest_key": "tags"}
input_data_list = [
    {"id": 1, "tags": ["a", "b", "c"]},
    {"id": 2, "tags": ["d", "e"]}
]

unnest_op_list = UnnestOperation(config_list)
result_list, _ = unnest_op_list.execute(input_data_list)

# Result will be:
# [
#     {"id": 1, "tags": "a"},
#     {"id": 1, "tags": "b"},
#     {"id": 1, "tags": "c"},
#     {"id": 2, "tags": "d"},
#     {"id": 2, "tags": "e"}
# ]

# Unnesting a dictionary
config_dict = {"unnest_key": "user", "expand_fields": ["name", "age"]}
input_data_dict = [
    {"id": 1, "user": {"name": "Alice", "age": 30, "email": "alice@example.com"}},
    {"id": 2, "user": {"name": "Bob", "age": 25, "email": "bob@example.com"}}
]

unnest_op_dict = UnnestOperation(config_dict)
result_dict, _ = unnest_op_dict.execute(input_data_dict)

# Result will be:
# [
#     {"id": 1, "name": "Alice", "age": 30, "user": {"name": "Alice", "age": 30, "email": "alice@example.com"}},
#     {"id": 2, "name": "Bob", "age": 25, "user": {"name": "Bob", "age": 25, "email": "bob@example.com"}}
# ]
```

### Configuration Schema
- **type** (str) - Required - Must be set to "unnest".
- **unnest_key** (str) - Required - The key in the input dictionaries whose value will be unnested.
- **keep_empty** (bool | None) - Optional - Whether to keep empty values during unnesting.
- **expand_fields** (list[str] | None) - Optional - Fields to expand from a nested dictionary into the parent dictionary.
- **recursive** (bool | None) - Optional - Whether to recursively unnest nested structures.
- **depth** (int | None) - Optional - The maximum depth to unnest when recursive is enabled.
```

--------------------------------

### EquijoinOperation.execute source code

Source: https://ucbepic.github.io/docetl/api-reference/operations

Source code of the execute method from docetl/operations/equijoin.py. It includes the method signature, docstring with usage example, and a comment describing the four-step process: initial blocking, embedding-based blocking, LLM-based comparison, and result aggregation/validation.

```python
def execute(
    self, left_data: list[dict], right_data: list[dict]
) -> tuple[list[dict], float]:
    """
    Executes the equijoin operation on the provided datasets.

    Args:
        left_data (list[dict]): The left dataset to join.
        right_data (list[dict]): The right dataset to join.

    Returns:
        tuple[list[dict], float]: A tuple containing the joined results and the total cost of the operation.

    Usage:
    ```python
    from docetl.operations import EquijoinOperation

    config = {
        "blocking_keys": {
            "left": ["id"],
            "right": ["user_id"]
        },
        "limits": {
            "left": 1,
            "right": 1
        },
        "comparison_prompt": "Compare {{left}} and {{right}} and determine if they match.",
        "blocking_threshold": 0.8,
        "blocking_conditions": ["left['id'] == right['user_id']"],
        "limit_comparisons": 1000
    }
    equijoin_op = EquijoinOperation(config)
    left_data = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
    right_data = [{"user_id": 1, "age": 30}, {"user_id": 2, "age": 25}]
    results, cost = equijoin_op.execute(left_data, right_data)
    print(f"Joined results: {results}")
    print(f"Total cost: {cost}")
    ```

    This method performs the following steps:
    1. Initial blocking based on specified conditions (if any)
    2. Embedding-based blocking (if threshold is provided)
    3. LLM-based comparison for blocked pairs
    4. Result aggregation and validation
    """
```

--------------------------------

### Basic output schema example in YAML

Source: https://ucbepic.github.io/docetl/concepts/operators

Defines an output schema with a list of objects and a string field. Use this as a simple starting point for structured LLM responses.

```yaml
output:
  schema:
    insights: "list[{insight: string, supporting_actions: string}]"
    detailed_summary: string

```

--------------------------------

### ClusterOperation class definition and plan traits

Source: https://ucbepic.github.io/docetl/api-reference/operations

Definition of the ClusterOperation class, including schema transformation, plan trait methods (cardinality, fields_written, is_llm, preserves_order), and the constructor that sets max_batch_size and calls ensure_non_jinja_prompt_confirmed. The syntax_check method is started but its body is not shown in the source.

```python
class ClusterOperation(BaseOperation):
    @classmethod
    def transform_schema(cls, schema, config):
        result = super().transform_schema(schema, config)
        result[config.get("output_key", "clusters")] = "list"
        return result

    # ── plan traits ────────────────────────────────────────────────
    # Annotates every row in place (order kept) but cluster assignments
    # depend on the whole dataset, so not row-local. fields_read stays
    # None: summaries render whole cluster members.

    @classmethod
    def cardinality(cls, config):
        return Cardinality.ONE_TO_ONE

    @classmethod
    def fields_written(cls, config):
        return frozenset({config.get("output_key", "clusters")})

    @classmethod
    def is_llm(cls, config):
        return True

    @classmethod
    def preserves_order(cls, config):
        return True

    def __init__(
        self,
        *args,
        **kwargs,
    ):
        super().__init__(*args, **kwargs)
        self.max_batch_size: int = self.config.get(
            "max_batch_size", kwargs.get("max_batch_size", float("inf"))
        )
        ensure_non_jinja_prompt_confirmed(
            self.config,
            "summary_prompt",
            "_append_document_to_prompt",
            console=self.console,
            status=self.status,
            extra_flags={"_is_reduce_operation": True},
        )

    def syntax_check(self) -> None:
        """
        Checks the configuration of the ClusterOperation for required keys and valid structure.
        """
```

--------------------------------

### render_chunk_with_context method in docetl/operations/gather.py

Source: https://ucbepic.github.io/docetl/api-reference/operations

Renders a chunk with its peripheral context and headers. If no peripheral configuration is provided, returns the main chunk content directly. Otherwise, builds a combined string with previous and next context sections, optional hierarchy headers, and the main chunk delimited by the provided start and end markers.

```python
def render_chunk_with_context(
    self,
    chunks: list[dict],
    current_index: int,
    peripheral_config: dict,
    content_key: str,
    order_key: str,
    main_chunk_start: str,
    main_chunk_end: str,
    doc_header_key: str,
) -> str:
    """
    Render a chunk with its peripheral context and headers.

    Args:
        chunks (list[dict]): List of all chunks in the document.
        current_index (int): Index of the current chunk being processed.
        peripheral_config (dict): Configuration for peripheral chunks.
        content_key (str): Key for the content in each chunk.
        order_key (str): Key for the order of each chunk.
        main_chunk_start (str): String to mark the start of the main chunk.
        main_chunk_end (str): String to mark the end of the main chunk.
        doc_header_key (str): The key for the headers in the current chunk.

    Returns:
        str: Renderted chunk with context and headers.
    """

    # If there are no peripheral chunks, return the main chunk
    if not peripheral_config:
        return chunks[current_index][content_key]

    combined_parts = ["--- Previous Context ---"]

    combined_parts.extend(
        self.process_peripheral_chunks(
            chunks[:current_index],
            peripheral_config.get("previous", {}),
            content_key,
            order_key,
        )
    )
    combined_parts.append("--- End Previous Context ---\n")

    # Process main chunk
    main_chunk = chunks[current_index]
    if headers := self.render_hierarchy_headers(
        main_chunk, chunks[: current_index + 1], doc_header_key
    ):
        combined_parts.append(headers)
    combined_parts.extend(
        (
            f"{main_chunk_start}",
            f"{main_chunk[content_key]}",
            f"{main_chunk_end}",
            "\n--- Next Context ---",
        )
    )
    combined_parts.extend(
        self.process_peripheral_chunks(
            chunks[current_index + 1 :],
            peripheral_config.get("next", {}),
            content_key,
            order_key,
        )
    )
    combined_parts.append("--- End Next Context ---")

    return "\n".join(combined_parts)

```

--------------------------------

### Aggregate common themes with pipeline.reduce and write JSON

Source: https://ucbepic.github.io/docetl/examples/mining-product-reviews

This snippet shows the final part of a DocETL pipeline that aggregates common themes from multiple video game reviews. It includes a prompt template for consolidating themes and a reduce operation that produces a JSON output file. The code is incomplete as it starts mid-prompt and lacks the initial pipeline setup.

```plaintext
Theme 2: {{ input2.theme }}
Summary 2: {{ input2.summary }}

```

```python
"""
Given the following themes, please come up with a theme that best captures the essence of all the themes:

{% for input in inputs %}
Theme {{ loop.index }}: {{ input.theme }}
{% if not loop.last %}
---
{% endif %}
{% endfor %}

Based on these themes, provide a consolidated theme that captures the essence of all the above themes. Ensure that the consolidated theme is concise yet comprehensive.
""",
output={"schema": {"theme": "str"}},
)
```

```python
pipeline = pipeline.reduce(
    name="aggregate_common_themes",
    reduce_key="theme",
    prompt="""
    You are given a theme and summary that appears across multiple video games, along with various apps and review quotes related to this theme. Your task is to consolidate this information into a comprehensive report.

    For each input, you will receive:
    - theme: A specific polarizing theme
    - summary: A brief summary of the theme
    - app_name: The name of the game
    - positive_quotes: List of supporting quotes from positive perspectives
    - negative_quotes: List of supporting quotes from negative perspectives

    Create a report that includes:
    1. The name of the common theme
    2. A summary of the theme and why it's common across games
    3. Representative quotes from different games, both positive and negative

    Here's the information for the theme:
    Theme: {{ inputs[0].theme }}
    Summary: {{ inputs[0].summary }}

    {% for app in inputs %}
    Game: {{ app.app_name }}
    Positive Quotes: {{ app.positive_quotes }}
    Negative Quotes: {{ app.negative_quotes }}
    {% if not loop.last %}
    ----------------------------------------
    {% endif %}
    {% endfor %}
    """,
    output={
        "schema": {
            "theme_summary": "str",
            "representative_quotes": "list[{game: str, quote: str, sentiment: str}]",
        }
    },
)

pipeline.write_json("/path/to/output/output_polarizing_themes.json")

```

--------------------------------

### Run basic test suite

Source: https://ucbepic.github.io/docetl/installation

Run the basic test suite to ensure everything is working. This costs less than $0.01 with OpenAI.

```bash
make tests-basic
```

--------------------------------

### Optimizer.__init__ - Initialize optimizer with runner and agent models

Source: https://ucbepic.github.io/docetl/api-reference/docetl

Initializes the Optimizer with a DSLRunner, default rewrite agent model gpt-5.1, judge agent model gpt-4o-mini, and optional litellm kwargs. Reads rate limits from optimizer_config, creates an LLMClient, and sets up a cache directory under DOCETL_HOME_DIR or the user's home directory.

```python
class Optimizer:

    def __init__(
        self,
        runner: "DSLRunner",
        rewrite_agent_model: str = "gpt-5.1",
        judge_agent_model: str = "gpt-4o-mini",
        litellm_kwargs: dict[str, Any] = {},
        resume: bool = False,
        timeout: int = 60,
    ):
        self.config = runner.config
        self.console = runner.console
        self.max_threads = runner.max_threads

        self.base_name = runner.base_name
        self.yaml_file_suffix = runner.yaml_file_suffix
        self.runner = runner
        self.status = runner.status

        rate_limits = self.config.get("optimizer_config", {}).get("rate_limits", {})

        self.llm_client = LLMClient(
            runner,
            rewrite_agent_model,
            judge_agent_model,
            rate_limits,
            **litellm_kwargs,
        )
        self.timeout = timeout
        self.resume = resume
        self.captured_output = CapturedOutput()

        self.sample_cache = {}

        home_dir = os.environ.get("DOCETL_HOME_DIR", os.path.expanduser("~"))
        cache_dir = os.path.join(home_dir, f".docetl/cache/{runner.yaml_file_suffix}")
        os.makedirs(cache_dir, exist_ok=True)

```

--------------------------------

### UnnestOperation class source with schema

Source: https://ucbepic.github.io/docetl/api-reference/operations

The full source code of the UnnestOperation class, including its schema definition. The schema sets type to "unnest" and defines optional fields: keep_empty, expand_fields, recursive, and depth. The class docstring contains the usage example.

```python
class UnnestOperation(BaseOperation):
    """
    A class that represents an operation to unnest a list-like or dictionary value in a dictionary into multiple dictionaries.

    This operation takes a list of dictionaries and a specified key, and creates new dictionaries based on the value type:
    - For list-like values: Creates a new dictionary for each element in the list, copying all other key-value pairs.
    - For dictionary values: Expands specified fields from the nested dictionary into the parent dictionary.

    Inherits from:
        BaseOperation

    Usage:
    ```python
    from docetl.operations import UnnestOperation

    # Unnesting a list
    config_list = {"unnest_key": "tags"}
    input_data_list = [
        {"id": 1, "tags": ["a", "b", "c"]},
        {"id": 2, "tags": ["d", "e"]}
    ]

    unnest_op_list = UnnestOperation(config_list)
    result_list, _ = unnest_op_list.execute(input_data_list)

    # Result will be:
    # [
    #     {"id": 1, "tags": "a"},
    #     {"id": 1, "tags": "b"},
    #     {"id": 1, "tags": "c"},
    #     {"id": 2, "tags": "d"},
    #     {"id": 2, "tags": "e"}
    # ]

    # Unnesting a dictionary
    config_dict = {"unnest_key": "user", "expand_fields": ["name", "age"]}
    input_data_dict = [
        {"id": 1, "user": {"name": "Alice", "age": 30, "email": "alice@example.com"}},
        {"id": 2, "user": {"name": "Bob", "age": 25, "email": "bob@example.com"}}
    ]

    unnest_op_dict = UnnestOperation(config_dict)
    result_dict, _ = unnest_op_dict.execute(input_data_dict)

    # Result will be:
    # [
    #     {"id": 1, "name": "Alice", "age": 30, "user": {"name": "Alice", "age": 30, "email": "alice@example.com"}},
    #     {"id": 2, "name": "Bob", "age": 25, "user": {"name": "Bob", "age": 25, "email": "bob@example.com"}}
    # ]
    """

    class schema(BaseOperation.schema):
        type: str = "unnest"
        unnest_key: str
        keep_empty: bool | None = None
        expand_fields: list[str] | None = None
        recursive: bool | None = None
        depth: int | None = None
```

--------------------------------

### Clear and Concise Prompt for summarize_prescriptions

Source: https://ucbepic.github.io/docetl/best-practices

Example prompt for the summarize_prescriptions operation in the tutorial. It provides specific instructions, including quantities (2-3 side effects, 1-2 primary uses) and constraints (based solely on transcripts, focus only on the reduce_key, include quotes).

```yaml
prompt: |
  Here are some transcripts of conversations between a doctor and a patient:

  {% for value in inputs %}
  Transcript {{ loop.index }}:
  {{ value.src }}
  {% endfor %}

  For the medication {{ reduce_key }}, please provide the following information based on all the transcripts above:

  1. Side Effects: Summarize all mentioned side effects of {{ reduce_key }}. List 2-3 main side effects.
  2. Therapeutic Uses: Explain the medical conditions or symptoms for which {{ reduce_key }} was prescribed or recommended. Provide 1-2 primary uses.

  Ensure your summary:
  - Is based solely on information from the provided transcripts
  - Focuses only on {{ reduce_key }}, not other medications
  - Includes relevant details from all transcripts
  - Is clear and concise
  - Includes quotes from the transcripts

```

```python
prompt="""Here are some transcripts of conversations between a doctor and a patient:

{% for value in inputs %}
Transcript {{ loop.index }}:
{{ value.src }}
{% endfor %}

For the medication {{ reduce_key }}, please provide the following information based on all the transcripts above:

1. Side Effects: Summarize all mentioned side effects of {{ reduce_key }}. List 2-3 main side effects.
2. Therapeutic Uses: Explain the medical conditions or symptoms for which {{ reduce_key }} was prescribed or recommended. Provide 1-2 primary uses.

Ensure your summary:
- Is based solely on information from the provided transcripts
- Focuses only on {{ reduce_key }}, not other medications
- Includes relevant details from all transcripts
- Is clear and concise
- Includes quotes from the transcripts"""

```

--------------------------------

### UnnestOperation execute method

Source: https://ucbepic.github.io/docetl/api-reference/operations

The execute method for UnnestOperation. It processes input data by unnesting list-like values (each element becomes a separate row) or expanding dictionary fields. Raises KeyError, TypeError, or ValueError as documented. The method supports recursive and depth configuration, and the docstring includes examples for both list and dictionary unnesting.

```python
    def execute(self, input_data: list[dict]) -> tuple[list[dict], float]:
        """
        Executes the unnest operation on the input data.

        Args:
            input_data (list[dict]): A list of dictionaries to process.

        Returns:
            tuple[list[dict], float]: A tuple containing the processed list of dictionaries
            and a float value (always 0 in this implementation).

        Raises:
            KeyError: If the specified unnest_key is not found in an input dictionary.
            TypeError: If the value of the unnest_key is not iterable (list, tuple, set, or dict).
            ValueError: If unnesting a dictionary and 'expand_fields' is not provided in the config.

        The operation supports unnesting of both list-like values and dictionary values:

        1. For list-like values (list, tuple, set):
           Each element in the list becomes a separate dictionary in the output.

        2. For dictionary values:
           The operation expands specified fields from the nested dictionary into the parent dictionary.
           The 'expand_fields' config parameter must be provided to specify which fields to expand.

        Examples:
        ```python
        # Unnesting a list
        unnest_op = UnnestOperation({"unnest_key": "colors"})
        input_data = [
            {"id": 1, "colors": ["red", "blue"]},
            {"id": 2, "colors": ["green"]}
        ]
        result, _ = unnest_op.execute(input_data)
        # Result will be:
        # [
        #     {"id": 1, "colors": "red"},
        #     {"id": 1, "colors": "blue"},
        #     {"id": 2, "colors": "green"}
        # ]

        # Unnesting a dictionary
        unnest_op = UnnestOperation({"unnest_key": "details", "expand_fields": ["color", "size"]})
        input_data = [
            {"id": 1, "details": {"color": "red", "size": "large", "stock": 5}},
            {"id": 2, "details": {"color": "blue", "size": "medium", "stock": 3}}
        ]
        result, _ = unnest_op.execute(input_data)
        # Result will be:
        # [
        #     {"id": 1, "details": {"color": "red", "size": "large", "stock": 5}, "color": "red", "size": "large"},
        #     {"id": 2, "details": {"color": "blue", "size": "medium", "stock": 3}, "color": "blue", "size": "medium"}
        # ]
        ```

        Note: When unnesting dictionaries, the original nested dictionary is preserved in the output,
        and the specified fields are expanded into the parent dictionary.
        """

        unnest_key = self.config["unnest_key"]
        recursive = self.config.get("recursive", False)
        depth = self.config.get("depth", None)
        if not depth:
            depth = 1 if not recursive else float("inf")
        results = []

        def unnest_recursive(item, key, level=0):
            if level == 0 and not isinstance(item[key], (list, tuple, set, dict)):
                raise TypeError(f"Value of unnest key '{key}' is not iterable")

            if level > 0 and not isinstance(item[key], (list, tuple, set, dict)):
                return [item]

            if level >= depth:
                return [item]
```

--------------------------------

### Output of docetl run pipeline_opt.yaml

Source: https://ucbepic.github.io/docetl/examples/presidential-debate-themes

Example output from running the optimized pipeline. Shows syntax check, operation progress, intermediate saves, resolution statistics (e.g., 56002 comparisons saved by blocking, 339 keys reduced to 152), and final cost and time metrics. Total cost was $0.29 and total time was 97.25 seconds.

```bash
$ docetl run pipeline_opt.yaml
[09:28:17] Performing syntax check on all operations...
           Syntax check passed for all operations.
           Running Operation:
             Type: map
             Name: extract_themes_and_viewpoints
⠧ Running step debate_analysis...
[09:28:36] Intermediate saved for operation 'extract_themes_and_viewpoints'
           Running Operation:
             Type: unnest
             Name: unnest_themes
           Intermediate saved for operation 'unnest_themes'
           Running Operation:
             Type: resolve
             Name: synthesized_resolve_0
[09:28:38] Comparisons saved by blocking: 56002 (97.75%)
⠋ Running step debate_analysis...
[09:29:02] Number of keys before resolution: 339
           Number of distinct keys after resolution: 152
⠋ Running step debate_analysis...
[09:29:04] Self-join selectivity: 0.0390
           Intermediate saved for operation 'synthesized_resolve_0'
           Running Operation:
             Type: reduce
             Name: summarize_theme_evolution
⠼ Running step debate_analysis...
[09:29:54] Intermediate saved for operation 'summarize_theme_evolution'
           Flushing cache to disk...
           Cache flushed to disk.
  Step debate_analysis completed. Cost: $0.29
  Operation extract_themes_and_viewpoints completed. Cost: $0.16
  Operation unnest_themes completed. Cost: $0.00
  Operation synthesized_resolve_0 completed. Cost: $0.04
  Operation summarize_theme_evolution completed. Cost: $0.09
            Output saved to theme_evolution_analysis_baseline.json
           Total cost: $0.29
           Total time: 97.25 seconds

```

--------------------------------

### PDF Parsing Output Example

Source: https://ucbepic.github.io/docetl/examples/custom-parsing

Example output from the paddleocr_pdf_to_string function for a PDF file, showing extracted text from a receipt.

```text
RECEIPT
Store: Example Store
Date: 2023-05-15
Items:

1. Product A - $10.99
2. Product B - $15.50
3. Product C - $7.25
4. Product D - $22.00
   Subtotal: $55.74
   Tax (8%): $4.46
   Total: $60.20

Payment Method: Credit Card
Card Number: \***\* \**\* \***\* 1234

Thank you for your purchase!

```

--------------------------------

### Quick Start Pipeline

Source: https://ucbepic.github.io/docetl/api-reference/python

Shows a complete DocETL pipeline: reads JSON, maps to classify, filters by topic, reduces by category, and collects results. Sets the default model to gpt-4o-mini.

```python
import docetl

docetl.default_model = "gpt-4o-mini"

results = (
    docetl.read_json("input.json")
    .map(
        prompt="Classify this document: {{ input.text }}",
        output={"schema": {"category": "string"}},
    )
    .filter(prompt="Is this document about technology? {{ input.text }}")
    .reduce(
        reduce_key="category",
        prompt="Summarize these documents: {% for item in inputs %}{{ item.text }}{% endfor %}",
        output={"schema": {"summary": "string"}},
    )
    .collect()
)

```

--------------------------------

### Using Function Arguments with Parsing Tools

Source: https://ucbepic.github.io/docetl/examples/custom-parsing

Example of how to pass additional arguments to parsing tools in DocETL configuration, using xlsx_to_string as an example.

```APIDOC
## Using Function Arguments with Parsing Tools

### Description
When using parsing tools in your DocETL configuration, you can pass additional arguments to the parsing functions. For example, when using the xlsx_to_string parsing tool, you can specify options like the orientation of the data, the order of columns, or whether to process each sheet separately.

### Configuration Example (YAML)
```yaml
datasets:
  my_sales:
    type: file
    source: local
    path: "sales_data/sales_paths.json"
    parsing_tools:
      - name: excel_parser
        function: xlsx_to_string
        orientation: row
        col_order: ["Date", "Product", "Quantity", "Price"]
        doc_per_sheet: true
```

### Configuration Example (Python)
```python
my_sales = docetl.read_json(
    "sales_data/sales_paths.json",
    parsing=[
        {
            "function": "xlsx_to_string",
            "orientation": "row",
            "col_order": ["Date", "Product", "Quantity", "Price"],
            "doc_per_sheet": True,
        }
    ],
)
```
```

--------------------------------

### docetl.tools.Sandbox.create

Source: https://ucbepic.github.io/docetl/api-reference/python

Documentation for the docetl.tools.Sandbox.create method, used to create a persistent hosted container on OpenAI.

```APIDOC
## [SDK] docetl.tools.Sandbox.create

### Description
Creates a persistent hosted container/sandbox filesystem on OpenAI. This is OpenAI-specific and calls the OpenAI Containers API.

### Method
`docetl.tools.Sandbox.create(name, network, memory_limit)`

### Parameters
- **name** (str) - Required - Name of the sandbox.
- **network** (str) - Optional - Network configuration (e.g., "disabled").
- **memory_limit** (str) - Optional - Memory limit (e.g., "1g").

### Usage Example
```python
sandbox = docetl.tools.Sandbox.create(
    name="docetl-research",
    network="disabled",
    memory_limit="1g",
)
bash = sandbox.bash()

agent = docetl.Agent(
    tools=[bash],
    max_turns=4,
    max_tool_calls=6,
)
```

### Notes
- Not portable to Claude, Together, or other LiteLLM providers.
- For provider-portable tooling, use `@docetl.tool` Python functions, MCP tools, or provider-native SDK tools.
```

--------------------------------

### Work with MOAR results: best, cheapest, frontier, and DataFrame

Source: https://ucbepic.github.io/docetl/optimization/moar/results

Shows how to work with MOAR results: select the best or cheapest pipeline, iterate over the frontier, and convert all explored configurations to a pandas DataFrame for analysis.

```python
# Choose based on your priorities
optimized = frame.optimize(eval_fn=evaluate, metric_key="score")
results = optimized.search_results

# Highest accuracy
best = results.best()
print(f"Best accuracy: {best.accuracy}, cost: ${best.cost:.4f}")
best.run()

# Lowest cost
cheap = results.cheapest()
print(f"Cheapest cost: ${cheap.cost:.4f}, accuracy: {cheap.accuracy}")

# Explore the full frontier
for plan in results.frontier:
    print(f"Cost: ${plan.cost:.4f}, Accuracy: {plan.accuracy}")

# Analyze all explored configurations as a DataFrame
df = results.to_df()
print(df[["cost", "accuracy", "on_frontier"]].sort_values("accuracy", ascending=False))

```

--------------------------------

### Example directory structure (contracts/)

Source: https://ucbepic.github.io/docetl/concepts/datasets

Every non-hidden file under the directory (recursively) becomes one row. PDF, Word, PowerPoint, and Excel files are converted to text; other files are read as UTF-8; binary files with no extractor are skipped with a warning.

```text
contracts/
  acme_msa.pdf
  globex_nda.docx
  notes/renewal_2026.txt

```

--------------------------------

### Input data for PDF processing (papers/urls.json)

Source: https://ucbepic.github.io/docetl/operators/map

Example input data for the PDF processing example. The file `papers/urls.json` should contain documents with PDF URLs, each with a `url` field.

```json
[
  {"url": "https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card.pdf"},
  ...
]
```

--------------------------------

### Run Docker with make docker

Source: https://ucbepic.github.io/docetl/playground

Builds the DocETL image, creates a Docker volume for persistent data, and runs the container with the UI at http://localhost:3000 and API at http://localhost:8000.

```bash
make docker

```
Tokens:
Raw