### Run Basic DocETL Tests

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Executes the basic test suite using the 'make' command to verify that the DocETL installation and setup are functioning correctly.

```bash
make tests-basic
```

--------------------------------

### Install DocETL Dependencies via Poetry

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Installs the project's dependencies and DocETL itself within a virtual environment managed by Poetry.

```bash
poetry install
```

--------------------------------

### Verify DocETL Installation

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Runs the DocETL command-line interface to display the installed version, confirming that the installation was successful and the command is accessible.

```bash
docetl version
```

--------------------------------

### Install DocETL via pip

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Installs the base DocETL package and its core dependencies using Python's package installer, pip.

```bash
pip install docetl
```

--------------------------------

### Install DocETL with Parsing Extra via Poetry

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Installs DocETL and its dependencies, including the optional parsing tools, using Poetry by specifying the 'parsing' extra.

```bash
poetry install --extras "parsing"
```

--------------------------------

### Clone DocETL Repository from Source

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Clones the DocETL source code repository from GitHub and changes the current directory to the newly cloned project folder.

```bash
git clone https://github.com/ucbepic/docetl.git
cd docetl
```

--------------------------------

### Install DocETL with Parsing Extra via pip

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Installs DocETL along with the optional dependencies required for utilizing the parsing tools by specifying the 'parsing' extra.

```bash
pip install docetl[parsing]
```

--------------------------------

### Install Poetry Dependency Manager

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Installs Poetry, a dependency management and packaging tool for Python, which is required for installing DocETL from source.

```bash
pip install poetry
```

--------------------------------

### Running the Next.js Development Server (Bash)

Source: https://github.com/ucbepic/docetl/blob/main/website/README.md

Provides commands to start the Next.js development server using different popular Node.js package managers (npm, yarn, pnpm, bun). The server will typically run on `http://localhost:3000`.

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

```

--------------------------------

### Install DocETL Frontend Dependencies

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Run this make command in the root directory to install the necessary dependencies for the DocETL Next.js frontend.

```Bash
make install-ui
```

--------------------------------

### Start DocETL Local Development Server

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Execute this make command in the root directory to start the local development server for the DocETL Playground UI.

```Bash
make run-ui-dev
```

--------------------------------

### Full Pipeline Configuration Example - YAML

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/mining-product-reviews.md

This comprehensive example demonstrates a complete pipeline configuration, including default model, system prompt, dataset definition, multiple operation definitions (map, unnest, resolve, reduce), and the pipeline steps with input and output.

```yaml
default_model: gpt-4o-mini

system_prompt:
  dataset_description: a collection of reviews for video games
  persona: a marketing analyst analyzing player opinions and themes

datasets:
  steam_reviews:
    type: file
    path: "path/to/top_apps_steam_sample.json"

operations:
  - name: identify_polarizing_themes
    optimize: true
    type: map
    prompt: |
      Analyze the following concatenated reviews for a video game and identify polarizing themes that divide player opinions. A polarizing theme is one that some players love while others strongly dislike.

      Game: {{ input.app_name }}
      Reviews: {{ input.concatenated_reviews }}

      For each polarizing theme you identify:
      1. Provide a summary of the theme
      2. Explain why it's polarizing
      3. Include supporting quotes from both positive and negative perspectives

      Aim to identify ~10 polarizing themes, if present.

    output:
      schema:
        polarizing_themes: "list[{theme: str, summary: str, polarization_reason: str, positive_quotes: str, negative_quotes: str}]"

  - name: unnest_polarizing_themes
    type: unnest
    unnest_key: polarizing_themes
    recursive: true
    depth: 2

  - name: resolve_themes
    type: resolve
    optimize: true
    comparison_prompt: |
      Are the themes "{{ input1.theme }}" and "{{ input2.theme }}" the same? Here is some context to help you decide:

      Theme 1: {{ input1.theme }}
      Summary 1: {{ input1.summary }}

      Theme 2: {{ input2.theme }}
      Summary 2: {{ input2.summary }}
    resolution_prompt: |
      Given the following themes, please come up with a theme that best captures the essence of all the themes:

      {% for input in inputs %}
      Theme {{ loop.index }}: {{ input.theme }}
      {% if not loop.last %}
      ---
      {% endif %}
      {% endfor %}

      Based on these themes, provide a consolidated theme that captures the essence of all the above themes. Ensure that the consolidated theme is concise yet comprehensive.
    output:
      schema:
        theme: str

  - name: aggregate_common_themes
    type: reduce
    optimize: true
    reduce_key: theme
    prompt: |
      You are given a theme and summary that appears across multiple video games, along with various apps and review quotes related to this theme. Your task is to consolidate this information into a comprehensive report.

      For each input, you will receive:
      - theme: A specific polarizing theme
      - summary: A brief summary of the theme
      - app_name: The name of the game
      - positive_quotes: List of supporting quotes from positive perspectives
      - negative_quotes: List of supporting quotes from negative perspectives

      Create a report that includes:
      1. The name of the common theme
      2. A summary of the theme and why it's common across games
      3. Representative quotes from different games, both positive and negative

      Here's the information for the theme:
      Theme: {{ inputs[0].theme }}
      Summary: {{ inputs[0].summary }}

      {% for app in inputs %}
      Game: {{ app.app_name }}
      Positive Quotes: {{ app.positive_quotes }}
      Negative Quotes: {{ app.negative_quotes }}
      {% if not loop.last %}
      ----------------------------------------
      {% endif %}
      {% endfor %}

    output:
      schema:
        theme_summary: str
        representative_quotes: "list[{game: str, quote: str, sentiment: str}]"

pipeline:
  steps:
    - name: game_analysis
      input: steam_reviews
      operations:
        - identify_polarizing_themes
        - unnest_polarizing_themes
        - resolve_themes
        - aggregate_common_themes

  output:
    type: file
    path: "path/to/output_polarizing_themes.json"
    intermediate_dir: "path/to/intermediates"
```

--------------------------------

### Install DocETL Python Dependencies

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Run this make command in the root directory to install the necessary Python dependencies for the DocETL backend.

```Bash
make install
```

--------------------------------

### Set OpenAI API Key Environment Variable

Source: https://github.com/ucbepic/docetl/blob/main/docs/installation.md

Sets the OPENAI_API_KEY environment variable, typically in a .env file or shell, which is required for DocETL to interact with the OpenAI API.

```bash
OPENAI_API_KEY=your_api_key_here
```

--------------------------------

### Sample Dataset Structure (JSON)

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/split-gather.md

Provides an example structure for the input dataset, showing a list containing a single document object with a `pdf_url` field. This dataset serves as the input for the ETL pipeline.

```JSON
[
  {
    "pdf_url": "https://storage.courtlistener.com/recap/gov.uscourts.dcd.258148/gov.uscourts.dcd.258148.252.0.pdf"
  }
]
```

--------------------------------

### Example Rendered Reduce Prompt (Text)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/reduce.md

Illustrates how a reduce operation prompt template is rendered with actual data before being processed by a language model. This example shows a prompt for summarizing product reviews, populated with review text for a specific product.

```text
Summarize the reviews for product PROD123:

Review 1: This laptop is amazing! The battery life is incredible, lasting me a full day of work without needing to charge. The display is crisp and vibrant, perfect for both work and entertainment. The only minor drawback is that it can get a bit warm during intensive tasks.

Review 2: I'm disappointed with this purchase. While the laptop looks sleek, its performance is subpar. It lags when running multiple applications, and the fan noise is quite noticeable. On the positive side, the keyboard is comfortable to type on.

Review 3: Decent laptop for the price. It handles basic tasks well, but struggles with more demanding software. The build quality is solid, and I appreciate the variety of ports. Battery life is average, lasting about 6 hours with regular use.

Review 4: Absolutely love this laptop! It's lightweight yet powerful, perfect for my needs as a student. The touchpad is responsive, and the speakers produce surprisingly good sound. My only wish is that it had a slightly larger screen.

Review 5: Mixed feelings about this product. The speed and performance are great for everyday use and light gaming. However, the webcam quality is poor, which is a letdown for video calls. The design is sleek, but the glossy finish attracts fingerprints easily.
```

--------------------------------

### Clone DocETL Repository

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Use these git commands to clone the DocETL repository from GitHub and navigate into the project directory for local setup.

```Bash
git clone https://github.com/ucbepic/docetl.git
cd docetl
```

--------------------------------

### Install DocETL Project Dependencies (Bash)

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Executes `make` commands to install project dependencies. `make install` installs the main Python package dependencies, and `make install-ui` installs the dependencies required for the user interface.

```bash
make install      # Install Python package
make install-ui   # Install UI dependencies
```

--------------------------------

### Multi-step Analysis Pipeline with DocETL Pandas

Source: https://github.com/ucbepic/docetl/blob/main/docs/pandas/examples.md

This snippet illustrates building a multi-step analysis pipeline using chained DocETL operations on a pandas DataFrame. It starts by using `map` to extract structured information from social media posts, then uses `filter` to select relevant posts (e.g., about Apple products), and finally uses `agg` with fuzzy matching to group the filtered posts by product and summarize the feedback.

```python
# Social media posts
posts = pd.DataFrame({
    "text": ["Just tried the new iPhone 15!", "Having issues with iOS 17", "Android is better"],
    "timestamp": ["2024-01-01", "2024-01-02", "2024-01-03"]
})

# 1. Extract structured information
analyzed = posts.semantic.map(
    prompt="""Analyze this social media post and extract:
    1. Product mentioned
    2. Sentiment
    3. Issues/Praise points
    
    Post: {{input.text}}""",
    output_schema={
        "product": "str",
        "sentiment": "str",
        "points": "list[str]"
    }
)

# 2. Filter relevant posts
relevant = analyzed.semantic.filter(
    prompt="Is this post about Apple products? {{input}}"
)


# 3. Group by issue and summarize
summaries = relevant.semantic.agg(
    fuzzy=True,
    reduce_keys=["product"],
    comparison_prompt="Do these posts discuss the same product?",
    reduce_prompt="Summarize the feedback about this product",
    output_schema={
        "summary": "str",
        "frequency": "int",
        "severity": "str"
    }
)

# Summaries will be a df with the following columns:
# - product: str (because this was the reduce_keys)
# - summary: str
# - frequency: int
# - severity: str
```

--------------------------------

### Example Prompt for Summarizing Prescriptions (Jinja/YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/best-practices.md

A detailed prompt template using Jinja to summarize information about a specific medication across multiple patient transcripts, providing clear instructions and formatting requirements.

```yaml
prompt: |
  Here are some transcripts of conversations between a doctor and a patient:

  {% for value in inputs %}
  Transcript {{ loop.index }}:
  {{ value.src }}
  {% endfor %}

  For the medication {{ reduce_key }}, please provide the following information based on all the transcripts above:

  1. Side Effects: Summarize all mentioned side effects of {{ reduce_key }}. List 2-3 main side effects.
  2. Therapeutic Uses: Explain the medical conditions or symptoms for which {{ reduce_key }} was prescribed or recommended. Provide 1-2 primary uses.

  Ensure your summary:
  - Is based solely on information from the provided transcripts
  - Focuses only on {{ reduce_key }}, not other medications
  - Includes relevant details from all transcripts
  - Is clear and concise
  - Includes quotes from the transcripts
```

--------------------------------

### Start DocETL UI Development Server (Bash)

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Runs the `make run-ui-dev` command to start the DocETL user interface in development mode. This command typically launches a local server for accessing the UI.

```bash
make run-ui-dev
```

--------------------------------

### Configure Backend Environment Variables for Docker

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Create the '.env' file in the root directory to configure backend settings for the Docker setup, including the LLM API key, allowed origins, host, and port.

```Environment Variables
# Required: API key for your preferred LLM provider (OpenAI, Anthropic, etc)
# The key format will depend on your chosen provider (sk-..., anthro-...)
OPENAI_API_KEY=your_api_key_here
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True
FRONTEND_HOST=localhost
FRONTEND_PORT=3000
```

--------------------------------

### Setting Environment Variables for DocWrangler (Bash)

Source: https://github.com/ucbepic/docetl/blob/main/website/README.md

Defines the required environment variables for the DocWrangler frontend application. This includes API keys for OpenAI and the backend host/port. These variables should be copied to a `.env.local` file.

```bash
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8008

```

--------------------------------

### Configure .env for DocWrangler Docker Setup

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Creates the main .env file in the root directory for configuring the DocWrangler Docker setup. It defines backend and frontend hosts, ports, reload settings, Docker compose port mappings, and supported text file encodings.

```Bash
OPENAI_API_KEY=your_api_key_here
# BACKEND configuration
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True

# FRONTEND configuration
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000

# Host port mapping for docker-compose (if not set, defaults are used in docker-compose.yml)
FRONTEND_DOCKER_COMPOSE_PORT=3031
BACKEND_DOCKER_COMPOSE_PORT=8081

# Supported text file encodings
TEXT_FILE_ENCODINGS=utf-8,latin1,cp1252,iso-8859-1
```

--------------------------------

### Example Infrastructure Development Report in Markdown

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/presidential-debate-themes.md

This Markdown snippet provides an example of a generated report analyzing the evolution of Democratic and Republican viewpoints on infrastructure development from 1992 to 2023. It details historical perspectives, recent stances, areas of agreement and disagreement, and influencing factors. The example is presented to illustrate the format and content of the pipeline's output, which was found to be lacking in recent quotes.

```markdown
# Infrastructure Development: A Comparative Analysis of Democratic and Republican Viewpoints from 1992 to 2023\n\n## Introduction\nInfrastructure development has long been a pivotal theme in American political discourse, with varying perspectives presented by major party candidates. This report analyzes shifts and trends in Democratic and Republican viewpoints from 1992, during the second presidential debate between George Bush, Bill Clinton, and Ross Perot, to 2023.\n\n## Republican Viewpoints\n### Early 1990s\nIn 1992, President George Bush emphasized a forward-looking approach to infrastructure, stating, "We passed this year the most furthest looking transportation bill in the history of this country...$150 billion for improving the infrastructure." This statement indicated a commitment to substantial federal investment in infrastructure aimed at enhancing transportation networks.\n\n### 2000s\nMoving into the early 2000s, the Republican party maintained a focus on infrastructure but began to frame it within the context of economic growth and public-private partnerships. However, after the 2008 financial crisis, there was a noticeable shift. The party emphasized tax cuts and reducing regulation over large public investments in infrastructure.\n\n### Recent Years\nBy 2020 and 2021, under the Trump administration, the emphasis returned to infrastructure. However, the tone shifted towards emphasizing private sector involvement and deregulation rather than large public spending. The Republican approach became more fragmented, with some factions calling for aggressive infrastructure investment, while others remained cautious about expenditures.\n\n## Democratic Viewpoints\n### Early 1990s\nIn contrast, Governor Bill Clinton in 1992 proposed a more systematic investment strategy, noting, "My plan would dedicate $20 billion a year in each of the next 4 years for investments in new transportation." This highlighted a stronger emphasis on direct federal involvement in infrastructure as a means of fostering economic opportunity and job creation.\n\n### 2000s\nThrough the late 1990s and early 2000s, the Democratic party continued to push for comprehensive federal infrastructure plans, often attached to broader economic initiatives aimed at reducing inequality and spurring job growth. The party emphasized sustainable infrastructure and investments that address climate change.\n\n### Recent Years\nBy 2020, under the Biden administration, the Democrat viewpoint strongly advocated for significant infrastructure investments, combining traditional infrastructure with climate resilience. The American Jobs Plan symbolized this shift, proposing vast funds for transit systems, renewable energy projects, and rural broadband internet. The framing increasingly included social equity as a core component of infrastructure, influenced by movements advocating for racial and economic justice.\n\n## Agreements and Disagreements\n### Agreements\nDespite inherent differences, both parties have historically acknowledged the necessity of infrastructure investments for economic growth. Both Bush and Clinton in 1992 recognized infrastructure as vital for job creation, but diverged on the scope and funding mechanisms.\n\n### Disagreements\nOver the years, major disagreements have surfaced, particularly in funding approaches. The Republican party has increasingly favored private sector involvement and tax incentives, while Democrats have consistently pushed for robust federal spending and the incorporation of progressive values into infrastructure projects.\n\n## Influencing Factors\nThe evolution of viewpoints has often mirrored external events such as economic recessions, technological advancement, and social movements. The post-9/11 era and the 2008 financial crisis notably shifted priorities, with bipartisan discussions centered around recovery through infrastructure spending. Additionally, increasing awareness of climate change and social justice has over the years significantly influenced Democratic priorities, leading to a more inclusive and sustainable approach to infrastructure development.\n\n## Conclusion\n
```

--------------------------------

### Docetl Simple Document Analysis Pipeline YAML

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/split-gather.md

This YAML configuration defines a basic Docetl pipeline. It specifies a default model, a dataset source with parsing instructions, a map operation to extract people and involvements using a prompt, and the pipeline steps including input and output file paths. This simple pipeline is used as an example to demonstrate compilation into a split-gather pipeline.

```yaml
default_model: gpt-4o-mini

datasets:
  legal_doc: # (1)!
    path: /path/to/dataset.json
    type: file
    parsing: # (2)!
      - input_key: pdf_url
        function: azure_di_read
        output_key: extracted_text
        function_kwargs:
          use_url: true
          include_line_numbers: true

operations:
  - name: find_people_and_involvements
    type: map
    optimize: true
    prompt: |
      Given this document, extract all the people and their involvements in the case described by the document.

      {{ input.extracted_text }}

      Return a list of people and their involvements in the case.
    output:
      schema:
        people_and_involvements: list[str]

pipeline:
  steps:
    - name: analyze_document
      input: legal_doc
      operations:
        - find_people_and_involvements

  output:
    type: file
    path: "/path/to/output/people_and_involvements.json"
```

--------------------------------

### Configure Frontend Environment Variables for Docker

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Create the '.env.local' file specifically in the 'website' directory to configure frontend settings for the Docker setup. This includes optional OpenAI keys for AI assistants and backend connection details.

```Environment Variables
# Optional: These are only needed if you want to use the AI assistant chatbot
# and prompt engineering tools. Must be OpenAI API keys specifically.
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
```

--------------------------------

### Sample Input Data for link_resolve

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/link-resolve.md

An example of the input data structure for the `link_resolve` operation, consisting of a list of items, each with a title (used as ID), a list of related links, and a description.

```json
[
  {
    "title": "Sailing vessel",
    "related_to": ["Main sail", "Main mast", "Rudder"],
    "description": "A boat or ship propelled by sails. Typically with a very hydrodynamical hull."
  },
  {
    "title": "Catamaran",
    "related_to": ["Sailing boat", "Hull"],
    "description": "A type of vessel with two parallel hulls"
  },
  {
    "title": "Sail (main)",
    "related_to": ["Sheet"],
    "description": "A cloth set up to catch the force of the wind"
  },
  {
    "title": "Sheet (sailing)",
    "related_to": [],
    "description": "Ropes used to trim the angle of the sails, deciding the (angle of) wind force applied to the masts"
  },
  {
    "title": "Rudder angle",
    "related_to": [],
    "description": "The rudder angle together with the wind force on the mast(s) decides the rate of turn of the vessel"
  }
]
```

--------------------------------

### Gather Operation Peripheral Chunks Configuration Example (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/gather.md

Provides an example configuration for the 'peripheral_chunks' setting within the 'gather' operation. It demonstrates how to specify the number and content key ('full_content' or 'summary_content') for including preceding and succeeding chunks (head, middle, tail) as context.

```YAML
peripheral_chunks:
  previous:
    head:
      count: 1
      content_key: full_content
    middle:
      content_key: summary_content
    tail:
      count: 2
      content_key: full_content
  next:
    head:
      count: 1
      content_key: full_content
```

--------------------------------

### Run DocWrangler Locally with Docker

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Executes the make command to build and run the DocWrangler Docker container. This provides the easiest way to get the interactive UI playground running locally.

```Bash
make docker
```

--------------------------------

### Example Pipeline Configuration with Optimizer Settings - YAML

Source: https://github.com/ucbepic/docetl/blob/main/docs/optimization/configuration.md

This YAML snippet demonstrates how to configure the `optimizer_config` within a pipeline definition. It shows how to set global parameters like retry count, agent models, and default sample sizes, as well as operation-specific settings for 'reduce' and 'map' operations.

```yaml
optimizer_config:
  rewrite_agent_model: gpt-4o-mini
  judge_agent_model: gpt-4o-mini
  litellm_kwargs:
    temperature: 0.5
  num_retries: 2
  sample_sizes:
    map: 10
    reduce: 50
  reduce:
    synthesize_resolve: false
  map:
    plan_types: # Considers all these plan types
      - chunk
      - proj_synthesis
      - glean

operations:
  - name: extract_medications
    type: map
    optimize: true
    recursively_optimize: true # Recursively optimize the map operation (i.e., optimize any new operations that are synthesized)
    # ... other configuration ...

  - name: summarize_prescriptions
    type: reduce
    optimize: true
    # ... other configuration ...
# ... rest of the pipeline configuration ...
```

--------------------------------

### Full ETL Pipeline Configuration (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/split-gather.md

Presents the complete YAML configuration for the ETL pipeline. It defines the dataset source, default model, system prompt, and a sequence of operations including metadata extraction, splitting, and header extraction. This configuration orchestrates the document processing workflow.

```YAML
datasets:
  legal_doc:
    type: file
    path: /path/to/your/dataset.json
    parsing: # (1)!
      - function: azure_di_read
        input_key: pdf_url
        output_key: extracted_text
        function_kwargs:
          use_url: true
          include_line_numbers: true

default_model: gpt-4o-mini

system_prompt:
  dataset_description: the Trump vs. United States case
  persona: a legal analyst

operations:
  - name: extract_metadata_find_people_and_involvements
    type: map
    model: gpt-4o-mini
    prompt: |-
      Given the document excerpt: {{ input.extracted_text }}
      Extract all the people mentioned and summarize their involvements in the case described.
    output:
      schema:
        metadata: str

  - name: split_find_people_and_involvements
    type: split
    method: token_count
    method_kwargs:
      num_tokens: 3993
    split_key: extracted_text

  - name: header_extraction_extracted_text_find_people_and_involvements
    type: map
    model: gpt-4o-mini
    output:
      schema:
        headers: "list[{header: string, level: integer}]"
    prompt: |-
      Analyze the following chunk of a document and extract any headers you see.

      { input.extracted_text_chunk }

      Examples of headers and their levels based on the document structure:
      - "GOVERNMENT'S MOTION FOR IMMUNITY DETERMINATIONS" (level 1)
      - "Legal Framework" (level 1)
      - "Section I" (level 2)
      - "Section II" (level 2)
      - "Section III" (level 2)
```

--------------------------------

### Analyzing PDFs from URLs with DocETL Map (Python)

Source: https://github.com/ucbepic/docetl/blob/main/docs/pandas/examples.md

Illustrates how to use the `pdf_url_key` parameter in a DocETL semantic map operation to process PDF content directly from URLs specified in a dataframe column.

```Python
df = pd.DataFrame({
    "PdfPath": ["https://docetl.blob.core.windows.net/ntsb-reports/Report_N617GC.pdf", "https://docetl.blob.core.windows.net/ntsb-reports/Report_CEN25LA075.pdf"]
})

result_df = df.semantic.map(
    prompt="Summarize the air crash report and determine any contributing factors",
    output_schema={"summary": "str", "contributing_factors": "list[str]"},
    pdf_url_key="PdfPath", # This is the column with the PDF paths
)

print(result_df.head()) # The result will have the same number of rows as the input dataframe, with the summary and contributing factors added
```

--------------------------------

### Configuring Uniform Sampling Example in DocETL (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/sample.md

This YAML snippet shows how to configure the DocETL Sample operation for uniform sampling. It uses the 'uniform' method and specifies to sample a fixed count of 100 items.

```yaml
- name: uniform_sample
  type: sample
  method: uniform
  samples: 100
```

--------------------------------

### Reduce Prompt for Compiling People and Involvements (Jinja2)

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/split-gather.md

A Jinja2 template used as the prompt for the 'subreduce_find_people_and_involvements' operation. It takes a list of extracted involvements from multiple input chunks and compiles them into a single, comprehensive list, ensuring all people and their involvements are included and grouped together if a person has multiple involvements.

```Jinja2
Given the following extracted information about individuals involved in the case, compile a comprehensive list of people and their specific involvements in the case:

{% for chunk in inputs %}
{% for involvement in chunk.people_and_involvements %}
- {{ involvement }}
{% endfor %}
{% endfor %}

Make sure to include all the people and their involvements. If a person has multiple involvements, group them together.
```

--------------------------------

### Map Prompt for Extracting People and Involvements (Jinja2)

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/split-gather.md

A Jinja2 template used as the prompt for the 'submap_find_people_and_involvements' operation. It instructs the language model to extract all mentioned people and summarize their involvements from a given document excerpt, specifically processing only the main chunk.

```Jinja2
Given the document excerpt: {{ input.extracted_text_chunk_rendered }}
Extract all the people mentioned and summarize their involvements in the case described. Only process the main chunk.
```

--------------------------------

### Two-Step Ranking Example Configuration

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/rank.md

YAML configuration demonstrating a two-step ranking process. It defines a 'map' operation to extract hostility details from debate transcripts and a 'rank' operation to order transcripts based on the extracted data. The pipeline then applies these operations sequentially.

```yaml
operations:
  - name: extract_hostile_exchanges
    type: map
    output:
      schema:
        meanness_summary: "str"
        hostility_level: "int"
        key_examples: "list[str]"
    prompt: |
      Analyze the following debate transcript for {{ input.title }} on {{ input.date }}:

      {{ input.content }}

      Extract and summarize exchanges where candidates are mean or hostile to each other.
      [... prompt details ...]

  - name: rank_by_meanness
    type: rank
    prompt: |
      Order these debate transcripts based on how mean or hostile the candidates are to each other.
      Focus on the meanness summaries and examples that have been extracted.
      
      Consider:
      - The overall hostility level rating
      - Severity of personal attacks in the key examples
      [... prompt details ...]
    input_keys: ["meanness_summary", "hostility_level", "key_examples", "title", "date"]
    direction: desc
    rerank_call_budget: 10

pipeline:
  steps:
    - name: meanness_analysis
      input: debates
      operations:
        - extract_hostile_exchanges
        - rank_by_meanness
```

--------------------------------

### Configuring Stratified Sampling Example in DocETL (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/sample.md

This YAML snippet demonstrates configuring the DocETL Sample operation using the 'stratify' method. It samples 10% of the data, stratifies based on the 'category' key, and uses a fixed random state (42) for reproducibility.

```yaml
- name: cluster_concepts
  type: sample
  method: stratify
  samples: 0.1
  method_kwargs:
    stratify_key: category
  random_state: 42
```

--------------------------------

### Custom Comparison Prompt for Entity Resolution (Jinja/YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/best-practices.md

Provides a specific comparison prompt template using Jinja for a `resolve` operation, guiding the LLM to determine if two medication entries are the same or closely related based on criteria like brand names, drug class, or common use.

```yaml
- name: resolve_medications
  type: resolve
  blocking_keys:
    - medication
  blocking_threshold: 0.6162
  comparison_prompt: |
    Compare the following two medication entries:
    Entry 1: {{ input1.medication }}
    Entry 2: {{ input2.medication }}

    Are these medications the same or closely related? Consider the following:
    1. Are they different brand names for the same active ingredient?
    2. Are they in the same drug class with similar effects?
    3. Are they commonly used as alternatives for the same condition?

    Respond with YES if they are the same or closely related, and NO if they are distinct medications.
```

--------------------------------

### Summarizing Themes with Reduce Operation (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/concepts/optimization.md

Defines a pipeline step using the 'reduce' type in the original pipeline example. It groups inputs by the 'theme' key and uses a prompt to generate a summary for each theme based on the associated responses. The output schema expects a 'summary' string.

```yaml
- name: summarize_themes
  type: reduce
  reduce_key: theme
  prompt: |
    Summarize the responses for the theme: {{ reduce_key }}

    Responses:
    {% for item in inputs %}
    - {{ item.response }}
    {% endfor %}

    Provide a summary of the main points expressed about this theme.
  output:
    schema:
      summary: string
```

--------------------------------

### Configure .env.local for DocWrangler Docker Frontend

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Creates the .env.local file in the 'website' directory for configuring frontend-specific settings in the DocWrangler Docker setup. It includes API key, base URL, model name, and backend host/port for the frontend.

```Bash
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
NEXT_PUBLIC_HOSTED_DOCWRANGLER=false
```

--------------------------------

### Configuring xlsx_to_string Parsing Tool with Arguments (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/examples/custom-parsing.md

Example YAML configuration snippet showing how to define a dataset and configure the 'xlsx_to_string' parsing tool, including passing specific arguments like 'orientation', 'col_order', and 'doc_per_sheet' to customize the parsing behavior.

```yaml
datasets:
  my_sales:
    type: file
    source: local
    path: "sales_data/sales_paths.json"
    parsing_tools:
      - name: excel_parser
        function: xlsx_to_string
        orientation: row
        col_order: ["Date", "Product", "Quantity", "Price"]
        doc_per_sheet: true
```

--------------------------------

### Defining and Running DocETL Pipeline for Review Analysis

Source: https://github.com/ucbepic/docetl/blob/main/docs/python/examples.md

This snippet defines a DocETL pipeline to process product reviews. It includes operations to extract themes and quotes using a MapOp and then summarize themes across all documents using a ReduceOp with the special _all key. It sets up the dataset, operations, steps, output, and runs the pipeline.

```Python
from docetl.api import Pipeline, Dataset, MapOp, ReduceOp, PipelineStep, PipelineOutput

# Define dataset - A CSV of product reviews
dataset = Dataset(
    type="file",
    path="product_reviews.csv"  # Contains columns: review_id, product_id, rating, review_text
)

# Define operations
operations = [
    # Extract themes and quotes from each review
    MapOp(
        name="extract_themes",
        type="map",
        prompt="""
        Analyze this product review and extract the key themes and representative quotes:
        
        Review: {{ input.review_text }}
        Rating: {{ input.rating }}
        
        Identify 2-3 major themes (e.g., usability, quality, value) and extract direct quotes that best represent each theme.
        """,
        output={
            "schema": {
                "themes": "list[string]",
                "quotes": "list[string]",
                "sentiment": "string"
            }
        }
    ),
    # Summarize all themes across reviews using _all key
    ReduceOp(
        name="summarize_themes",
        type="reduce",
        reduce_key="_all",  # Special key to reduce across all items
        prompt="""
        Analyze and synthesize the themes and quotes from these product reviews:
        
        {% for item in inputs %}
        Review ID: {{ item.review_id }}
        Product: {{ item.product_id }}
        Rating: {{ item.rating }}
        Themes: {{ item.themes | join(", ") }}
        Quotes: 
        {% for quote in item.quotes %}
        - "{{ quote }}"
        {% endfor %}
        Sentiment: {{ item.sentiment }}
        
        {% endfor %}
        
        Summarize the most frequent themes, the most representative quotes for each theme, and the overall sentiment.
        """,
        output={
            "schema": {
                "summary": "string"
            }
        }
    )
]

# Define pipeline step (can consist of multiple operations)
step = PipelineStep(
    name="review_analysis",
    input="product_reviews",
    operations=["extract_themes", "summarize_themes"]
)

# Define output
output = PipelineOutput(type="file", path="review_analysis_summary.json")

# Create and run pipeline
pipeline = Pipeline(
    name="review_analysis_pipeline",
    datasets={"product_reviews": dataset},
    operations=operations,
    steps=[step],
    output=output,
    default_model="gpt-4o-mini"
)

# Run the pipeline
cost = pipeline.run()
print(f"Pipeline execution completed. Total cost: ${cost:.2f}")
```

--------------------------------

### Configure Extract Operation for Findings (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/extract.md

This YAML snippet configures an 'extract' operation named 'findings'. It uses a prompt to guide an LLM ('gpt-4.1-mini') to identify and extract sections discussing key findings, results, or conclusions from the 'report_text' field. The extracted content will be added to a new field suffixed '_extracted_findings'.

```yaml
- name: findings
  type: extract
  prompt: |
    Extract all sections that discuss key findings, results, or conclusions from this research report.
    Focus on paragraphs that:
    - Summarize experimental outcomes
    - Present statistical results
    - Describe discovered insights
    - State conclusions drawn from the research
    
    Only extract the most important and substantive findings.
  document_keys: ["report_text"]
  model: "gpt-4.1-mini"
```

--------------------------------

### Define and Run Product Review Analysis Pipeline - Python

Source: https://github.com/ucbepic/docetl/blob/main/docs/python/examples.md

This snippet defines a data processing pipeline using custom operations (`MapOp`, `UnnestOp`, `ResolveOp`, `ReduceOp`) to analyze product reviews. It extracts theme-quote pairs, unnests them, resolves similar themes, aggregates results by theme, and then runs the defined pipeline, printing the execution cost.

```python
# Define operations
operations = [
    # Extract theme-quote pairs from each review
    MapOp(
        name="extract_theme_quotes",
        type="map",
        prompt="""
        Extract theme and quote pairs from this product review:
        
        Review: {{ input.review_text }}
        Product: {{ input.product_name }}
        Rating: {{ input.rating }}
        
        For each distinct theme in the review, extract a direct quote that best represents that theme.
        Return each theme and its representative quote as a separate object in the "theme_quotes" array.
        """,
        output={
            "schema": {
                "theme_quotes": "array"  # Array of objects with theme and quote properties
            }
        }
    ),
    # Unnest to create separate items for each theme-quote pair
    UnnestOp(
        name="unnest_theme_quotes",
        type="unnest",
        array_path="theme_quotes"
    ),
    # Resolve similar themes using fuzzy matching
    ResolveOp(
        name="resolve_themes",
        type="resolve",
        comparison_prompt="""
        Determine if these two themes are the same or closely related:
        
        Theme 1: {{ input1.theme }}
        Theme 2: {{ input2.theme }}
        
        Consider semantic similarity, synonyms, and conceptual overlap.
        """,
        resolution_prompt="""
        Given the following list of similar themes, determine a canonical name that best represents all of them:
        
        {% for item in inputs %}
        Theme: {{ item.theme }}
        {% endfor %}
        
        Choose a clear, concise name that accurately captures the core concept shared across all these related themes.
        """
    ),
    # Reduce by theme to aggregate quotes and insights
    ReduceOp(
        name="aggregate_by_theme",
        type="reduce",
        reduce_key="theme",
        prompt="""
        Analyze all quotes related to the theme "{{ reduce_key }}":
        
        {% for item in inputs %}
        Product: {{ item.product_name }}
        Rating: {{ item.rating }}
        Quote: "{{ item.quote }}"
        
        {% endfor %}
        
        Summarize the key insights about this theme across all products and ratings.
        """,
        output={
            "schema": {
                "summary": "string"
            }
        }
    )
]

# Define pipeline with a single step
step = PipelineStep(
    name="theme_analysis",
    input="product_reviews",
    operations=["extract_theme_quotes", "unnest_theme_quotes", "resolve_themes", "aggregate_by_theme"]
)

# Define output
output = PipelineOutput(type="file", path="theme_analysis_results.json")

# Create the pipeline
pipeline = Pipeline(
    name="theme_analysis_pipeline",
    datasets={"product_reviews": dataset},
    operations=operations,
    steps=[step],
    output=output,
    default_model="gpt-4o"
)

# Optimize the pipeline before running
optimized_pipeline = pipeline.optimize()

# Run the optimized pipeline
cost = optimized_pipeline.run()
print(f"Pipeline execution completed. Total cost: ${cost:.2f}")

```

--------------------------------

### Configuring System Prompt in YAML

Source: https://github.com/ucbepic/docetl/blob/main/docs/best-practices.md

Sets up the system prompt for an LLM pipeline, defining the dataset description and the persona the LLM should adopt for operations.

```yaml
system_prompt:
  dataset_description: a collection of transcripts of doctor visits
  persona: a medical practitioner analyzing patient symptoms and reactions to medications
```

--------------------------------

### Install DocETL with Pandas Integration

Source: https://github.com/ucbepic/docetl/blob/main/docs/pandas/index.md

Provides the command to install the DocETL package, which includes the pandas integration feature.

```bash
pip install docetl
```

--------------------------------

### Example Input Data for DocETL PDF Processing (JSON)

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/map.md

Provides an example of the JSON structure required for the input dataset when processing PDFs by URL. It shows an array of objects, where each object contains a key (`url`) holding the URL of a PDF document.

```JSON
[
  {"url": "https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card.pdf"},
  ...
]
```

--------------------------------

### Example Output Structure for Map Operation - JSON

Source: https://github.com/ucbepic/docetl/blob/main/docs/operators/map.md

This JSON snippet shows an example of the structured output generated by a Map operation. It includes an array of identified entities with names and descriptions, overall sentiment, a list of identified biases, relevant categories, and a credibility score.

```JSON
[
  {
    "name": "Ursula von der Leyen",
    "description": "European Commission President"
  },
  {
    "name": "Poland",
    "description": "Eastern European country critical of the plan"
  },
  {
    "name": "Hungary",
    "description": "Eastern European country critical of the plan"
  },
  {
    "name": "Greenpeace",
    "description": "Environmental organization supporting the initiative"
  }
],
"sentiment": "positive",
"biases": [
  "Slight bias towards environmental concerns over economic impacts",
  "More emphasis on supportive voices than critical ones"
],
"categories": [
  "Environment",
  "Politics",
  "Economy"
],
"credibility_score": 8

```

--------------------------------

### Build and Run DocETL Docker Container

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Execute this make command in the root directory to build the DocETL Docker image, create a volume for persistent data, and run the container with the UI and API accessible locally.

```Bash
make docker
```

--------------------------------

### Install DocETL Python Package

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Installs the DocETL Python package using pip. This is required to use DocETL programmatically or run production pipelines from the command line.

```Bash
pip install docetl
```

--------------------------------

### Configure UI Environment Variables (.env.local)

Source: https://github.com/ucbepic/docetl/blob/main/README.md

Sets up environment variables specifically for the DocETL UI (website) in the `.env.local` file located in the `website` directory. Includes OpenAI API key, base URL, model name, and backend host/port for the UI to connect to.

```bash
OPENAI_API_KEY=sk-xxx
OPENAI_API_BASE=https://api.openai.com/v1
MODEL_NAME=gpt-4o-mini

NEXT_PUBLIC_BACKEND_HOST=localhost
NEXT_PUBLIC_BACKEND_PORT=8000
NEXT_PUBLIC_HOSTED_DOCWRANGLER=false
```

--------------------------------

### Example of Complex Nested Output Schema (YAML)

Source: https://github.com/ucbepic/docetl/blob/main/docs/concepts/operators.md

Shows a complex nested structure with a list containing objects that contain another list. This example is presented as a structure to generally avoid for better LLM consistency and parsing reliability.

```yaml
output:
  schema:
    insights: "list[{insight: string, supporting_actions: list[{action: string, priority: integer}]}]"
```

--------------------------------

### Configure Backend Environment Variables for Local Development

Source: https://github.com/ucbepic/docetl/blob/main/docs/playground/index.md

Create the '.env' file in the root directory to configure backend settings for local development, including the LLM API key, allowed origins, host, and port.

```Environment Variables
LLM_API_KEY=your_api_key_here
BACKEND_ALLOW_ORIGINS=http://localhost:3000,http://127.0.0.1:3000
BACKEND_HOST=localhost
BACKEND_PORT=8000
BACKEND_RELOAD=True
FRONTEND_HOST=0.0.0.0
FRONTEND_PORT=3000
```