### Example DocETL Pipeline Workflow

Source: https://www.docetl.org/llms-full.txt

A comprehensive example of a DocETL pipeline YAML file demonstrating dataset configuration, multiple operations (map, resolve, reduce) with prompts and schemas, and pipeline steps. It includes 'optimize: true' and 'sample: 10' for development purposes.

```yaml
default_model: gpt-5-nano

datasets:
  documents:
    path: data.json
    type: file

operations:
  - name: extract_entities
    type: map
    optimize: true
    sample: 10  # For development
    output:
      schema: 
        entities: "list[{name: string, type: string}]"
    prompt: |
      Extract named entities from: {{ input.text }}

  - name: resolve_entities
    type: resolve
    comparison_prompt: |
      Are these entities the same?
      Entity 1: {{ input1.name }} ({{ input1.type }})
      Entity 2: {{ input2.name }} ({{ input2.type }})
    resolution_prompt: |
      Create a canonical name for the following entities:
      {% for entity in inputs %}
      - {{ entity.name }} ({{ entity.type }})
      {% endfor %}
    output:
      schema:
        name: string

  - name: summarize_entities
    type: reduce
    reduce_key: name
    optimize: true
    prompt: |
      Summarize mentions of entity {{ reduce_key }}:
      {% for entity in inputs %}
      - {{ entity.text }}
      {% endfor %}

pipeline:
  steps:
    - name: entity_analysis
      input: documents
      operations:
        - extract_entities
        - resolve_entities
        - summarize_entities
  output:
    type: file
    path: entity_summary.json
    intermediate_dir: debug_outputs
```

--------------------------------

### Simple YAML Schema for Non-GPT Models

Source: https://www.docetl.org/llms-full.txt

Provides an example of a simple output schema suitable for LLM models other than GPT, Claude, or Gemini. It emphasizes single string outputs or simple key-value pairs.

```yaml
output:
  schema:
    category: string
    confidence: string  # Use string instead of number for better reliability
```

--------------------------------

### DocETL Pipeline Configuration - Optimization

Source: https://www.docetl.org/llms-full.txt

Example of configuring optimization within a DocETL pipeline YAML file. Setting 'optimize: true' for specific operations allows the 'docetl build' command to generate more efficient execution plans for those operations.

```yaml
# Set optimize: true for operations you want to optimize
operations:
  - name: extract_info
    type: map
    optimize: true  # This operation will be rewritten into smaller, better-scoped operations
    ...
```

--------------------------------

### Reduce Operation for Analyzing Product Feedback using YAML

Source: https://www.docetl.org/llms-full.txt

This YAML example showcases a 'reduce' operation for analyzing customer feedback. It groups feedback by 'product_id' and uses a prompt to identify quality issues, reliability concerns, and improvement suggestions, outputting a structured summary.

```yaml
- name: analyze_product_feedback
  type: reduce
  reduce_key: product_id
  prompt: |
    Analyze these customer reviews for product {{ reduce_key }}:

    {% for review in inputs %}
    Review {{ loop.index }}:
    Rating: {{ review.rating }}
    Text: {{ review.review_text }}
    {% endfor %}

    Identify:
    1. Common quality issues
    2. Reliability concerns
    3. Suggested improvements
  output:
    schema:
      quality_issues: "list[{issue: string, frequency: string, severity: string}]"
      reliability_concerns: "list[string]"
      improvement_suggestions: "list[string]"

```

--------------------------------

### Example JSON Data Format for DocETL

Source: https://context7_llms

This is an example of the JSON data format that DocETL can process. It represents a list of objects, where each object is a document with fields like 'text', 'date', and 'metadata'. Fields are accessible in operations using the `input.field_name` syntax.

```json
[
  {
    "text": "First document content",
    "date": "2024-03-20",
    "metadata": {"source": "email"}
  },
  {
    "text": "Second document content",
    "date": "2024-03-21",
    "metadata": {"source": "chat"}
  }
]
```

--------------------------------

### Filter Operation with Validation using YAML

Source: https://www.docetl.org/llms-full.txt

This example shows a 'filter' operation in YAML that analyzes news articles based on predefined criteria. It uses a prompt to determine if an article is 'high-impact' and outputs a boolean schema.

```yaml
- name: filter_high_impact_articles
  type: filter
  prompt: |
    Analyze the following news article:
    Title: "{{ input.title }}"
    Content: "{{ input.content }}"

    Determine if this article is high-impact based on the following criteria:
    1. Covers a significant global or national event
    2. Has potential long-term consequences
    3. Affects a large number of people
    4. Is from a reputable source

    Respond with 'true' if the article meets at least 3 of these criteria, otherwise respond with 'false'.

  output:
    schema:
      is_high_impact: boolean

```

--------------------------------

### Resolve Operation for Standardizing Patient Names using YAML

Source: https://www.docetl.org/llms-full.txt

This YAML example demonstrates a 'resolve' operation designed to standardize patient names. It includes a comparison prompt for matching entries and a resolution prompt for standardization, outputting a standardized patient name.

```yaml
- name: standardize_patient_names
  type: resolve
  optimize: true
  comparison_prompt: |
    Compare the following two patient name entries:

    Patient 1: {{ input1.patient_name }}
    Date of Birth 1: {{ input1.date_of_birth }}

    Patient 2: {{ input2.patient_name }}
    Date of Birth 2: {{ input2.date_of_birth }}

    Are these entries likely referring to the same patient? Consider name similarity and date of birth. 
    Respond with "True" if they are likely the same patient, or "False" if they are likely different patients.
  resolution_prompt: |
    Standardize the following patient name entries into a single, consistent format:

    {% for entry in inputs %}
    Patient Name {{ loop.index }}: {{ entry.patient_name }}
    {% endfor %}

    Provide a single, standardized patient name that represents all the matched entries. 
    Use the format "LastName, FirstName MiddleInitial" if available.
  output:
    schema:
      patient_name: string

```

--------------------------------

### Example CSV Data Format for DocETL

Source: https://context7_llms

This snippet illustrates the CSV data format supported by DocETL. The first row contains column headers, which become field names accessible via `input.column_name`. Each subsequent row represents a document to be processed.

```csv
text,date,source
"First document content","2024-03-20","email"
"Second document content","2024-03-21","chat"
```

--------------------------------

### Advanced Gleaning Validation in DocETL

Source: https://www.docetl.org/llms-full.txt

Illustrates advanced validation using gleaning in a DocETL map operation. Gleaning is an iterative LLM refinement process guided by a validation prompt to improve output quality. This can increase LLM calls, cost, and latency.

```yaml
- name: extract_insights
  type: map
  gleaning:
    num_rounds: 1
    validation_prompt: |
      Evaluate the extraction for completeness and relevance:
      1. Are all key user behaviors and pain points from the log addressed in the insights?
      2. Are the supporting actions practical and relevant to the insights?
      3. Is there any important information missing or any irrelevant information included?

```

--------------------------------

### Advanced Map Operation for Medical Information Extraction using YAML

Source: https://www.docetl.org/llms-full.txt

This YAML configuration illustrates an advanced 'map' operation to extract structured medical information from text. It defines schemas for medications, symptoms, and recommendations, with a detailed prompt guiding the extraction process.

```yaml
- name: extract_medical_info
  type: map
  optimize: true
  output:
    schema:
      medications: "list[{name: string, dosage: string, frequency: string}]"
      symptoms: "list[{description: string, severity: string, duration: string}]"
      recommendations: "list[string]"
  prompt: |
    Analyze the following medical record and extract key information:
    {{ input.text }}

    For each medication mentioned:
    1. Extract the name, dosage, and frequency
    2. Ensure dosage includes units (mg, ml, etc.)
    3. Standardize frequency to times per day/week

    For each symptom:
    1. Provide a clear description
    2. Rate severity (mild/moderate/severe)
    3. Note duration if mentioned

    Finally, list any doctor's recommendations.

```

--------------------------------

### DocETL Pipeline Configuration - Data Sampling

Source: https://www.docetl.org/llms-full.txt

Demonstrates how to use the 'sample' parameter within an operation in a DocETL pipeline YAML. This is useful during development to process only a subset of the data, speeding up testing and debugging.

```yaml
operations:
  - name: first_operation
    type: map
    sample: 10  # Only process 10 documents
    ...
```

--------------------------------

### Basic Pipeline Components: System Prompt Configuration (YAML)

Source: https://www.docetl.org/llms-full.txt

An optional configuration for setting a system prompt in DocETL pipelines, allowing definition of the dataset description and the LLM's persona.

```yaml
system_prompt:
  dataset_description: description of your data
  persona: role the LLM should assume
```

--------------------------------

### Run DocETL Pipeline

Source: https://www.docetl.org/llms-full.txt

Executes a DocETL pipeline. Use 'pipeline_opt.yaml' if optimization was applied, otherwise use the original 'pipeline.yaml'. This command processes the data according to the defined steps and operations.

```bash
# If you used the optimizer:
docetl run pipeline_opt.yaml

# If you didn't use the optimizer:
docetl run pipeline.yaml
```

--------------------------------

### Define Datasets in DocETL Pipeline (YAML)

Source: https://context7_llms

This snippet shows how to configure datasets within a DocETL pipeline using YAML. It specifies the dataset type as 'file' and provides the path to the data file, which can be in JSON or CSV format. Ensure JSON files contain a list of objects and CSV files have a header row.

```yaml
datasets:
  documents:
    type: file
    path: "data.json"  # or "data.csv"
```

--------------------------------

### Basic Pipeline Components: Datasets Configuration (YAML)

Source: https://www.docetl.org/llms-full.txt

Defines the basic structure for configuring datasets within a DocETL pipeline using YAML, specifying the path and type of the input data file.

```yaml
datasets:
  input_data:
    path: data.json
    type: file
```

--------------------------------

### Map-Only Pipeline Pattern

Source: https://www.docetl.org/llms-full.txt

Illustrates a simple DocETL pipeline pattern using only map operations for independent document transformations.

```yaml
operations:
  - extract_info  # map operation
```

--------------------------------

### Jinja2 Template Access for Resolve Operations

Source: https://www.docetl.org/llms-full.txt

Illustrates how to access two input documents for comparison prompts in DocETL's Resolve operations using `input1` and `input2` variables.

```jinja
Compare {{ input1.field_name }} and {{ input2.field_name }}
```

--------------------------------

### Map-Resolve-Reduce Pipeline Pattern

Source: https://www.docetl.org/llms-full.txt

Demonstrates a complex DocETL pipeline pattern involving map, resolve, and reduce operations for entity resolution and aggregation.

```yaml
operations:
  - extract_entities  # map operation
  - standardize_entities  # resolve operation
  - summarize_by_entity  # reduce operation
```

--------------------------------

### Basic Pipeline Components: Model Configuration (YAML)

Source: https://www.docetl.org/llms-full.txt

Specifies the default LLM model to be used in the DocETL pipeline. DocETL supports various providers through LiteLLM.

```yaml
default_model: gpt-5-nano
```

--------------------------------

### YAML Schema with List and Nested Objects for LLM Output

Source: https://www.docetl.org/llms-full.txt

Illustrates how to define schemas that include lists of strings and lists of objects with nested fields. This schema is more complex and should be used judiciously.

```yaml
output:
  schema:
    tags: "list[string]"
    users: "list[{name: string, age: integer}]"
```

--------------------------------

### Preprocess and Filter Customer Reviews using Python

Source: https://www.docetl.org/llms-full.txt

This snippet includes Python code for preprocessing customer reviews by cleaning text, tokenizing, and extracting relevant metadata. It then filters out reviews that are too short, ensuring only substantial feedback is processed further. The code relies on basic Python string manipulation and list operations.

```python
def transform(doc) -> dict:
    # Clean and tokenize text
    text = doc['text'].strip().lower()
    words = text.split()
    return {
        'text': text,
        'word_count': len(words),
        'rating': doc['rating'],
        'processed_date': doc['date'][:10]  # Extract date only
    }
```

```python
def transform(doc) -> bool:
    return doc['word_count'] >= 20  # Keep only substantial reviews
```

--------------------------------

### Build Optimized DocETL Pipeline

Source: https://www.docetl.org/llms-full.txt

Builds an optimized DocETL pipeline YAML file. This is required if your pipeline includes 'resolve' operations to generate efficient blocking rules. It takes the input pipeline YAML and outputs an optimized version, typically named 'pipeline_opt.yaml'.

```bash
docetl build pipeline.yaml
```

--------------------------------

### Dynamic Dataset Loading with Parsing Tools (YAML)

Source: https://www.docetl.org/llms-full.txt

Configures dynamic data loading for non-standard formats like audio, using parsing tools to convert files into structured data. It specifies the input file path and the function to use for parsing.

```yaml
datasets:
  audio_transcripts:
    type: file
    source: local
    path: "audio_files/audio_paths.json"  # JSON list of paths to audio files
    parsing_tools:
      - input_key: audio_path    # Field containing file path
        function: whisper_speech_to_text
        output_key: transcript   # Field where transcript will be stored
```

--------------------------------

### DocETL Pipeline Configuration - Intermediate Output Directory

Source: https://www.docetl.org/llms-full.txt

Shows how to configure an intermediate output directory in a DocETL pipeline YAML. This directory will store the output of each operation separately, aiding in debugging by allowing inspection of intermediate results.

```yaml
pipeline:
  output:
    type: file
    path: output.json
    intermediate_dir: intermediate_results  # Each operation's output will be saved here
```

--------------------------------

### Jinja2 Template Iteration for Reduce Operations

Source: https://www.docetl.org/llms-full.txt

Shows how to iterate over a list of documents using Jinja2 templates in DocETL's Reduce operations. The `inputs` variable holds the list of documents.

```jinja
{% for item in inputs %}
  {{ item.field_name }}
{% endfor %}
```

--------------------------------

### Analyze Customer Review Sentiment and Themes using LLM

Source: https://www.docetl.org/llms-full.txt

This LLM operation analyzes individual customer reviews to determine sentiment, extract key themes, and identify any mentioned products. It takes the review text and rating as input and outputs a structured JSON object containing these insights. The LLM is prompted to categorize sentiment and list themes/products.

```prompt
Analyze this customer review:
Rating: {{ input.rating }}
Review: {{ input.text }}

1. Identify the main sentiment (one of positive/negative/neutral)
2. Extract key themes or topics
3. Note any specific product mentions

```

--------------------------------

### Sample Data Subset

Source: https://www.docetl.org/llms-full.txt

The sample operation selects a uniform subset of data for iterative development. It uses `category` for stratification and `random_state` for reproducibility, recommending removal before production.

```yaml
- name: observe_subset
  type: sample
  method: uniform
  samples: 0.1
  stratify_key: category
  random_state: 42
```

--------------------------------

### Basic Validation in DocETL Pipeline

Source: https://www.docetl.org/llms-full.txt

Demonstrates basic validation using Python statements within a DocETL map operation. Validation rules are applied to the output schema, and the operation retries on failure. Input documents are not directly accessible in validation.

```yaml
- name: extract_info
  type: map
  output:
    schema: "{\"insights\": \"list[{insight: string, supporting_actions: list[string]}]\"}"
  validate:
    - "len(output[\"insights\"]) >= 2"
    - "all(len(insight[\"supporting_actions\"]) >= 1 for insight in output[\"insights\"])"
  num_retries_on_validate_failure: 3

```

--------------------------------

### Parallel Map for Concurrent LLM Prompts with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'parallel_map' operation allows multiple prompts to run concurrently on input data, enriching the document with results from each prompt. It's ideal for tasks like skill extraction, experience calculation, and cultural fit evaluation.

```yaml
- name: process_job_application
  type: parallel_map
  prompts:
    - name: extract_skills
      prompt: "List the top 5 relevant engineering skills from {{ input.resume fileresume }}."
      output_keys:
        - skills
      model: gpt-5-nano
      gleaning:
        num_rounds: 1
        validation_prompt: |
          Confirm the skills list contains exactly five items.
    - name: calculate_experience
      prompt: "Estimate total years of relevant experience from {{ input.resume fileresume }}."
      output_keys:
        - years_experience
      model: gpt-5-nano
    - name: evaluate_cultural_fit
      prompt: "Rate the cultural fit from {{ input.cover_letter fileresume }} on a 1-10 scale."
      output_keys:
        - cultural_fit_score
      model: gpt-5-nano
  output:
    schema:
      skills: "list[string]"
      years_experience: number
      cultural_fit_score: integer
```

--------------------------------

### Summarize Daily Customer Feedback Insights using LLM

Source: https://www.docetl.org/llms-full.txt

This LLM operation summarizes daily customer feedback by consolidating statistics and individual review analyses. It uses a reduce operation to group feedback by date, then generates a narrative summary highlighting key trends, concerns, and positive aspects. The prompt incorporates statistical data and sentiment analysis results.

```prompt
Summarize the customer feedback for {{ reduce_key }}:

Statistics:
- Average Rating: {{ inputs[0].avg_rating }}
- Number of Reviews: {{ inputs[0].review_count }}
- Rating Range: {{ inputs[0].min_rating }} to {{ inputs[0].max_rating }}

Reviews and Sentiments:
{% for review in inputs %}
- Sentiment: {{ review.sentiment }}
- Themes: {{ review.themes | join(", ") }}
- Products: {{ review.products | join(", ") }}
{% endfor %}

Provide:
1. Key trends and patterns
2. Notable customer concerns
3. Positive highlights

```

--------------------------------

### Code Operator for Custom Python Logic

Source: https://www.docetl.org/llms-full.txt

Demonstrates the use of the 'Code' operator in DocETL, which allows execution of custom Python code within the pipeline for deterministic logic. This is useful for operations not covered by standard operators.

```yaml
code_operation:
  type: code
  function: |-
    def process_document(input):
      # Custom Python logic here
      output = input.copy()
      output['processed_text'] = input['text'].upper()
      return output
```

--------------------------------

### Simple YAML Schema for LLM Output

Source: https://www.docetl.org/llms-full.txt

Defines a basic output schema for LLM operations, specifying string and number types for fields like summary, sentiment, and confidence. This is a recommended simple structure.

```yaml
output:
  schema:
    summary: string
    sentiment: string  # Use string if prompt doesn't list exact values
    confidence: number
```

--------------------------------

### TopK for Semantic Retrieval with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'topk' operation performs semantic retrieval using embeddings to find the most similar items based on a query. It's suitable for tasks like finding relevant support tickets.

```yaml
- name: find_relevant_tickets
  type: topk
  method: embedding
  k: 5
  keys:
    - subject
    - description
    - customer_feedback
  query: "payment processing errors with international transactions"
  embedding_model: text-embedding-3-small
```

--------------------------------

### Equijoin for Semantic Joins with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'equijoin' operation performs semantic joins between datasets based on specified blocking keys and a comparison prompt. It's powerful for matching related items like candidates to jobs but can be computationally intensive.

```yaml
- name: match_candidates_to_jobs
  type: equijoin
  blocking_keys:
    left: [skills]
    right: [required_skills]
  blocking_threshold: 0.4
  embedding_model: text-embedding-3-small
  comparison_prompt: |
    Compare the candidate (skills: {{ left.skills }}, experience: {{ left.years_experience }})
    to the job requirements (skills: {{ right.required_skills }}, experience: {{ right.desired_experience }}).
    Respond with "True" if the alignment is strong, otherwise "False".
```

--------------------------------

### YAML Enum Schema for LLM Output (Conditional)

Source: https://www.docetl.org/llms-full.txt

Shows the correct and incorrect usage of enum types in DocETL schemas. Enums are only appropriate when the prompt explicitly lists all possible values.

```yaml
# Good - prompt explicitly says "respond with positive, negative, or neutral"
sentiment: "enum[positive, negative, neutral]"

# Bad - prompt doesn't explicitly list all category values
category: "enum[news, opinion, analysis]"  # Should be string instead
```

--------------------------------

### Calculate Review Statistics using Python

Source: https://www.docetl.org/llms-full.txt

This Python code snippet performs a reduce operation to calculate basic statistics for customer reviews, grouped by date. It computes the average rating, total review count, and the minimum and maximum ratings within each group. This operation is useful for summarizing feedback trends over time.

```python
def transform(items) -> dict:
    ratings = [item['rating'] for item in items]
    return {
        'avg_rating': sum(ratings) / len(ratings),
        'review_count': len(items),
        'min_rating': min(ratings),
        'max_rating': max(ratings)
    }
```

--------------------------------

### Jinja2 Template Access for Map/Filter Operations

Source: https://www.docetl.org/llms-full.txt

Demonstrates how to access document fields within Jinja2 templates for Map and Filter operations in DocETL. It uses the `input.field_name` syntax.

```jinja
{{ input.field_name }}
```

--------------------------------

### Complete Medical Transcript Processing Pipeline (YAML)

Source: https://www.docetl.org/llms-full.txt

This YAML configuration defines a complete DocETL pipeline for processing medical transcripts. It includes operations for extracting medical information, unnesting medication data, resolving medication names using a comparison model, and summarizing medication usage. The pipeline is designed to output a structured analysis of medications found in the transcripts.

```yaml
default_model: gpt-5-nano

system_prompt:
  dataset_description: a collection of medical transcripts from doctor-patient conversations
  persona: a medical practitioner analyzing patient symptoms and medications

datasets:
  transcripts:
    path: medical_transcripts.json
    type: file

operations:
  - name: extract_medical_info
    type: map
    optimize: true
    output:
      schema:
        medications: "list[{name: string, dosage: string, frequency: string}]"
        symptoms: "list[{description: string, severity: string, duration: string}]"
    prompt: |
      Extract medications and symptoms from:
      {{ input.text }}

  - name: unnest_medications
    type: unnest
    unnest_key: medications
    recursive: true # This is a recursive unnest, so it will unnest the medications list into individual medications

  - name: resolve_medications
    type: resolve
    blocking_keys:
      - name
    blocking_threshold: 0.35
    comparison_model: gpt-5-nano
    comparison_prompt: |
      Are these medications the same or closely related?
      Med 1: {{ input1.name }} ({{ input1.dosage }})
      Med 2: {{ input2.name }} ({{ input2.dosage }})
    resolution_prompt: |
      Create a canonical name for the following medications:
      {% for med in inputs %}
      - {{ med.name }} ({{ med.dosage }})
      {% endfor %}
    embedding_model: text-embedding-3-small
    output:
      schema:
        name: string
        standard_dosage: string

  - name: summarize_medications
    type: reduce
    reduce_key: name
    prompt: |
      Summarize the usage pattern for {{ reduce_key }}:
      {% for med in inputs %}
      - Dosage: {{ med.dosage }}
      - Frequency: {{ med.frequency }}
      {% endfor %}
    output:
      schema:
        usage_summary: string
        common_dosage: string
        side_effects: "list[string]"

pipeline:
  steps:
    - name: medical_analysis
      input: transcripts
      operations:
        - extract_medical_info
        - unnest_medications
        - resolve_medications
        - summarize_medications
  output:
    type: file
    path: medication_analysis.json
    intermediate_dir: intermediate_results

```

--------------------------------

### Rank Operation

Source: https://www.docetl.org/llms-full.txt

The 'rank' operation orders items based on a provided prompt that defines the ranking criteria. It allows for nuanced, semantic ordering beyond simple sorting.

```APIDOC
## Rank Operation

### Description
Orders items based on a semantic understanding defined by a prompt, allowing for nuanced ordering.

### Method
rank

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'rank'.
- **prompt** (string) - Required - The prompt defining the ranking criteria.
- **input_keys** (list[string]) - Required - The keys from the input documents to be used for ranking.
- **direction** (string) - Optional - The ordering direction ('asc' or 'desc'). Defaults to 'asc'.
- **rerank_call_budget** (integer) - Optional - Budget for reranking calls.
- **initial_ordering_method** (string) - Optional - Method for initial ordering (e.g., 'likert').
- **model** (string) - Required - The model to use for ranking.
```

--------------------------------

### Rank Operation for Nuanced Ordering with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'rank' operation provides a semantic ordering of items based on a prompt, considering factors like controversy or sentiment. It's useful for nuanced sorting when deterministic numeric sorting is not required.

```yaml
- name: rank_by_controversy
  type: rank
  prompt: |
    Order these debates by how controversial they are.
    Consider disagreement, divisive topics, emotional language, and public reaction.
  input_keys: ["content", "title", "date"]
  direction: desc
  rerank_call_budget: 10
  initial_ordering_method: likert
  model: gpt-5-nano
```

--------------------------------

### Cluster Operation for Hierarchical Grouping with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'cluster' operation groups semantically similar items into a hierarchical structure by generating summaries for clusters. It's useful for organizing concepts and requires careful monitoring of token usage.

```yaml
- name: cluster_concepts
  type: cluster
  max_batch_size: 5
  embedding_keys:
    - concept
    - description
  output_key: categories
  summary_schema:
    concept: str
    description: str
  summary_prompt: |
    Given these related concepts, name the overarching concept and summarize why they belong together.
    {% for input in inputs %}
    {{ input.concept }}: {{ input.description }}
    {% endfor %}
```

--------------------------------

### Parallel Map Operation

Source: https://www.docetl.org/llms-full.txt

The 'parallel_map' operation allows multiple prompts to be executed concurrently on input documents. Each prompt can extract different information, and the results are combined into a single enriched document.

```APIDOC
## Parallel Map Operation

### Description
Executes multiple prompts concurrently on input documents, combining their outputs into a single enriched document.

### Method
parallel_map

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'parallel_map'.
- **prompts** (list[object]) - Required - A list of prompt configurations to run in parallel.
  - **name** (string) - Required - Name of the individual prompt.
  - **prompt** (string) - Required - The prompt text.
  - **output_keys** (list[string]) - Required - Keys where the output should be stored.
  - **model** (string) - Required - The model to use.
  - **gleaning** (object) - Optional - Configuration for gleaning (e.g., validation).
    - **num_rounds** (integer) - Required - Number of gleaning rounds.
    - **validation_prompt** (string) - Required - Prompt for validation.
- **output** (object) - Required - Defines the output schema.
  - **schema** (object) - Required - The schema of the output.
```

--------------------------------

### Unnest Operation for Recursive Category Processing using YAML

Source: https://www.docetl.org/llms-full.txt

This YAML sequence demonstrates an 'unnest' operation for recursively processing nested data, specifically product categories. It first uses a 'map' operation to extract categories and then 'unnests' them recursively to a specified depth for further analysis.

```yaml
# First, extract nested data
- name: extract_product_details
  type: map
  prompt: |
    Extract product details from this catalog entry:
    {{ input.text }}

    Include:
    1. Product categories (main and sub-categories)
    2. Product features
  output:
    schema:
      categories: "list[{main: string, subcategories: list[string]}]"

# Unnest categories recursively
- name: unnest_categories
  type: unnest
  unnest_key: categories
  recursive: true  # Must set recursive: true when unnesting a list[dict] type key. Not needed for list[string] or other simple list types.
  depth: 2  # Limit recursion to 2 levels (main category and subcategories)

# Analyze individual categories
- name: analyze_category
  type: map
  prompt: |
    Analyze this product category:
    Main Category: {{ input.main }}
    {% if input.subcategories %}
    Subcategories:
    {% for subcat in input.subcategories %}
    - {{ subcat }}
    {% endfor %}
    {% endif %}

    Provide:
    1. Market size (one of large/medium/small)
    2. Competition level
    3. Growth potential
  output:
    schema:
      market_size: "enum[large, medium, small]"
      competition: string
      growth_potential: string

```

--------------------------------

### Equijoin Operation

Source: https://www.docetl.org/llms-full.txt

The 'equijoin' operation performs semantic joins between two sets of documents based on specified blocking keys and a comparison prompt. It's useful for matching related items like candidates to jobs.

```APIDOC
## Equijoin Operation

### Description
Performs semantic joins between two sets of documents using blocking keys and a comparison prompt to find aligned items.

### Method
equijoin

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'equijoin'.
- **blocking_keys** (object) - Required - Keys used for blocking.
  - **left** (list[string]) - Required - Blocking keys for the left dataset.
  - **right** (list[string]) - Required - Blocking keys for the right dataset.
- **blocking_threshold** (number) - Required - Threshold for blocking.
- **embedding_model** (string) - Required - The model for embeddings.
- **comparison_prompt** (string) - Required - Prompt for comparing items.
```

--------------------------------

### Extract Text Verbatim with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'extract' operation in DocETL selects specific text sections based on a prompt and appends them to the document with an '_extracted_' suffix. It's useful for pulling out key information like findings or results.

```yaml
- name: findings
  type: extract
  prompt: |
    Extract all sections that discuss key findings, results, or conclusions.
    Focus on paragraphs that mention outcomes, statistics, or discovered insights.
  document_keys:
    - report_text
  model: gpt-5-nano
```

--------------------------------

### TopK Operation

Source: https://www.docetl.org/llms-full.txt

The 'topk' operation performs semantic retrieval using embeddings to find the 'k' most similar items based on a query. It's ideal for finding relevant documents or data points.

```APIDOC
## TopK Operation

### Description
Retrieves the 'k' most semantically similar items to a given query using embedding methods.

### Method
topk

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'topk'.
- **method** (string) - Required - The retrieval method (e.g., 'embedding').
- **k** (integer) - Required - The number of top items to retrieve.
- **keys** (list[string]) - Required - The keys within the documents to consider for similarity.
- **query** (string) - Required - The query string for retrieval.
- **embedding_model** (string) - Required - The model to use for generating embeddings.
```

--------------------------------

### Gather Transcript Context

Source: https://www.docetl.org/llms-full.txt

The gather operation collects transcript chunks, adding adjacent context from previous and next chunks. It uses `split_transcript_id` for document identification and `split_transcript_chunk_num` for ordering, preparing data for LLM processing.

```yaml
- name: gather_transcript
  type: gather
  content_key: transcript_chunk
  doc_id_key: split_transcript_id
  order_key: split_transcript_chunk_num
  peripheral_chunks:
    previous:
      tail:
        count: 1
        content_key: transcript_chunk
    next:
      head:
        count: 1
        content_key: transcript_chunk
```

--------------------------------

### Cluster Operation

Source: https://www.docetl.org/llms-full.txt

The 'cluster' operation groups semantically similar items into hierarchical clusters. It uses embeddings and a summary prompt to name and describe each cluster.

```APIDOC
## Cluster Operation

### Description
Groups semantically similar items into hierarchical clusters, generating summaries for each cluster.

### Method
cluster

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'cluster'.
- **max_batch_size** (integer) - Optional - Maximum batch size for processing.
- **embedding_keys** (list[string]) - Required - Keys to use for generating embeddings.
- **output_key** (string) - Required - The key where the cluster results will be stored.
- **summary_schema** (object) - Required - Schema for the cluster summaries.
- **summary_prompt** (string) - Required - Prompt used to generate cluster summaries.
```

--------------------------------

### Python Code Reduce Operation

Source: https://www.docetl.org/llms-full.txt

Defines a DocETL code reduce operation using Python for aggregating data. The `transform` function calculates the total, average, and count for items grouped by a `reduce_key`.

```yaml
- name: aggregate_stats
  type: code_reduce
  reduce_key: category
  code: |
    def transform(items) -> dict:
        # Aggregate multiple items into a single result
        total = sum(item['value'] for item in items)
        avg = total / len(items)
        return {
            'total': total,
            'average': avg,
            'count': len(items)
        }
```

--------------------------------

### Link Resolve for Fixing References with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'link_resolve' operation fixes broken references in documents by comparing link values to canonical item IDs. It assumes item IDs are authoritative and only updates incorrect references.

```yaml
- name: fix_links
  type: link_resolve
  id_key: title
  link_key: related_to
  blocking_threshold: 0.85
  embedding_model: text-embedding-3-small
  comparison_model: gpt-5-nano
  comparison_prompt: |
    Compare the link value "{{ link_value }}" with the canonical record "{{ id_value }}".
    Consider the description: {{ item.description filerescription }}.
    Respond with "True" if they refer to the same concept.
```

--------------------------------

### Python Code Map Operation

Source: https://www.docetl.org/llms-full.txt

Implements a DocETL code map operation using Python for deterministic processing. The `transform` function processes each document independently, extracting keywords and their count from the 'text' field.

```yaml
- name: extract_keywords
  type: code_map
  code: |
    def transform(doc) -> dict:
        # Process each document independently
        keywords = doc['text'].lower().split()
        return {
            'keywords': keywords,
            'keyword_count': len(keywords)
        }
```

--------------------------------

### Split Operation for Long Documents with DocETL

Source: https://www.docetl.org/llms-full.txt

The 'split' operation divides long documents into smaller chunks based on specified criteria, such as token count. This is essential for processing documents that exceed model context limits.

```yaml
- name: split_transcript
  type: split
  split_key: transcript
  method: token_count
  method_kwargs:
    num_tokens: 600
```

--------------------------------

### Split Operation

Source: https://www.docetl.org/llms-full.txt

The 'split' operation divides long documents into smaller chunks based on a specified method, such as token count. This is useful for processing documents that exceed model context limits.

```APIDOC
## Split Operation

### Description
Divides long documents into smaller, manageable chunks based on specified criteria like token count.

### Method
split

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'split'.
- **split_key** (string) - Required - The key of the document content to split.
- **method** (string) - Required - The method for splitting (e.g., 'token_count').
- **method_kwargs** (object) - Optional - Keyword arguments for the splitting method (e.g., `num_tokens`).
```

--------------------------------

### Link Resolve Operation

Source: https://www.docetl.org/llms-full.txt

The 'link_resolve' operation fixes broken references within documents by comparing potential link values against canonical IDs using a comparison prompt and embeddings.

```APIDOC
## Link Resolve Operation

### Description
Resolves and rewrites broken references in documents by comparing link values against canonical IDs.

### Method
link_resolve

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (string) - Required - A name for this operation.
- **type** (string) - Required - Must be 'link_resolve'.
- **id_key** (string) - Required - The key containing the canonical ID.
- **link_key** (string) - Required - The key containing the reference to be resolved.
- **blocking_threshold** (number) - Required - Threshold for matching.
- **embedding_model** (string) - Required - The model for embeddings.
- **comparison_model** (string) - Required - The model for comparison.
- **comparison_prompt** (string) - Required - Prompt for comparing link values and IDs.
```