Parallel LLMs.txt (parallel-web/parallel-llms-txt)

Parallel LLMs.txt

https://github.com/parallel-web/parallel-llms-txt
Admin
A repository that provides an agent-friendly, machine-readable context of all public Parallel...

Tokens:259,587
Snippets:1,040
Trust Score:5.5
Update:5 months ago
Show doc for...
Context Summary (auto-generated)
Raw
# Parallel llms.txt - Agent-Friendly Navigation System

## Introduction

Parallel llms.txt is a specialized repository that generates and maintains a comprehensive machine-readable index for Parallel Web Systems' APIs and documentation. The project creates an `llms.txt` file containing all public Parallel content, making it equally accessible to both humans and AI agents. This repository serves as an agent-friendly navigation and context document, enabling programmatic discovery of Parallel's deep research APIs, search services, and integration patterns.

The core mission is to build infrastructure for the "Programmatic Web" designed specifically for AIs, enabling declarative, composable layers built around reasoning, computation, verifiable provenance, and open markets. The project aggregates content from Parallel's main website, documentation portal, and SDK repositories, deploying it via Cloudflare Workers with automated re-extraction every 6 hours to ensure content freshness while resetting git history for privacy.

---

## APIs and Key Functions

### Task API - Automated Deep Research

Enterprise-grade API for structured web research with async processing, supporting 5-second to 30-minute operations. Provides multiple processor tiers (base, core, lite, pro, ultra2x, ultra4x, ultra8x) with structured input/output schemas, MCP server integration, and comprehensive citation tracking.

```bash
# Create a task run for company research
curl -X POST https://api.parallel.ai/v1/tasks/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "processor": "base",
    "input": "Research the latest financial performance and key executives of Anthropic",
    "task_spec": {
      "output_schema": {
        "type": "json",
        "json_schema": {
          "type": "object",
          "properties": {
            "company_name": {"type": "string"},
            "revenue": {"type": "string"},
            "executives": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": {"type": "string"},
                  "title": {"type": "string"}
                }
              }
            },
            "recent_news": {"type": "array", "items": {"type": "string"}}
          },
          "required": ["company_name", "executives"]
        }
      }
    },
    "source_policy": {
      "include_domains": [".com", ".org"],
      "exclude_domains": ["reddit.com", "x.com"]
    },
    "webhook": {
      "url": "https://your-app.com/webhooks/parallel",
      "headers": {"x-webhook-secret": "your_secret"}
    }
  }'

# Response (202 Accepted):
{
  "run_id": "run_abc123xyz",
  "status": "queued",
  "created_at": "2025-10-30T14:24:00Z",
  "processor": "base",
  "estimated_cost": 0.15
}

# Retrieve result after processing
curl -X GET https://api.parallel.ai/v1/tasks/runs/run_abc123xyz/result \
  -H "Authorization: Bearer YOUR_API_KEY"

# Response (200 OK):
{
  "run_id": "run_abc123xyz",
  "status": "completed",
  "output": {
    "company_name": "Anthropic",
    "revenue": "Estimated $500M+ ARR (2024)",
    "executives": [
      {"name": "Dario Amodei", "title": "CEO"},
      {"name": "Daniela Amodei", "title": "President"}
    ],
    "recent_news": [
      "Raised $4B from Amazon",
      "Released Claude 3.5 Sonnet"
    ]
  },
  "basis": {
    "citations": [
      {
        "url": "https://www.anthropic.com/about",
        "title": "About Anthropic",
        "excerpt": "Led by CEO Dario Amodei and President Daniela Amodei..."
      }
    ],
    "reasoning": "Gathered information from company website and recent news articles",
    "confidence": 0.92
  },
  "usage": {
    "total_cost": 0.12
  }
}
```

### Task API with Server-Sent Events (SSE)

Stream real-time progress updates for long-running research tasks, enabling responsive UIs and immediate feedback on task processing stages.

```python
from parallel import Parallel
from parallel.types import TaskSpecParam

client = Parallel(api_key="YOUR_API_KEY")

# Create task with event streaming enabled
task_run = client.task_run.create(
    processor="pro",
    input="Analyze top 5 competitors in the autonomous vehicle space",
    task_spec=TaskSpecParam(
        output_schema="List each competitor with their market position, funding, and key technology differentiation"
    ),
    enable_events=True
)

# Stream events
print(f"Task started: {task_run.run_id}")
for event in client.task_run.stream_events(task_run.run_id):
    if event.event_type == "progress":
        print(f"Progress: {event.data.get('stage')} - {event.data.get('message')}")
    elif event.event_type == "completed":
        print("Task completed!")
        break
    elif event.event_type == "error":
        print(f"Error: {event.data.get('error')}")
        break

# Retrieve final result
result = client.task_run.result(task_run.run_id)
print(f"Output: {result.output}")
print(f"Citations: {len(result.basis.citations)} sources")

# Expected output:
# Task started: run_def456abc
# Progress: search - Identifying relevant sources
# Progress: extract - Extracting content from 12 URLs
# Progress: analyze - Analyzing competitive landscape
# Progress: synthesize - Generating structured output
# Task completed!
# Output: [Structured competitor analysis]
# Citations: 12 sources
```

### Task Groups - Batch Processing

Process multiple research tasks in parallel with unified monitoring, intelligent failure handling, and real-time event aggregation for high-throughput workflows.

```python
from parallel import Parallel
from parallel.types import TaskGroupCreateParams, TaskSpecParam

client = Parallel(api_key="YOUR_API_KEY")

# Create a batch of 50 company research tasks
companies = ["Anthropic", "OpenAI", "Google DeepMind", "Meta AI", "Cohere"]

task_group = client.task_group.create(
    tasks=[
        TaskGroupCreateParams.Task(
            processor="base",
            input=f"Research {company}: latest product launches and market position",
            task_spec=TaskSpecParam(
                output_schema={
                    "type": "json",
                    "json_schema": {
                        "type": "object",
                        "properties": {
                            "company": {"type": "string"},
                            "products": {"type": "array", "items": {"type": "string"}},
                            "market_share": {"type": "string"}
                        }
                    }
                }
            ),
            metadata={"company": company, "batch": "q4_2024"}
        )
        for company in companies
    ],
    max_concurrent=10,
    webhook={
        "url": "https://your-app.com/webhooks/batch-complete",
        "headers": {"x-api-key": "webhook_secret"}
    }
)

print(f"Task group created: {task_group.group_id}")
print(f"Total tasks: {task_group.total_tasks}")

# Stream aggregated events
for event in client.task_group.stream_events(task_group.group_id):
    if event.event_type == "task_completed":
        print(f"Completed: {event.task_id} ({event.data.get('metadata').get('company')})")
    elif event.event_type == "group_completed":
        print(f"All tasks completed! Success: {event.data.get('successful')}, Failed: {event.data.get('failed')}")
        break

# Retrieve all results
results = client.task_group.results(task_group.group_id)
for result in results:
    print(f"{result.metadata['company']}: {result.output}")

# Expected output:
# Task group created: tg_789xyz123
# Total tasks: 5
# Completed: run_001 (Anthropic)
# Completed: run_002 (OpenAI)
# Completed: run_003 (Google DeepMind)
# Completed: run_004 (Meta AI)
# Completed: run_005 (Cohere)
# All tasks completed! Success: 5, Failed: 0
```

### Task API with MCP Tool Calling

Integrate external tools via Model Context Protocol servers, enabling tasks to call APIs, query databases, or execute custom functions during research workflows.

```bash
curl -X POST https://api.parallel.ai/v1/tasks/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "processor": "base",
    "input": "Find the current weather in San Francisco and book a restaurant reservation for tonight",
    "mcp_servers": [
      {
        "type": "url",
        "url": "https://weather-api.example.com/mcp",
        "name": "weather_service",
        "headers": {"x-api-key": "weather_key"},
        "allowed_tools": ["get_current_weather", "get_forecast"]
      },
      {
        "type": "url",
        "url": "https://opentable-mcp.example.com/mcp",
        "name": "restaurant_booking",
        "headers": {"authorization": "Bearer opentable_token"},
        "allowed_tools": ["search_restaurants", "create_reservation"]
      }
    ],
    "task_spec": {
      "output_schema": {
        "type": "json",
        "json_schema": {
          "type": "object",
          "properties": {
            "weather": {
              "type": "object",
              "properties": {
                "temperature": {"type": "number"},
                "conditions": {"type": "string"}
              }
            },
            "reservation": {
              "type": "object",
              "properties": {
                "restaurant_name": {"type": "string"},
                "time": {"type": "string"},
                "confirmation_code": {"type": "string"}
              }
            }
          }
        }
      }
    }
  }'

# Response includes tool calls made during execution:
{
  "run_id": "run_mcp789",
  "status": "completed",
  "output": {
    "weather": {
      "temperature": 62,
      "conditions": "Partly cloudy"
    },
    "reservation": {
      "restaurant_name": "The Slanted Door",
      "time": "19:30",
      "confirmation_code": "OT-ABC123"
    }
  },
  "basis": {
    "tool_calls": [
      {
        "server": "weather_service",
        "tool": "get_current_weather",
        "input": {"location": "San Francisco, CA"},
        "output": {"temp": 62, "conditions": "Partly cloudy"}
      },
      {
        "server": "restaurant_booking",
        "tool": "search_restaurants",
        "input": {"location": "San Francisco", "time": "tonight", "party_size": 2}
      },
      {
        "server": "restaurant_booking",
        "tool": "create_reservation",
        "input": {"restaurant_id": "12345", "time": "19:30", "party_size": 2}
      }
    ]
  }
}
```

### Search API - Real-time Web Search

Low-latency web search API optimized for LLM consumption, returning ranked URLs with compressed excerpts in under 3 seconds. Supports natural language objectives and keyword queries with configurable result quality via processors.

```typescript
import Parallel from "parallel-web";

const client = new Parallel({ apiKey: process.env.PARALLEL_API_KEY });

async function searchWithContext() {
  try {
    // Perform web search with natural language objective
    const searchResult = await client.search.create({
      objective: "Find recent benchmarks comparing Claude, GPT-4, and Gemini on coding tasks",
      processor: "pro", // Use 'pro' for freshness and quality
      max_results: 10,
      max_chars_per_result: 1500,
      source_policy: {
        include_domains: [".edu", ".org", "arxiv.org"],
        exclude_domains: ["reddit.com", "quora.com"]
      }
    });

    console.log(`Search ID: ${searchResult.search_id}`);
    console.log(`Found ${searchResult.results.length} results\n`);

    // Process results
    searchResult.results.forEach((result, index) => {
      console.log(`${index + 1}. ${result.title}`);
      console.log(`   URL: ${result.url}`);
      console.log(`   Excerpts: ${result.excerpts.length} relevant passages`);
      console.log(`   Preview: ${result.excerpts[0].substring(0, 150)}...\n`);
    });

    // Use excerpts to build context for LLM
    const context = searchResult.results
      .map(r => `Source: ${r.title}\n${r.excerpts.join('\n')}`)
      .join('\n\n---\n\n');

    return { searchId: searchResult.search_id, context };

  } catch (error) {
    if (error.status === 429) {
      console.error("Rate limit exceeded. Max 600 requests/min for Search API.");
    } else {
      console.error("Search error:", error.message);
    }
    throw error;
  }
}

// Execute search
searchWithContext().then(({ searchId, context }) => {
  console.log("Context ready for LLM processing");
  console.log(`Total context length: ${context.length} characters`);
});

// Expected output:
// Search ID: search_xyz789
// Found 10 results
//
// 1. LLM Coding Benchmark Results Q4 2024
//    URL: https://arxiv.org/abs/2024.12345
//    Excerpts: 3 relevant passages
//    Preview: We evaluated Claude 3.5 Sonnet, GPT-4 Turbo, and Gemini 1.5 Pro on the HumanEval benchmark...
//
// 2. Comparative Analysis of AI Coding Assistants
//    URL: https://cs.stanford.edu/research/...
//    Excerpts: 2 relevant passages
//    Preview: In our study of 500 software engineers, Claude demonstrated...
```

### Chat API - OpenAI-Compatible Completions

Low-latency LLM completions with real-time web research, providing OpenAI-compatible endpoints with streaming support and JSON schema response formatting. Ideal for interactive applications requiring up-to-date information.

```python
from openai import OpenAI

# Initialize client with Parallel base URL
client = OpenAI(
    api_key="YOUR_PARALLEL_API_KEY",
    base_url="https://api.parallel.ai"
)

# Stream chat completion with web research
response = client.chat.completions.create(
    model="speed",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful AI assistant with access to real-time web information."
        },
        {
            "role": "user",
            "content": "What are the latest announcements from Anthropic about Claude 4?"
        }
    ],
    stream=True,
    temperature=0.7,
    max_tokens=500
)

print("Streaming response:")
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print("\n")

# JSON schema response format
structured_response = client.chat.completions.create(
    model="speed",
    messages=[
        {
            "role": "user",
            "content": "Research the top 3 electric vehicle companies by market cap"
        }
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ev_companies",
            "schema": {
                "type": "object",
                "properties": {
                    "companies": {
                        "type": "array",
                        "items": {
                            "type": "object",
                            "properties": {
                                "name": {"type": "string"},
                                "market_cap": {"type": "string"},
                                "country": {"type": "string"}
                            },
                            "required": ["name", "market_cap"]
                        }
                    },
                    "as_of_date": {"type": "string"}
                },
                "required": ["companies", "as_of_date"]
            }
        }
    }
)

result = eval(structured_response.choices[0].message.content)
print("Structured output:")
for company in result["companies"]:
    print(f"- {company['name']}: {company['market_cap']} ({company.get('country', 'N/A')})")

# Expected output:
# Streaming response:
# Based on the latest information, Anthropic has announced several updates to Claude...
# [streaming continues]
#
# Structured output:
# - Tesla: $800B (USA)
# - BYD: $120B (China)
# - Rivian: $15B (USA)
```

### MCP Integration - Search MCP Server

Model Context Protocol server providing Parallel Search as a tool for AI agents in Claude, VS Code, Cursor, Windsurf, and other MCP-compatible environments.

```json
{
  "mcpServers": {
    "parallel-search": {
      "command": "npx",
      "args": ["-y", "@parallel/mcp-server"],
      "env": {
        "PARALLEL_API_KEY": "your_api_key_here"
      }
    }
  }
}
```

```python
# Example: Using Parallel Search MCP in a Claude Desktop conversation
# Claude automatically detects the MCP tool and can call it

# User: "What are the latest developments in quantum computing?"

# Claude internally calls:
# Tool: parallel_search
# Input: {
#   "objective": "Latest developments in quantum computing 2024-2025",
#   "processor": "pro",
#   "max_results": 8
# }

# Tool returns ranked URLs with excerpts
# Claude synthesizes the information and responds:

"""
Based on recent developments in quantum computing:

1. **IBM's Quantum System Two** (December 2024): IBM unveiled a new quantum
   processor with 1,121 qubits, showing significant error correction improvements.
   [Source: IBM Research Blog]

2. **Google's Willow Chip**: Achieved a major breakthrough in quantum error
   correction, reducing errors exponentially as more qubits are added.
   [Source: Nature Journal]

3. **Microsoft Azure Quantum**: Launched commercial access to topological qubits,
   promising more stable quantum computations.
   [Source: Microsoft Blog]

These advances mark significant progress toward practical quantum computing applications.
"""
```

### Ingest API - Private Knowledge Integration

Upload and index private documents for inclusion in Task API research, enabling hybrid search across public web and proprietary knowledge bases.

```bash
# Upload a document to your private knowledge base
curl -X POST https://api.parallel.ai/v1/ingest/documents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "documents": [
      {
        "id": "internal_policy_001",
        "content": "Company travel policy: All international travel requires VP approval. Maximum daily per diem: $150 for meals, $300 for lodging. Economy class for flights under 6 hours, business class allowed for longer flights.",
        "metadata": {
          "type": "policy",
          "department": "finance",
          "last_updated": "2024-10-01"
        }
      },
      {
        "id": "internal_policy_002",
        "content": "Expense reporting: Submit all expenses within 30 days. Attach receipts for amounts over $25. Use Expensify for submission.",
        "metadata": {
          "type": "policy",
          "department": "finance"
        }
      }
    ],
    "index_name": "company_policies"
  }'

# Response:
{
  "index_name": "company_policies",
  "documents_indexed": 2,
  "index_id": "idx_abc123"
}

# Use ingested content in Task API
curl -X POST https://api.parallel.ai/v1/tasks/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "processor": "base",
    "input": "What is our company policy for international travel expenses?",
    "ingest_index": "company_policies",
    "source_policy": {
      "include_private": true
    }
  }'

# Response includes both web research and private knowledge:
{
  "run_id": "run_ingest456",
  "status": "completed",
  "output": "According to your company travel policy, international travel requires VP approval. For expenses, business class is allowed for flights over 6 hours, with a daily per diem of $150 for meals and $300 for lodging. All expenses must be submitted within 30 days using Expensify, with receipts required for amounts over $25.",
  "basis": {
    "citations": [
      {
        "source": "private",
        "document_id": "internal_policy_001",
        "excerpt": "Company travel policy: All international travel requires VP approval..."
      },
      {
        "source": "private",
        "document_id": "internal_policy_002",
        "excerpt": "Expense reporting: Submit all expenses within 30 days..."
      }
    ]
  }
}
```

### Python SDK - Complete Integration

Full-featured Python SDK providing type-safe access to all Parallel APIs with async support, pagination, and comprehensive error handling.

```python
from parallel import Parallel
from parallel.types import TaskSpecParam, SourcePolicy
import asyncio

# Initialize client
client = Parallel(api_key="YOUR_API_KEY")

# Synchronous task creation
def sync_research():
    task_run = client.task_run.create(
        processor="base",
        input="What are the key differences between Cloudflare Workers and AWS Lambda?",
        task_spec=TaskSpecParam(
            output_schema="Provide a comparison table with: pricing, cold start time, supported languages, edge locations"
        ),
        source_policy=SourcePolicy(
            include_domains=["aws.amazon.com", "cloudflare.com", ".dev"]
        )
    )

    print(f"Task created: {task_run.run_id}")

    # Poll for completion
    while True:
        status = client.task_run.get(task_run.run_id)
        if status.status == "completed":
            break
        elif status.status == "failed":
            raise Exception(f"Task failed: {status.error}")
        time.sleep(2)

    result = client.task_run.result(task_run.run_id)
    return result.output

# Asynchronous batch processing
async def async_batch_research():
    from parallel import AsyncParallel

    async_client = AsyncParallel(api_key="YOUR_API_KEY")

    queries = [
        "Latest funding rounds in AI startups Q4 2024",
        "Top 10 YC companies from W24 batch",
        "Recent AI acquisitions by big tech companies"
    ]

    # Create tasks concurrently
    tasks = await asyncio.gather(*[
        async_client.task_run.create(
            processor="lite",
            input=query,
            task_spec=TaskSpecParam(output_schema="Brief summary with key points")
        )
        for query in queries
    ])

    print(f"Created {len(tasks)} tasks")

    # Wait for all to complete
    results = await asyncio.gather(*[
        async_client.task_run.result(task.run_id)
        for task in tasks
    ])

    return [result.output for result in results]

# Search API usage
def quick_search():
    search_result = client.search.create(
        objective="Find official documentation for Next.js 15 app router",
        processor="base",
        max_results=5
    )

    return [
        {"url": r.url, "title": r.title, "preview": r.excerpts[0][:100]}
        for r in search_result.results
    ]

# Error handling
def robust_task_execution():
    try:
        task_run = client.task_run.create(
            processor="pro",
            input="Complex research query...",
            metadata={"user_id": "user_123", "session": "session_456"}
        )

        result = client.task_run.result(task_run.run_id)
        return result

    except client.RateLimitError as e:
        print(f"Rate limit hit. Retry after: {e.retry_after} seconds")
        time.sleep(e.retry_after)
        # Retry logic

    except client.TaskFailedError as e:
        print(f"Task failed: {e.message}")
        # Handle failure

    except client.APIError as e:
        print(f"API error: {e.status_code} - {e.message}")
        # Handle API errors

# Execute examples
if __name__ == "__main__":
    print("Sync research:", sync_research())
    print("Quick search:", quick_search())
    print("Async batch:", asyncio.run(async_batch_research()))
```

### TypeScript SDK - Complete Integration

Type-safe TypeScript/JavaScript SDK with full async/await support, streaming capabilities, and comprehensive type definitions for all API endpoints.

```typescript
import Parallel from "parallel-web";
import type { TaskRun, SearchResult, SourcePolicy } from "parallel-web";

// Initialize client
const client = new Parallel({
  apiKey: process.env.PARALLEL_API_KEY,
  timeout: 60000, // 60 second timeout
});

// Task API with streaming events
async function streamTaskWithProgress(query: string): Promise<void> {
  const taskRun = await client.taskRun.create({
    processor: "base",
    input: query,
    enableEvents: true,
    taskSpec: {
      outputSchema: {
        type: "json",
        jsonSchema: {
          type: "object",
          properties: {
            summary: { type: "string" },
            key_points: { type: "array", items: { type: "string" } },
          },
        },
      },
    },
  });

  console.log(`Task started: ${taskRun.runId}`);

  // Stream events
  const eventStream = await client.taskRun.streamEvents(taskRun.runId);

  for await (const event of eventStream) {
    switch (event.eventType) {
      case "progress":
        console.log(`[${event.data.stage}] ${event.data.message}`);
        break;
      case "completed":
        console.log("Task completed!");
        break;
      case "error":
        console.error("Task failed:", event.data.error);
        throw new Error(event.data.error);
    }
  }

  const result = await client.taskRun.result(taskRun.runId);
  console.log("Output:", JSON.stringify(result.output, null, 2));
  console.log(`Citations: ${result.basis.citations.length}`);
}

// Parallel task execution with Promise.all
async function parallelResearch(queries: string[]): Promise<TaskRun[]> {
  const taskPromises = queries.map((query) =>
    client.taskRun.create({
      processor: "lite",
      input: query,
      metadata: { query, timestamp: new Date().toISOString() },
    })
  );

  const taskRuns = await Promise.all(taskPromises);
  console.log(`Created ${taskRuns.length} tasks in parallel`);

  // Wait for all results
  const resultPromises = taskRuns.map((task) =>
    client.taskRun.result(task.runId)
  );

  return Promise.all(resultPromises);
}

// Search API with source filtering
async function advancedSearch(
  query: string,
  sourcePolicy: SourcePolicy
): Promise<SearchResult> {
  try {
    const result = await client.search.create({
      objective: query,
      processor: "pro",
      maxResults: 15,
      maxCharsPerResult: 2000,
      sourcePolicy,
    });

    console.log(`Search ${result.searchId}: ${result.results.length} results`);

    // Process results
    const processedResults = result.results.map((r, idx) => ({
      rank: idx + 1,
      url: r.url,
      title: r.title,
      relevance: r.excerpts.length,
      firstExcerpt: r.excerpts[0]?.substring(0, 200),
    }));

    return { ...result, processedResults };
  } catch (error) {
    if (error.status === 429) {
      console.error("Rate limit: 600 requests/min exceeded");
      throw error;
    }
    throw error;
  }
}

// Task Groups for batch processing
async function batchProcessCompanies(
  companies: string[]
): Promise<void> {
  const taskGroup = await client.taskGroup.create({
    tasks: companies.map((company) => ({
      processor: "base",
      input: `Research ${company}: revenue, employee count, and recent news`,
      taskSpec: {
        outputSchema: {
          type: "json",
          jsonSchema: {
            type: "object",
            properties: {
              name: { type: "string" },
              revenue: { type: "string" },
              employees: { type: "number" },
              news: { type: "array", items: { type: "string" } },
            },
          },
        },
      },
      metadata: { company },
    })),
    maxConcurrent: 10,
  });

  console.log(`Task group ${taskGroup.groupId}: ${taskGroup.totalTasks} tasks`);

  // Stream group events
  const groupStream = await client.taskGroup.streamEvents(taskGroup.groupId);

  let completed = 0;
  for await (const event of groupStream) {
    if (event.eventType === "task_completed") {
      completed++;
      console.log(
        `Progress: ${completed}/${taskGroup.totalTasks} (${event.data.metadata.company})`
      );
    } else if (event.eventType === "group_completed") {
      console.log(
        `All tasks done! Success: ${event.data.successful}, Failed: ${event.data.failed}`
      );
      break;
    }
  }

  // Retrieve all results
  const results = await client.taskGroup.results(taskGroup.groupId);
  results.forEach((result) => {
    console.log(`${result.metadata.company}:`, result.output);
  });
}

// Webhook configuration
async function createTaskWithWebhook(query: string): Promise<TaskRun> {
  return client.taskRun.create({
    processor: "base",
    input: query,
    webhook: {
      url: "https://your-app.com/api/webhooks/parallel",
      headers: {
        "x-webhook-secret": process.env.WEBHOOK_SECRET!,
        "content-type": "application/json",
      },
    },
  });
}

// Example usage
async function main() {
  // Stream a single task
  await streamTaskWithProgress(
    "What are the latest AI safety research developments?"
  );

  // Parallel research
  const results = await parallelResearch([
    "Latest GPT-5 rumors",
    "Claude 4 release date",
    "Gemini Ultra features",
  ]);

  // Advanced search
  const searchResults = await advancedSearch(
    "Best practices for vector databases in production",
    {
      includeDomains: [".dev", ".io", "github.com"],
      excludeDomains: ["reddit.com", "medium.com"],
    }
  );

  // Batch processing
  await batchProcessCompanies([
    "Anthropic",
    "OpenAI",
    "Cohere",
    "Mistral AI",
  ]);
}

main().catch(console.error);
```

### Cloudflare Workers Deployment Configuration

Automated deployment configuration using Cloudflare Workers for static asset hosting with custom domain routing and logging enabled for monitoring.

```json
{
  "name": "parallel-llmstext",
  "main": "public",
  "compatibility_date": "2024-10-14",
  "observability": {
    "enabled": true,
    "logs": {
      "enabled": true
    }
  },
  "routes": [
    {
      "pattern": "llm.parallel.ai/*",
      "custom_domain": true
    }
  ]
}
```

### GitHub Actions CI/CD Pipeline

Automated re-extraction workflow running every 6 hours to keep content fresh while resetting git history for privacy. Includes Cloudflare deployment integration.

```yaml
name: Build and Reset History

on:
  schedule:
    - cron: "0 */6 * * *"  # Every 6 hours
  workflow_dispatch:

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm ci

      - name: Extract content from sitemaps
        run: npm run build
        # Runs: extract-from-sitemap (reads llmtext.json config)

      - name: Reset git history
        run: |
          git checkout --orphan latest_branch
          git add -A
          git commit -m "Fresh extraction $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
          git branch -D main
          git branch -m main
          git push -f origin main

      - name: Deploy to Cloudflare
        uses: cloudflare/wrangler-action@v3
        with:
          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          command: pages deploy public --project-name=parallel-llmstext

      - name: Notify deployment
        run: |
          echo "Deployed to https://llm.parallel.ai"
          echo "Total files: $(find public -type f | wc -l)"
          echo "llms.txt size: $(stat -f%z public/llms.txt 2>/dev/null || stat -c%s public/llms.txt)"
```

### llmtext.json Configuration

Source configuration for extracting and aggregating content from multiple Parallel web properties into a unified llms.txt index.

```json
{
  "sources": [
    {
      "url": "https://parallel.ai/sitemap.xml",
      "name": "Parallel Main Site",
      "exclude_patterns": [
        "/legal/",
        "/privacy/",
        "/terms/"
      ]
    },
    {
      "url": "https://docs.parallel.ai/sitemap.xml",
      "name": "Parallel Documentation",
      "include_patterns": [
        "/api-reference/",
        "/task-api/",
        "/search-api/",
        "/chat-api/",
        "/integrations/"
      ]
    },
    {
      "url": "https://github.com/parallel-ai/python-sdk/blob/main/README.md",
      "name": "Python SDK",
      "type": "markdown"
    },
    {
      "url": "https://github.com/parallel-ai/typescript-sdk/blob/main/README.md",
      "name": "TypeScript SDK",
      "type": "markdown"
    }
  ],
  "output": {
    "path": "public/llms.txt",
    "format": "markdown",
    "include_metadata": true
  },
  "processing": {
    "extract_code_examples": true,
    "preserve_formatting": true,
    "generate_table_of_contents": true
  }
}
```

---

## Summary

Parallel llms.txt serves as a comprehensive reference system for developers and AI agents integrating with Parallel's enterprise web research APIs. The primary use cases include CRM data enrichment pipelines processing thousands of company records, compliance and due diligence workflows requiring structured verification, competitive intelligence platforms aggregating market data, and AI agent tool calling systems leveraging MCP integration. The project demonstrates production-grade deployment patterns with Cloudflare Workers, automated content freshness via scheduled GitHub Actions, and privacy-preserving git history management.

Integration patterns center around three API tiers: Task API for deep asynchronous research with structured schemas and citation tracking, Search API for real-time web discovery with LLM-optimized excerpts, and Chat API for interactive completions with web context. The repository provides complete SDK implementations in Python and TypeScript with async support, streaming capabilities, and comprehensive error handling. MCP server integration enables seamless tool calling in Claude Desktop, VS Code, and other compatible environments. The llms.txt format itself serves as a blueprint for building agent-friendly documentation, combining human readability with machine parseability through structured markdown, enabling programmatic discovery and reduced hallucination through verifiable source attribution.