### Check Environment Configuration - Bash API Source: https://context7.com/firecrawl/fire-enrich/llms.txt Verifies the configured API keys in the environment using a simple curl command. This is useful for initial setup validation. ```bash # Check environment configuration curl http://localhost:3000/api/check-env # Response # { # "environmentStatus": { # "FIRECRAWL_API_KEY": true, # "OPENAI_API_API_KEY": true, # "ANTHROPIC_API_KEY": false, # "FIRESTARTER_DISABLE_CREATION_DASHBOARD": false # } # } ``` -------------------------------- ### Environment Check API - GET /api/check-env Source: https://context7.com/firecrawl/fire-enrich/llms.txt Verifies which API keys are configured in the environment. This endpoint is useful for setup validation. ```APIDOC ## GET /api/check-env ### Description Verifies which API keys are configured in the environment. Useful for setup validation. ### Method GET ### Endpoint /api/check-env ### Parameters #### Query Parameters None #### Request Body None ### Response #### Success Response (200) - **environmentStatus** (object) - An object indicating the status of configured API keys. - **FIRECRAWL_API_KEY** (boolean) - Status of Firecrawl API key. - **OPENAI_API_KEY** (boolean) - Status of OpenAI API key. - **ANTHROPIC_API_KEY** (boolean) - Status of Anthropic API key. - **FIRESTARTER_DISABLE_CREATION_DASHBOARD** (boolean) - Status of Firestarter dashboard creation. #### Response Example ```json { "environmentStatus": { "FIRECRAWL_API_KEY": true, "OPENAI_API_KEY": true, "ANTHROPIC_API_KEY": false, "FIRESTARTER_DISABLE_CREATION_DASHBOARD": false } } ``` ``` -------------------------------- ### Email Data Enrichment Example Source: https://github.com/firecrawl/fire-enrich/blob/main/README.md Demonstrates the transformation of a simple email object into a comprehensive company profile. The enrichment process adds details such as company name, industry, employee count, funding information, and relevant sources. ```json { "email": "erez@wiz.io" } ``` ```json { "email": "erez@wiz.io", "companyName": "Wiz", "industry": "Cybersecurity", "employeeCount": "1001-5000", "yearFounded": 2020, "headquarters": "New York, NY", "fundingStage": "Series D", "totalRaised": "$900M", "website": "https://www.wiz.io", "sources": [ "https://www.wiz.io/about", "https://techcrunch.com/2023/02/27/wiz-confirms-300m-at-a-10b-valuation-to-build-out-its-cloud-security-platform/" ] } ``` -------------------------------- ### Environment Variables Setup for Fire Enrich Source: https://github.com/firecrawl/fire-enrich/blob/main/README.md This snippet shows how to configure the necessary API keys for Fire Enrich in a local environment file. It requires keys for both Firecrawl and OpenAI to enable web scraping and data extraction functionalities. ```dotenv FIRECRAWL_API_KEY=your_firecrawl_key OPENAI_API_KEY=your_openai_key ``` -------------------------------- ### OpenAI Service: Generate Search Queries (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Shows how to use the OpenAIService to generate optimized search queries based on provided company information and a target topic. It can also take previously tried queries as input. ```typescript // Generate optimized search queries const queries = await openai.generateSearchQueries( { email: 'ceo@stripe.com', company: 'Stripe' }, 'employee count', ['Stripe employees'] // Previously tried queries ); // Returns: ['Stripe workforce size 2024', 'site:stripe.com team careers', 'Stripe company headcount'] ``` -------------------------------- ### OpenAI Service: Extract Structured Data with Corroboration (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Demonstrates how to use the OpenAIService to extract structured data from text, including corroboration from multiple sources to improve accuracy. It also shows how to provide a progress callback. ```typescript import { OpenAIService } from '@/lib/services/openai'; const openai = new OpenAIService('sk-your-api-key'); // Extract structured data with source corroboration const enrichments = await openai.extractStructuredDataWithCorroboration( `URL: https://techcrunch.com/article Stripe has raised over $2.2 billion in total funding... URL: https://crunchbase.com/stripe Total Funding Amount: $2.2B, Latest Round: Series I...`, [ { name: 'totalFunding', displayName: 'Total Funding', description: 'Total amount raised', type: 'string', required: false }, { name: 'fundingStage', displayName: 'Funding Stage', description: 'Latest round', type: 'string', required: false } ], { companyName: 'Stripe', targetDomain: 'stripe.com' }, (message, type) => console.log(`[${type}] ${message}`) // Progress callback ); // Result includes corroboration data: // { // totalFunding: { // field: 'totalFunding', // value: '$2.2B', // confidence: 0.95, // sourceContext: [ // { url: 'https://techcrunch.com/article', snippet: 'Stripe has raised over $2.2 billion in total funding' }, // { url: 'https://crunchbase.com/stripe', snippet: 'Total Funding Amount: $2.2B' } // ], // corroboration: { // evidence: [...], // sources_agree: true // } // } // } ``` -------------------------------- ### OpenAI Service: Answer Questions from Table Data (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Illustrates how to use the OpenAIService to answer questions based on structured table data. It takes the question, the table data in CSV format, and optional conversation history as input. ```typescript // Answer questions from table data const answer = await openai.answerFromTableData( 'What is the average funding for Series B companies?', 'company,stage,funding\nAcme,Series B,$50M\nBeta,Series B,$30M', [] // Conversation history ); // Returns: { found: true, answer: 'The average Series B funding is $40M (Acme: $50M, Beta: $30M)' } ``` -------------------------------- ### Configuration: Environment Variables Source: https://context7.com/firecrawl/fire-enrich/llms.txt Lists essential environment variables required for Fire Enrich, including API keys for Firecrawl and OpenAI, and a flag to enable unlimited mode. ```env # .env.local FIRECRAWL_API_KEY=fc-your-key OPENAI_API_KEY=sk-your-key FIRE_ENRICH_UNLIMITED=true # Enable unlimited mode for production ``` -------------------------------- ### Extending FundingAgent with Zod Schema Source: https://github.com/firecrawl/fire-enrich/blob/main/README.md Demonstrates how to add a new field, 'debtFinancing', to the FundingAgent's Zod schema to extend its data extraction capabilities. This ensures type safety and maintainability when adding new data points. ```typescript const FundingResult = z.object({ fundingStage: z.string().optional(), totalRaised: z.string().optional(), lastRoundAmount: z.string().optional(), investors: z.array(z.string()).optional(), // Add your new field here: debtFinancing: z.string().optional(), }); ``` -------------------------------- ### Configuration: Application Limits (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Sets application-level limits for Fire Enrich, such as maximum rows and columns for CSV processing, delays between processing rows, and request limits. ```typescript // app/fire-enrich/config.ts - Application limits export const FIRE_ENRICH_CONFIG = { CSV_LIMITS: { MAX_ROWS: Infinity, // Unlimited in dev/self-hosted mode MAX_COLUMNS: Infinity, }, PROCESSING: { DELAY_BETWEEN_ROWS_MS: 1000, MAX_RETRIES: 3, }, REQUEST_LIMITS: { MAX_BODY_SIZE_MB: 50, MAX_FIELDS_PER_ENRICHMENT: 50, }, }; ``` -------------------------------- ### Configuration: Enrichment Processing Limits (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Defines the processing configuration for enrichment tasks, including the maximum number of concurrent rows to process and the delay between batches. ```typescript // lib/config/enrichment.ts - Processing configuration export const ENRICHMENT_CONFIG = { CONCURRENT_ROWS: 10, // Process up to 10 rows simultaneously BATCH_DELAY_MS: 1000, // 1 second delay between batches }; ``` -------------------------------- ### Cancel Enrichment Session with API (Bash) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Command-line interface to cancel an ongoing data enrichment process using the DELETE /api/enrich endpoint. Requires the unique sessionId of the active enrichment session to terminate it. ```bash # Cancel an active enrichment session curl -X DELETE "http://localhost:3000/api/enrich?sessionId=1699123456789-abc123" # Response on success # {"success": true} # Response if session not found # {"error": "Session not found"} ``` -------------------------------- ### Chat API - POST /api/chat Source: https://context7.com/firecrawl/fire-enrich/llms.txt This API offers a conversational interface for querying enriched data or searching the web. It prioritizes enriched table data and falls back to web search if necessary, supporting conversation history for context. ```APIDOC ## POST /api/chat ### Description Conversational interface for querying enriched data or searching the web. First checks the enriched table data, then falls back to web search if needed. Supports conversation history for contextual follow-up questions. ### Method POST ### Endpoint /api/chat ### Parameters #### Request Body - **question** (string) - Required - The user's query. - **context** (object) - Optional - Contextual information for the query. - **tableData** (string) - Optional - Structured data in a string format (e.g., CSV). - **conversationHistory** (array of objects) - Optional - Previous messages in the conversation. - **role** (string) - The role of the speaker ('user' or 'assistant'). - **content** (string) - The message content. - **sessionId** (string) - Optional - An identifier for the chat session, useful for cancellation. ### Request Example ```json { "question": "What is the total funding raised by companies in the cybersecurity industry?", "context": { "tableData": "companyName,industry,fundingStage,totalRaised\nWiz,Cybersecurity,Series D,$900M\nSnyk,Cybersecurity,Series G,$849M" }, "conversationHistory": [ { "role": "user", "content": "Show me cybersecurity companies" }, { "role": "assistant", "content": "Found 2 cybersecurity companies: Wiz and Snyk" } ], "sessionId": "chat-session-123" } ``` ### Response #### Success Response (200) - Streamed The response is streamed as a series of Server-Sent Events (SSE). - **event.type** (string) - The type of event ('status', 'response', 'complete'). - **status**: Indicates the current step or status of the process. - **step** (string) - The current step (e.g., 'querying_enriched_data', 'searching_web'). - **message** (string) - A message describing the current status. - **response**: Contains the answer and its source. - **message** (string) - The answer to the question. - **source** (string) - The source of the information (e.g., 'enriched_data', 'web_search'). - **complete**: Indicates the end of the stream. #### Response Example (Streamed Events) ``` data: {"type": "status", "step": "querying_enriched_data", "message": "Checking enriched data for cybersecurity companies."} data: {"type": "response", "message": "Companies in the cybersecurity industry include Wiz with $900M raised and Snyk with $849M.", "source": "enriched_data"} data: {"type": "complete"} ``` ``` -------------------------------- ### Chat with Enriched Data or Web Search (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt A conversational interface for querying data or searching the web. It prioritizes enriched table data before falling back to web search and supports conversation history for context. Responses are streamed. ```typescript // Chat with enriched data or search the web const askQuestion = async (question: string, tableData: string, history: Array<{role: string, content: string}>) => { const response = await fetch('/api/chat', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ question: 'What is the total funding raised by companies in the cybersecurity industry?', context: { tableData: 'companyName,industry,fundingStage,totalRaised\nWiz,Cybersecurity,Series D,$900M\nSnyk,Cybersecurity,Series G,$849M' }, conversationHistory: [ { role: 'user', content: 'Show me cybersecurity companies' }, { role: 'assistant', content: 'Found 2 cybersecurity companies: Wiz and Snyk' } ], sessionId: 'chat-session-123' // Optional: for cancellation }) }); // Stream response const reader = response.body?.getReader(); const decoder = new TextDecoder(); while (true) { const { done, value } = await reader!.read(); if (done) break; const chunk = decoder.decode(value); const lines = chunk.split('\n\n').filter(l => l.startsWith('data: ')); for (const line of lines) { const event = JSON.parse(line.replace('data: ', '')); switch (event.type) { case 'status': console.log(`[${event.step}] ${event.message}`); break; case 'response': console.log('Answer:', event.message); console.log('Source:', event.source); break; case 'complete': console.log('Done'); break; } } } }; ``` -------------------------------- ### Agent Enrichment Strategy: Enrich Row with Email Filtering (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Demonstrates the AgentEnrichmentStrategy for enriching data rows, specifically filtering out personal email domains like gmail.com and yahoo.com. It includes callbacks for field value retrieval and agent messaging. ```typescript import { AgentEnrichmentStrategy } from '@/lib/strategies/agent-enrichment-strategy'; const strategy = new AgentEnrichmentStrategy( 'sk-your-openai-key', 'fc-your-firecrawl-key' ); // Enrich a row (skips personal emails like gmail.com, yahoo.com) const result = await strategy.enrichRow( { email: 'ceo@acme.com', name: 'John Doe' }, [ { name: 'companyName', displayName: 'Company Name', description: 'Company name', type: 'string', required: true } ], 'email', (field, value) => console.log(`Got ${field}: ${value}`), (message, type) => console.log(`Agent: ${message}`) ); // Personal emails are automatically skipped const skippedResult = await strategy.enrichRow( { email: 'user@gmail.com' }, [...], 'email' ); // Returns: { status: 'skipped', error: 'Personal email domain (gmail.com) - skipped' } ``` -------------------------------- ### Generate Fields from Natural Language (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Uses AI to generate structured field definitions from natural language prompts. It sends a POST request to the /api/generate-fields endpoint with a prompt and returns an array of field schemas. ```typescript const generateFields = async (prompt: string) => { const response = await fetch('/api/generate-fields', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ prompt: 'I need the company name, industry, employee count, funding stage, total amount raised, and the tech stack they use' }) }); const result = await response.json(); // Example response: // { // success: true, // data: { // fields: [ // { displayName: 'Company Name', description: 'Official registered name of the company', type: 'string', examples: ['Stripe', 'OpenAI'] }, // { displayName: 'Industry', description: 'Primary industry or sector the company operates in', type: 'string', examples: ['Fintech', 'AI/ML'] }, // { displayName: 'Employee Count', description: 'Total number of employees', type: 'number', examples: ['500', '1000'] }, // { displayName: 'Funding Stage', description: 'Most recent funding round', type: 'string', examples: ['Series A', 'Series B'] }, // { displayName: 'Total Raised', description: 'Total funding amount raised', type: 'string', examples: ['$50M', '$100M'] }, // { displayName: 'Tech Stack', description: 'Technologies and frameworks used', type: 'array', examples: ['React', 'Node.js', 'PostgreSQL'] } // ], // interpretation: 'Generated 6 fields for company profiling including basic info, funding details, and technical infrastructure' // } // } return result.data.fields; }; ``` -------------------------------- ### Enrichment API - POST /api/enrich Source: https://context7.com/firecrawl/fire-enrich/llms.txt Processes CSV rows through the multi-agent system, streaming results in real-time via Server-Sent Events. It accepts rows of data, requested fields, and the email column identifier, then orchestrates specialized agents to gather company information. ```APIDOC ## POST /api/enrich ### Description Processes CSV rows through the multi-agent system, streaming results in real-time via Server-Sent Events. It accepts rows of data, requested fields, and the email column identifier, then orchestrates specialized agents to gather company information. ### Method POST ### Endpoint /api/enrich ### Parameters #### Query Parameters - **sessionId** (string) - Optional - The ID of the enrichment session to cancel. #### Request Body - **rows** (array) - Required - An array of objects, where each object represents a row of data to be enriched. Each object should contain at least an email field. - **fields** (array) - Required - An array of enrichment field definitions. Each field object should include `name`, `displayName`, `description`, `type`, and `required` properties. - **emailColumn** (string) - Required - The name of the column in the `rows` that contains the email addresses. - **nameColumn** (string) - Optional - The name of the column in the `rows` that contains the company or person name, used to improve enrichment accuracy. ### Request Example ```json { "rows": [ { "email": "ceo@wiz.io", "name": "Assaf Rappaport" }, { "email": "founder@firecrawl.dev", "name": "Eric Ciarla" } ], "fields": [ { "name": "companyName", "displayName": "Company Name", "description": "Official company name", "type": "string", "required": true }, { "name": "industry", "displayName": "Industry", "description": "Primary industry classification", "type": "string", "required": false }, { "name": "fundingStage", "displayName": "Funding Stage", "description": "Latest funding round (Seed, Series A, etc)", "type": "string", "required": false }, { "name": "employeeCount", "displayName": "Employee Count", "description": "Number of employees", "type": "number", "required": false }, { "name": "techStack", "displayName": "Tech Stack", "description": "Technologies used", "type": "array", "required": false } ], "emailColumn": "email", "nameColumn": "name" } ``` ### Headers - **Content-Type**: `application/json` - **X-OpenAI-API-Key**: `your-openai-key` (Optional if set in env) - **X-Firecrawl-API-Key**: `your-firecrawl-key` (Optional if set in env) ### Response #### Success Response (200 OK) - The response is a Server-Sent Events (SSE) stream. Each event is a JSON object with a `type` field indicating the event type: - **session**: `{ "type": "session", "sessionId": "string" }` - **pending**: `{ "type": "pending", "rowIndex": number, "totalRows": number }` - **processing**: `{ "type": "processing", "rowIndex": number }` - **agent_progress**: `{ "type": "agent_progress", "messageType": "string", "message": "string" }` - **result**: `{ "type": "result", "rowIndex": number, "result": { "enrichments": object } }` - **complete**: `{ "type": "complete" }` - **error**: `{ "type": "error", "error": "string" }` #### Response Example (for `result` event) ```json { "type": "result", "rowIndex": 0, "result": { "enrichments": { "companyName": { "value": "Wiz", "confidence": 0.95, "source": "https://wiz.io/about" }, "industry": { "value": "Cybersecurity", "confidence": 0.9, "source": "https://techcrunch.com/..." }, "fundingStage": { "value": "Series D", "confidence": 0.85, "source": "https://crunchbase.com/..." }, "employeeCount": { "value": 1500, "confidence": 0.8, "source": "https://wiz.io/careers" } } } } ``` ``` -------------------------------- ### Scrape URL(s) with Firecrawl (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Provides a direct interface to Firecrawl for scraping web content. Supports scraping single URLs or multiple URLs in batch, returning content in markdown, HTML, and metadata formats. ```typescript // Scrape a single URL const scrapeSingle = async (url: string) => { const response = await fetch('/api/scrape', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://stripe.com/about', formats: ['markdown', 'html'] // Optional: defaults to markdown }) }); const result = await response.json(); // { // success: true, // data: { // markdown: '# About Stripe\n\nStripe is a financial infrastructure platform...', // html: '...', // metadata: { title: 'About - Stripe', description: '...' } // } // } return result; }; // Batch scrape multiple URLs const scrapeBatch = async (urls: string[]) => { const response = await fetch('/api/scrape', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ urls: [ 'https://stripe.com/about', 'https://stripe.com/customers', 'https://stripe.com/blog' ] }) }); return response.json(); }; ``` -------------------------------- ### Agent Orchestrator - enrichRow Source: https://context7.com/firecrawl/fire-enrich/llms.txt The AgentOrchestrator coordinates multiple specialized agents in a sequential pipeline to enrich data rows. ```APIDOC ## POST /agent-orchestrator/enrichRow ### Description Coordinates multiple specialized agents in a sequential pipeline. Each agent phase builds upon the context gathered by previous phases for maximum accuracy. ### Method POST ### Endpoint /agent-orchestrator/enrichRow ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **apiKey** (string) - Your Firecrawl API key. - **openaiApiKey** (string) - Your OpenAI API key. - **rowData** (object) - The data for the row to enrich. - **schema** (array) - An array of field definitions for enrichment. - **name** (string) - The name of the field. - **displayName** (string) - The display name of the field. - **description** (string) - A description of the field. - **type** (string) - The data type of the field. - **required** (boolean) - Whether the field is required. - **emailColumnName** (string) - The name of the column containing the email address. - **progressCallback** (function, optional) - A callback function for progress updates. - **statusCallback** (function, optional) - A callback function for agent status updates. ### Request Example ```typescript import { AgentOrchestrator } from '@/lib/agent-architecture'; const orchestrator = new AgentOrchestrator( 'fc-your-firecrawl-api-key', 'sk-your-openai-api-key' ); const result = await orchestrator.enrichRow( { email: 'founder@openai.com', name: 'Sam Altman' }, // Row data [ { name: 'companyName', displayName: 'Company Name', description: 'Official company name', type: 'string', required: true }, { name: 'industry', displayName: 'Industry', description: 'Industry classification', type: 'string', required: false }, { name: 'headquarters', displayName: 'Headquarters', description: 'HQ location', type: 'string', required: false }, { name: 'fundingStage', displayName: 'Funding Stage', description: 'Latest funding round', type: 'string', required: false }, { name: 'ceo', displayName: 'CEO', description: 'Chief Executive Officer name', type: 'string', required: false } ], 'email', // Email column name (field, value) => console.log(`Progress: ${field} = ${value}`), // Optional progress callback (message, type) => console.log(`[${type}] ${message}`) // Optional agent status callback ); ``` ### Response #### Success Response (200) - **rowIndex** (number) - The index of the enriched row. - **originalData** (object) - The original data of the row. - **enrichments** (object) - An object containing the enrichment results for each field. - **field** (string) - The name of the enriched field. - **value** (any) - The enriched value. - **confidence** (number) - The confidence score of the enrichment. - **source** (string, optional) - The source URL for the enrichment. - **sourceContext** (array, optional) - Context from the source. - **status** (string) - The status of the enrichment process (e.g., 'completed'). #### Response Example ```json { "rowIndex": 0, "originalData": { "email": "founder@openai.com", "name": "Sam Altman" }, "enrichments": { "companyName": { "field": "companyName", "value": "OpenAI", "confidence": 0.95, "source": "https://openai.com", "sourceContext": [...] }, "industry": { "field": "industry", "value": "Artificial Intelligence", "confidence": 0.9, "source": "https://..." }, "headquarters": { "field": "headquarters", "value": "San Francisco, CA", "confidence": 0.85 }, "fundingStage": { "field": "fundingStage", "value": "Series Unknown (Private)", "confidence": 0.7 }, "ceo": { "field": "ceo", "value": "Sam Altman", "confidence": 0.95 } }, "status": "completed" } ``` ``` -------------------------------- ### Web Scraping and Search Service - TypeScript Source: https://context7.com/firecrawl/fire-enrich/llms.txt Utilizes the FirecrawlService for web scraping and search operations. It includes features like automatic retries, rate limit handling, and options for scraping content. ```typescript import { FirecrawlService } from '@/lib/services/firecrawl'; const firecrawl = new FirecrawlService('fc-your-api-key'); // Search the web const searchResults = await firecrawl.search( '"Stripe" funding "series" investors', { limit: 5, scrapeContent: true } ); // Returns: Array<{ url, title, description, markdown, html, links, metadata }> // Search with multiple queries (deduplicates results) const combinedResults = await firecrawl.searchWithMultipleQueries( [ 'Stripe company funding round', 'site:stripe.com about investors', 'Stripe Series H valuation 2023' ], { limit: 3, scrapeContent: true } ); // Scrape a specific URL const pageContent = await firecrawl.scrapeUrl('https://stripe.com/about'); // Returns: { data: { markdown: '...', html: '...' } } ``` -------------------------------- ### Configure Fire Enrich Limits (TypeScript) Source: https://github.com/firecrawl/fire-enrich/blob/main/README.md Defines configuration for Fire Enrich, including CSV row and column limits, and the maximum number of fields per enrichment. It enables 'Unlimited Mode' based on environment variables (FIRE_ENRICH_UNLIMITED or NODE_ENV). ```typescript const isUnlimitedMode = process.env.FIRE_ENRICH_UNLIMITED === 'true' || process.env.NODE_ENV === 'development'; export const FIRE_ENRICH_CONFIG = { CSV_LIMITS: { MAX_ROWS: isUnlimitedMode ? Infinity : 15, MAX_COLUMNS: isUnlimitedMode ? Infinity : 5, }, REQUEST_LIMITS: { MAX_FIELDS_PER_ENRICHMENT: isUnlimitedMode ? 50 : 10, }, } as const; ``` -------------------------------- ### Field Generation API - POST /api/generate-fields Source: https://context7.com/firecrawl/fire-enrich/llms.txt This API endpoint uses AI to generate structured field definitions from natural language descriptions. It converts user requests into properly typed field schemas. ```APIDOC ## POST /api/generate-fields ### Description Uses AI to generate structured field definitions from natural language descriptions. Converts user requests like "I want company size and funding info" into properly typed field schemas. ### Method POST ### Endpoint /api/generate-fields ### Parameters #### Request Body - **prompt** (string) - Required - A natural language description of the fields to generate. ### Request Example ```json { "prompt": "I need the company name, industry, employee count, funding stage, total amount raised, and the tech stack they use" } ``` ### Response #### Success Response (200) - **success** (boolean) - Indicates if the operation was successful. - **data** (object) - Contains the generated fields and interpretation. - **fields** (array) - An array of field objects. - **displayName** (string) - The user-friendly name of the field. - **description** (string) - A description of the field. - **type** (string) - The data type of the field (e.g., 'string', 'number', 'array'). - **examples** (array) - Example values for the field. - **interpretation** (string) - A summary of the generated fields. #### Response Example ```json { "success": true, "data": { "fields": [ { "displayName": "Company Name", "description": "Official registered name of the company", "type": "string", "examples": ["Stripe", "OpenAI"] }, { "displayName": "Industry", "description": "Primary industry or sector the company operates in", "type": "string", "examples": ["Fintech", "AI/ML"] }, { "displayName": "Employee Count", "description": "Total number of employees", "type": "number", "examples": ["500", "1000"] }, { "displayName": "Funding Stage", "description": "Most recent funding round", "type": "string", "examples": ["Series A", "Series B"] }, { "displayName": "Total Raised", "description": "Total funding amount raised", "type": "string", "examples": ["$50M", "$100M"] }, { "displayName": "Tech Stack", "description": "Technologies and frameworks used", "type": "array", "examples": ["React", "Node.js", "PostgreSQL"] } ], "interpretation": "Generated 6 fields for company profiling including basic info, funding details, and technical infrastructure" } } ``` ``` -------------------------------- ### Orchestrate Agent for Row Enrichment - TypeScript Source: https://context7.com/firecrawl/fire-enrich/llms.txt Initializes and uses the AgentOrchestrator to enrich a single row of data. It takes row data, field definitions, and optional callbacks for progress and status updates. ```typescript import { AgentOrchestrator } from '@/lib/agent-architecture'; // Initialize the orchestrator const orchestrator = new AgentOrchestrator( 'fc-your-firecrawl-api-key', 'sk-your-openai-api-key' ); // Enrich a single row const result = await orchestrator.enrichRow( { email: 'founder@openai.com', name: 'Sam Altman' }, // Row data [ { name: 'companyName', displayName: 'Company Name', description: 'Official company name', type: 'string', required: true }, { name: 'industry', displayName: 'Industry', description: 'Industry classification', type: 'string', required: false }, { name: 'headquarters', displayName: 'Headquarters', description: 'HQ location', type: 'string', required: false }, { name: 'fundingStage', displayName: 'Funding Stage', description: 'Latest funding round', type: 'string', required: false }, { name: 'ceo', displayName: 'CEO', description: 'Chief Executive Officer name', type: 'string', required: false } ], 'email', // Email column name (field, value) => console.log(`Progress: ${field} = ${value}`), // Optional progress callback (message, type) => console.log(`[${type}] ${message}`) // Optional agent status callback ); // Result structure: // { // rowIndex: 0, // originalData: { email: 'founder@openai.com', name: 'Sam Altman' }, // enrichments: { // companyName: { field: 'companyName', value: 'OpenAI', confidence: 0.95, source: 'https://openai.com', sourceContext: [...] }, // industry: { field: 'industry', value: 'Artificial Intelligence', confidence: 0.9, source: 'https://...' }, // headquarters: { field: 'headquarters', value: 'San Francisco, CA', confidence: 0.85 }, // fundingStage: { field: 'fundingStage', value: 'Series Unknown (Private)', confidence: 0.7 }, // ceo: { field: 'ceo', value: 'Sam Altman', confidence: 0.95 } // }, // status: 'completed' // } ``` -------------------------------- ### Web Scraping API - POST /api/scrape Source: https://context7.com/firecrawl/fire-enrich/llms.txt This API provides a direct interface to Firecrawl for scraping single URLs or batch scraping multiple URLs. It returns markdown content, HTML, and metadata. ```APIDOC ## POST /api/scrape ### Description Direct interface to Firecrawl for scraping single URLs or batch scraping multiple URLs. Returns markdown content, HTML, and metadata. ### Method POST ### Endpoint /api/scrape ### Parameters #### Request Body - **url** (string) - Optional - The URL to scrape (use either `url` or `urls`). - **urls** (array of strings) - Optional - An array of URLs to scrape in batch (use either `url` or `urls`). - **formats** (array of strings) - Optional - An array specifying the desired output formats (e.g., `['markdown', 'html']`). Defaults to `['markdown']`. ### Request Example (Single URL) ```json { "url": "https://stripe.com/about", "formats": ["markdown", "html"] } ``` ### Request Example (Batch URLs) ```json { "urls": [ "https://stripe.com/about", "https://stripe.com/customers", "https://stripe.com/blog" ] } ``` ### Response #### Success Response (200) - **success** (boolean) - Indicates if the operation was successful. - **data** (object) - Contains the scraped content and metadata. - **markdown** (string) - The scraped content in Markdown format. - **html** (string) - The scraped content in HTML format. - **metadata** (object) - Metadata about the scraped page. - **title** (string) - The title of the page. - **description** (string) - The meta description of the page. #### Response Example ```json { "success": true, "data": { "markdown": "# About Stripe\n\nStripe is a financial infrastructure platform...", "html": "...", "metadata": { "title": "About - Stripe", "description": "..." } } } ``` ``` -------------------------------- ### Firecrawl Service - Web Scraping and Search Source: https://context7.com/firecrawl/fire-enrich/llms.txt The FirecrawlService provides web scraping and search capabilities with automatic retry logic and rate limit handling. ```APIDOC ## Firecrawl Service API ### Description Provides web scraping and search capabilities with automatic retry logic and rate limit handling. ### Method POST (for search and scrapeUrl) ### Endpoint - `/search` - `/searchWithMultipleQueries` - `/scrapeUrl` ### Parameters #### Query Parameters (for search and searchWithMultipleQueries) - **query** (string) - The search query. - **options** (object, optional) - Search options. - **limit** (number) - The maximum number of results to return. - **scrapeContent** (boolean) - Whether to scrape the content of the search results. #### Request Body (for scrapeUrl) - **url** (string) - The URL to scrape. ### Request Example ```typescript import { FirecrawlService } from '@/lib/services/firecrawl'; const firecrawl = new FirecrawlService('fc-your-api-key'); // Search the web const searchResults = await firecrawl.search( '"Stripe" funding "series" investors', { limit: 5, scrapeContent: true } ); // Returns: Array<{ url, title, description, markdown, html, links, metadata }> // Search with multiple queries (deduplicates results) const combinedResults = await firecrawl.searchWithMultipleQueries( [ 'Stripe company funding round', 'site:stripe.com about investors', 'Stripe Series H valuation 2023' ], { limit: 3, scrapeContent: true } ); // Scrape a specific URL const pageContent = await firecrawl.scrapeUrl('https://stripe.com/about'); // Returns: { data: { markdown: '...', html: '...' } } ``` ### Response #### Success Response (200) - **search**: Returns an array of search result objects, each potentially containing `url`, `title`, `description`, `markdown`, `html`, `links`, and `metadata`. - **searchWithMultipleQueries**: Returns a deduplicated array of search result objects similar to `search`. - **scrapeUrl**: Returns an object with a `data` property, which contains `markdown` and `html` content of the scraped page. #### Response Example (scrapeUrl) ```json { "data": { "markdown": "# About Stripe\n\nStripe is a financial services and software as a service company...", "html": "
Stripe is a financial services...
" } } ``` ``` -------------------------------- ### Enrich Data with Firecrawl API (TypeScript) Source: https://context7.com/firecrawl/fire-enrich/llms.txt Client-side function to send data to the /api/enrich endpoint for AI-powered company data enrichment. It handles POST requests with CSV rows and desired fields, and processes real-time results streamed via Server-Sent Events. ```typescript const enrichData = async (csvRows: Record