### Install Firecrawl SDK for JavaScript/TypeScript Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build/references/sdk-installation.md Use this command to install the JavaScript/TypeScript SDK using npm. Ensure Node.js and npm are installed. ```bash npm install @mendable/firecrawl-js ``` -------------------------------- ### Install Firecrawl CLI and Build Skills Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/SKILL.md Run this command to install the Firecrawl CLI, CLI skills, and build skills. It also initiates the browser authentication flow. ```bash npx -y firecrawl-cli@latest init --all --browser ``` -------------------------------- ### Install Firecrawl SDK for Python Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build/references/sdk-installation.md Use this command to install the Python SDK using pip. Ensure Python and pip are installed. ```bash pip install firecrawl-py ``` -------------------------------- ### Install Firecrawl SDK and Set API Key Source: https://context7.com/firecrawl/skills/llms.txt Begin your integration by installing the appropriate Firecrawl SDK for your project (Node.js or Python) and setting your API key in a .env file. ```bash # 1. Confirm target language/stack and install SDK npm install @mendable/firecrawl-js # or pip install firecrawl-py # 2. Set up environment echo "FIRECRAWL_API_KEY=fc-your-key" >> .env ``` -------------------------------- ### Install Firecrawl Build Skills Directly Source: https://github.com/firecrawl/skills/blob/main/README.md Installs only the Firecrawl build skills directly, without the CLI tools. Use this if you only need the application integration capabilities. ```bash npx skills add firecrawl/skills ``` -------------------------------- ### Self-Hosted Firecrawl API Key and URL Setup Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/references/project-setup.md For self-hosted Firecrawl, configure both the API key and the API URL in your .env file. This setup is crucial for connecting to your custom Firecrawl instance. ```dotenv FIRECRAWL_API_KEY=fc-... FIRECRAWL_API_URL=https://your-firecrawl-instance.example.com ``` -------------------------------- ### Create Minimal JavaScript Integration Source: https://context7.com/firecrawl/skills/llms.txt This snippet demonstrates a basic JavaScript integration using the Firecrawl SDK. Ensure you have the '@mendable/firecrawl-js' package installed and your API key configured in environment variables. ```javascript import FirecrawlApp from '@mendable/firecrawl-js'; import 'dotenv/config'; const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY }); const result = await app.scrapeUrl('https://example.com', { formats: ['markdown'] }); console.log(result.markdown); ``` -------------------------------- ### JavaScript: Paginated Results Extraction with Interact Source: https://context7.com/firecrawl/skills/llms.txt Handle paginated content by first scraping the initial page and then escalating to the interact method to navigate through subsequent pages. Requires API key setup. ```javascript import FirecrawlApp from '@mendable/firecrawl-js'; const app = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY }); // Start with scrape to assess the page const initialScrape = await app.scrapeUrl('https://example.com/products', { formats: ['markdown'] }); // If pagination detected, escalate to interact const interactResult = await app.interact('https://example.com/products', { actions: [ { type: 'click', selector: '.next-page' }, { type: 'wait', milliseconds: 1000 } ], formats: ['markdown'] }); console.log('Page 2 content:', interactResult.markdown); ``` -------------------------------- ### Generate Browser Auth Parameters Source: https://context7.com/firecrawl/skills/llms.txt Generates PKCE parameters and constructs an authentication URL for browser-based authorization when no API key is available. ```bash # Step 1: Generate session ID and PKCE parameters SESSION_ID=$(openssl rand -hex 32) CODE_VERIFIER=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '= ' | head -c 43) CODE_CHALLENGE=$(printf '%s' "$CODE_VERIFIER" | openssl dgst -sha256 -binary | openssl base64 -A | tr '+/' '-_' | tr -d '=') # Step 2: Construct the auth URL for the user to open echo "https://www.firecrawl.dev/cli-auth?code_challenge=$CODE_CHALLENGE&source=coding-agent#session_id=$SESSION_ID" ``` -------------------------------- ### Python: Ingest Documentation from Known URL Source: https://context7.com/firecrawl/skills/llms.txt Use the scrape_url method for documentation ingestion pipelines. Specify formats and optionally use `onlyMainContent` for cleaner extraction. Ensure your API key is loaded from environment variables. ```python from firecrawl import FirecrawlApp import os app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY")) # Scrape documentation page result = app.scrape_url( "https://docs.example.com/api-reference", params={ "formats": ["markdown"], "onlyMainContent": True } ) # Use extracted content for RAG, summarization, or indexing markdown_content = result.get("markdown", "") metadata = result.get("metadata", {}) print(f"Title: {metadata.get('title')}") print(f"Content length: {len(markdown_content)} chars") ``` -------------------------------- ### Search for Web Content Source: https://context7.com/firecrawl/skills/llms.txt Discovers web sources based on a query using the /search endpoint. Can limit the number of results returned. ```bash # Search request via REST API curl -X POST https://api.firecrawl.dev/v1/search \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -d '{ "query": "firecrawl web scraping API documentation", "limit": 5 }' # Response includes search results: # { # "success": true, # "data": [ # { # "url": "https://docs.firecrawl.dev/", # "title": "Firecrawl Documentation", # "snippet": "Firecrawl is a web scraping API..." # } # ] # } ``` -------------------------------- ### Generate Auth Parameters Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/references/auth-flow.md Generates a session ID, code verifier, and code challenge required for the authentication flow. These are used to securely initiate the authorization process. ```bash SESSION_ID=$(openssl rand -hex 32) CODE_VERIFIER=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '= ' | head -c 43) CODE_CHALLENGE=$(printf '%s' "$CODE_VERIFIER" | openssl dgst -sha256 -binary | openssl base64 -A | tr '+/' '-_' | tr -d '=') ``` -------------------------------- ### Python: Research Workflow with Search and Scrape Source: https://context7.com/firecrawl/skills/llms.txt Combine search and scrape for research workflows. Discover relevant URLs using the search method and then scrape content from selected results. Ensure API key is set. ```python from firecrawl import FirecrawlApp import os app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY")) # Step 1: Discover sources via search search_results = app.search( "latest AI agent development frameworks 2024", params={"limit": 3} ) # Step 2: Selectively scrape relevant results for result in search_results.get("data", []): url = result.get("url") scraped = app.scrape_url(url, params={"formats": ["markdown"]}) print(f"Scraped {url}: {len(scraped.get('markdown', ''))} chars") ``` -------------------------------- ### Configure Hosted Firecrawl API Key Source: https://context7.com/firecrawl/skills/llms.txt Sets the environment variable for hosted Firecrawl API access. Avoid hardcoding API keys. ```dotenv # .env file for hosted Firecrawl FIRECRAWL_API_KEY=fc-your-api-key-here ``` -------------------------------- ### Configure Self-Hosted Firecrawl Source: https://context7.com/firecrawl/skills/llms.txt Sets environment variables for self-hosted Firecrawl, including a custom API URL. ```dotenv # .env file for self-hosted Firecrawl FIRECRAWL_API_KEY=fc-your-api-key-here FIRECRAWL_API_URL=https://your-firecrawl-instance.example.com ``` -------------------------------- ### Save API Key to .env Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/references/auth-flow.md Appends the obtained Firecrawl API key to the .env file for use in subsequent operations. Replace 'fc-...' with the actual API key received. ```bash echo "FIRECRAWL_API_KEY=fc-..." >> .env ``` -------------------------------- ### Smoke Test Firecrawl /scrape Endpoint Source: https://context7.com/firecrawl/skills/llms.txt Verify authentication, networking, and SDK wiring with a basic /scrape request. Ensure your API key is correctly configured. ```bash curl -X POST https://api.firecrawl.dev/v1/scrape \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -d '{"url": "https://example.com", "formats": ["markdown"]}' ``` -------------------------------- ### Interact with Dynamic Content using Firecrawl API Source: https://context7.com/firecrawl/skills/llms.txt Use the /interact endpoint for pages requiring user actions like filling forms, clicking buttons, or waiting for content. Ensure the API key is set in your environment. ```bash curl -X POST https://api.firecrawl.dev/v1/interact \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -d '{ "url": "https://example.com/search", "actions": [ {"type": "fill", "selector": "#search-input", "value": "firecrawl"}, {"type": "click", "selector": "#search-button"}, {"type": "wait", "milliseconds": 2000} ], "formats": ["markdown"] }' ``` -------------------------------- ### Set Firecrawl API Key Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build/references/auth-and-env.md Set the `FIRECRAWL_API_KEY` environment variable for hosted Firecrawl. Never hardcode API keys in source files; use `.env` or a secret manager. ```dotenv FIRECRAWL_API_KEY=fc-... ``` -------------------------------- ### Set Firecrawl API URL for Self-Hosted Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build/references/auth-and-env.md For self-hosted Firecrawl, set both `FIRECRAWL_API_KEY` and `FIRECRAWL_API_URL`. Only set `FIRECRAWL_API_URL` if not using the default `https://api.firecrawl.dev`. ```dotenv FIRECRAWL_API_URL=https://your-firecrawl-instance.example.com ``` -------------------------------- ### User Authorization URL Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/references/auth-flow.md The URL to present to the user for completing the browser authorization flow. Ensure the generated code_challenge and session_id are correctly appended. ```text https://www.firecrawl.dev/cli-auth?code_challenge=$CODE_CHALLENGE&source=coding-agent#session_id=$SESSION_ID ``` -------------------------------- ### Smoke Test Firecrawl /search Endpoint Source: https://context7.com/firecrawl/skills/llms.txt Perform a quick test of the /search endpoint with a single query to confirm its functionality. This helps verify basic integration. ```bash curl -X POST https://api.firecrawl.dev/v1/search \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -d '{"query": "test query", "limit": 1}' ``` -------------------------------- ### Poll for API Key after Browser Auth Source: https://context7.com/firecrawl/skills/llms.txt Polls the status endpoint to retrieve the API key after the user completes browser authorization. Saves the key to the .env file upon completion. ```bash # Poll for completion (repeat until status is "complete") curl -X POST https://www.firecrawl.dev/api/auth/cli/status \ -H "Content-Type: application/json" \ -d '{"session_id":"' ``` ```bash "$SESSION_ID"'","code_verifier":"' ``` ```bash "$CODE_VERIFIER""}' # Response when pending: # {"status":"pending"} # Response when complete: # {"status":"complete","apiKey":"fc-...","teamName":"..."} # Step 4: Save the key to .env echo "FIRECRAWL_API_KEY=fc-..." >> .env ``` -------------------------------- ### Scrape Single Page Content Source: https://context7.com/firecrawl/skills/llms.txt Extracts content from a single URL using the /scrape endpoint. Supports markdown, HTML, links, screenshots, or structured data. Use onlyMainContent for cleaner extractions. ```bash # Basic scrape request via REST API curl -X POST https://api.firecrawl.dev/v1/scrape \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -d '{ "url": "https://example.com/docs/getting-started", "formats": ["markdown"], "onlyMainContent": true }' # Response includes: # { # "success": true, # "data": { # "markdown": "# Getting Started\n\nWelcome to the documentation...", # "metadata": { # "title": "Getting Started", # "description": "Learn how to get started..." # } # } # } ``` -------------------------------- ### Poll for Auth Completion Source: https://github.com/firecrawl/skills/blob/main/skills/firecrawl-build-onboarding/references/auth-flow.md Sends a POST request to the status endpoint with the session ID and code verifier to check the authorization status. Continue polling if the status is 'pending'. ```http POST https://www.firecrawl.dev/api/auth/cli/status Content-Type: application/json {"session_id":"$SESSION_ID","code_verifier":"$CODE_VERIFIER"} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.