### Starting Firecrawl Main API Server with pnpm

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command launches the main Firecrawl API server, which handles incoming requests and orchestrates the overall crawling process. It needs to be run from the `apps/api/` directory after ensuring pnpm is correctly installed.

```bash
pnpm run start
```

--------------------------------

### Starting Firecrawl Services with Docker Compose

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command uses Docker Compose to start all necessary Firecrawl services, including Redis, the API server, and workers, in a single, orchestrated setup. It provides a simpler alternative to manually starting each service, requiring Docker and Docker Compose to be installed and the `.env` file to be configured.

```bash
docker compose up
```

--------------------------------

### Installing Firecrawl API Dependencies with pnpm

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command installs all necessary project dependencies for the Firecrawl API service using pnpm. It is crucial to navigate to the `apps/api/` directory before running this command and ensure that pnpm version 9 or higher is installed on your system.

```bash
pnpm install
```

--------------------------------

### Starting Firecrawl API Workers with pnpm

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command starts the worker processes that are responsible for handling and processing all crawl jobs within the Firecrawl API. It must be executed from the `apps/api/` directory. An optional `OPENAI_API_KEY` export is mentioned for enabling LLM-dependent features.

```bash
pnpm run workers
# if you are going to use the [llm-extract feature](https://github.com/mendableai/firecrawl/pull/586/), you should also export OPENAI_API_KEY=sk-______
```

--------------------------------

### Starting Redis Server for Firecrawl

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command initiates the Redis server, which is a fundamental prerequisite for Firecrawl's local operation. Redis is used for managing queues and rate limiting within the application. This command can be executed from any directory within your project.

```bash
redis-server
```

--------------------------------

### Running Playwright Scrape API (Build & Start) - Bash

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/playwright-service-ts/README.md

These commands first build the Playwright Scrape API project for production and then start the application, making it ready to receive scraping requests.

```bash
npm run build
npm start
```

--------------------------------

### Running Firecrawl Production Tests With Authentication

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command runs the production-level test suite for Firecrawl, which includes tests that require authentication. It is used for more comprehensive testing scenarios where authentication features are enabled and need to be validated.

```bash
npm run test:prod
```

--------------------------------

### Installing Firecrawl Python SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet demonstrates how to install the Firecrawl Python SDK using pip, the Python package installer. This is the first step required to use the Firecrawl library in a Python project.

```bash
pip install firecrawl-py
```

--------------------------------

### Installing Dependencies for Playwright Hero Scraper (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/hero-service-ts/README.md

Installs project dependencies using pnpm and sets up Playwright browsers required for the scraping service to function correctly.

```bash
pnpm install
npx playwright install
```

--------------------------------

### Testing Firecrawl API Server with GET Request

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This `curl` command sends a simple GET request to the `/test` endpoint of the locally running Firecrawl API server. Its purpose is to verify the basic functionality and responsiveness of the server. A successful response should return 'Hello, world!'.

```curl
curl -X GET http://localhost:3002/test
```

--------------------------------

### Building and Running Docker Containers

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

These commands are used to build the Docker images defined in `docker-compose.yml` and then start all the services as defined in the same file. This initiates the Firecrawl application locally.

```bash
docker compose build
docker compose up
```

--------------------------------

### Running Firecrawl Local Tests Without Authentication

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This command executes the local test suite for Firecrawl, specifically configured to run without requiring authentication. It is the recommended method for general development and contribution testing, allowing developers to quickly verify changes.

```bash
npm run test:local-no-auth
```

--------------------------------

### Configuring Firecrawl Environment Variables

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This snippet outlines the required and optional environment variables for the Firecrawl API service, typically set in a `.env` file within the `/apps/api/` directory. It includes configurations for worker concurrency, network settings, Redis connections, and flags for database authentication. Optional variables cover integrations with Supabase, various API keys for external services (e.g., Scraping Bee, OpenAI, LlamaParse), and logging/monitoring tools like Slack and Posthog.

```env
# ===== Required ENVS ======
NUM_WORKERS_PER_QUEUE=8
PORT=3002
HOST=0.0.0.0
REDIS_URL=redis://localhost:6379
REDIS_RATE_LIMIT_URL=redis://localhost:6379

## To turn on DB authentication, you need to set up supabase.
USE_DB_AUTHENTICATION=false

# ===== Optional ENVS ======

# Supabase Setup (used to support DB authentication, advanced logging, etc.)
SUPABASE_ANON_TOKEN=
SUPABASE_URL=
SUPABASE_SERVICE_TOKEN=

# Other Optionals
TEST_API_KEY= # use if you've set up authentication and want to test with a real API key
SCRAPING_BEE_API_KEY= #Set if you'd like to use scraping Bee to handle JS blocking
OPENAI_API_KEY= # add for LLM dependednt features (image alt generation, etc.)
BULL_AUTH_KEY= @
PLAYWRIGHT_MICROSERVICE_URL=  # set if you'd like to run a playwright fallback
LLAMAPARSE_API_KEY= #Set if you have a llamaparse key you'd like to use to parse pdfs
SLACK_WEBHOOK_URL= # set if you'd like to send slack server health status messages
POSTHOG_API_KEY= # set if you'd like to send posthog events like job logs
POSTHOG_HOST= # set if you'd like to send posthog events like job logs

```

--------------------------------

### Testing Firecrawl Crawl Endpoint with POST Request

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/CONTRIBUTING.md

This `curl` command demonstrates how to send a POST request to the `/v1/crawl` endpoint, initiating a web crawl for a specified URL. It correctly sets the `Content-Type` header to `application/json` and includes the target URL in the request body, showcasing how to interact with the core crawling functionality.

```curl
curl -X POST http://localhost:3002/v1/crawl \
    -H 'Content-Type: application/json' \
    -d '{
      "url": "https://mendable.ai"
    }'
```

--------------------------------

### Installing Playwright Scrape API Dependencies - Bash

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/playwright-service-ts/README.md

This snippet provides the commands to install the necessary Node.js dependencies for the Playwright Scrape API, including project packages and Playwright browser binaries required for scraping.

```bash
npm install
npx playwright install
```

--------------------------------

### Building and Running Playwright Hero Scraper (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/hero-service-ts/README.md

Builds the Playwright Hero Scrape API service from source and then starts it, making it ready to accept incoming scrape requests on the configured port.

```bash
pnpm run build
pnpm start
```

--------------------------------

### Installing Firecrawl Node.js SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet provides the npm command to install the Firecrawl Node.js SDK. This is the prerequisite step for developing web scraping and crawling applications using Firecrawl in a Node.js environment.

```bash
npm install @mendable/firecrawl-js
```

--------------------------------

### Building Docker Images (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/README.md

This command builds the Docker images for all services defined in the `docker-compose.yml` file. It's a prerequisite before starting the services to ensure all dependencies are correctly packaged and ready for deployment.

```bash
docker-compose build
```

--------------------------------

### Scraping and Crawling Websites with Firecrawl Python SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This Python snippet initializes the FirecrawlApp and demonstrates both `scrape_url` for single-page content extraction and `crawl_url` for multi-page website traversal. It shows how to specify output formats and crawling limits, providing a basic usage example for web data collection.

```python
from firecrawl.firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

# Scrape a website:
scrape_status = app.scrape_url(
  'https://firecrawl.dev', 
  params={'formats': ['markdown', 'html']}
)
print(scrape_status)

# Crawl a website:
crawl_status = app.crawl_url(
  'https://firecrawl.dev', 
  params={
    'limit': 100, 
    'scrapeOptions': {'formats': ['markdown', 'html']}
  },
  poll_interval=30
)
print(crawl_status)
```

--------------------------------

### Scraping and Crawling Websites with Firecrawl Node.js SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This Node.js snippet demonstrates how to initialize the FirecrawlApp and perform both single-page scraping (`scrapeUrl`) and multi-page crawling (`crawlUrl`). It showcases setting output formats and crawl limits, providing a fundamental example for web data retrieval in Node.js applications.

```javascript
import FirecrawlApp, { CrawlParams, CrawlStatusResponse } from '@mendable/firecrawl-js';

const app = new FirecrawlApp({apiKey: "fc-YOUR_API_KEY"});

// Scrape a website
const scrapeResponse = await app.scrapeUrl('https://firecrawl.dev', {
  formats: ['markdown', 'html'],
});

if (scrapeResponse) {
  console.log(scrapeResponse)
}

// Crawl a website
const crawlResponse = await app.crawlUrl('https://firecrawl.dev', {
  limit: 100,
  scrapeOptions: {
    formats: ['markdown', 'html'],
  }
} satisfies CrawlParams, true, 30) satisfies CrawlStatusResponse;

if (crawlResponse) {
  console.log(crawlResponse)
}
```

--------------------------------

### Starting Services in Detached Mode (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/README.md

This command starts all services defined in the `docker-compose.yml` file in detached mode, meaning they will run in the background. This allows the user to continue using the terminal while the services operate, providing a non-blocking execution.

```bash
docker-compose up -d
```

--------------------------------

### Running Playwright Scrape API (Development) - Bash

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/playwright-service-ts/README.md

This command starts the Playwright Scrape API in development mode, which typically includes features like hot-reloading for more efficient development and debugging.

```bash
npm run dev
```

--------------------------------

### Example JSON Response for Firecrawl Search

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object illustrates a typical successful response from the Firecrawl API's search endpoint. It provides an array of search results, each containing the URL, title, and a brief description of the found page.

```json
{
  "success": true,
  "data": [
    {
      "url": "https://mendable.ai",
      "title": "Mendable | AI for CX and Sales",
      "description": "AI for CX and Sales"
    }
  ]
}
```

--------------------------------

### Configuring Environment Variables for Firecrawl

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet illustrates the initial required and optional environment variables for self-hosting Firecrawl. It sets up core parameters like worker count, port, host, and Redis URLs, and shows how to disable database authentication for a basic setup, omitting advanced features like Supabase integration.

```Configuration
# ===== Required ENVS ======
NUM_WORKERS_PER_QUEUE=8
PORT=3002
HOST=0.0.0.0
REDIS_URL=redis://redis:6379
REDIS_RATE_LIMIT_URL=redis://redis:6379

## To turn on DB authentication, you need to set up Supabase.
USE_DB_AUTHENTICATION=false

# ===== Optional ENVS ======

# Supabase Setup (used to support DB authentication, advanced logging, etc.)
SUPABASE_ANON_TOKEN=
SUPABASE_URL=
SUPABASE_SERVICE_TOKEN=
```

--------------------------------

### Extracting Data with Prompt using Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet demonstrates how to use the Firecrawl API's LLM extraction feature without a predefined schema. Instead, it uses a natural language prompt to guide the LLM in extracting relevant information, allowing the LLM to determine the output structure.

```bash
curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev/",
      "formats": ["json"],
      "jsonOptions": {
        "prompt": "Extract the company mission from the page."
      }
    }'
```

--------------------------------

### Checking Docker Container Logs

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This command is used to view the logs of a specific Docker container. It is a crucial troubleshooting step for diagnosing issues when containers exit unexpectedly or fail to start, providing insights into runtime errors.

```bash
docker logs [container_name]
```

--------------------------------

### Extracting Structured Data with Pydantic and Firecrawl Python SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This Python example illustrates how to extract structured data from a URL using Firecrawl's LLM extraction capabilities, integrated with Pydantic schemas. It defines `ArticleSchema` and `TopArticlesSchema` to specify the desired data structure, then uses `scrape_url` with `jsonOptions` to apply the schema for precise data extraction.

```python
from firecrawl.firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="fc-YOUR_API_KEY")

class ArticleSchema(BaseModel):
    title: str
    points: int
    by: str
    commentsURL: str

class TopArticlesSchema(BaseModel):
    top: List[ArticleSchema] = Field(..., max_items=5, description="Top 5 stories")

data = app.scrape_url('https://news.ycombinator.com', {
    'formats': ['json'],
    'jsonOptions': {
        'schema': TopArticlesSchema.model_json_schema()
    }
})
print(data["json"])
```

--------------------------------

### Example JSON Response for LLM Extraction with Schema

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object represents a successful response from the Firecrawl API's LLM extraction endpoint when a schema is used. It includes the raw content, metadata about the scraped page, and the extracted structured data conforming to the specified schema.

```json
{
  "success": true,
  "data": {
    "content": "Raw Content",
    "metadata": {
      "title": "Mendable",
      "description": "Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide",
      "robots": "follow, index",
      "ogTitle": "Mendable",
      "ogDescription": "Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide",
      "ogUrl": "https://mendable.ai/",
      "ogImage": "https://mendable.ai/mendable_new_og1.png",
      "ogLocaleAlternate": [],
      "ogSiteName": "Mendable",
      "sourceURL": "https://mendable.ai/"
    },
    "json": {
      "company_mission": "Train a secure AI on your technical resources that answers customer and employee questions so your team doesn't have to",
      "supports_sso": true,
      "is_open_source": false,
      "is_in_yc": true
    }
  }
}
```

--------------------------------

### Extracting Structured Data with Zod and Firecrawl Node.js SDK

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This Node.js example demonstrates extracting structured data from a URL using Firecrawl's LLM extraction capabilities, integrated with Zod schemas. It defines a Zod schema to specify the desired data structure for the top 5 Hacker News stories, then uses `scrapeUrl` with `jsonOptions` to apply the schema for precise data extraction.

```javascript
import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";

const app = new FirecrawlApp({
  apiKey: "fc-YOUR_API_KEY"
});

// Define schema to extract contents into
const schema = z.object({
  top: z
    .array(
      z.object({
        title: z.string(),
        points: z.number(),
        by: z.string(),
        commentsURL: z.string(),
      })
    )
    .length(5)
    .describe("Top 5 stories on Hacker News"),
});

const scrapeResult = await app.scrapeUrl("https://news.ycombinator.com", {
  jsonOptions: { extractionSchema: schema },
});

console.log(scrapeResult.data["json"]);
```

--------------------------------

### Supabase Client Not Configured Error Symptom

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet displays the error messages that appear in the logs when the Supabase client is not configured. It indicates that Firecrawl attempted to access Supabase for operations like inserting scrape events without proper setup, though scraping and crawling may still function.

```bash
[YYYY-MM-DDTHH:MM:SS.SSSz]ERROR - Attempted to access Supabase client when it's not configured.
[YYYY-MM-DDTHH:MM:SS.SSSz]ERROR - Error inserting scrape event: Error: Supabase client is not configured.
```

--------------------------------

### Authentication Bypass Warning Symptom

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet shows a warning message indicating that authentication is being bypassed. This typically occurs in self-hosted Firecrawl instances when the Supabase client setup is not completed, as API keys are optional for self-hosted usage.

```bash
[YYYY-MM-DDTHH:MM:SS.SSSz]WARN - You're bypassing authentication
```

--------------------------------

### Checking Firecrawl Crawl Job Status (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This cURL GET request demonstrates how to check the status and retrieve the results of a previously initiated Firecrawl job. The request uses the job ID obtained from the initial crawl request to query the API.

```bash
curl -X GET https://api.firecrawl.dev/v1/crawl/123-456-789 \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY'
```

--------------------------------

### Cloning the Repository (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/README.md

This snippet demonstrates how to clone the `localfirecrawl` repository from GitHub using Git and then navigate into the newly cloned directory. This is the essential first step to set up the project locally.

```bash
git clone git@github.com:Ozamatash/localfirecrawl.git
cd localfirecrawl
```

--------------------------------

### Building Go Library as Shared Object - Bash

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/api/sharedLibs/go-html-to-md/README.md

This snippet provides the necessary bash commands to navigate to the Go HTML to Markdown library directory, compile the Go source file into a shared object library (`html-to-markdown.so`) using `go build -buildmode=c-shared`, and then make the resulting shared object executable. This process is crucial for integrating the Go library into applications that require a shared library.

```bash
cd apps/api/src/lib/go-html-to-md
go build -o html-to-markdown.so -buildmode=c-shared html-to-markdown.go
chmod +x html-to-markdown.so
```

--------------------------------

### Initiating a Web Crawl with Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet demonstrates how to initiate a web crawl using the Firecrawl API via a cURL POST request. It specifies the target URL, a limit on the number of pages to crawl, and desired output formats (markdown, HTML). The API returns a job ID to track the crawl's progress.

```bash
curl -X POST https://api.firecrawl.dev/v1/crawl \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer fc-YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev",
      "limit": 100,
      "scrapeOptions": {
        "formats": ["markdown", "html"]
      }
    }'
```

--------------------------------

### Performing Page Actions with Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet illustrates how to use Firecrawl's 'actions' feature to interact with a web page before scraping. It includes a sequence of actions like waiting, clicking, typing text, pressing keys, and taking a screenshot, useful for dynamic content or navigation.

```bash
curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
        "url": "google.com",
        "formats": ["markdown"],
        "actions": [
            {"type": "wait", "milliseconds": 2000},
            {"type": "click", "selector": "textarea[title=\"Search\"]"},
            {"type": "wait", "milliseconds": 2000},
            {"type": "write", "text": "firecrawl"},
            {"type": "wait", "milliseconds": 2000},
            {"type": "press", "key": "ENTER"},
            {"type": "wait", "milliseconds": 3000},
            {"type": "click", "selector": "h3"},
            {"type": "wait", "milliseconds": 3000},
            {"type": "screenshot"}
        ]
    }'
```

--------------------------------

### Mapping Website URLs with Firecrawl API (cURL)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This cURL command demonstrates how to use the Firecrawl API's `/map` endpoint to discover and retrieve most links present on a specified website. It requires an API key for authentication and a target URL.

```bash
curl -X POST https://api.firecrawl.dev/v1/map \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://firecrawl.dev"
    }'
```

--------------------------------

### Configuring Optional Environment Variables

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet lists various optional environment variables for Firecrawl, such as API keys for external services (OpenAI, ScrapingBee, LLamaParse), authentication (BULL_AUTH_KEY), Playwright service URL, and logging/monitoring (Slack, Posthog). These variables allow for extended functionality and integration.

```plaintext
TEST_API_KEY= # use if you've set up authentication and want to test with a real API key
SCRAPING_BEE_API_KEY= # use if you'd like to use as a fallback scraper
OPENAI_API_KEY= # add for LLM-dependent features (e.g., image alt generation)
BULL_AUTH_KEY= @
PLAYWRIGHT_MICROSERVICE_URL=  # set if you'd like to run a playwright fallback
LLAMAPARSE_API_KEY= #Set if you have a llamaparse key you'd like to use to parse pdfs
SLACK_WEBHOOK_URL= # set if you'd like to send slack server health status messages
POSTHOG_API_KEY= # set if you'd like to send posthog events like job logs
POSTHOG_HOST= # set if you'd like to send posthog events like job logs
```

--------------------------------

### Scraping URL Content with Firecrawl API (cURL)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This cURL command demonstrates how to use the Firecrawl API's `/scrape` endpoint to retrieve content from a specified URL. It supports multiple output formats like markdown and HTML, requiring an API key for authentication.

```bash
curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://docs.firecrawl.dev",
      "formats" : ["markdown", "html"]
    }'
```

--------------------------------

### Batch Scraping Multiple URLs with Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet demonstrates how to use the Firecrawl API's batch scraping endpoint to process multiple URLs concurrently. It initiates a batch job, allowing for efficient scraping of several pages and specifying desired output formats like markdown and HTML.

```bash
curl -X POST https://api.firecrawl.dev/v1/batch/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "urls": ["https://docs.firecrawl.dev", "https://docs.firecrawl.dev/sdks/overview"],
      "formats" : ["markdown", "html"]
    }'
```

--------------------------------

### Scraping a URL using Curl (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/hero-service-ts/README.md

Demonstrates how to make a POST request to the /scrape endpoint using curl, providing a target URL and optional parameters such as wait time after load, a total timeout, custom headers, and a CSS selector to check for page readiness.

```bash
curl -X POST http://localhost:3000/scrape \
-H "Content-Type: application/json" \
-d '{
  "url": "https://example.com",
  "wait_after_load": 1000,
  "timeout": 60000,
  "headers": {
    "Custom-Header": "value"
  },
  "check_selector": "#content"
}'
```

--------------------------------

### Scraping URL via Playwright API with cURL - Bash

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/apps/playwright-service-ts/README.md

This cURL command demonstrates how to make a POST request to the Playwright Scrape API endpoint, specifying the target URL, wait times, timeout, custom headers, and a CSS selector to ensure content is loaded.

```bash
curl -X POST http://localhost:3000/scrape \
-H "Content-Type: application/json" \
-d '{
  "url": "https://example.com",
  "wait_after_load": 1000,
  "timeout": 15000,
  "headers": {
    "Custom-Header": "value"
  },
  "check_selector": "#content"
}'
```

--------------------------------

### Performing Web Search with Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet shows how to use the Firecrawl API's search endpoint to combine web search with scraping capabilities. It sends a query and by default returns SERP results (URL, title, description), but can be configured to scrape full page content.

```bash
curl -X POST https://api.firecrawl.dev/v1/search \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "query": "What is Mendable?"
    }'
```

--------------------------------

### Firecrawl Crawl Job Initiation Response (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON snippet illustrates the expected response after successfully initiating a crawl job. It confirms the success of the request and provides a unique job ID along with a URL to monitor the status of the ongoing crawl.

```json
{
  "success": true,
  "id": "123-456-789",
  "url": "https://api.firecrawl.dev/v1/crawl/123-456-789"
}
```

--------------------------------

### Pulling Firecrawl SearXNG Docker Image (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This Bash command is used to pull the latest Docker image of the Firecrawl fork, which includes SearXNG integration, directly from Docker Hub. It provides a straightforward method for users to acquire and deploy the pre-built application without manual compilation.

```bash
docker pull loorisr/firecrawl-searxng:latest
```

--------------------------------

### Scraping API Response Format (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object illustrates the successful response from the Firecrawl `/scrape` endpoint. It includes the scraped content in specified formats (markdown, html) and comprehensive metadata about the source URL, such as title, description, and Open Graph properties.

```json
{
  "success": true,
  "data": {
    "markdown": "Launch Week I is here! [See our Day 2 Release 🚀](https://www.firecrawl.dev/blog/launch-week-i-day-2-doubled-rate-limits)[💥 Get 2 months free...",
    "html": "<!DOCTYPE html><html lang=\"en\" class=\"light\" style=\"color-scheme: light;\"><body class=\"__variable_36bd41 __variable_d7dc5d font-inter ...",
    "metadata": {
      "title": "Home - Firecrawl",
      "description": "Firecrawl crawls and converts any website into clean markdown.",
      "language": "en",
      "keywords": "Firecrawl,Markdown,Data,Mendable,Langchain",
      "robots": "follow, index",
      "ogTitle": "Firecrawl",
      "ogDescription": "Turn any website into LLM-ready data.",
      "ogUrl": "https://www.firecrawl.dev/",
      "ogImage": "https://www.firecrawl.dev/og.png?123",
      "ogLocaleAlternate": [],
      "ogSiteName": "Firecrawl",
      "sourceURL": "https://firecrawl.dev",
      "statusCode": 200
    }
  }
}
```

--------------------------------

### Configuring SearXNG for JSON Output (YAML)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This YAML snippet illustrates how to configure a SearXNG instance to enable JSON as an output format for search results. This setting is crucial for the Firecrawl fork to correctly process data received from SearXNG, ensuring structured data is available.

```yaml
search:
  formats:
    - html
    - json
```

--------------------------------

### Extracting Structured Data with Firecrawl API (cURL)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This cURL command demonstrates how to use the Firecrawl API's `/extract` endpoint to retrieve structured data from one or multiple URLs, including wildcard patterns. It allows specifying a natural language `prompt` and/or a JSON `schema` to define the desired data structure, requiring an API key for authentication.

```bash
curl -X POST https://api.firecrawl.dev/v1/extract \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "urls": [
        "https://firecrawl.dev/*", 
        "https://docs.firecrawl.dev/", 
        "https://www.ycombinator.com/companies"
      ],
      "prompt": "Extract the company mission, whether it is open source, and whether it is in Y Combinator from the page.",
      "schema": {
        "type": "object",
        "properties": {
          "company_mission": {
            "type": "string"
          },
          "is_open_source": {
            "type": "boolean"
          },
          "is_in_yc": {
            "type": "boolean"
          }
        },
        "required": [
          "company_mission",
          "supports_sso",
          "is_open_source",
          "is_in_yc"
        ]
      }
    }'
```

--------------------------------

### Testing Firecrawl API with cURL

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This cURL command demonstrates how to send a POST request to the local Firecrawl `/v1/crawl` endpoint. It includes the necessary content type header and a JSON payload specifying the URL to be crawled, allowing for basic API functionality testing.

```bash
curl -X POST http://localhost:3002/v1/crawl \
      -H 'Content-Type: application/json' \
      -d '{
        "url": "https://mendable.ai"
      }'
```

--------------------------------

### Generating SearXNG Secret Key (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/README.md

This command uses `openssl` to generate a 32-character hexadecimal string, which is required as a secure secret key for the SearXNG service. The generated output should be copied and set as the `SEARXNG_SECRET_KEY` environment variable in the `.env` file.

```bash
openssl rand -hex 32  # Copy the output to SEARXNG_SECRET_KEY in .env
```

--------------------------------

### Updating Docker Compose for TypeScript Playwright Service

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet provides instructions to modify the `docker-compose.yml` file, changing the Playwright service build target from the default `apps/playwright-service` to `apps/playwright-service-ts` to enable the TypeScript version of the Playwright service.

```plaintext
build: apps/playwright-service
        TO
        build: apps/playwright-service-ts
```

--------------------------------

### Testing Firecrawl Search Endpoint (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/README.md

This `curl` command sends a POST request to the Firecrawl API's search endpoint (`http://localhost:3002/v1/search`) with a JSON payload. It tests the search functionality by querying for 'test' and limiting the results to 10, verifying the API's responsiveness and correct operation.

```bash
curl -X POST -H "Content-Type: application/json" -d '{"query": "test", "limit": 10}' http://localhost:3002/v1/search
```

--------------------------------

### Mapping Website URLs with Search Parameter (cURL)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This cURL command demonstrates how to use the Firecrawl API's `/map` endpoint with a `search` parameter to filter discovered URLs. It allows users to find specific URLs within a website that contain the specified search term, returning results ordered by relevance.

```bash
curl -X POST https://api.firecrawl.dev/v1/map \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://firecrawl.dev",
      "search": "docs"
    }'
```

--------------------------------

### Setting Playwright Microservice URL

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/SELF_HOST.md

This snippet shows how to set the `PLAYWRIGHT_MICROSERVICE_URL` in the `.env` file. This configuration is essential for connecting Firecrawl to the TypeScript Playwright service, typically running locally at `http://localhost:3000/scrape`.

```plaintext
PLAYWRIGHT_MICROSERVICE_URL=http://localhost:3000/scrape
```

--------------------------------

### Extracting Structured Data with Schema using Firecrawl API (Bash)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This snippet demonstrates how to use the Firecrawl API's LLM extraction feature to scrape a URL and extract structured data based on a provided JSON schema. It specifies the desired data types (string, boolean) and required fields for the extracted object.

```bash
curl -X POST https://api.firecrawl.dev/v1/scrape \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer YOUR_API_KEY' \
    -d '{
      "url": "https://www.mendable.ai/",
      "formats": ["json"],
      "jsonOptions": {
        "schema": {
          "type": "object",
          "properties": {
            "company_mission": {
                      "type": "string"
            },
            "supports_sso": {
                      "type": "boolean"
            },
            "is_open_source": {
                      "type": "boolean"
            },
            "is_in_yc": {
                      "type": "boolean"
            }
          },
          "required": [
            "company_mission",
            "supports_sso",
            "is_open_source",
            "is_in_yc"
          ]
        }
      }
    }'
```

--------------------------------

### Extract API Initial Response (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object represents the immediate response from the Firecrawl `/extract` endpoint after submitting a request. It indicates the `success` of the request and provides an `id` for tracking the asynchronous extraction process, along with an empty `urlTrace`.

```json
{
  "success": true,
  "id": "44aa536d-f1cb-4706-ab87-ed0386685740",
  "urlTrace": []
}
```

--------------------------------

### Firecrawl Crawl Job Status and Result Response (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON snippet shows the detailed response when checking a crawl job's status. It includes the job's completion status, total pages processed, credits used, expiration time, and an array containing the scraped data (markdown, HTML, and metadata) for each URL.

```json
{
  "status": "completed",
  "total": 36,
  "creditsUsed": 36,
  "expiresAt": "2024-00-00T00:00:00.000Z",
  "data": [
    {
      "markdown": "[Firecrawl Docs home page![light logo](https://mintlify.s3-us-west-1.amazonaws.com/firecrawl/logo/light.svg)!...",
      "html": "<!DOCTYPE html><html lang=\"en\" class=\"js-focus-visible lg:[--scroll-mt:9.5rem]\" data-js-focus-visible=\"\">...",
      "metadata": {
        "title": "Build a 'Chat with website' using Groq Llama 3 | Firecrawl",
        "language": "en",
        "sourceURL": "https://docs.firecrawl.dev/learn/rag-llama3",
        "description": "Learn how to use Firecrawl, Groq Llama 3, and Langchain to build a 'Chat with your website' bot.",
        "ogLocaleAlternate": [],
        "statusCode": 200
      }
    }
  ]
}
```

--------------------------------

### Mapping API Response with Search Results (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object shows the response from the Firecrawl `/map` endpoint when a `search` parameter is used. It returns a filtered list of `links` that match the search term, ordered by relevance, along with the `status` of the operation.

```json
{
  "status": "success",
  "links": [
    "https://docs.firecrawl.dev",
    "https://docs.firecrawl.dev/sdks/python",
    "https://docs.firecrawl.dev/learn/rag-llama3"
  ]
}
```

--------------------------------

### Mapping API Response Format (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object shows the successful response from the Firecrawl `/map` endpoint. It returns a list of URLs discovered on the target website, indicating the `status` of the operation.

```json
{
  "status": "success",
  "links": [
    "https://firecrawl.dev",
    "https://www.firecrawl.dev/pricing",
    "https://www.firecrawl.dev/blog",
    "https://www.firecrawl.dev/playground",
    "https://www.firecrawl.dev/smart-crawl"
  ]
}
```

--------------------------------

### Extract API Final Structured Data Response (JSON)

Source: https://github.com/ozamatash/localfirecrawl/blob/main/firecrawl-searxng/README.md

This JSON object shows the final, structured data response from the Firecrawl `/extract` endpoint, typically retrieved automatically when using SDKs. It contains the extracted data (`company_mission`, `supports_sso`, `is_open_source`, `is_in_yc`) conforming to the specified schema, along with a `success` status.

```json
{
  "success": true,
  "data": {
    "company_mission": "Firecrawl is the easiest way to extract data from the web. Developers use us to reliably convert URLs into LLM-ready markdown or structured data with a single API call.",
    "supports_sso": false,
    "is_open_source": true,
    "is_in_yc": true
  }
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.