FireScraper

https://firescraper.com/docs
Admin
FireScraper is a web scraping API service for crawling and extracting clean Markdown content from...
Tokens:5,402
Snippets:42
Trust Score:5
Update:3 months ago
Show doc for...
Tokens:
Raw
### Start a Web Crawl with FireScraper Python SDK

Source: https://firescraper.com/docs

Initialize the FireScraper client and start a web crawl using Python. This example demonstrates crawl configuration and result retrieval.

```python
from firescraper import FireScraper

client = FireScraper("fsk_your_api_key")

# Start a crawl
session = client.scrape(
    name="Docs crawl",
    urls=["https://docs.example.com/"],
    max_depth=2,
    scraper="article",
)

# Wait for it to finish
result = client.wait_for_completion(
    session.id,
    on_progress=lambda s: print(f"{s.counts.success} pages scraped"),
)

# Download clean Markdown for your RAG pipeline
download = client.get_results(session.id, format="markdown")
```

--------------------------------

### Install FireScraper SDK for Node.js

Source: https://firescraper.com/docs

Install the FireScraper SDK using npm. This is the first step to using the TypeScript client.

```bash
npm install @firescraper/sdk
```

--------------------------------

### Start a Web Crawl with FireScraper TypeScript SDK

Source: https://firescraper.com/docs

Initialize the FireScraper client and start a web crawl. This example shows how to specify crawl parameters like URLs and maximum depth.

```typescript
import { FireScraper } from '@firescraper/sdk';

const client = new FireScraper('fsk_your_api_key');

// Start a crawl
const session = await client.scrape({
  name: 'Docs crawl',
  urls: ['https://docs.example.com/'],
  maxDepth: 2,
  scraper: 'article',
});

// Wait for it to finish
const result = await client.waitForCompletion(session.id, {
  onProgress: (s) => console.log(`${s.counts.success} pages scraped`),
});

// Download clean Markdown for your RAG pipeline
const download = await client.getResults(session.id, 'markdown');
```

--------------------------------

### Install FireScraper for Python

Source: https://firescraper.com/docs

Install the FireScraper Python package using pip. This is required to use the Python client.

```bash
pip install firescraper
```

--------------------------------

### Webhook Payload Example

Source: https://firescraper.com/docs

This is an example of the JSON payload sent to a configured webhook URL when a crawl session is completed. It includes event details, session information, and a list of generated files.

```json
{
  "event": "session.completed",
  "occurredAt": "2026-05-18T14:30:00.000Z",
  "sessionId": "SESSION_ID",
  "session": {
    "id": "SESSION_ID",
    "name": "Vendor docs crawl",
    "status": "done"
  },
  "files": [
    { "format": "csv", "fileName": "corpus-csv.csv" },
    { "format": "zip", "fileName": "corpus-zip.zip" }
  ]
}
```

--------------------------------

### Start a crawl (Python SDK)

Source: https://firescraper.com/docs

Initiates a web crawl with specified parameters using the Python SDK. You can define the crawl name, target URLs, maximum depth, and the scraper type.

```APIDOC
## Start a crawl (Python SDK)

### Description
Initiates a web crawl with specified parameters using the Python SDK. You can define the crawl name, target URLs, maximum depth, and the scraper type.

### Method
`client.scrape(name, urls, max_depth, scraper)`

### Parameters
- **name** (string) - Required - A name for the crawl.
- **urls** (list) - Required - A list of URLs to start crawling from.
- **max_depth** (int) - Optional - The maximum depth to crawl.
- **scraper** (string) - Optional - The type of scraper to use (e.g., 'article').

### Request Example
```python
session = client.scrape(
    name="Docs crawl",
    urls=["https://docs.example.com/"],
    max_depth=2,
    scraper="article",
)
```

### Response
#### Success Response (200)
- **session** (object) - An object containing information about the initiated crawl session.
  - **id** (string) - The unique identifier for the crawl session.
```

--------------------------------

### Start a crawl (TypeScript SDK)

Source: https://firescraper.com/docs

Initiates a web crawl with specified parameters using the TypeScript SDK. You can define the crawl name, target URLs, maximum depth, and the scraper type.

```APIDOC
## Start a crawl (TypeScript SDK)

### Description
Initiates a web crawl with specified parameters using the TypeScript SDK. You can define the crawl name, target URLs, maximum depth, and the scraper type.

### Method
`client.scrape(options)`

### Parameters
- **options** (object) - Required - Configuration for the crawl.
  - **name** (string) - Required - A name for the crawl.
  - **urls** (string[]) - Required - An array of URLs to start crawling from.
  - **maxDepth** (number) - Optional - The maximum depth to crawl.
  - **scraper** (string) - Optional - The type of scraper to use (e.g., 'article').

### Request Example
```typescript
const session = await client.scrape({
  name: 'Docs crawl',
  urls: ['https://docs.example.com/'],
  maxDepth: 2,
  scraper: 'article',
});
```

### Response
#### Success Response (200)
- **session** (object) - An object containing information about the initiated crawl session.
  - **id** (string) - The unique identifier for the crawl session.
```

--------------------------------

### Initiate a scrape session via REST API

Source: https://firescraper.com/docs

Use this curl command to start a new web scraping session. It includes common options like name, URLs, depth, scraper type, and content selection. A webhook can be configured for completion notifications.

```bash
curl -X POST https://firescraper.com/api/v1/scrape \
  -H "Authorization: Bearer fsk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Vendor docs crawl",
    "urls": ["https://docs.vendor.com/"],
    "maxDepth": 2,
    "scraper": "article",
    "respectRobotsTxt": true,
    "contentSelector": "main article",
    "webhookUrl": "https://example.com/webhook"
  }'
```

--------------------------------

### API Key Authentication Header

Source: https://firescraper.com/docs

Every request to the API requires an API key for authentication. Ensure your key starts with `fsk_`.

```http
Authorization: Bearer fsk_your_api_key
```

--------------------------------

### Wait for crawl completion (Python SDK)

Source: https://firescraper.com/docs

Waits for a previously started crawl session to complete and provides progress updates.

```APIDOC
## Wait for crawl completion (Python SDK)

### Description
Waits for a previously started crawl session to complete and provides progress updates.

### Method
`client.wait_for_completion(session_id, on_progress)`

### Parameters
- **session_id** (string) - Required - The ID of the crawl session to wait for.
- **on_progress** (function) - Optional - A callback function that receives progress updates.

### Request Example
```python
result = client.wait_for_completion(
    session.id,
    on_progress=lambda s: print(f"{s.counts.success} pages scraped"),
)
```

### Response
#### Success Response (200)
- **result** (object) - The final result of the crawl.
```

--------------------------------

### GET /api/v1/sessions/:id

Source: https://firescraper.com/docs

Retrieves the status, page counts, and queue depth for a specific scraping session.

```APIDOC
## GET /api/v1/sessions/:id

### Description
Get session status, page counts, and queue depth.

### Method
GET

### Endpoint
/api/v1/sessions/:id

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the session to retrieve.

### Response
#### Success Response (200)
- **session** (object) - Information about the session.
  - **id** (string) - The session ID.
  - **name** (string) - The name of the session.
  - **status** (string) - The current status of the session.
  - **downloadFilesReady** (boolean) - Indicates if download files are ready.
- **counts** (object) - Counts of pages processed.
  - **success** (number) - Number of successfully processed pages.
  - **warning** (number) - Number of pages with warnings.
  - **error** (number) - Number of pages with errors.
  - **total** (number) - Total number of pages processed.
- **processing** (object) - Information about ongoing processing.
  - **serverInstancesCount** (number) - Number of server instances working on the session.
  - **queueLength** (number) - The number of items in the processing queue.

#### Response Example
```json
{
  "session": {
    "id": "SESSION_ID",
    "name": "Vendor docs crawl",
    "status": "in-progress",
    "downloadFilesReady": false
  },
  "counts": {
    "success": 124, "warning": 3,
    "error": 1, "total": 128
  },
  "processing": {
    "serverInstancesCount": 3,
    "queueLength": 41
  }
}
```
```

--------------------------------

### GET /api/v1/sessions/:id/results

Source: https://firescraper.com/docs

Lists available export files for a completed session or downloads results in a specified format. Supports partial downloads mid-crawl.

```APIDOC
## GET /api/v1/sessions/:id/results

### Description
List available export files after crawl completes or download results in a specific format. Supports partial downloads mid-crawl.

### Method
GET

### Endpoint
/api/v1/sessions/:id/results

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the session.

#### Query Parameters
- **format** (string) - Optional - The desired format for the results (e.g., "markdown", "csv", "documents", "extracted").
- **partial** (boolean) - Optional - If true, downloads pages scraped so far mid-crawl.

### Request Example
```bash
# List available files
curl https://firescraper.com/api/v1/sessions/SESSION_ID/results \
  -H "Authorization: Bearer fsk_your_api_key"

# Download a specific format
curl -L "https://firescraper.com/api/v1/sessions/SESSION_ID/results?format=markdown" \
  -H "Authorization: Bearer fsk_your_api_key" \
  -o corpus.md

# Partial export (mid-crawl)
curl -L "https://firescraper.com/api/v1/sessions/SESSION_ID/results?partial=true&format=csv" \
  -H "Authorization: Bearer fsk_your_api_key" \
  -o partial.csv
```
```

--------------------------------

### Wait for crawl completion (TypeScript SDK)

Source: https://firescraper.com/docs

Waits for a previously started crawl session to complete and provides progress updates.

```APIDOC
## Wait for crawl completion (TypeScript SDK)

### Description
Waits for a previously started crawl session to complete and provides progress updates.

### Method
`client.waitForCompletion(sessionId, options)`

### Parameters
- **sessionId** (string) - Required - The ID of the crawl session to wait for.
- **options** (object) - Optional - Options for waiting.
  - **onProgress** (function) - Optional - A callback function that receives progress updates.

### Request Example
```typescript
const result = await client.waitForCompletion(session.id, {
  onProgress: (s) => console.log(`${s.counts.success} pages scraped`),
});
```

### Response
#### Success Response (200)
- **result** (object) - The final result of the crawl.
```

--------------------------------

### Get scrape session status via REST API

Source: https://firescraper.com/docs

This curl command retrieves the current status, counts, and queue depth of a specific scrape session using its ID.

```bash
curl https://firescraper.com/api/v1/sessions/SESSION_ID \
  -H "Authorization: Bearer fsk_your_api_key"
```

--------------------------------

### Feed a RAG pipeline with scraped data

Source: https://firescraper.com/docs

Use this snippet to scrape articles, wait for completion, and then process the results by feeding them into a vector store. Ensure the 'documents' format is available and the vector store is configured.

```typescript
const session = await client.scrape({
  name: 'Knowledge base',
  urls: ['https://docs.example.com/'],
  maxDepth: 4,
  scraper: 'article',
  respectRobotsTxt: true,
});

await client.waitForCompletion(session.id);
const docs = await client.getResults(session.id, 'documents');
const text = new TextDecoder().decode(docs.data);

for (const line of text.split('\n').filter(Boolean)) {
  const doc = JSON.parse(line);
  await vectorStore.upsert(doc.document_id, doc.text);
}
```

--------------------------------

### POST /api/v1/scrape

Source: https://firescraper.com/docs

Initiates a new web scraping session. You can specify URLs, depth, scraper type, and other options. A session ID is returned immediately.

```APIDOC
## POST /api/v1/scrape

### Description
Start a new crawl. Returns the session ID immediately.

### Method
POST

### Endpoint
/api/v1/scrape

### Parameters
#### Request Body
- **name** (string) - Required - Project name.
- **urls** (string[]) - Required - One or more seed URLs.
- **ignoreUrls** (string[]) - Optional - URLs to exclude.
- **maxDepth** (number) - Optional - Link-hop depth (0 = seed only).
- **minTextLength** (number) - Optional - Minimum word count per page.
- **scraper** (string) - Optional - Extraction mode. Options: "article", "full". Default: article.
- **uniqueTextDownloads** (boolean) - Optional - Deduplicate text content.
- **respectRobotsTxt** (boolean) - Optional - Honour robots.txt rules.
- **contentSelector** (string) - Optional - CSS selector to restrict extraction. Max 500 chars.
- **webhookUrl** (string) - Optional - POST callback when crawl finishes.
- **extractionSchema** (string (JSON)) - Optional - JSON Schema for structured extraction.

### Request Example
```json
{
  "name": "Vendor docs crawl",
  "urls": ["https://docs.vendor.com/"],
  "maxDepth": 2,
  "scraper": "article",
  "respectRobotsTxt": true,
  "contentSelector": "main article",
  "webhookUrl": "https://example.com/webhook"
}
```

### Response
#### Success Response (201)
- **id** (string) - The session ID.
- **status** (string) - The status of the session (e.g., "in-progress").
- **message** (string) - A confirmation message.
- **webhookSecret** (string) - The secret for webhook verification.

#### Response Example
```json
{
  "id": "SESSION_ID",
  "status": "in-progress",
  "message": "Scrape session created successfully.",
  "webhookSecret": "whsec_abc123..."
}
```
```

--------------------------------

### List and download scrape results via REST API

Source: https://firescraper.com/docs

These curl commands show how to list available result files for a session, download results in a specific format (e.g., markdown), or retrieve partial results mid-crawl.

```bash
# List available files
curl https://firescraper.com/api/v1/sessions/SESSION_ID/results \
  -H "Authorization: Bearer fsk_your_api_key"

# Download a specific format
curl -L "https://firescraper.com/api/v1/sessions/SESSION_ID/results?format=markdown" \
  -H "Authorization: Bearer fsk_your_api_key" \
  -o corpus.md

# Partial export (mid-crawl)
curl -L "https://firescraper.com/api/v1/sessions/SESSION_ID/results?partial=true&format=csv" \
  -H "Authorization: Bearer fsk_your_api_key" \
  -o partial.csv
```

--------------------------------

### Perform structured data extraction

Source: https://firescraper.com/docs

This snippet demonstrates how to scrape a product catalog and extract structured data based on a defined JSON schema. It waits for the crawl to finish and then retrieves the extracted data.

```typescript
const session = await client.scrape({
  name: 'Product catalog',
  urls: ['https://shop.example.com/products'],
  maxDepth: 2,
  extractionSchema: {
    type: 'object',
    properties: {
      product_name: { type: 'string' },
      price: { type: 'number' },
      in_stock: { type: 'boolean' },
    },
  },
});

await client.waitForCompletion(session.id);
const extracted = await client.getResults(session.id, 'extracted');
```

--------------------------------

### Add MCP Server with API Key

Source: https://firescraper.com/docs

Use this command to add the firescraper MCP server to your AI client, including your API key. This command is run directly in the terminal.

```bash
claude mcp add firescraper \
  --env FIRESCRAPER_API_KEY=fsk_your_api_key \
  -- npx -y @firescraper/mcp
```

--------------------------------

### Download crawl results (Python SDK)

Source: https://firescraper.com/docs

Downloads the results of a completed crawl in a specified format.

```APIDOC
## Download crawl results (Python SDK)

### Description
Downloads the results of a completed crawl in a specified format.

### Method
`client.get_results(session_id, format)`

### Parameters
- **session_id** (string) - Required - The ID of the crawl session.
- **format** (string) - Required - The desired format for the results (e.g., 'markdown').

### Request Example
```python
download = client.get_results(session.id, format="markdown")
```

### Response
#### Success Response (200)
- **download** (any) - The downloaded crawl results in the specified format.
```

--------------------------------

### Configure MCP Server in JSON

Source: https://firescraper.com/docs

Add this configuration to your MCP config file to integrate the firescraper server. Restart your client after updating the config.

```json
{
  "mcpServers": {
    "firescraper": {
      "command": "npx",
      "args": ["-y", "@firescraper/mcp"],
      "env": {
        "FIRESCRAPER_API_KEY": "fsk_your_api_key"
      }
    }
  }
}
```

--------------------------------

### Download crawl results (TypeScript SDK)

Source: https://firescraper.com/docs

Downloads the results of a completed crawl in a specified format.

```APIDOC
## Download crawl results (TypeScript SDK)

### Description
Downloads the results of a completed crawl in a specified format.

### Method
`client.getResults(sessionId, format)`

### Parameters
- **sessionId** (string) - Required - The ID of the crawl session.
- **format** (string) - Required - The desired format for the results (e.g., 'markdown').

### Request Example
```typescript
const download = await client.getResults(session.id, 'markdown');
```

### Response
#### Success Response (200)
- **download** (any) - The downloaded crawl results in the specified format.
```

--------------------------------

### Webhook Signature Verification

Source: https://firescraper.com/docs

This describes the process of verifying webhook delivery signatures. It involves computing an HMAC-SHA256 hash of the timestamp and raw body using a secret and comparing it with the provided signature.

```text
t=<unix_timestamp>,v1=<hmac_sha256_hex>
```

--------------------------------

### Rotate webhook signing secret via REST API

Source: https://firescraper.com/docs

This curl command demonstrates how to rotate the webhook signing secret for a given session by sending a PATCH request.

```bash
curl -X PATCH https://firescraper.com/api/v1/sessions/SESSION_ID \
  -H "Authorization: Bearer fsk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "action": "rotate_webhook_secret" }'
```

--------------------------------

### PATCH /api/v1/sessions/:id

Source: https://firescraper.com/docs

Allows for rotating the webhook signing secret for a given session.

```APIDOC
## PATCH /api/v1/sessions/:id

### Description
Rotate the webhook signing secret for a session.

### Method
PATCH

### Endpoint
/api/v1/sessions/:id

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the session.

#### Request Body
- **action** (string) - Required - The action to perform. Must be "rotate_webhook_secret".

### Request Example
```bash
curl -X PATCH https://firescraper.com/api/v1/sessions/SESSION_ID \
  -H "Authorization: Bearer fsk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{ "action": "rotate_webhook_secret" }'
```
```
Tokens:
Raw