Spider

https://spider.cloud/docs/api
Admin
Spider is a web crawling and scraping API designed to extract clean data from websites, handle...
Tokens:42,575
Snippets:252
Trust Score:7.9
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Example response from GET /data/scraper-directory

Source: https://spider.cloud/docs/api/scraper-directory

Example JSON response from the GET /data/scraper-directory endpoint. Shows a single scraper configuration object with fields like domain, path_pattern, display_name, description, category, tags, confidence_score, validation_count, slug, and timestamps, along with pagination metadata.

```json
{
  "data": [
    {
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "domain": "example.com",
      "path_pattern": "/blog/*",
      "display_name": "Example Blog Scraper",
      "description": "Extracts blog posts with title, author, and content.",
      "category": "news",
      "tags": ["blog", "articles"],
      "confidence_score": 0.95,
      "validation_count": 12,
      "slug": "example-com-blog",
      "created_at": "2025-12-01T10:00:00+00:00",
      "updated_at": "2026-01-15T08:30:00+00:00"
    }
  ],
  "total": 1,
  "page": 1,
  "limit": 20,
  "total_pages": 1
}
```

--------------------------------

### Response JSON for GET /data/credits

Source: https://spider.cloud/docs/api/credits

Example JSON response from the credits endpoint, showing the credit balance and user ID.

```json
{
  "data": {
    "id": "8d662167-5a5f-41aa-9cb8-0cbb7d536891",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "credits": 53334,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  }
}
```

--------------------------------

### Screenshot response example

Source: https://spider.cloud/docs/api/screenshot

Example response showing a base64-encoded screenshot in the content field. With binary: true, raw image bytes are returned instead.

```json
[
  {
    "url": "https://example.com",
    "content": "iVBORw0KGgoAAAANSUhEUgAA...",
    "status": 200
  }
]
```

--------------------------------

### cdp_params example for screenshot settings

Source: https://spider.cloud/docs/api/screenshot

Example cdp_params object for adjusting screenshot settings. Sets JPEG format, quality 90, a clip region with scale, and disables fromSurface while enabling captureBeyondViewport.

```json
{
  "format": "jpeg",
  "quality": 90,
  "clip": { "x": 10, "y": 20, "width": 500, "height": 400, "scale": 1.5 },
  "fromSurface": false,
  "captureBeyondViewport": true
}
```

--------------------------------

### Configure multiple provider credentials

Source: https://spider.cloud/docs/api/router

Example JSON body showing how to provide multiple API keys within the router credentials map.

```json
{
  "url": "https://example.com",
  "router": {
    "mode": "fallback",
    "credentials": {
      "ZYTE_API_KEY": "YOUR_ZYTE_API_KEY",
      "FIRECRAWL_API_KEY": "YOUR_FIRECRAWL_API_KEY"
    }
  }
}
```

--------------------------------

### Sample response from /links endpoint

Source: https://spider.cloud/docs/api/links

Example JSON response showing an array of objects, each with url, status, duration_elasped_ms, error, links array, and costs breakdown.

```json
[
  {
    "url": "https://spider.cloud",
    "status": 200,
    "duration_elasped_ms": 698,
    "error": null,
    "links": [
      "https://spider.cloud/docs/api",
      "https://spider.cloud/pricing",
      "https://spider.cloud/docs/quickstart",
      "https://spider.cloud/credits",
      "https://spider.cloud/guides/spider",
      "https://spider.cloud/blog",
      "https://spider.cloud/faq"
    ],
    "costs": {
      "file_cost": 0.0001284,
      "bytes_transferred_cost": 0.0000022,
      "compute_cost": 0.0000013,
      "total_cost": 0.0001319
    }
  },
  {
    "url": "https://spider.cloud/pricing",
    "status": 200,
    "duration_elasped_ms": 412,
    "error": null,
    "links": [
      "https://spider.cloud/credits",
      "https://spider.cloud/docs/api"
    ],
    "costs": {
      "file_cost": 0.0000948,
      "bytes_transferred_cost": 0.0000022,
      "compute_cost": 0.0000012,
      "total_cost": 0.0000982
    }
  }
]
```

--------------------------------

### Python requests proxy example

Source: https://spider.cloud/docs/api/proxy-mode

Python example using the requests library to make requests through the Spider proxy. It reads the API key from the SPIDER_API_KEY environment variable and configures both HTTP and HTTPS proxies. The function get_via_proxy makes a GET request and prints the status code and response body.

```python
import requests, os


# Proxy configuration
proxies = {
    'http': f"http://{os.getenv('SPIDER_API_KEY')}:proxy=residential@proxy.spider.cloud:8888",
    'https': f"https://{os.getenv('SPIDER_API_KEY')}:proxy=residential@proxy.spider.cloud:8889"
}

# Function to make a request through the proxy
def get_via_proxy(url):
    try:
        response = requests.get(url, proxies=proxies)
        response.raise_for_status()
        print('Response HTTP Status Code: ', response.status_code)
        print('Response HTTP Response Body: ', response.content)
        return response.text
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")
        return None

# Example usage
if __name__ == "__main__":
     get_via_proxy("https://www.example.com")
     get_via_proxy("https://www.example.com/community")
```

--------------------------------

### Fetch API Response Example

Source: https://spider.cloud/docs/api/fetch

Shows the JSON response from the Fetch API request, including business data, costs, credits, and a list of reviews with author, rating, and text.

```json
{
  "business": {
    "data_id": "0x88e7699d11dc124d:0xc0dd4b6ae44ddf17"
  },
  "chain_exhausted": true,
  "costs": {
    "ai_cost": 0,
    "ai_cost_formatted": "0",
    "bytes_transferred_cost": 0,
    "bytes_transferred_cost_formatted": "0",
    "compute_cost": 0.0021,
    "compute_cost_formatted": "0.0021",
    "file_cost": 0,
    "file_cost_formatted": "0",
    "total_cost": 0.0021,
    "total_cost_formatted": "0.0021",
    "transform_cost": 0,
    "transform_cost_formatted": "0"
  },
  "credits": 21,
  "has_more": false,
  "next_cursor": null,
  "next_offset": null,
  "query": {
    "keyword": null,
    "limit": 25,
    "max_rating": null,
    "min_rating": null,
    "sort": "newest"
  },
  "reviews": [
    {
      "author": "Priya Raghunathan",
      "owner_response": "Thank you, Dr. Raghunathan. Same turnaround on the next case.",
      "rating": 5,
      "relative_time": "2 weeks ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSUR4c2RqZ3FRRRAB",
      "text": "Zirconia crowns came back in four days and the margins fit on the first try."
    },
    {
      "author": "Marcus Oyelaran",
      "rating": 5,
      "relative_time": "a month ago",
      "review_id": "ChZDSUhNMG9nS0VJQ0FnSURoNHAzMkZBEAE"
    },
    {
      "author": "Elena Kowalczyk",
      "rating": 4,
      "relative_time": "3 months ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSURKd3ZLbzdBRRAB",
      "text": "Shade match on the anterior bridge was spot on. Pickup hours could run later."
    },
    {
      "author": "Tomas Herrera",
      "owner_response": "Sorry about the delay. The remake went out at no charge.",
      "rating": 2,
      "relative_time": "6 months ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSUM3a3FyRkNnRRAB",
      "text": "A partial needed a remake and the second round took another week."
    }
  ],
  "reviews_collected": 21,
  "reviews_scanned": 21,
  "stopped_early": null
}
```

--------------------------------

### Python request to GET /data/scraper-directory

Source: https://spider.cloud/docs/api/scraper-directory

Python example using the requests library to call the GET /data/scraper-directory endpoint. Requires an API key set as $SPIDER_API_KEY and uses the 'application/jsonl' content type. The query parameters include limit=5, return_format=markdown, and a URL-encoded url parameter.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/scraper-directory?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

--------------------------------

### Fetch API Python Request Example

Source: https://spider.cloud/docs/api/fetch

Sends a POST request to the Fetch API endpoint with an Authorization header using a Bearer token and a JSON payload specifying limit and reviews parameters. The example targets a Google Maps URL and prints the JSON response.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":1,"reviews":{"limit":25,"sort":"newest"}}

response = requests.post('https://api.spider.cloud/fetch/www.google.com/maps/place/Team+Solutions+Dental+Lab/@28.7307747,-81.2843008,15z/data=!4m6!3m5!1s0x88e7699d11dc124d:0xc0dd4b6ae44ddf17!8m2!3d28.7289405!4d-81.2973214!16s%2Fg%2F11b637_f60?entry=ttu&g_ep=EgoyMDI2MDkwMi4wIKXMDSoASAFQAw%3D%3D', 
  headers=headers, json=json_data)

print(response.json())
```

--------------------------------

### POST /fetch/{url}

Source: https://spider.cloud/docs/api/fetch

Fetches a web page and extracts structured data based on the provided configuration. The example shows how to fetch Google Maps reviews with a limit and sorting.

```APIDOC
## POST /fetch/{url}

### Description
Fetches a web page and extracts structured data based on the provided configuration. The example shows how to fetch Google Maps reviews with a limit and sorting.

### Method
POST

### Endpoint
https://api.spider.cloud/fetch/{url}

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL of the page to fetch.

#### Request Body
- **limit** (number) - Optional - The maximum number of items to return.
- **reviews** (object) - Optional - Configuration for extracting reviews.
  - **limit** (number) - Optional - Maximum number of reviews to return.
  - **sort** (string) - Optional - Sort order for reviews (e.g., "newest").

### Request Example
{
  "limit": 1,
  "reviews": {
    "limit": 25,
    "sort": "newest"
  }
}

### Response
#### Success Response (200)
- **business** (object) - Extracted business data.
  - **data_id** (string) - Unique identifier for the business.
- **chain_exhausted** (boolean) - Indicates if all data has been exhausted.
- **costs** (object) - Cost breakdown.
  - **ai_cost** (number) - Cost for AI processing.
  - **ai_cost_formatted** (string) - Formatted AI cost.
  - **bytes_transferred_cost** (number) - Cost for bytes transferred.
  - **bytes_transferred_cost_formatted** (string) - Formatted bytes transferred cost.
  - **compute_cost** (number) - Compute cost.
  - **compute_cost_formatted** (string) - Formatted compute cost.
  - **file_cost** (number) - File storage cost.
  - **file_cost_formatted** (string) - Formatted file cost.
  - **total_cost** (number) - Total cost.
  - **total_cost_formatted** (string) - Formatted total cost.
  - **transform_cost** (number) - Cost for data transformation.
  - **transform_cost_formatted** (string) - Formatted transform cost.
- **credits** (number) - Credits used.
- **has_more** (boolean) - Indicates if more results are available.
- **next_cursor** (string|null) - Cursor for pagination.
- **next_offset** (number|null) - Offset for pagination.
- **query** (object) - The query parameters used.
  - **keyword** (string|null) - Search keyword.
  - **limit** (number) - Limit applied.
  - **max_rating** (number|null) - Maximum rating filter.
  - **min_rating** (number|null) - Minimum rating filter.
  - **sort** (string) - Sort order.
- **reviews** (array) - List of reviews.
  - **author** (string) - Review author.
  - **owner_response** (string) - Response from the owner (if any).
  - **rating** (number) - Star rating.
  - **relative_time** (string) - Relative time of the review.
  - **review_id** (string) - Unique review identifier.
  - **text** (string) - Review text.
- **reviews_collected** (number) - Number of reviews collected.
- **reviews_scanned** (number) - Number of reviews scanned.
- **stopped_early** (boolean|null) - Indicates if the fetch stopped early.

#### Response Example
{
  "business": {
    "data_id": "0x88e7699d11dc124d:0xc0dd4b6ae44ddf17"
  },
  "chain_exhausted": true,
  "costs": {
    "ai_cost": 0,
    "ai_cost_formatted": "0",
    "bytes_transferred_cost": 0,
    "bytes_transferred_cost_formatted": "0",
    "compute_cost": 0.0021,
    "compute_cost_formatted": "0.0021",
    "file_cost": 0,
    "file_cost_formatted": "0",
    "total_cost": 0.0021,
    "total_cost_formatted": "0.0021",
    "transform_cost": 0,
    "transform_cost_formatted": "0"
  },
  "credits": 21,
  "has_more": false,
  "next_cursor": null,
  "next_offset": null,
  "query": {
    "keyword": null,
    "limit": 25,
    "max_rating": null,
    "min_rating": null,
    "sort": "newest"
  },
  "reviews": [
    {
      "author": "Priya Raghunathan",
      "owner_response": "Thank you, Dr. Raghunathan. Same turnaround on the next case.",
      "rating": 5,
      "relative_time": "2 weeks ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSUR4c2RqZ3FRRRAB",
      "text": "Zirconia crowns came back in four days and the margins fit on the first try."
    },
    {
      "author": "Marcus Oyelaran",
      "rating": 5,
      "relative_time": "a month ago",
      "review_id": "ChZDSUhNMG9nS0VJQ0FnSURoNHAzMkZBEAE"
    },
    {
      "author": "Elena Kowalczyk",
      "rating": 4,
      "relative_time": "3 months ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSURKd3ZLbzdBRRAB",
      "text": "Shade match on the anterior bridge was spot on. Pickup hours could run later."
    },
    {
      "author": "Tomas Herrera",
      "owner_response": "Sorry about the delay. The remake went out at no charge.",
      "rating": 2,
      "relative_time": "6 months ago",
      "review_id": "ChdDSUhNMG9nS0VJQ0FnSUM3a3FyRkNnRRAB",
      "text": "A partial needed a remake and the second round took another week."
    }
  ],
  "reviews_collected": 21,
  "reviews_scanned": 21,
  "stopped_early": null
}
```

--------------------------------

### Search API Response JSON

Source: https://spider.cloud/docs/api/search

Example JSON response from the Search endpoint. Contains an array of results, each with a description, title, and URL.

```json
{
  "content": [
      {
          "description": "Visit ESPN for live scores, highlights and sports news. Stream exclusive games on ESPN+ and play fantasy sports.",
          "title": "ESPN - Serving Sports Fans. Anytime. Anywhere.",
          "url": "https://www.espn.com/"
      },
      {
          "description": "Sports Illustrated, SI.com provides sports news, expert analysis, highlights, stats and scores for the NFL, NBA, MLB, NHL, college football, soccer,&nbsp;...",
          "title": "Sports Illustrated",
          "url": "https://www.si.com/"
      },
      {
          "description": "CBS Sports features live scoring, news, stats, and player info for NFL football, MLB baseball, NBA basketball, NHL hockey, college basketball and football.",
          "title": "CBS Sports - News, Live Scores, Schedules, Fantasy ...",
          "url": "https://www.cbssports.com/"
      },
      {
          "description": "Sport is a form of physical activity or game. Often competitive and organized, sports use, maintain, or improve physical ability and skills.",
          "title": "Sport",
          "url": "https://en.wikipedia.org/wiki/Sport"
      },
      {
          "description": "Watch FOX Sports and view live scores, odds, team news, player news, streams, videos, stats, standings &amp; schedules covering NFL, MLB, NASCAR, WWE, NBA, NHL,&nbsp;...",
          "title": "FOX Sports News, Scores, Schedules, Odds, Shows, Streams ...",
          "url": "https://www.foxsports.com/"
      },
      {
          "description": "Founded in 1974 by tennis legend, Billie Jean King, the Women's Sports Foundation is dedicated to creating leaders by providing girls access to sports.",
          "title": "Women's Sports Foundation: Home",
          "url": "https://www.womenssportsfoundation.org/"
      },
      {
          "description": "List of sports · Running. Marathon · Sprint · Mascot race · Airsoft · Laser tag · Paintball · Bobsleigh · Jack jumping · Luge · Shovel racing · Card stacking&nbsp;...",
          "title": "List of sports",
          "url": "https://en.wikipedia.org/wiki/List_of_sports"
      },
      {
          "description": "Stay up-to-date with the latest sports news and scores from NBC Sports.",
          "title": "NBC Sports - news, scores, stats, rumors, videos, and more",
          "url": "https://www.nbcsports.com/"
      },
      {
          "description": "r/sports: Sports News and Highlights from the NFL, NBA, NHL, MLB, MLS, and leagues around the world.",
          "title": "r/sports",
          "url": "https://www.reddit.com/r/sports/"
      },
      {
          "description": "The A-Z of sports covered by the BBC Sport team. Find all the latest live sports coverage, breaking news, results, scores, fixtures, tables,&nbsp;...",
          "title": "AZ Sport",
          "url": "https://www.bbc.com/sport/all-sports"
      }
  ]
}
```

--------------------------------

### POST /unlimited/scrape with curl

Source: https://spider.cloud/docs/api/unlimited

Example curl command to send a scrape request to the Unlimited endpoint. Uses the same parameters as the standard scrape endpoint. Requires an active Unlimited subscription and a valid API key.

```bash
curl -X POST https://api.spider.cloud/unlimited/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown"
  }'
```

--------------------------------

### POST /fetch/{domain}/{path}

Source: https://spider.cloud/docs/api

Per-website APIs with AI-discovered configs.

```APIDOC
## POST /fetch/{domain}/{path}

### Description
Per-website APIs with AI-discovered configs.

### Method
POST

### Endpoint
/fetch/{domain}/{path}

### Parameters
#### Path Parameters
- **domain** (string) - Required - The domain to fetch.
- **path** (string) - Required - The path on the domain.

#### Query Parameters
- **request** (string) - Optional - Request mode, e.g., `browser`.
- **return_format** (string) - Optional - Response format.
- **readability** (boolean) - Optional - Extract readable content.
- **evaluate_on_new_document** (string) - Optional - Custom script to evaluate on new document creation.
- **scroll** (number) - Optional - Infinite scroll duration in milliseconds.
- **viewport** (object) - Optional - Viewport configuration for browser requests.
- **country_code** (string) - Optional - ISO country code for proxy connections.
- **locale** (string) - Optional - Locale for request, e.g., `en-US`.

### Response
#### Success Response (200)
- **content** (object) - Fetched content.

#### Response Example
{
  "content": {}
}
```

--------------------------------

### Python request to GET /data/credits

Source: https://spider.cloud/docs/api/credits

Use this Python snippet to query the remaining credits. It sends a GET request to https://api.spider.cloud/data/credits with an Authorization header using the $SPIDER_API_KEY environment variable. The response is printed as JSON.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/credits?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

--------------------------------

### Proxy Connection

Source: https://spider.cloud/docs/api/proxy-mode

Documentation for using Spider as a standard HTTP proxy with connection details, parameters, and examples.

```APIDOC
## [PROXY] proxy.spider.cloud

### Description
Spider acts as a standard HTTP proxy. Set `proxy.spider.cloud` as your proxy address, put your API key in the username, and pass parameters in the password field. Every request goes through the same rendering and stealth pipeline as the REST API.

### Method
PROXY (HTTP, HTTPS, SOCKS5)

### Endpoint
- HTTP: `http://YOUR-API-KEY:proxy=residential@proxy.spider.cloud:8888`
- HTTPS: `https://YOUR-API-KEY:proxy=residential@proxy.spider.cloud:8889`
- SOCKS5: `socks5://YOUR-API-KEY:proxy=residential@proxy.spider.cloud:8887`

### Parameters
#### Username
- **API Key** (string) - Required - Your Spider API key.

#### Password (parameters)
- **proxy** (string) - Optional - Picks the proxy pool. Values: `residential` (default) or `isp`.
- **country_code** (string) - Optional - Two-letter country code for per-request geo-routing.
- **trackers_disabled** (boolean) - Optional - Default `true`. Blocks trackers and ad networks. Set `false` to allow all traffic through.

### Request Example
```
curl -x "http://YOUR-API-KEY:proxy=residential@proxy.spider.cloud:8888" \
  https://example.com
```

### Response
#### Success Response
- Returns the response from the target URL through the proxy.

#### Response Example
```
HTTP/1.1 200 OK
Content-Type: text/html
...
```
```

--------------------------------

### POST /ai/*

Source: https://spider.cloud/docs/api

Prompt-guided crawl, scrape, search, and browser automation. Requires an AI plan.

```APIDOC
## POST /ai/*

### Description
Prompt-guided crawl, scrape, search, and browser automation. Requires an AI plan.

### Method
POST

### Endpoint
/ai/*

### Parameters
#### Query Parameters
- **limit** (number) - Optional - Maximum number of pages.
- **depth** (number) - Optional - Maximum crawl depth.
- **delay** (number) - Optional - Delay between requests in milliseconds.
- **request** (string) - Optional - Request mode, e.g., `browser`.
- **return_format** (string) - Optional - Response format.
- **readability** (boolean) - Optional - Extract readable content.
- **evaluate_on_new_document** (string) - Optional - Custom script to evaluate on new document creation.
- **scroll** (number) - Optional - Infinite scroll duration in milliseconds.
- **viewport** (object) - Optional - Viewport configuration for browser requests.
- **country_code** (string) - Optional - ISO country code for proxy connections.
- **locale** (string) - Optional - Locale for request, e.g., `en-US`.

### Response
#### Success Response (200)
- **content** (object) - AI-generated output.

#### Response Example
{
  "content": {}
}
```

--------------------------------

### Interpret vendor cost response

Source: https://spider.cloud/docs/api/router

Example of the vendor object returned in the costs field when an outside provider serves the page.

```json
{
  "costs": {
    "total_cost": 0.0010292944,
    "vendor": {
      "provider": "zyte",
      "route": "zyte.api",
      "vendor_cost": 0.0008,
      "billed_cost": 0.0002,
      "byok": true,
      "attempts": 2
    }
  }
}
```

--------------------------------

### GET /data/scraper-directory

Source: https://spider.cloud/docs/api/scraper-directory

Retrieves a list of curated scraper configurations. Supports filtering by domain, category, or search term. No authentication required.

```APIDOC
## GET /data/scraper-directory

### Description
Browse optimized scraper configs for popular websites. Filter by domain, category, or search term. Each config is curated to deliver the best extraction results out of the box. No authentication required.

### Method
GET

### Endpoint
https://api.spider.cloud/data/scraper-directory

### Parameters
#### Query Parameters
- **url** (string) - Optional - Filter a single url record.
- **limit** (string) - Optional - The limit of records to get.
- **domain** (string) - Optional - Filter a single domain record.
- **page** (number) - Optional - The current page to get.

### Request Example
```
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/scraper-directory?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

### Response
#### Success Response (200)
- **data** (array) - List of scraper configurations.
  - **id** (string) - Unique identifier.
  - **domain** (string) - Domain of the scraper.
  - **path_pattern** (string) - URL path pattern.
  - **display_name** (string) - Display name.
  - **description** (string) - Description of the scraper.
  - **category** (string) - Category of the scraper.
  - **tags** (array) - Tags associated.
  - **confidence_score** (number) - Confidence score.
  - **validation_count** (number) - Number of validations.
  - **slug** (string) - Slug identifier.
  - **created_at** (string) - Creation timestamp.
  - **updated_at** (string) - Last update timestamp.
- **total** (number) - Total number of records.
- **page** (number) - Current page.
- **limit** (number) - Limit of records per page.
- **total_pages** (number) - Total number of pages.

#### Response Example
```
{
  "data": [
    {
      "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "domain": "example.com",
      "path_pattern": "/blog/*",
      "display_name": "Example Blog Scraper",
      "description": "Extracts blog posts with title, author, and content.",
      "category": "news",
      "tags": ["blog", "articles"],
      "confidence_score": 0.95,
      "validation_count": 12,
      "slug": "example-com-blog",
      "created_at": "2025-12-01T10:00:00+00:00",
      "updated_at": "2026-01-15T08:30:00+00:00"
    }
  ],
  "total": 1,
  "page": 1,
  "limit": 20,
  "total_pages": 1
}
```
```

--------------------------------

### Request a provider first using Python

Source: https://spider.cloud/docs/api/router

Uses the Python standard library to send a request with a specific provider preference. Requires SPIDER_API_KEY and ZYTE_API_KEY environment variables.

```python
import json
import os
from urllib.request import Request, urlopen

body = {
    "url": "https://example.com",
    "return_format": "markdown",
    "router": {
        "mode": "first",
        "provider": "zyte",
        "token": os.environ["ZYTE_API_KEY"],
    },
}
request = Request(
    "https://api.spider.cloud/scrape",
    data=json.dumps(body).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
)
with urlopen(request) as response:
    print(json.load(response))
```

--------------------------------

### POST /crawl

Source: https://spider.cloud/docs/api/crawl

This code snippet documents the POST /crawl endpoint, including its request body parameters, request example, and response structure.

```APIDOC
## POST /crawl

### Description
Start crawling website(s) to collect resources. You can pass an array of objects for the request body. Results stream back as they are found, so you can process pages before the full crawl completes.

### Method
POST

### Endpoint
https://api.spider.cloud/crawl

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **url** (string) - Required - The URI resource to crawl. This can be a comma split list for multiple URLs.
- **limit** (number) - Optional - The maximum number of pages to crawl. Default is 0 (no limit).
- **return_format** (string | array) - Optional - The format of the returned content. Default is 'raw'. Possible values include 'markdown'.
- **depth** (number) - Optional - The maximum depth to crawl. Default is 25.
- **metadata** (boolean) - Optional - Whether to include metadata. Default is false.
- **external_domains** (array) - Optional - List of external domains to allow.
- **session** (boolean) - Optional - Whether to use a session. Default is true.
- **subdomains** (boolean) - Optional - Whether to include subdomains. Default is false.
- **tld** (boolean) - Optional - Whether to include top-level domains. Default is false.
- **concurrency_limit** (number) - Optional - Maximum number of concurrent requests.
- **chunking_alg** (object) - Optional - Chunking algorithm configuration.
- **redirect_policy** (string) - Optional - Redirect policy. Default is 'Loose'.
- **user_agent** (string) - Optional - User agent string.
- **blacklist** (array) - Optional - List of URLs to blacklist.
- **whitelist** (array) - Optional - List of URLs to whitelist.
- **root_selector** (string) - Optional - CSS selector for root element.
- **exclude_selector** (string) - Optional - CSS selector to exclude elements.
- **full_resources** (boolean) - Optional - Whether to include full resources.
- **budget** (object) - Optional - Budget configuration.
- **max_credits_per_page** (number) - Optional - Maximum credits per page.
- **max_credits_allowed** (number) - Optional - Maximum credits allowed.
- **request_timeout** (number) - Optional - Request timeout in seconds. Default is 60.
- **webhooks** (object) - Optional - Webhook configuration.
- **data_connectors** (object) - Optional - Data connector configuration.
- **sitemap** (boolean) - Optional - Whether to use sitemap. Default is false.
- **sitemap_only** (boolean) - Optional - Whether to only use sitemap. Default is false.
- **sitemap_path** (string) - Optional - Path to sitemap. Default is 'sitemap.xml'.
- **wait_for** (object) - Optional - Wait for condition.
- **crawl_timeout** (object) - Optional - Crawl timeout configuration.
- **execution_scripts** (object) - Optional - Execution scripts.
- **preserve_host** (boolean) - Optional - Whether to preserve host. Default is false.
- **event_tracker** (object) - Optional - Event tracker configuration.
- **disable_intercept** (boolean) - Optional - Whether to disable intercept. Default is false.
- **block_ads** (boolean) - Optional - Whether to block ads. Default is true.
- **block_analytics** (boolean) - Optional - Whether to block analytics. Default is true.
- **block_stylesheets** (boolean) - Optional - Whether to block stylesheets. Default is true.
- **disable_first_party_stylesheets** (boolean) - Optional - Whether to disable first-party stylesheets. Default is false.
- **disable_first_party_javascript** (boolean) - Optional - Whether to disable first-party JavaScript. Default is false.
- **disable_first_party_visuals** (boolean) - Optional - Whether to disable first-party visuals. Default is false.
- **run_in_background** (boolean) - Optional - Whether to run in background. Default is false.
- **max_size** (number) - Optional - Maximum size of the response.
- **network_blacklist** (string[]) - Optional - List of network URLs to blacklist.
- **disable_hints** (boolean) - Optional - Whether to disable hints.
- **network_whitelist** (string[]) - Optional - List of network URLs to whitelist.
- **readability** (boolean) - Optional - Whether to use readability. Default is false.
- **return_page_links** (boolean) - Optional - Whether to return page links. Default is false.
- **return_json_data** (boolean) - Optional - Whether to return JSON data. Default is false.
- **return_headers** (boolean) - Optional - Whether to return headers. Default is false.
- **return_cookies** (boolean) - Optional - Whether to return cookies. Default is false.
- **css_extraction_map** (object) - Optional - CSS extraction map.
- **filter_output_images** (boolean) - Optional - Whether to filter output images.
- **filter_output_svg** (boolean) - Optional - Whether to filter output SVG.
- **filter_output_main_only** (boolean) - Optional - Whether to filter output to main only. Default is true.
- **clean_html** (boolean) - Optional - Whether to clean HTML.
- **filter_svg** (boolean) - Optional - Whether to filter SVG.
- **filter_images** (boolean) - Optional - Whether to filter images.
- **link_rewrite** (json) - Optional - Link rewrite configuration.
- **encoding** (string) - Optional - Encoding of the response.
- **return_embeddings** (boolean) - Optional - Whether to return embeddings. Default is false.
- **proxy_enabled** (boolean) - Optional - Whether to enable proxy. Default is false.
- **proxy** ('residential' | 'mobile' | 'isp') - Optional - Proxy type.
- **remote_proxy** (string) - Optional - Remote proxy URL.
- **provider** (string) - Optional - Provider name.
- **vendor_credentials** (Record<string, string>) - Optional - Vendor credentials.
- **router** (object) - Optional - Router configuration.
- **provider_options** (object) - Optional - Provider options.
- **fingerprint** (boolean) - Optional - Whether to use fingerprint. Default is true.
- **cookies** (string) - Optional - Cookies string.
- **stealth** (boolean) - Optional - Whether to use stealth mode. Default is true.
- **headers** (object) - Optional - Custom headers.
- **respect_robots** (boolean) - Optional - Whether to respect robots.txt. Default is true.
- **cache** (boolean | { maxAge?: number; allowStale?: boolean; period?: string; skipBrowser?: boolean }) - Optional - Cache configuration. Default is true.
- **service_worker_enabled** (boolean) - Optional - Whether to enable service worker. Default is true.
- **delay** (number) - Optional - Delay between requests in milliseconds. Default is 0.
- **skip_config_checks** (boolean) - Optional - Whether to skip config checks. Default is true.
- **evaluate_on_new_document** (string) - Optional - JavaScript to evaluate on new document.
- **scroll** (number) - Optional - Scroll amount.
- **viewport** (object) - Optional - Viewport configuration.
- **automation_scripts** (object) - Optional - Automation scripts.
- **country_code** (string) - Optional - Country code for geolocation.
- **locale** (string) - Optional - Locale for geolocation.

### Request Example
```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data)

print(response.json())
```

### Response
#### Success Response (200)
- **content** (string) - The content of the crawled page.
- **error** (null) - Error message, if any.
- **status** (number) - HTTP status code.
- **duration_elapsed_ms** (number) - Time taken in milliseconds.
- **costs** (object) - Cost breakdown.
  - **ai_cost** (number) - AI cost.
  - **compute_cost** (number) - Compute cost.
  - **file_cost** (number) - File cost.
  - **bytes_transferred_cost** (number) - Bytes transferred cost.
  - **total_cost** (number) - Total cost.
  - **transform_cost** (number) - Transform cost.
- **url** (string) - The URL that was crawled.

#### Response Example
```json
[
  {
    "content": "<resource>...",
    "error": null,
    "status": 200,
    "duration_elapsed_ms": 122,
    "costs": {
      "ai_cost": 0,
      "compute_cost": 0.00001,
      "file_cost": 0.00002,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.00004,
      "transform_cost": 0.0001
    },
    "url": "https://spider.cloud"
  }
]
```
```

--------------------------------

### Initialize SpiderBrowser with API key

Source: https://spider.cloud/docs/api/browser

Initializes the SpiderBrowser client with an API key from the SPIDER_API_KEY environment variable. The init() method handles sessions, stealth, and geo-routing automatically.

```typescript
import { SpiderBrowser } from "spider-browser"

const browser = new SpiderBrowser({ apiKey: process.env.SPIDER_API_KEY! })
await browser.init() // sessions, stealth and geo-routing handled for you
```

--------------------------------

### POST /unlimited/*

Source: https://spider.cloud/docs/api

Flat-rate Scrape, Crawl, and Links, limited by purchased concurrency. Requires an Unlimited plan.

```APIDOC
## POST /unlimited/*

### Description
Flat-rate Scrape, Crawl, and Links, limited by purchased concurrency. Requires an Unlimited plan.

### Method
POST

### Endpoint
/unlimited/*

### Parameters
#### Query Parameters
- **limit** (number) - Optional - Maximum number of pages.
- **depth** (number) - Optional - Maximum crawl depth.
- **delay** (number) - Optional - Delay between requests in milliseconds.
- **request** (string) - Optional - Request mode, e.g., `browser`.
- **return_format** (string) - Optional - Response format.
- **readability** (boolean) - Optional - Extract readable content.
- **evaluate_on_new_document** (string) - Optional - Custom script to evaluate on new document creation.
- **scroll** (number) - Optional - Infinite scroll duration in milliseconds.
- **viewport** (object) - Optional - Viewport configuration for browser requests.
- **country_code** (string) - Optional - ISO country code for proxy connections.
- **locale** (string) - Optional - Locale for request, e.g., `en-US`.

### Response
#### Success Response (200)
- **content** (object) - Scraped or crawled content.

#### Response Example
{
  "content": {}
}
```

--------------------------------

### GET /data/credits

Source: https://spider.cloud/docs/api/credits

Retrieves the remaining credits available for the authenticated user. Use this endpoint to monitor usage and set up alerts before credits run low.

```APIDOC
## GET /data/credits

### Description
Query your current credit balance. Use this endpoint to monitor usage and set up alerts before credits run low.

### Method
GET

### Endpoint
https://api.spider.cloud/data/credits

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Limits the number of results returned.
- **return_format** (string) - Optional - Specifies the format of the response (e.g., markdown).
- **url** (string) - Optional - The URL to process.

### Request Example
```
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/credits?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

### Response
#### Success Response (200)
- **data** (object) - The credit balance data.
  - **id** (string) - The unique identifier for the credit record.
  - **user_id** (string) - The unique identifier of the user.
  - **credits** (integer) - The number of remaining credits.
  - **created_at** (string) - The timestamp when the credit record was created.
  - **updated_at** (string) - The timestamp when the credit record was last updated.

#### Response Example
```
{
  "data": {
    "id": "8d662167-5a5f-41aa-9cb8-0cbb7d536891",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "credits": 53334,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  }
}
```

```
Tokens:
Raw