Spider

https://spider.cloud/docs
Admin
Spider is a web scraping and crawling platform that provides APIs and clients for efficient data...
Tokens:79,003
Snippets:592
Trust Score:7.9
Update:1 month ago
Show doc for...
Tokens:
Raw
### Python SDK usage with Spider

Source: https://spider.cloud/docs/core/authentication

Python SDK example. Install spider_client and set SPIDER_API_KEY environment variable; the SDK handles authentication automatically.

```python
# pip install spider_client
from spider import Spider

# Reads SPIDER_API_KEY from environment automatically
app = Spider()
result = app.crawl_url('https://example.com', params={'limit': 1})
print(result)
```

--------------------------------

### CLI: Install, authenticate, and crawl

Source: https://spider.cloud/docs/libraries

Install the CLI, authenticate with an API key, and run a crawl command.

```bash
cargo install spider-cloud-cli
```

```bash
spider-cloud-cli auth --api_key YOUR_API_KEY
```

```bash
spider-cloud-cli crawl --url https://example.com --limit 5
```

--------------------------------

### Node.js SDK usage with Spider

Source: https://spider.cloud/docs/core/authentication

Node.js SDK example. Install @spider-cloud/spider-client and set SPIDER_API_KEY environment variable; the SDK handles authentication automatically.

```javascript
// npm install @spider-cloud/spider-client
import { Spider } from "@spider-cloud/spider-client";

// Reads SPIDER_API_KEY from environment automatically
const app = new Spider();
const result = await app.crawlUrl("https://example.com", { limit: 1 });
console.log(result);
```

--------------------------------

### Install Agno, Spider, and OpenAI

Source: https://spider.cloud/docs/integrations/agno

Use this pip command to install the required Python packages. The command installs agno, spider-client, and openai in one line.

```bash
pip install agno spider-client openai 
```

--------------------------------

### Go SDK: Install and crawl a URL

Source: https://spider.cloud/docs/libraries

Install the Go SDK and use the Spider client to crawl a URL with a limit parameter.

```bash
go get github.com/spider-rs/spider-clients/go
```

```go
package main

import (
    "context"
    "fmt"
    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    client := spider.New("")
    pages, _ := client.CrawlURL(
        context.Background(),
        "https://example.com",
        &spider.SpiderParams{Limit: 5},
    )
    for _, page := range pages {
        fmt.Printf("%s: %d chars\n", page.URL, len(page.Content))
    }
}
```

--------------------------------

### Rust SDK: Install and crawl a URL

Source: https://spider.cloud/docs/libraries

Install the Rust SDK and use the Spider client to crawl a URL with a limit parameter.

```bash
cargo add spider-client
```

```rust
let spider = spider_client::Spider::new(None).expect("API required");
let crawler_params = RequestParams { limit: Some(5), ..Default::default() };

let crawl_result = spider
    .crawl_url("https://example.com", Some(crawler_params), false, "application/json", None)
    .await
    .expect("Failed to crawl the URL");
```

--------------------------------

### Python SDK: Install and crawl a URL

Source: https://spider.cloud/docs/libraries

Install the Python SDK and use the Spider client to crawl a URL with a limit parameter.

```bash
pip install spider_client
```

```python
from spider import Spider

app = Spider()
params = {"limit": 5}
crawl_result = app.crawl_url("https://example.com", params=params)
```

--------------------------------

### Install x402 client with EVM support

Source: https://spider.cloud/docs/integrations/x402

Install the x402 client SDK and a wallet library. Requires an EVM wallet with USDC on Base and the private key set as EVM_PRIVATE_KEY in the environment.

```bash
npm install @x402/core @x402/evm @x402/fetch viem
```

--------------------------------

### Response for GET /data/credits

Source: https://spider.cloud/docs/api/credits

Example JSON response from the GET /data/credits endpoint. It contains the credit balance and metadata such as user_id and timestamps.

```JSON
{
  "data": {
    "id": "8d662167-5a5f-41aa-9cb8-0cbb7d536891",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "credits": 53334,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  }
}
```

--------------------------------

### Install spider-browser SDK

Source: https://spider.cloud/docs/core/spider-browser

Installs the TypeScript SDK for the Spider Browser. All SDKs connect to the same infrastructure over wss://browser.spider.cloud.

```bash
npm install spider-browser
```

--------------------------------

### Install Spider client with pip

Source: https://spider.cloud/docs/integrations/langchain

Installs the Spider client package using pip. Required before using the SpiderLoader.

```bash
pip install spider_client
```

--------------------------------

### Fetch crawl logs with Python requests

Source: https://spider.cloud/docs/api/logs

Python example using the requests library to call the GET /data/crawl_logs endpoint. Requires an API key set as $SPIDER_API_KEY. The query parameters include limit=5, return_format=markdown, and a URL-encoded url parameter.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/crawl_logs?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

--------------------------------

### Node.js fetch example for Spider Cloud API

Source: https://spider.cloud/docs/core/authentication

Node.js example using fetch to call the Spider Cloud API. Reads the API key from the SPIDER_API_KEY environment variable.

```javascript
const response = await fetch('https://api.spider.cloud/crawl', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.SPIDER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({ url: 'https://example.com', limit: 1 }),
});

const data = await response.json();
console.log(data);
```

--------------------------------

### Install Spider client and CrewAI tools

Source: https://spider.cloud/docs/integrations/crewai

Install the Spider client and CrewAI tools using pip. Set the SPIDER_API_KEY environment variable before using the tool.

```bash
pip install spider_client 'crewai[tools]'
```

--------------------------------

### JavaScript/TypeScript SDK: Install and crawl a URL

Source: https://spider.cloud/docs/libraries

Install the JavaScript/TypeScript SDK and use the Spider client to crawl a URL with a limit parameter.

```bash
npm install @spider-cloud/spider-client
```

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const params = { limit: 5 };
const result = await app.crawlUrl("https://example.com", params);
```

--------------------------------

### Python requests proxy example

Source: https://spider.cloud/docs/api/proxy-mode

Python example using the requests library to make requests through the Spider Cloud proxy. It reads the API key from the SPIDER_API_KEY environment variable and configures both HTTP and HTTPS proxies. The function get_via_proxy sends a GET request and prints the status code and response body.

```python
import requests, os


# Proxy configuration
proxies = {
    'http': f"http://{os.getenv('SPIDER_API_KEY')}:proxy=residential@proxy.spider.cloud:8888",
    'https': f"https://{os.getenv('SPIDER_API_KEY')}:proxy=residential@proxy.spider.cloud:8889"
}

# Function to make a request through the proxy
def get_via_proxy(url):
    try:
        response = requests.get(url, proxies=proxies)
        response.raise_for_status()
        print('Response HTTP Status Code: ', response.status_code)
        print('Response HTTP Response Body: ', response.content)
        return response.text
    except requests.exceptions.RequestException as e:
        print(f"Error: {e}")
        return None

# Example usage
if __name__ == "__main__":
     get_via_proxy("https://www.example.com")
     get_via_proxy("https://www.example.com/community")
```

--------------------------------

### Example Response JSON

Source: https://spider.cloud/docs/advanced/json-scraping

Example response from the Spider Cloud API showing extracted JSON-LD data under the json_data.other_scripts array, with cost details.

```json
{
  "costs": {
    "ai_cost": 0,
    "file_cost": 0.0005,
    "total_cost": 0.0005
  },
  "error": null,
  "json_data": {
    "other_scripts": [
      {
        "@context": "http://schema.org",
        "@type": ["Recipe"],
        "aggregateRating": {
          "@type": "AggregateRating",
          "ratingCount": "102",
          "ratingValue": "4.7"
        },
        "author": [
          {
            "@type": "Person",
            "name": "Carmella DiNardo"
          }
        ],
        "cookTime": "PT10M",
        "datePublished": "2020-06-18T23:50:15.000-04:00",
        "description": "Nutella cookies made with chocolate-hazelnut spread...",
        "headline": "Nutella Cookies",
        "name": "Nutella Cookies"
      }
    ]
  }
}
```

--------------------------------

### Screenshot API response example

Source: https://spider.cloud/docs/api/screenshot

Example response array with a base64-encoded image in the content field and a status code.

```json
[
  {
    "url": "https://example.com",
    "content": "iVBORw0KGgoAAAANSUhEUgAA...",
    "status": 200
  }
]
```

--------------------------------

### cdp_params example for JPEG with clip

Source: https://spider.cloud/docs/api/screenshot

Example cdp_params object to set format to jpeg, quality 90, a clip region with scale, and captureBeyondViewport true.

```json
{
  "format": "jpeg",
  "quality": 90,
  "clip": { "x": 10, "y": 20, "width": 500, "height": 400, "scale": 1.5 },
  "fromSurface": false,
  "captureBeyondViewport": true
}
```

--------------------------------

### Python requests example for Spider Cloud API

Source: https://spider.cloud/docs/core/authentication

Python example using the requests library to call the Spider Cloud API. Reads the API key from the SPIDER_API_KEY environment variable.

```python
import requests
import os

headers = {
    'Authorization': f'Bearer {os.getenv("SPIDER_API_KEY")}',
    'Content-Type': 'application/json',
}

response = requests.post(
    'https://api.spider.cloud/crawl',
    headers=headers,
    json={"url": "https://example.com", "limit": 1}
)

print(response.json())
```

--------------------------------

### Crawl logs API response example

Source: https://spider.cloud/docs/api/logs

Example JSON response from the GET /data/crawl_logs endpoint. Contains a data object with log details such as id, domain, url, credits_used, crawl_duration, status_code, and timestamps, plus an error field.

```json
{
  "data": {
    "id": "195bf2f2-2821-421d-b89c-f27e57ca71fh",
    "user_id": "6bd06efa-bb0a-4f1f-a29f-05db0c4b1bfg",
    "domain": "spider.cloud",
    "url": "https://spider.cloud",
    "links": 1,
    "credits_used": 3,
    "mode": 2,
    "crawl_duration": 340,
    "message": null,
    "request_user_agent": "Spider",
    "level": "UI",
    "status_code": 0,
    "created_at": "2024-04-21T01:21:32.886863+00:00",
    "updated_at": "2024-04-21T01:21:32.886863+00:00"
  },
  "error": null
}
```

--------------------------------

### Request a provider first with router.token (Python)

Source: https://spider.cloud/docs/api/router

Use this Python example to request a specific provider first with mode "first". Set SPIDER_API_KEY and ZYTE_API_KEY in your environment. Spider authentication stays in the header, and the provider key goes in router.token. The request still has to meet the route's capability and budget checks; "first" does not force a specific provider or bypass account availability.

```python
import json
import os
from urllib.request import Request, urlopen

body = {
    "url": "https://example.com",
    "return_format": "markdown",
    "router": {
        "mode": "first",
        "provider": "zyte",
        "token": os.environ["ZYTE_API_KEY"],
    },
}
request = Request(
    "https://api.spider.cloud/scrape",
    data=json.dumps(body).encode(),
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
)
with urlopen(request) as response:
    print(json.load(response))
```

--------------------------------

### 401 Unauthorized response example

Source: https://spider.cloud/docs/core/authentication

Example 401 Unauthorized response returned when the API key is missing, invalid, or expired.

```json
{
  "error": "Unauthorized",
  "status": 401
}
```

--------------------------------

### Try Spider first with fallback mode (curl)

Source: https://spider.cloud/docs/api/router

Use this curl command to try Spider first with fallback routing. Set SPIDER_API_KEY in your shell before running. With no provider preference, the router chooses among eligible routes, and without a usable key of your own, a provider attempt can use Spider's credentials and pooled billing.

```bash
curl https://api.spider.cloud/scrape \
  -H "Authorization: Bearer $SPIDER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "return_format": "markdown",
    "router": { "mode": "fallback" }
  }'
```

--------------------------------

### Simple Instantiation of SpiderTool

Source: https://spider.cloud/docs/integrations/crewai

Initialize SpiderTool with your API key. If not provided, it looks for SPIDER_API_KEY in environment variables.

```python
from crewai_tools import SpiderTool

# To enable scraping any website it finds during its execution
spider_tool = SpiderTool(api_key='YOUR_API_KEY')
```

--------------------------------

### POST /ai/browser with wait_for and response

Source: https://spider.cloud/docs/ai-studio

Sends a natural-language prompt to the AI Browser endpoint, using a wait_for object with selector, idle_network, and page_navigations to wait for the pricing section and network idle. The response shows the extracted markdown content, metadata with the billing period and plan, and a detailed cost breakdown with formatted cost fields.

```python
import os

import requests

payload = {
  "url": "https://spider.cloud/pricing",
  "prompt": "Switch the billing toggle to annual, then read back each plan name with its price.",
  "wait_for": {
    "selector": {
      "timeout": {
        "secs": 10,
        "nanos": 0
      },
      "selector": "#pricing"
    },
    "idle_network": {
      "timeout": {
        "secs": 15,
        "nanos": 0
      }
    },
    "page_navigations": True
  },
  "return_format": "markdown",
  "metadata": True
}

response = requests.post(
    "https://api.spider.cloud/ai/browser",
    headers={
        "Authorization": f"Bearer {os.environ['SPIDER_API_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    timeout=120,
)

print(response.json())
```

```json
[
  {
    "url": "https://spider.cloud/pricing",
    "error": null,
    "status": 200,
    "content": "# Pricing\n\nAnnual billing selected.",
    "metadata": {
      "title": "Pricing",
      "extracted_data": {
        "billing_period": "annual",
        "plans": [
          { "plan": "Pro", "price": 79 }
        ]
      }
    },
    "costs": {
      "file_cost": 0.00021,
      "transform_cost": 0.00004,
      "compute_cost": 0.00038,
      "ai_cost": 0.00265,
      "bytes_transferred_cost": 0.00002,
      "total_cost": 0.0033,
      "file_cost_formatted": "$0.00021",
      "transform_cost_formatted": "$0.00004",
      "compute_cost_formatted": "$0.00038",
      "ai_cost_formatted": "$0.00265",
      "bytes_transferred_cost_formatted": "$0.00002",
      "total_cost_formatted": "$0.0033"
    },
    "duration_elasped_ms": 9310
  }
]
```

--------------------------------

### Python SDK: Streaming with stream and callback

Source: https://spider.cloud/docs/core/concurrent-streaming

Use the Python SDK's built-in streaming support via the stream and callback parameters. The callback function receives each page as it is crawled. Set the Content-Type header to application/jsonl and enable streaming in your HTTP client.

```python
from spider import Spider

app = Spider()

def handle_page(page: dict) -> None:
    print(f"Crawled: {page['url']} ({page['status']})")

result = app.crawl_url(
    "https://www.example.com",
    params={
        "limit": 100,
        "depth": 3,
        "request": "smart",
        "return_format": "markdown"
    },
    stream=True,
    callback=handle_page,
)
```

--------------------------------

### Browser automation workflow with spider_browser_* tools

Source: https://spider.cloud/docs/integrations/mcp

Illustrates a full browser automation workflow: open a session, navigate, fill, click, wait, navigate again, screenshot, and close. The session_id is passed to every tool after spider_browser_open, and spider_browser_close stops billing.

```text
User: "Log into my dashboard and screenshot the analytics page"

Agent uses spider_browser_open
  → returns session_id: "abc-123"

Agent uses spider_browser_navigate with:
  session_id: "abc-123"
  url: "https://app.example.com/login"

Agent uses spider_browser_fill with:
  session_id: "abc-123"
  selector: "input[name='email']"
  value: "user@example.com"

Agent uses spider_browser_click with:
  session_id: "abc-123"
  selector: "button[type='submit']"

Agent uses spider_browser_wait_for with:
  session_id: "abc-123"
  selector: ".dashboard-loaded"

Agent uses spider_browser_navigate with:
  session_id: "abc-123"
  url: "https://app.example.com/analytics"

Agent uses spider_browser_screenshot with:
  session_id: "abc-123"
  → returns base64 PNG image

Agent uses spider_browser_close with:
  session_id: "abc-123"
```

--------------------------------

### Navigate pages with goto() and goto_fast()

Source: https://spider.cloud/docs/core/spider-browser

Shows the navigation methods: goto() waits for DOM content, goto_fast() returns as soon as possible, and content() waits for full readiness with a minimum content length. Spider automatically detects anti-bot challenges, rotates fingerprints, and solves CAPTCHAs.

```typescript
// Standard navigation, waits for DOM content
await browser.page.goto("https://www.amazon.com/dp/B0DGJHM7QN");

// Fast navigation, returns as soon as possible
await browser.page.goto_fast("https://news.ycombinator.com");

// Wait for full readiness with a minimum content length
const html = await browser.page.content({
  wait_ms: 5000,
  min_length: 500,
});
```

--------------------------------

### POST /links with Python requests

Source: https://spider.cloud/docs/api/links

Python example using the requests library to call the /links endpoint. It sends a POST request with an Authorization header and JSON body containing limit, return_format, and url parameters, then prints the JSON response.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"return_format":"markdown","url":"https://spider.cloud"}

response = requests.post('https://api.spider.cloud/links', 
  headers=headers, json=json_data)

print(response.json())
```

--------------------------------

### Fetch API Response Example

Source: https://spider.cloud/docs/api/fetch

Example JSON response from the Fetch API. It includes the crawled URL, HTTP status, extracted content as a JSON string, error field, and cost details.

```json
[
  {
    "url": "https://example.com/",
    "status": 200,
    "content": "{\"title\": \"Example Domain\", \"description\": \"This domain is for use in illustrative examples.\", \"links\": [\"https://www.iana.org/domains/example\"]}",
    "error": null,
    "costs": {
      "total_cost": 0.001,
      "total_cost_formatted": "0.0010"
    }
  }
]
```

--------------------------------

### Python request to GET /data/credits

Source: https://spider.cloud/docs/api/credits

Use this Python code to query your credit balance. It sends a GET request to https://api.spider.cloud/data/credits with an Authorization header using your API key. The response is printed as JSON.

```Python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/jsonl',
}

response = requests.get('https://api.spider.cloud/data/credits?limit=5&return_format=markdown&url=https%253A%252F%252Fspider.cloud', 
  headers=headers)

print(response.json())
```

--------------------------------

### Response fields example

Source: https://spider.cloud/docs/core/scraping-crawling

Example response object for a crawled page, showing the fields returned for each page in the response array. Includes the URL, HTTP status, content, error, and a costs breakdown with formatted values.

```json
[
  {
    "url": "https://example.com",
    "status": 200,
    "content": "<html>...</html>",
    "error": null,
    "costs": {
      "ai_cost": 0.0,
      "ai_cost_formatted": "0",
      "bytes_transferred_cost": 3.165e-9,
      "bytes_transferred_cost_formatted": "0.0000000032",
          "compute_cost": 0.0,
          "compute_cost_formatted": "0",
          "file_cost": 0.000029,
          "file_cost_formatted": "0.0000290000",
          "total_cost": 0.000029,
          "total_cost_formatted": "0.0000290000",
          "transform_cost": 0.0,
          "transform_cost_formatted": "0"
        }
      }
    ]
```

--------------------------------

### Open parallel sessions with Promise.all

Source: https://spider.cloud/docs/core/spider-browser

Open up to 100 parallel browser sessions, each fully isolated with a separate fingerprint, cookie jar, and proxy. This example scrapes three sites concurrently and collects their content.

```TypeScript
const urls = [
  "https://news.ycombinator.com",
  "https://reddit.com/r/programming",
  "https://lobste.rs",
];

// Open 3 sessions in parallel
const results = await Promise.all(
  urls.map(async (url) => {
    const browser = new SpiderBrowser({
      apiKey: process.env.SPIDER_API_KEY,
    });
    await browser.init();
    await browser.page.goto(url);
    const content = await browser.page.content();
    await browser.close();
    return { url, content };
  })
);

console.log(`Scraped ${results.length} sites`);
```

--------------------------------

### Discord webhook example

Source: https://spider.cloud/docs/core/webhooks

Example request body for a crawl that sends a Discord webhook. Spider detects Discord webhook URLs and formats the payload as a rich embed with file attachments for HTML content and screenshot.

```json
{
  "url": "https://example.com",
  "limit": 5,
  "webhook": {
    "url": "https://discord.com/api/webhooks/123456/abcdef",
    "on_find": true
  }
}
```

--------------------------------

### Basic crawl request with requests

Source: https://spider.cloud/docs/quickstart

Sends a basic crawl request to the Spider Cloud API using the requests library. The example crawls https://example.com with a limit of 5 pages and prints the raw JSON response. Requires the SPIDER_API_KEY environment variable to be set.

```python
import requests

headers = {
    'Authorization': 'Bearer $SPIDER_API_KEY',
    'Content-Type': 'application/json',
}

json_data = {"limit":5,"url":"https://example.com"}

response = requests.post('https://api.spider.cloud/crawl', 
  headers=headers, json=json_data)

print(response.json())
```
Tokens:
Raw