### Authentication

Source: https://spider.cloud/llms.txt

Details on how to authenticate API requests using a Bearer token.

```APIDOC
## Authentication

All requests require a Bearer token:

```
Authorization: Bearer YOUR_API_KEY
```
```

--------------------------------

### AI Endpoints (Subscription Required)

Source: https://spider.cloud/llms.txt

This section details the AI-powered endpoints for natural language web data extraction.

```APIDOC
## POST /ai/crawl

### Description
AI-guided crawling with natural language prompts.

### Method
POST

### Endpoint
/ai/crawl

### Request Body
- **prompt** (string) - Required - Natural language prompt describing the crawl.
- **urls** (array[string]) - Optional - Starting URLs.

### Response
#### Success Response (200)
- **results** (array) - AI-guided crawl results.

## POST /ai/scrape

### Description
Extract structured data using plain English.

### Method
POST

### Endpoint
/ai/scrape

### Request Body
- **prompt** (string) - Required - Natural language prompt describing the data to extract.
- **url** (string) - Required - The URL to scrape.

### Response
#### Success Response (200)
- **data** (object) - Extracted structured data.

## POST /ai/search

### Description
AI-enhanced semantic web search.

### Method
POST

### Endpoint
/ai/search

### Request Body
- **prompt** (string) - Required - Natural language search query.

### Response
#### Success Response (200)
- **results** (array) - AI-enhanced search results.

## POST /ai/browser

### Description
Automate browser interactions with natural language.

### Method
POST

### Endpoint
/ai/browser

### Request Body
- **prompt** (string) - Required - Natural language instruction for browser interaction.
- **url** (string) - Required - The starting URL.

### Response
#### Success Response (200)
- **output** (string) - The result of the browser interaction.

## POST /ai/links

### Description
Intelligent link extraction and filtering.

### Method
POST

### Endpoint
/ai/links

### Request Body
- **prompt** (string) - Required - Natural language prompt for link extraction.
- **url** (string) - Required - The URL to extract links from.

### Response
#### Success Response (200)
- **links** (array[string]) - Intelligently extracted links.
```

--------------------------------

### Core API Endpoints

Source: https://spider.cloud/llms.txt

This section details the core API endpoints for web crawling, scraping, searching, and data transformation.

```APIDOC
## POST /crawl

### Description
Crawl websites and extract content from multiple pages.

### Method
POST

### Endpoint
/crawl

### Request Body
- **urls** (array[string]) - Required - List of URLs to crawl.
- **output_format** (string) - Optional - Desired output format (e.g., "markdown", "json").

### Response
#### Success Response (200)
- **results** (array) - List of crawl results.

## POST /scrape

### Description
Scrape a single page.

### Method
POST

### Endpoint
/scrape

### Request Body
- **url** (string) - Required - The URL of the page to scrape.
- **output_format** (string) - Optional - Desired output format.

### Response
#### Success Response (200)
- **content** (string) - The scraped content.

## POST /search

### Description
Search the web, optionally crawl results.

### Method
POST

### Endpoint
/search

### Request Body
- **query** (string) - Required - The search query.
- **crawl_results** (boolean) - Optional - Whether to crawl the search results.

### Response
#### Success Response (200)
- **results** (array) - List of search results.

## POST /links

### Description
Extract all links from a page.

### Method
POST

### Endpoint
/links

### Request Body
- **url** (string) - Required - The URL of the page.

### Response
#### Success Response (200)
- **links** (array[string]) - List of extracted links.

## POST /screenshot

### Description
Capture page screenshots.

### Method
POST

### Endpoint
/screenshot

### Request Body
- **url** (string) - Required - The URL of the page.

### Response
#### Success Response (200)
- **image_url** (string) - URL of the captured screenshot.

## POST /unblocker

### Description
Access bot-protected content.

### Method
POST

### Endpoint
/unblocker

### Request Body
- **url** (string) - Required - The URL of the protected page.

### Response
#### Success Response (200)
- **content** (string) - The content of the protected page.

## POST /transform

### Description
Convert HTML to markdown/text/other formats.

### Method
POST

### Endpoint
/transform

### Request Body
- **html** (string) - Required - The HTML content to transform.
- **output_format** (string) - Required - The desired output format (e.g., "markdown", "text").

### Response
#### Success Response (200)
- **content** (string) - The transformed content.

## GET /data/credits

### Description
Check available credits.

### Method
GET

### Endpoint
/data/credits

### Response
#### Success Response (200)
- **credits** (integer) - The number of available credits.
```

--------------------------------

### Crawl Web Pages with Spider Cloud API

Source: https://spider.cloud/llms.txt

Sends a POST request to the Spider Cloud crawl endpoint to retrieve web content in markdown format. Requires a valid API key in the Authorization header and a JSON body specifying the target URL and configuration limits.

```python
import requests

response = requests.post(
    "https://api.spider.cloud/crawl",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://example.com",
        "limit": 10,
        "return_format": "markdown"
    }
)

for page in response.json():
    print(page["url"], len(page["content"]), "chars")
```

```javascript
const response = await fetch("https://api.spider.cloud/crawl", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com",
    limit: 10,
    return_format: "markdown"
  })
});

const pages = await response.json();
```

```bash
curl -X POST https://api.spider.cloud/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "limit": 10, "return_format": "markdown"}'
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.