### Install Spider Go SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Use 'go get' to install the Spider Go SDK.

```bash
go get github.com/spider-rs/spider-clients/go
```

--------------------------------

### Chain Website Configurations

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Chain multiple configuration methods together for simplified setup. This example sets subdomains, TLDs, user agent, and robots.txt respect.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_subdomains(true).with_tlds(true).with_user_agent("mybot/v1").with_respect_robots_txt(true)

asyncio.run(main())
```

--------------------------------

### Install Spider Client with Yarn

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/getting-started.md

Install the Spider client library using Yarn.

```bash
yarn add @spider-cloud/spider-client
```

--------------------------------

### Install Spider Cloud Python SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Install the SDK using pip. Ensure you have Python and pip installed.

```bash
pip install spider_client
```

--------------------------------

### Install Spider Cloud JavaScript SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Install the SDK using npm or yarn.

```bash
npm install @spider-cloud/spider-client
```

```bash
yarn add @spider-cloud/spider-client
```

--------------------------------

### Install Spider Client with npm

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/getting-started.md

Install the Spider client library using npm.

```bash
npm install @spider-cloud/spider-client
```

--------------------------------

### Install Spider Python SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/getting-started.md

Install the spider-client package using pip. This is the first step to using the SDK.

```bash
pip install spider-client
```

--------------------------------

### Install Spider Cloud CLI with Homebrew

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Install the Spider Cloud CLI using Homebrew by first tapping the repository and then installing the package.

```sh
brew tap spider-rs/spider-cloud-cli
brew install spider-cloud-cli
```

--------------------------------

### Install Spider Client

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Install the Python client using pip. Set your API key as an environment variable.

```bash
pip install spider-client
```

```bash
export SPIDER_API_KEY="sk-your-key"
```

--------------------------------

### Install JavaScript/TypeScript Spider Client

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Provides instructions for installing the `@spider-cloud/spider-client` package using npm or yarn.

```bash
npm install @spider-cloud/spider-client
# or
yarn add @spider-cloud/spider-client
```

--------------------------------

### Initialize and Use Spider SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Example demonstrating how to set the API key, initialize the Spider struct, scrape a single URL, and crawl a website.

```rust
use serde_json::json;
use std::env;

#[tokio::main]
async fn main() {
    // Set the API key as an environment variable
    env::set_var("SPIDER_API_KEY", "your_api_key");

    // Initialize the Spider with your API key
    let spider = Spider::new(None).expect("API key must be provided");

    let url = "https://spider.cloud";

    // Scrape a single URL
    let scraped_data = spider.scrape_url(url, None, false, "application/json").await.expect("Failed to scrape the URL");

    println!("Scraped Data: {:?}", scraped_data);

    // Crawl a website
    let crawler_params = RequestParams {
        limit: Some(1),
        proxy_enabled: Some(true),
        metadata: Some(false),
        request: Some(RequestType::Http),
        ..Default::default()
    };

    let crawl_result = spider.crawl_url(url, Some(crawler_params), false, "application/json", None::<fn(serde_json::Value)>).await.expect("Failed to crawl the URL");

    println!("Crawl Result: {:?}", crawl_result);
}

```

--------------------------------

### Rust SDK Installation

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Specifies the dependencies required for the Rust SDK in the Cargo.toml file.

```toml
[dependencies]
spider_client = "0.1"
tokio = { version = "1", features = ["full"] }
```

--------------------------------

### Install Spider Cloud CLI with Cargo

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Install the Spider Cloud CLI using Cargo from crates.io.

```sh
cargo install spider-cloud-cli
```

--------------------------------

### Install Spider CLI via Homebrew or Cargo

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Instructions for installing the Spider Cloud CLI using Homebrew for macOS/Linux or directly via Cargo.

```bash
# Homebrew (macOS / Linux)
brew install spider-rs/tap/spider-cloud-cli

# or via Cargo

```

```bash
cargo install spider-cloud-cli
```

--------------------------------

### Example: Search With Page Fetching

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/search.md

Demonstrates how to perform a search, limit results, fetch page content, and specify return format.

```APIDOC
## Example: Search With Page Fetching

```python
from spider import Spider

app = Spider()

results = app.search(
    "best rust web crawlers",
    params={
        "search_limit": 5,
        "fetch_page_content": True,
        "base": {
            "request": "smart",
            "return_format": "markdown",
            "stealth": True
        }
    }
)

print(results)
```

This example:

* Performs a search query
* Limits results to 5
* Fetches and scrapes each result
* Uses `smart` request mode
* Returns content in Markdown format
```

--------------------------------

### Configuration Options

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Examples of configuring the Spider client with an API key, custom HTTP client, custom base URL, or AI Studio tier.

```go
// Explicit API key
client := spider.New("sk-your-api-key")

// Custom HTTP client
client := spider.New("", spider.WithHTTPClient(&http.Client{
    Timeout: 10 * time.Minute,
}))

// Custom base URL
client := spider.New("", spider.WithBaseURL("https://custom-api.example.com"))

// AI Studio tier for rate limiting
client := spider.New("", spider.WithAIStudioTier(spider.TierStandard))
```

--------------------------------

### Initialize Website Crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Initializes the Website object with a target URL and starts the crawling process. Retrieves all links found on the page.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com")
    website.crawl()
    print(website.get_links())

asyncio.run(main())
```

--------------------------------

### Quick Start Crawl URL

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Initiates a web crawl for a given URL using default API key from environment variables. Handles potential errors and iterates through results.

```go
package main

import (
    "context"
    "fmt"
    "log"

    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    // Uses SPIDER_API_KEY env var by default
    client := spider.New("")

    pages, err := client.CrawlURL(context.Background(), "https://example.com", &spider.SpiderParams{
        Limit:        10,
        ReturnFormat: spider.FormatMarkdown,
    })
    if err != nil {
        log.Fatal(err)
    }

    for _, page := range pages {
        fmt.Printf("%s: %d chars\n", page.URL, len(page.Content))
    }
}
```

--------------------------------

### Install Spider Cloud Rust SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Add the spider-client dependency to your Cargo.toml file to include the SDK in your project.

```toml
[
dependencies]
spider-client = "0.1"

```

--------------------------------

### Basic Website Scraping

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/scrape.md

Scrape a website and get its content in markdown format by default. Ensure the Spider package is installed and the API key is exported as an environment variable.

```python
from spider import Spider

app = Spider()
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)

print(scraped_data)
```

--------------------------------

### search(q, params)

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Perform a search and gather a list of websites to start crawling and collect resources.

```APIDOC
## search(q, params)

### Description
Performs a search query to find a list of websites suitable for crawling and data collection.

### Parameters
#### Path Parameters
- **q** (string) - Required - The search query.
- **params** (object) - Optional - An object containing parameters to customize the search.

### Request Example
```javascript
app.search('latest AI trends', { limit: 10 })
```

### Response
#### Success Response (200)
- **websites** (array) - A list of websites found based on the search query.
```

--------------------------------

### Get Account Credits

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Fetch the remaining account credits. This command requires authentication.

```sh
spider-cloud-cli get_credits
```

--------------------------------

### Set Crawl Budget

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Limits the number of pages the spider will crawl. The example sets a budget of 1 page for all URLs.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_budget({
    "*": 1,
  })

asyncio.run(main())
```

--------------------------------

### AI Scrape Example

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Utilizes the `aiScrape` method for AI-guided scraping, extracting specific product information based on a natural language prompt.

```javascript
// AI Scrape example
const result = await app.aiScrape(
  "https://example.com/products",
  "Extract all product names, prices, and descriptions"
);
```

--------------------------------

### Setup Cron Job for Crawling

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Schedules the website crawl to run at specified intervals using cron syntax. The crawl will execute in the background.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_cron("1/5 * * * * *")

asyncio.run(main())
```

--------------------------------

### Crawl a Website and Get Content

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/crawl.md

Initiates a crawl for a given URL and returns the scraped content. The default return format is markdown. The `limit` parameter restricts the number of pages crawled.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const url = "https://spider.cloud";
const scrapedData = await app.crawlUrl(url, { limit: 10 });
console.log(scrapedData);
```

--------------------------------

### Initialize Browser Automation Client

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Sets up a Spider client with an API key and initializes a browser instance with specified configurations like browser type and LLM settings.

```go
client := spider.New("your-api-key")

// Create a browser instance (inherits API key from client)
browser := client.Browser(
    spider.WithBrowserType("chrome"),
    spider.WithLLM(spider.LLMConfig{
        Provider: "openai",
        Model:    "gpt-4o",
        APIKey:   "your-openai-key",
    }),
)

if err := browser.Init(); err != nil {
    log.Fatal(err)
}
defer browser.Close()
```

--------------------------------

### Initialize Spider Client and Browser Instance

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Instantiate the Spider client with your API key and create a browser instance. Ensure to connect to the browser before interacting with it.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider({ apiKey: "YOUR_API_KEY" });

// Create a browser instance (inherits your API key)
const browser = app.browser();

// Connect and interact
await browser.connect();
```

--------------------------------

### Basic SDK Usage: Scrape and Crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Initialize the SDK with an API key and perform a basic URL scrape and crawl. Handles promises for asynchronous operations.

```javascript
import { Spider } from "@spider-cloud/spider-client";

// Initialize the SDK with your API key
const app = new Spider({ apiKey: "YOUR_API_KEY" });

// Scrape a URL
const url = "https://spider.cloud";
app
  .scrapeUrl(url)
  .then((data) => {
    console.log("Scraped Data:", data);
  })
  .catch((error) => {
    console.error("Scrape Error:", error);
  });

// Crawl a website
const crawlParams = {
  limit: 5,
  proxy_enabled: true,
  metadata: false,
  request: "http",
};
app
  .crawlUrl(url, crawlParams)
  .then((result) => {
    console.log("Crawl Result:", result);
  })
  .catch((error) => {
    console.error("Crawl Error:", error);
  });
```

--------------------------------

### Spider.CrawlURL

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Performs a batch crawl of a website, starting from a given URL and respecting depth and limit parameters.

```APIDOC
## Spider.CrawlURL — Batch crawl

### Description
Performs a batch crawl of a website starting from a specified URL. Allows setting limits, depth, return format, request type, and proxy usage.

### Method Signature
`client.CrawlURL(ctx context.Context, startURL string, params *SpiderParams) ([]SpiderResponse, error)`

### Parameters
#### `ctx` (context.Context)
Context for managing request lifecycle.

#### `startURL` (string)
The starting URL for the crawl.

#### `params` (*SpiderParams)
Optional parameters for crawling:
- `Limit` (int): Maximum number of pages to crawl.
- `Depth` (int): Maximum crawl depth.
- `ReturnFormat` (SpiderFormat): Desired output format (e.g., `FormatMarkdown`).
- `Request` (RequestType): Type of request to make (e.g., `RequestHTTP`).
- `ProxyEnabled` (bool): Whether to use a proxy.

### Request Example
```go
client.CrawlURL(context.Background(), "https://spider.cloud", &spider.SpiderParams{
    Limit:        20,
    Depth:        3,
    ReturnFormat: spider.FormatMarkdown,
    Request:      spider.RequestHTTP,
    ProxyEnabled: true,
})
```

### Response
#### Success Response ([]SpiderResponse)
An array of `SpiderResponse` objects for each crawled page, containing `URL` and `Content`.

#### Error Response (error)
An error if the crawl process fails.
```

--------------------------------

### Configure Spider Client

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Demonstrates various ways to configure the Spider client, including setting an explicit API key, a custom HTTP client with a timeout, a custom base URL, and specifying an AI Studio tier.

```go
// Explicit API key
client := spider.New("sk-your-api-key")
```

```go
// Custom HTTP client
client := spider.New("", spider.WithHTTPClient(&http.Client{
    Timeout: 10 * time.Minute,
}))
```

```go
// Custom base URL
client := spider.New("", spider.WithBaseURL("https://custom-api.example.com"))
```

```go
// AI Studio tier for rate limiting
client := spider.New("", spider.WithAIStudioTier(spider.TierStandard))
```

--------------------------------

### Get Credit Balance

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Retrieves the current credit balance for the Spider account. Displays the number of available credits.

```go
credits, err := client.GetCredits(ctx)
fmt.Printf("Credits: %d\n", credits.Credits)
```

--------------------------------

### Enable HTTP/2 Prior Knowledge

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Connect using HTTP/2 if the target website is known to support it for potential performance improvements.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_http2_prior_knowledge(True)

asyncio.run(main())
```

--------------------------------

### Crawl a Website with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Crawl a website using the CLI, setting limits, depth, return format, and enabling proxy.

```bash
spider-cloud-cli crawl \
  --url https://spider.cloud \
  --limit 20 \
  --depth 3 \
  --return-format markdown \
  --proxy-enabled
```

--------------------------------

### Perform Web Search with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Perform a web search using the CLI, specifying the query and the number of results.

```bash
spider-cloud-cli search --query "Rust async web scraping" --limit 5
```

--------------------------------

### Crawl a URL with Page Limit

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Crawl a website starting from a given URL, with an option to limit the number of pages visited. Authentication is required.

```sh
spider-cloud-cli crawl --url http://example.com --limit 10
```

--------------------------------

### Initialize Spider and Scrape URL

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Initialize the Spider client with an API key and scrape a single URL. The API key can be passed directly or set as an environment variable.

```python
from spider import Spider

# Initialize the Spider with your API key
app = Spider(api_key='your_api_key')

# Scrape a single URL
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)

# Crawl a website
crawler_params = {
    'limit': 1,
    'proxy_enabled': True,
    'metadata': False,
    'request': 'http'
}
crawl_result = app.crawl_url(url, params=crawler_params)
```

--------------------------------

### Set Request Timeout

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Configures the maximum time in seconds to wait for a response from a single page request. The example sets a 30-second timeout.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_request_timeout(30)

asyncio.run(main())
```

--------------------------------

### Import Spider Class

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/getting-started.md

Import the Spider class from the SDK. This is a prerequisite for initializing the client.

```javascript
import { Spider } from "@spider-cloud/spider-client";
```

--------------------------------

### Browser Automation Actions

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Demonstrates basic browser automation tasks: navigating to a URL, performing AI-powered actions like clicking, and retrieving page content.

```go
// Navigate
if err := browser.Goto("https://example.com"); err != nil {
    log.Fatal(err)
}

// AI-powered actions
browser.Act("Click the login button")

// Get page content
html, _ := browser.Page().Content(8000, 1000)
```

--------------------------------

### Batch crawl multiple URLs with Spider Go Client

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Utilize `CrawlURL` for batch crawling starting from a given URL. Supports depth-limited crawling and proxy usage.

```go
package main

import (
    "context"
    "fmt"
    "log"
    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    client := spider.New("")

    pages, err := client.CrawlURL(context.Background(), "https://spider.cloud", &spider.SpiderParams{
        Limit:        20,
        Depth:        3,
        ReturnFormat: spider.FormatMarkdown,
        Request:      spider.RequestHTTP,
        ProxyEnabled: true,
    })
    if err != nil {
        log.Fatalf("crawl error: %v", err) 
    }

    fmt.Printf("Crawled %d pages\n", len(pages))
    for _, p := range pages {
        fmt.Printf("  %s — %d chars\n", p.URL, len(p.Content))
    }
}
```

--------------------------------

### Perform web search with Spider Go Client

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Execute web searches using `Search` and optionally fetch the content of the search result pages. Specify the number of results and return format.

```go
package main

import (
    "context"
    "fmt"
    "log"
    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    client := spider.New("")

    results, err := client.Search(context.Background(), "Go web scraping", &spider.SearchParams{
        SearchLimit:      5,
        FetchPageContent: true,
        ReturnFormat:     spider.FormatMarkdown,
    })
    if err != nil {
        log.Fatalf("search error: %v", err) 
    }

    for _, r := range results {
        fmt.Println(r.URL)
    }
}
```

--------------------------------

### Perform a Basic Web Search

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/search.md

Initiates a web search and logs the results. Ensure your API key is exported as an environment variable.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();

const query = "site:spider.cloud web scraping";
const results = await app.search(query);

console.log(results);
```

--------------------------------

### Set Crawl Depth Limit

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Limits the maximum depth of pages the spider will crawl. A depth of 3 means it will crawl the starting page and up to 3 levels of linked pages.

```typescript
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_depth(3)

asyncio.run(main())
```

--------------------------------

### Crawl a Website with Spider::crawl_url (Batch and Streaming)

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Demonstrates `crawl_url` for both batch and streaming modes. Batch mode returns all crawled pages at once, while streaming invokes a callback for each page as it arrives. The API key is read from the SPIDER_API_KEY environment variable if not provided.

```rust
use spider_client::{Spider, RequestParams, ReturnFormat, RequestType};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let spider = Spider::new(None)?;

    let params = RequestParams {
        limit: Some(10),
        return_format: Some(ReturnFormat::Markdown),
        request: Some(RequestType::SmartMode),
        proxy_enabled: Some(true),
        ..Default::default()
    };

    // --- Batch ---
    let pages = spider.crawl_url("https://spider.cloud", Some(params.clone()), false, "").await?;
    println!("Crawled {} pages", pages.len());

    // --- Streaming with callback ---
    let mut count = 0usize;
    spider.crawl_url(
        "https://spider.cloud",
        Some(params),
        true,   // stream
        move |page| {
            count += 1;
            println!("[{{}}] {{}}", count, page.url.as_deref().unwrap_or(""));
        },
    ).await?;

    Ok(())
}
```

--------------------------------

### Spider.crawl_url

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Recursively crawls a website starting from the given URL, following internal links up to `limit` pages. Returns a list of page dicts. Pass `stream=True` with a `callback` for real-time processing.

```APIDOC
## Spider.crawl_url — Multi-page crawl

### Description
Recursively crawls a website starting from the given URL, following internal links up to `limit` pages. Returns a list of page dicts. Pass `stream=True` with a `callback` for real-time processing.

### Method
```python
app.crawl_url(
    "https://spider.cloud",
    params={
        "limit": 10,
        "depth": 2,
        "return_format": "markdown",
        "request": "smart",
        "proxy_enabled": True,
        "blacklist": [".*\\.pdf$", "/login"],
        "whitelist": ["/docs"],
        "metadata": False,
    },
    stream=True,
    callback=on_page,
)
```

### Parameters
#### Path Parameters
- **url** (string) - Required - The starting URL for the crawl.

#### Query Parameters
- **limit** (integer) - Optional - The maximum number of pages to crawl.
- **depth** (integer) - Optional - The maximum depth of links to follow.
- **return_format** (string) - Optional - Specifies the desired format for the returned content. Options include "raw", "markdown", "commonmark", "text", "bytes".
- **request** (string) - Optional - Specifies the request mode. Options include "http", "chrome", "smart".
- **proxy_enabled** (boolean) - Optional - If true, enables proxy usage.
- **blacklist** (list of strings) - Optional - A list of regular expressions to exclude URLs.
- **whitelist** (list of strings) - Optional - A list of regular expressions to include URLs.
- **metadata** (boolean) - Optional - If true, includes metadata in the response.

#### Callback Function (for streaming)
- **callback** (function) - Optional - A function to be called for each crawled page. It receives the page data as a dictionary.

### Response
#### Success Response (200)
- **pages** (list of dicts) - A list of dictionaries, where each dictionary represents a crawled page and contains `url`, `content`, `status`, and optional `metadata`.

### Request Example (Batch Crawl)
```python
from spider import Spider

app = Spider()

pages = app.crawl_url(
    "https://spider.cloud",
    params={
        "limit": 10,
        "depth": 2,
        "return_format": "markdown",
        "request": "smart",
        "proxy_enabled": True,
        "blacklist": [".*\\.pdf$", "/login"],
        "whitelist": ["/docs"],
        "metadata": False,
    }
)
for page in pages:
    print(page["url"], "—", len(page["content"]), "chars")
```

### Request Example (Streaming Crawl)
```python
from spider import Spider

app = Spider()

count = [0]
def on_page(data: dict) -> None:
    count[0] += 1
    print(f"[{count[0]}] {data['url']}")

app.crawl_url(
    "https://spider.cloud",
    params={"limit": 100, "return_format": "markdown"},
    stream=True,
    callback=on_page,
)
print(f"Total pages: {count[0]}")
```
```

--------------------------------

### AI-Guided Crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Perform an AI-guided crawl of a website using natural language prompts to specify the desired content. Requires an active AI Studio subscription.

```python
result = app.ai_crawl(
    url='https://example.com',
    prompt='Find all blog posts and extract titles and summaries'
)
```

--------------------------------

### Capture Screenshot with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Capture a screenshot of a URL and save it to a file using the CLI.

```bash
spider-cloud-cli screenshot --url https://spider.cloud --output shot.png
```

--------------------------------

### Clear Crawl Data

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use `website.clear()` to remove visited links and page data, or `website.drain_links()` to empty the visited links queue. This example demonstrates clearing and checking the link count.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com")
    website.crawl()
    print(website.getLinks())
    website.clear()
    print(website.getLinks())

asyncio.run(main())
```

--------------------------------

### Stop a Crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Stop a running crawl using `website.stopCrawl(id)`. Pass the crawl ID to stop a specific run, or leave empty to stop all crawls. The example shows stopping all crawls after a delay.

```python
import asyncio
from spider_rs import Website

class Subscription:
    def __init__(self):
        print("Subscription Created...")
    def __call__(self, page):
        print(page.url + " - status: " + str(page.status_code))

async def main():
    website = Website("https://choosealicense.com")
    website.crawl(Subscription())
    # sleep for 2s and stop etc
    website.stop()

asyncio.run(main())
```

--------------------------------

### AI Browser Automation

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Automate browser interactions using natural language prompts. This function guides the browser to perform actions like clicking buttons and filling forms based on the provided prompt.

```APIDOC
## AI Browser

AI-guided browser automation:

```python
result = app.ai_browser(
    url='https://example.com/login',
    prompt='Click the sign in button and fill the email field with test@example.com'
)
```
```

--------------------------------

### Import Browser Primitives

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Import specific browser automation primitives directly from the SDK for more granular control.

```javascript
import { SpiderBrowser, Agent, act, observe, extract, RetryEngine } from "@spider-cloud/spider-client";
```

--------------------------------

### Scrape URL with Python Spider Client

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/simple-example.md

Initialize the Spider client with your API key and use it to scrape a given URL. Replace 'your_api_key' with your actual key.

```python
from spider import Spider

app = Spider(api_key='your_api_key')
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)
```

--------------------------------

### Configure Proxy for Crawling

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Sets up a list of proxy servers to be used for crawling the website. This can help with IP rotation or accessing geo-restricted content.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_proxies(["https://www.myproxy.com"])

asyncio.run(main())
```

--------------------------------

### AI Browser Automation with Python

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Use `app.ai_browser` for AI-guided browser automation. Specify the URL and a prompt to instruct the AI on actions to perform.

```python
result = app.ai_browser(
    url='https://example.com/login',
    prompt='Click the sign in button and fill the email field with test@example.com'
)
```

--------------------------------

### Transform HTML to Markdown with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Convert HTML content from a URL to Markdown format using the CLI, with readability option.

```bash
spider-cloud-cli transform \
  --url https://example.com \
  --return-format markdown \
  --readability
```

--------------------------------

### Initialize Spider with API Key Argument

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/getting-started.md

Alternatively, pass your API key directly as an argument when creating an instance of the Spider class. This is useful if you prefer not to set environment variables.

```python
from spider import Spider
app = Spider(api_key='your_api_key')
```

--------------------------------

### Fetch Links from a URL

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Retrieve all the links present on a specified URL. This command requires prior authentication.

```sh
spider-cloud-cli links --url http://example.com
```

--------------------------------

### Perform Search for Websites

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Execute a search query to find websites or gather search results. Customizable with search-specific parameters.

```rust
let query = "a sports website";
let crawl_params = RequestParams {
    request: Some(RequestType::Smart),
    search_limit: Some(5),
    limit: Some(5),
    fetch_page_content: Some(true),
    ..Default::default()
};
let crawl_result = spider.search(query, Some(crawl_params), false, "application/json").await.expect("Failed to perform search");

```

--------------------------------

### Scraping with Custom Parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/scrape.md

Demonstrates how to scrape a website with specific return formats and request types.

```APIDOC
## scrapeUrl with Parameters

### Description
Scrapes a website with specified parameters for return format and request type.

### Method
`scrapeUrl(url: string, params?: SpiderParams): Promise<string | object>`

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL of the website to scrape.

#### Query Parameters
- **params** (SpiderParams) - Optional - An object containing scraping parameters.
  - **return_format** (string) - Optional - The format for the scraped data. Options: "raw", "markdown", "commonmark", "html2text", "text", "bytes". Defaults to "markdown".
  - **request** (string) - Optional - The type of request to make. Options: "http", "chrome", "smart". Defaults to "http".

### Request Example
```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const url = "https://spider.cloud";
const scrapedData = await app.scrapeUrl(url, {
  return_format: "raw",
});
console.log(scrapedData);
```

### Response
#### Success Response (200)
- **content** (string) - The scraped content of the website in the specified format.
```

--------------------------------

### Generate Dynamic Scripts with OpenAI

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use OpenAI to generate dynamic scripts for headless browsing. Ensure the OPENAI_API_KEY environment variable is set.

```python
import asyncio
from spider_rs import Website

async def main():
    website = (
        Website("https://google.com")
        .with_openai({
            "model": "gpt-3.5-turbo",
            "prompt": "Search for movies",
            "maxTokens": 300
        })
    )

asyncio.run(main())
```

--------------------------------

### Browser Automation with Go

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Initiate browser automation with stealth mode and country targeting. Manages page navigation and interaction via agent.

```go
package main

import (
    "context"
    "fmt"
    "log"
    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    client := spider.New("")

    browser, err := client.Browser(
        spider.WithStealth(true),
        spider.WithCountry("US"),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer browser.Close()

    page, _ := browser.NewPage()
    page.Goto(context.Background(), "https://example.com")

    agent := spider.Agent{Page: page}
    result, _ := agent.Act(context.Background(), "Click the sign-up button")
    fmt.Println(result)
}
```

--------------------------------

### Scrape a URL with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Scrape a single URL using the CLI, specifying return format and enabling readability.

```bash
spider-cloud-cli scrape \
  --url https://example.com \
  --return-format markdown \
  --readability
```

--------------------------------

### Async Spider Client Usage

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Demonstrates the asynchronous equivalent of the Spider client using `aiohttp`. Supports scraping, streaming crawls, and checking credits asynchronously.

```python
import asyncio
from spider import AsyncSpider

async def main():
    async with AsyncSpider(api_key="sk-your-key") as app:

        # Scrape
        pages = await app.scrape_url(
            "https://example.com",
            params={"return_format": "markdown"}
        )
        print(pages[0]["content"][:200])

        # Streaming crawl via async generator
        async for page in app.crawl_url(
            "https://spider.cloud",
            params={"limit": 20, "return_format": "markdown"},
            stream=True,
        ):
            print(page["url"])

        # Credits
        credits = await app.get_credits()
        print("Credits remaining:", credits)

asyncio.run(main())
```

--------------------------------

### Checking Available Credits

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Check the remaining credits on your account using the `get_credits` method. This is essential for monitoring your usage.

```APIDOC
## get_credits

### Description
Retrieves the number of available credits on the user's account.

### Method
`spider.get_credits() -> Result<String, SpiderError>`

### Request Example
```rust
let credits = spider.get_credits().await.expect("Failed to get credits");
println!("Remaining Credits: {:?}", credits);
```

### Response
#### Success Response (200)
- `credits` (string) - A string representing the remaining credits.
```

--------------------------------

### Crawl a URL with Default Parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/crawl.md

Initiates a website crawl using the default parameters. Ensure the Spider API key is exported as an environment variable.

```python
from spider import Spider

app = Spider()
url = "https://spider.cloud"
crawled_data = app.crawl_url(url, params={"limit": 10})
print(crawled_data)
```

--------------------------------

### Take Screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Captures a screenshot of a given URL. Accepts optional parameters for customization.

```go
pages, err := client.Screenshot(ctx, "https://example.com", nil)
```

--------------------------------

### Take Screenshot of URL

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Capture a screenshot of a given URL. Ensure the Spider client is initialized.

```rust
let url = "https://example.com";
let screenshot = spider.screenshot(url, None, false, "application/json").await.expect("Failed to take screenshot of URL");

```

--------------------------------

### Scrape URL with JavaScript Spider Client

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/simple-example.md

Initialize the Spider client with your API key and use it to scrape a given URL. Replace 'your-api-key' with your actual key. Logs the scraped data to the console.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider({ apiKey: "your-api-key" });
const url = "https://spider.cloud";
const scrapedData = await app.scrapeUrl(url);
console.log(scrapedData);
```

--------------------------------

### Streaming Web Requests with Python

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Enable streaming for web requests by setting the third parameter to `True` in the `app.links` function. Configure crawler parameters like limit and proxy settings.

```python
url = 'https://example.com'

crawler_params = {
    'limit': 1,
    'proxy_enabled': True,
    'metadata': False,
    'request': 'http'
}

links = app.links(url, crawler_params, True)
```

--------------------------------

### AI Studio methods

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Leverages large language models for intelligent crawling, scraping, search, and browser navigation. Requires an active AI Studio subscription.

```APIDOC
## AI Studio methods (Python)

### Description
AI Studio endpoints use large language models to power intelligent crawling, scraping, search, and browser navigation. Requires an active AI Studio subscription.

### Available Methods
- `ai_crawl(url: str, params: dict = None)`: AI-enhanced crawl that understands page structure semantically.
- `ai_scrape(url: str, params: dict = None)`: AI scrape to extract structured data via natural language prompt.
- `ai_search(query: str, params: dict = None)`: Performs an AI-powered search.
- `ai_browser(url: str, params: dict = None)`: Drives headless Chrome with natural language instructions.

### Parameters
Common parameters for AI Studio methods include:
- `"limit"` (int): The maximum number of results or pages to process.
- `"prompt"` (str): A natural language prompt describing the desired action or data.
- `"search_limit"` (int): For `ai_search`, the number of search results to retrieve.

### Request Example
```python
from spider import Spider, AIStudioSubscriptionRequired, AIStudioRateLimitExceeded

app = Spider()

try:
    # AI-enhanced crawl
    pages = app.ai_crawl(
        "https://example.com",
        params={"limit": 10, "prompt": "Find all product pages"}
    )

    # AI scrape
    result = app.ai_scrape(
        "https://example.com/product",
        params={"prompt": "Extract product name, price, and availability"}
    )

    # AI search
    results = app.ai_search(
        "latest AI research papers 2024",
        params={"search_limit": 5}
    )

    # AI browser
    actions = app.ai_browser(
        "https://example.com",
        params={"prompt": "Click the login button and take a screenshot"}
    )

    for page in pages:
        print(page["url"], page.get("extracted_data"))

except AIStudioSubscriptionRequired:
    print("Subscribe at https://spider.cloud/ai-studio")
except AIStudioRateLimitExceeded as e:
    print(f"Rate limited, retry after {e.retry_after_ms}ms")
```

### Response
Responses vary depending on the method, but typically include:
- `"url"` (str): The URL processed.
- `"content"` (str): The scraped or transformed content.
- `"extracted_data"` (dict, optional): Structured data extracted by AI models.

### Error Handling
- `AIStudioSubscriptionRequired`: Raised if an AI Studio subscription is not active.
- `AIStudioRateLimitExceeded`: Raised if the API rate limit is exceeded, providing a `retry_after_ms` attribute.
```

--------------------------------

### Search with Spider Cloud CLI

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Perform a search query using the CLI. This command requires authentication.

```sh
spider-cloud-cli search --query "example query"
```

--------------------------------

### Perform a Search

Source: https://github.com/spider-rs/spider-clients/blob/main/rust/README.md

Execute a search query to find websites or gather search results. The `RequestParams` can be configured to refine the search and crawling behavior.

```rust
let query = "a sports website";
let crawl_params = RequestParams {
    request: Some(RequestType::Smart),
    search_limit: Some(5),
    limit: Some(5),
    fetch_page_content: Some(true),
    ..Default::default()
};
let crawl_result = spider.search(query, Some(crawl_params), false, "application/json").await.expect("Failed to perform search");
```

--------------------------------

### aiBrowser(url, prompt, params)

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

AI-guided browser automation using natural language commands.

```APIDOC
## aiBrowser(url, prompt, params)

### Description
Automates browser interactions based on natural language commands, allowing for complex tasks to be performed programmatically.

### Parameters
#### Path Parameters
- **url** (string) - Required - The starting URL for the browser automation.
- **prompt** (string) - Required - The natural language command describing the action to perform in the browser.
- **params** (object) - Optional - An object containing additional parameters for browser automation.

### Request Example
```javascript
await app.aiBrowser('https://example.com', 'Fill out the form and submit it')
```

### Response
#### Success Response (200)
- **result** (any) - The outcome of the AI-guided browser automation.
```

--------------------------------

### Spider.screenshot

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Renders the page with headless Chrome and returns a base-64-encoded PNG. The `return_format="bytes"` option streams raw bytes instead.

```APIDOC
## Spider.screenshot — Capture full-page screenshot

### Description
Renders the page with headless Chrome and returns a base-64-encoded PNG. The `return_format="bytes"` option streams raw bytes instead.

### Method
```python
app.screenshot(
    "https://spider.cloud",
    params={
        "viewport": {"width": 1280, "height": 800},
        "full_page": True,
    }
)
```

### Parameters
#### Path Parameters
- **url** (string) - Required - The URL of the page to capture a screenshot of.

#### Query Parameters
- **viewport** (object) - Optional - Specifies the viewport dimensions for the screenshot. Should be an object with `width` and `height` properties.
- **full_page** (boolean) - Optional - If true, captures a screenshot of the entire scrollable page.
- **return_format** (string) - Optional - Specifies the desired format for the returned content. Options include "base64" (default) or "bytes".

### Response
#### Success Response (200)
- **content** (string or bytes) - The screenshot image data, either base-64 encoded string or raw bytes depending on `return_format`.
- **url** (string) - The URL of the page that was screenshotted.

### Request Example
```python
import base64
from spider import Spider

app = Spider()

result = app.screenshot(
    "https://spider.cloud",
    params={
        "viewport": {"width": 1280, "height": 800},
        "full_page": True,
    }
)

img_b64 = result[0]["content"]
with open("screenshot.png", "wb") as f:
    f.write(base64.b64decode(img_b64))
print("Saved screenshot.png")
```
```

--------------------------------

### Check API Credits with CLI

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Check remaining API credits using the CLI. Returns credits in JSON format.

```bash
spider-cloud-cli get_credits
# {"credits": 42000}
```

--------------------------------

### Enable Streaming with Callback

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Set the `stream` parameter to `true` and provide a callback function to handle incoming JSON data chunks.

```rust
fn handle_json(json_obj: serde_json::Value) {
    println!("Received chunk: {:?}", json_obj);
}

let url = "https://example.com";
let crawler_params = RequestParams {
    limit: Some(1),
    proxy_enabled: Some(true),
    metadata: Some(false),
    request: Some(RequestType::Http),
    ..Default::default()
};

spider.links(url, Some(crawler_params), true, "application/json").await.expect("Failed to retrieve links from URL");
```

--------------------------------

### Crawl with Streaming Response

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Demonstrates how to crawl a website and receive the response as a stream, processing data via a callback function.

```javascript
import { Spider } from "@spider-cloud/spider-client";

// Initialize the SDK with your API key
const app = new Spider({ apiKey: "YOUR_API_KEY" });

// The target URL
const url = "https://spider.cloud";

// Crawl a website
const crawlParams = {
  limit: 5,
  metadata: true,
  request: "http",
};

const stream = true;

const streamCallback = (data) => {
  console.log(data["url"]);
};

app.crawlUrl(url, crawlParams, stream, streamCallback);
```

--------------------------------

### AI-Guided Scrape

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Execute an AI-guided scrape of a URL with natural language prompts to extract specific data. This feature requires an active AI Studio subscription.

```python
result = app.ai_scrape(
    url='https://example.com/products',
    prompt='Extract all product names, prices, and descriptions'
)
```

--------------------------------

### Authenticate Spider Cloud CLI with API Key

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Authenticate the CLI with your Spider Cloud API key before using most commands. Replace YOUR_API_KEY with your actual key.

```sh
spider-cloud-cli auth --api_key YOUR_API_KEY
```

--------------------------------

### AI Studio Crawl with Go

Source: https://context7.com/spider-rs/spider-clients/llms.txt

Use the AICrawl method for AI-powered data extraction. Handles subscription and rate limit errors.

```go
package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    client := spider.New("", spider.WithAIStudioTier(spider.TierPro))

    pages, err := client.AICrawl(context.Background(), "https://example.com", &spider.AIParams{
        Limit:  10,
        Prompt: "Find all product pages",
    })

    var subErr *spider.AIStudioSubscriptionRequired
    var rlErr  *spider.AIStudioRateLimitExceeded
    switch {
    case errors.As(err, &subErr):
        log.Fatal("Subscribe at https://spider.cloud/ai-studio")
    case errors.As(err, &rlErr):
        log.Fatalf("Rate limited, retry after %dms", rlErr.RetryAfterMs)
    case err != nil:
        log.Fatal(err)
    }

    for _, p := range pages {
        fmt.Println(p.URL)
    }
}
```

--------------------------------

### Async Crawl with Parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/async-crawl.md

Crawl a URL asynchronously with specific parameters such as `limit`, `request_timeout`, `stealth`, and `return_format`. Ensure parameters are correctly formatted for the `crawl_url` method.

```python
import asyncio

from spider import AsyncSpider

url = "https://spider.cloud"


async def async_crawl_url(url, params):
    async with AsyncSpider() as app:
        crawled_data = []
        async for data in app.crawl_url(url, params=params):
            crawled_data.append(data)
    return crawled_data


result = asyncio.run(
    async_crawl_url(
        url,
        params={
            "limit": 10,
            "request_timeout": 10,
            "stealth": True,
            "return_format": "html",
        },
    )
)
print(result)
```

--------------------------------

### Take Screenshot of a URL

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Generate a screenshot for a given URL. Ensure you are authenticated before executing this command.

```sh
spider-cloud-cli screenshot --url http://example.com
```

--------------------------------

### Crawl URL with Streaming

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Initiates a web crawl and processes results as they arrive via a callback function. Useful for handling large amounts of data efficiently.

```go
err := client.CrawlURLStream(ctx, "https://example.com", &spider.SpiderParams{
    Limit:        100,
    ReturnFormat: spider.FormatMarkdown,
}, func(page spider.SpiderResponse) {
    fmt.Printf("Received: %s\n", page.URL)
})
```

--------------------------------

### AI Link Extraction and Filtering with Python

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Employ `app.ai_links` for AI-guided link extraction. Provide a URL and a prompt to find specific types of links, such as product or documentation pages.

```python
result = app.ai_links(
    url='https://example.com',
    prompt='Find all links to product pages and documentation'
)
```