Spider Clients

https://github.com/spider-rs/spider-clients
Admin
Spider Clients provides libraries for integrating the Spider web crawler into Python, JavaScript,...
Tokens:25,234
Snippets:342
Trust Score:7.8
License:MIT
Update:3 weeks ago
Show doc for...
Tokens:
Raw
### Quick Start: Crawl and Print Pages

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

A complete quick start example that initializes the client using the SPIDER_API_KEY environment variable by default, crawls a URL, and prints the URL and content length of each page.

```go
package main

import (
    "context"
    "fmt"
    "log"

    spider "github.com/spider-rs/spider-clients/go"
)

func main() {
    // Uses SPIDER_API_KEY env var by default
    client := spider.New("")

    pages, err := client.CrawlURL(context.Background(), "https://example.com", &spider.SpiderParams{
        Limit:        10,
        ReturnFormat: spider.FormatMarkdown,
    })
    if err != nil {
        log.Fatal(err)
    }

    for _, page := range pages {
        fmt.Printf("%s: %d chars\n", page.URL, len(page.Content))
    }
}
```

--------------------------------

### Install Spider Go SDK

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Install the Spider Go SDK using the go get command.

```bash
go get github.com/spider-rs/spider-clients/go
```

--------------------------------

### Complete usage example: scrape and crawl with Spider

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Shows the full setup: setting the SPIDER_API_KEY environment variable, creating a Spider instance, scraping a URL, and crawling with custom RequestParams. Requires the tokio runtime and serde_json.

```rust
use serde_json::json;
use std::env;

#[tokio::main]
async fn main() {
    // Set the API key as an environment variable
    env::set_var("SPIDER_API_KEY", "your_api_key");

    // Initialize the Spider with your API key
    let spider = Spider::new(None).expect("API key must be provided");

    let url = "https://spider.cloud";

    // Scrape a single URL
    let scraped_data = spider.scrape_url(url, None, false, "application/json").await.expect("Failed to scrape the URL");

    println!("Scraped Data: {:?}", scraped_data);

    // Crawl a website
    let crawler_params = RequestParams {
        limit: Some(1),
        proxy_enabled: Some(true),
        metadata: Some(false),
        request: Some(RequestType::Http),
        ..Default::default()
    };

    let crawl_result = spider.crawl_url(url, Some(crawler_params), false, "application/json", None::<fn(serde_json::Value)>).await.expect("Failed to crawl the URL");

    println!("Crawl Result: {:?}", crawl_result);
}
```

--------------------------------

### Install with Homebrew

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Install the Spider Cloud CLI using Homebrew. Requires Homebrew to be installed on the system.

```sh
brew tap spider-rs/spider-cloud-cli
brew install spider-cloud-cli
```

--------------------------------

### Install spider-client with pip

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/getting-started.md

Install the Python SDK package using pip. This is the first step to using the SDK.

```bash
pip install spider-client
```

--------------------------------

### Install @spider-cloud/spider-client with npm or yarn

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/getting-started.md

Installs the JavaScript SDK using npm or yarn. Choose the command that matches your package manager.

```bash
npm install @spider-cloud/spider-client
```

```bash
yarn add @spider-cloud/spider-client
```

--------------------------------

### Install @spider-cloud/spider-client with npm

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/simple-example.md

Use this command to install the JavaScript package @spider-cloud/spider-client via npm. Requires Node.js to be installed.

```bash
npm install @spider-cloud/spider-client
```

--------------------------------

### Basic scrape and crawl example

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Basic usage of the Spider SDK: initialize with an API key, scrape a URL, and crawl a website with parameters. The example uses promises and logs results or errors.

```javascript
import { Spider } from "@spider-cloud/spider-client";

// Initialize the SDK with your API key
const app = new Spider({ apiKey: "YOUR_API_KEY" });

// Scrape a URL
const url = "https://spider.cloud";
app
  .scrapeUrl(url)
  .then((data) => {
    console.log("Scraped Data:", data);
  })
  .catch((error) => {
    console.error("Scrape Error:", error);
  });

// Crawl a website
const crawlParams = {
  limit: 5,
  proxy_enabled: true,
  metadata: false,
  request: "http",
};
app
  .crawlUrl(url, crawlParams)
  .then((result) => {
    console.log("Crawl Result:", result);
  })
  .catch((error) => {
    console.error("Crawl Error:", error);
  });
```

--------------------------------

### Install with Cargo

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Install the Spider Cloud CLI using Cargo from crates.io. Requires Cargo to be installed.

```sh
cargo install spider-cloud-cli
```

--------------------------------

### Streaming crawl example

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Real-world crawl example that streams the response. Passes a stream flag and a callback that logs each data item's URL.

```javascript
import { Spider } from "@spider-cloud/spider-client";

// Initialize the SDK with your API key
const app = new Spider({ apiKey: "YOUR_API_KEY" });

// The target URL
const url = "https://spider.cloud";

// Crawl a website
const crawlParams = {
  limit: 5,
  metadata: true,
  request: "http",
};

const stream = true;

const streamCallback = (data) => {
  console.log(data["url"]);
};

app.crawlUrl(url, crawlParams, stream, streamCallback);
```

--------------------------------

### Chain configurations with with_subdomains, with_tlds, with_user_agent, with_respect_robots_txt

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Chain multiple configuration methods together for a simple setup, including subdomains, TLDs, user agent, and robots.txt respect.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_subdomains(true).with_tlds(true).with_user_agent("mybot/v1").with_respect_robots_txt(true)

asyncio.run(main())
```

--------------------------------

### AI Scrape example

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

AI-guided scraping using a natural language prompt. Requires an active AI Studio subscription.

```javascript
// AI Scrape example
const result = await app.aiScrape(
  "https://example.com/products",
  "Extract all product names, prices, and descriptions"
);
```

--------------------------------

### Get Account Credits

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Fetch the account credits left.

```sh
spider-cloud-cli get_credits
```

--------------------------------

### Schedule crawling with with_cron

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Set up a cron job that runs in the background using cron-syntax. The example uses a schedule of every 5 seconds. See the cron section for details.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_cron("1/5 * * * * *")

asyncio.run(main())
```

--------------------------------

### Unlimited scrape and crawl examples

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Examples of using the Unlimited plan methods: unlimitedScrape and unlimitedCrawl. Requires an active Unlimited subscription. The Unlimited plan bills a flat monthly rate by concurrency seats; requests are not queued and return 429 with Retry-After when all seats are in flight. AI/LLM extraction params are rejected with 400.

```javascript
// Unlimited scrape example
const data = await app.unlimitedScrape("https://spider.cloud");

// Unlimited crawl example
const result = await app.unlimitedCrawl("https://spider.cloud", { limit: 5 });
```

--------------------------------

### Get Links

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Extracts links from the given URL with default parameters.

```go
pages, err := client.Links(ctx, "https://example.com", nil)
```

--------------------------------

### Get Account Credits

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Retrieves the account's remaining credits and prints the number.

```go
credits, err := client.GetCredits(ctx)
fmt.Printf("Credits: %d\n", credits.Credits)
```

--------------------------------

### Limit crawl pages with with_budget

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Add a crawl budget that limits the number of pages crawled. The example sets a budget of 1 page for all paths.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_budget({
    "*": 1,
  })

asyncio.run(main())
```

--------------------------------

### Whitelist URLs with with_whitelist_url

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Only crawl specified paths, URLs, or patterns using regex. The example restricts crawling to /licenses.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_whitelist_url(["/licenses"])

asyncio.run(main())
```

--------------------------------

### Set request timeout with with_request_timeout

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Add a request timeout per page in milliseconds. The example shows 30 seconds.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_request_timeout(30)

asyncio.run(main())
```

--------------------------------

### Blacklist URLs with with_blacklist_url

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Prevent crawling specific paths, URLs, or patterns using regex. The example blocks /blog and /resume.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_blacklist_url(["/blog", "/resume"])

asyncio.run(main())
```

--------------------------------

### Set custom HTTP headers with with_headers

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Add custom HTTP headers to the crawl request. The example sets an authorization header with a token.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_headers({ "authorization": "mytoken"})

asyncio.run(main())
```

--------------------------------

### Stream crawl with callback

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Stream crawl a website in chunks, processing each JSON object with a callback function. Note: the example uses 'params' but the variable is defined as 'crawl_params'.

```python
    def handle_json(json_obj: dict) -> None:
        assert json_obj["url"] is not None

    url = 'https://example.com'
    crawl_params = {
        'limit': 200,
    }
    response = app.crawl_url(
        url,
        params=params,
        stream=True,
        callback=handle_json,
    )
```

--------------------------------

### Get Recording Video URL

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Retrieve the URL for a recorded browser session using the session ID.

```javascript
const videoUrl = Spider.getRecordingVideoUrl("session-id");
```

--------------------------------

### Scrape a website with Spider

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/scrape.md

Basic usage of the Spider client to scrape a website. The `scrapeUrl` method returns the content in markdown format by default. Requires the Spider package installed and the API key exported as an environment variable.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const url = "https://spider.cloud";
const scrapedData = await app.scrapeUrl(url);
console.log(scrapedData);
```

--------------------------------

### Async Crawl with Typed RequestParamsDict Variable

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/async-crawl.md

Uses a separate params variable typed as `spider_types.RequestParamsDict` to hold crawl parameters, making configuration cleaner when there are many options. This example sets limit, request_timeout, and stealth.

```python
import asyncio

from spider import AsyncSpider, spider_types

url = "https://spider.cloud"


async def async_crawl_url(url, params):
    async with AsyncSpider() as app:
        crawled_data = []
        async for data in app.crawl_url(url, params=params):
            crawled_data.append(data)
    return crawled_data


params: spider_types.RequestParamsDict = {
    "limit": 10,
    "request_timeout": 10,
    "stealth": True,
}

result = asyncio.run(async_crawl_url(url, params=params))
print(result)
```

--------------------------------

### Clear crawl data with website.clear and drain links with website.drain_links

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use `website.clear` to remove visited links and page data, or `website.drain_links` to drain the links visited. The example shows clearing after crawling and printing the link list before and after.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com")
    website.crawl()
    print(website.getLinks())
    website.clear()
    print(website.getLinks())

asyncio.run(main())
```

--------------------------------

### Async Crawl Basic Usage with AsyncSpider.crawl_url

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/async-crawl.md

Crawl a URL asynchronously using the AsyncSpider class and the crawl_url method, which yields data via an async generator. The example uses a limit of 10 pages and prints the result.

```python
import asyncio

from spider import AsyncSpider

url = "https://spider.cloud"


async def async_crawl_url(url, params):
    async with AsyncSpider() as app:
        crawled_data = []
        async for data in app.crawl_url(url, params=params):
            crawled_data.append(data)
    return crawled_data


result = asyncio.run(async_crawl_url(url, params={"limit": 10}))
print(result)
```

--------------------------------

### Initialize and Use Browser Automation

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Shows how to initialize a browser instance with LLM configuration, navigate to a URL, perform AI-powered actions, and retrieve page content.

```go
client := spider.New("your-api-key")

// Create a browser instance (inherits API key from client)
browser := client.Browser(
    spider.WithBrowserType("chrome"),
    spider.WithLLM(spider.LLMConfig{
        Provider: "openai",
        Model:    "gpt-4o",
        APIKey:   "your-openai-key",
    }),
)

if err := browser.Init(); err != nil {
    log.Fatal(err)
}
defer browser.Close()

// Navigate
if err := browser.Goto("https://example.com"); err != nil {
    log.Fatal(err)
}

// AI-powered actions
browser.Act("Click the login button")

// Get page content
html, _ := browser.Page().Content(8000, 1000)
```

--------------------------------

### Configure OpenAI with with_openai

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use OpenAI to generate dynamic scripts for headless browsing. Requires the OPENAI_API_KEY environment variable to be set.

```python
import asyncio
from spider_rs import Website

async def main():
    website = (
        Website("https://google.com")
        .with_openai({
            "model": "gpt-3.5-turbo",
            "prompt": "Search for movies",
            "maxTokens": 300
        })
    )

asyncio.run(main())
```

--------------------------------

### Client Configuration Options

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Shows four client configuration variations: explicit API key, custom HTTP client with a 10-minute timeout, custom base URL, and AI Studio tier for rate limiting.

```go
// Explicit API key
client := spider.New("sk-your-api-key")

// Custom HTTP client
client := spider.New("", spider.WithHTTPClient(&http.Client{
    Timeout: 10 * time.Minute,
}))

// Custom base URL
client := spider.New("", spider.WithBaseURL("https://custom-api.example.com"))

// AI Studio tier for rate limiting
client := spider.New("", spider.WithAIStudioTier(spider.TierStandard))
```

--------------------------------

### Crawl a website and print links

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Basic usage of the Website class: create a Website instance, crawl the target URL, and print the extracted links. Replace the URL with your target.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com")
    website.crawl()
    print(website.get_links())

asyncio.run(main())
```

--------------------------------

### Search with Query

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Search for a query using the CLI.

```sh
spider-cloud-cli search --query "example query"
```

--------------------------------

### Scrape a URL with spider-client in Python

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/simple-example.md

Initializes the Spider client with an API key and scrapes a URL. Replace 'your_api_key' with your actual key from spider.cloud.

```python
from spider import Spider

app = Spider(api_key='your_api_key')
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)
```

--------------------------------

### Enable HTTP/2 with with_http2_prior_knowledge

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Enable HTTP/2 prior knowledge when the target website is known to support HTTP/2.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_http2_prior_knowledge(True)

asyncio.run(main())
```

--------------------------------

### Collect full resources with with_full_resources

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Collect all resources found, not just valid web pages.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_full_resources(True)

asyncio.run(main())
```

--------------------------------

### AI Browser with app.ai_browser

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

AI-guided browser automation using a natural language prompt. Requires an active AI Studio subscription.

```python
result = app.ai_browser(
    url='https://example.com/login',
    prompt='Click the sign in button and fill the email field with test@example.com'
)
```

--------------------------------

### Async Crawl with Custom Parameters (limit, request_timeout, stealth, return_format)

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/async-crawl.md

Crawl with a dictionary of parameters including request_timeout, stealth, and return_format. Demonstrates how to configure the crawl_url method with additional options for more control.

```python
import asyncio

from spider import AsyncSpider

url = "https://spider.cloud"


async def async_crawl_url(url, params):
    async with AsyncSpider() as app:
        crawled_data = []
        async for data in app.crawl_url(url, params=params):
            crawled_data.append(data)
    return crawled_data


result = asyncio.run(
    async_crawl_url(
        url,
        params={
            "limit": 10,
            "request_timeout": 10,
            "stealth": True,
            "return_format": "html",
        },
    )
)
print(result)
```

--------------------------------

### Handle Unlimited Endpoint Errors

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Demonstrates how to handle errors specific to unlimited endpoints, such as plan requirements and concurrency limits.

```go
pages, err := client.UnlimitedCrawlURL(ctx, url, nil)
if err != nil {
    switch e := err.(type) {
    case *spider.UnlimitedPlanRequired:
        // Need an active plan: https://spider.cloud/pricing?plan=unlimited
    case *spider.UnlimitedConcurrencyLimitReached:
        // All e.Seats seats in flight — wait e.RetryAfterMs milliseconds, then retry
    case *spider.APIError:
        // HTTP error: e.StatusCode
    default:
        // Network or other error
    }
}
```

--------------------------------

### Stop a crawl with website.stopCrawl(id)

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Pass a crawl id to `website.stopCrawl(id)` to stop a specific run, or leave empty to stop all crawls. The example shows a subscription callback and a commented sleep before stopping.

```python
import asyncio
from spider_rs import Website

class Subscription:
    def __init__(self):
        print("Subscription Created...")
    def __call__(self, page):
        print(page.url + " - status: " + str(page.status_code))

async def main():
    website = Website("https://choosealicense.com")
    website.crawl(Subscription())
    # sleep for 2s and stop etc
    website.stop()

asyncio.run(main())
```

--------------------------------

### Enable raw content mode with Website constructor second parameter

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Set the second parameter of the Website constructor to `true` to return content without UTF-8 encoding. This populates `rawContent` and leaves `content` unchanged when using subscriptions or the Page Object.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com", True)
    website.scrape_url()

asyncio.run(main())
```

--------------------------------

### Check available credits with app.get_credits

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Check the remaining credits on your account.

```python
credits = app.get_credits()
```

--------------------------------

### Take Screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Captures a screenshot of the given URL with default parameters.

```go
pages, err := client.Screenshot(ctx, "https://example.com", nil)
```

--------------------------------

### Scrape a URL with @spider-cloud/spider-client in JavaScript

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/simple-example.md

Initializes the Spider client with an API key and scrapes a URL asynchronously. Replace 'your-api-key' with your actual key from spider.cloud.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider({ apiKey: "your-api-key" });
const url = "https://spider.cloud";
const scrapedData = await app.scrapeUrl(url);
console.log(scrapedData);
```

--------------------------------

### Browser Automation with Spider

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Create a browser instance using the SDK's built-in browser automation via spider-browser. The browser instance inherits your API key and provides WebSocket-based CDP/BiDi control.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider({ apiKey: "YOUR_API_KEY" });

// Create a browser instance (inherits your API key)
const browser = app.browser();

// Connect and interact
await browser.connect();
```

--------------------------------

### Crawl URL with Parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Crawls a URL with a limit of 50 pages, markdown format, and the smart request mode.

```go
pages, err := client.CrawlURL(ctx, "https://example.com", &spider.SpiderParams{
    Limit:        50,
    ReturnFormat: spider.FormatMarkdown,
    Request:      spider.RequestSmart,
})
```

--------------------------------

### Take a screenshot with app.screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Capture a screenshot of a given URL.

```python
url = 'https://example.com'
screenshot = app.screenshot(url)
```

--------------------------------

### Respect robots.txt with with_respect_robots_txt

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Respect the robots.txt file during crawling.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_respect_robots_txt(True)

asyncio.run(main())
```

--------------------------------

### AI Crawl with app.ai_crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

AI-guided crawling using a natural language prompt. Requires an active AI Studio subscription.

```python
result = app.ai_crawl(
    url='https://example.com',
    prompt='Find all blog posts and extract titles and summaries'
)
```

--------------------------------

### Enable HTTP caching with with_caching

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Enable HTTP caching, useful when using the spider on a server.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_caching(True)

asyncio.run(main())
```

--------------------------------

### Import Browser Primitives

Source: https://github.com/spider-rs/spider-clients/blob/main/javascript/README.md

Import browser primitives directly from the SDK for advanced browser automation.

```javascript
import { SpiderBrowser, Agent, act, observe, extract, RetryEngine } from "@spider-cloud/spider-client";
```

--------------------------------

### Use proxy with with_proxies

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use a proxy to crawl a website.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_proxies(["https://www.myproxy.com"])

asyncio.run(main())
```

--------------------------------

### AI Studio endpoints: SetAIStudioTier, AICrawl, AIScrape, AISearch, AIBrowser, AILinks

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

AI Studio endpoints require an active subscription. Set the client-side rate limiting tier with SetAIStudioTier, then call AICrawl, AIScrape, AISearch, AIBrowser, or AILinks with a context, URL, and prompt.

```go
// Set tier for client-side rate limiting
client.SetAIStudioTier(spider.TierStandard)

// AI Crawl
result, err := client.AICrawl(ctx, "https://example.com", "Extract all product names and prices", nil)

// AI Scrape
result, err := client.AIScrape(ctx, "https://example.com/pricing", "Get all plan names and monthly costs", nil)

// AI Search
result, err := client.AISearch(ctx, "Find the top 5 web scraping APIs with pricing", nil)

// AI Browser
result, err := client.AIBrowser(ctx, "https://example.com", "Click the login button and fill in the form", nil)

// AI Links
result, err := client.AILinks(ctx, "https://example.com", "Find all documentation links", nil)
```

--------------------------------

### Fetch Links from a URL

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Fetch links from the specified URL.

```sh
spider-cloud-cli links --url http://example.com
```

--------------------------------

### Authenticate with API Key

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Authenticate the CLI by providing your API key. Required before using most commands.

```sh
spider-cloud-cli auth --api_key YOUR_API_KEY
```

--------------------------------

### Take a screenshot of a URL with spider.screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Calls spider.screenshot on a URL to capture a screenshot. Expects the spider instance to be already created.

```rust
let url = "https://example.com";
let screenshot = spider.screenshot(url, None, false, "application/json").await.expect("Failed to take screenshot of URL");
```

--------------------------------

### AI Scrape with app.ai_scrape

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

AI-guided scraping using a natural language prompt. Requires an active AI Studio subscription.

```python
result = app.ai_scrape(
    url='https://example.com/products',
    prompt='Extract all product names, prices, and descriptions'
)
```

--------------------------------

### Search with Page Fetching and Parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/search.md

Searches with parameters to limit results to 5, fetch and scrape each result, use smart request mode, and return content in Markdown format. The base field accepts all scrape parameters.

```python
from spider import Spider

app = Spider()

results = app.search(
    "best rust web crawlers",
    params={
        "search_limit": 5,
        "fetch_page_content": True,
        "base": {
            "request": "smart",
            "return_format": "markdown",
            "stealth": True
        }
    }
)

print(results)
```

--------------------------------

### Take a Screenshot of a URL

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Take a screenshot of the specified URL.

```sh
spider-cloud-cli screenshot --url http://example.com
```

--------------------------------

### Search with fetch_page_content and base parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/search.md

Searches with parameters to limit results to 5, fetch and scrape each result's page content, use smart request mode, and return content in Markdown.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();

const results = await app.search("best rust web crawlers", {
  search_limit: 5,
  fetch_page_content: true,
  base: {
    request: "smart",
    return_format: "markdown",
    stealth: true,
  },
});

console.log(results);
```

--------------------------------

### Unlimited endpoints: UnlimitedScrapeURL, UnlimitedCrawlURL, UnlimitedCrawlURLStream, UnlimitedLinks

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Unlimited endpoints require an active Unlimited subscription. Use UnlimitedScrapeURL, UnlimitedCrawlURL, UnlimitedCrawlURLStream, and UnlimitedLinks. Unlimited plans bill a flat monthly rate for concurrency seats; requests are never queued, returning 429 with Retry-After when all seats are in flight. AI/LLM extraction parameters are rejected with 400.

```go
// Unlimited Scrape
pages, err := client.UnlimitedScrapeURL(ctx, "https://example.com/page", &spider.SpiderParams{
    ReturnFormat: spider.FormatMarkdown,
})

// Unlimited Crawl
pages, err := client.UnlimitedCrawlURL(ctx, "https://example.com", &spider.SpiderParams{
    Limit:        50,
    ReturnFormat: spider.FormatMarkdown,
})

// Unlimited Crawl with streaming
err := client.UnlimitedCrawlURLStream(ctx, "https://example.com", nil, func(page spider.SpiderResponse) {
    fmt.Printf("Received: %s\n", page.URL)
})

// Unlimited Links
pages, err := client.UnlimitedLinks(ctx, "https://example.com", nil)
```

--------------------------------

### AI Scrape and AI Search with Spider Rust Client

Source: https://github.com/spider-rs/spider-clients/blob/main/rust/README.md

Use ai_scrape to extract specific data from a URL and ai_search to find web content using natural language prompts. Requires an active AI Studio subscription, billed separately from credits.

```rust
let url = "https://example.com";
let scraped = spider.ai_scrape(url, "Extract the page title and main heading", None, "application/json").await.expect("Failed to AI scrape the URL");

let results = spider.ai_search("Find the top Rust web frameworks", None, "application/json").await.expect("Failed to AI search");
```

--------------------------------

### AI Search with app.ai_search

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

AI-enhanced web search using a natural language prompt. Requires an active AI Studio subscription.

```python
result = app.ai_search(prompt='Find the best Python web scraping libraries')
```

--------------------------------

### Authenticate with API Key

Source: https://github.com/spider-rs/spider-clients/blob/main/cli/README.md

Authenticate the CLI by providing your Spider Cloud API key. Required before using most commands.

```sh
spider-cloud-cli auth --api-key YOUR_API_KEY
```

--------------------------------

### Scrape Single Page

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Scrapes a single page and returns the content in markdown format.

```go
pages, err := client.ScrapeURL(ctx, "https://example.com/page", &spider.SpiderParams{
    ReturnFormat: spider.FormatMarkdown,
})
```

--------------------------------

### Set custom User-Agent with with_user_agent

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Use a custom User-Agent for the crawl.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_user_agent("mybot/v1")

asyncio.run(main())
```

--------------------------------

### Scrape a website with Spider

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/scrape.md

Basic usage of the Spider SDK to scrape a website. The `scrape_url` method returns the content in markdown format by default.

```python
from spider import Spider

app = Spider()
url = 'https://spider.cloud'
scraped_data = app.scrape_url(url)

print(scraped_data)
```

--------------------------------

### AI Links with app.ai_links

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

AI-guided link extraction and filtering using a natural language prompt. Requires an active AI Studio subscription.

```python
result = app.ai_links(
    url='https://example.com',
    prompt='Find all links to product pages and documentation'
)
```

--------------------------------

### Transform HTML to markdown with app.transform

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Transform HTML to markdown or text using the transform method.

```python
data = [ { 'html': '<html><body><h1>Hello world</h1></body></html>' } ]
params = {
    'readability': False,
    'return_format': 'markdown',
}
result = app.transform(data, params=params)
```

--------------------------------

### Set SPIDER_API_KEY environment variable

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/getting-started.md

Export your API key as an environment variable. This is the recommended way to configure the SDK.

```bash
export SPIDER_API_KEY=your_api_key
```

--------------------------------

### Streaming with Callback

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

To enable streaming, set the stream parameter to true and provide a callback function to process the received JSON chunks.

```rust
fn handle_json(json_obj: serde_json::Value) {
    println!("Received chunk: {:?}", json_obj);
}

let url = "https://example.com";
let crawler_params = RequestParams {
    limit: Some(1),
    proxy_enabled: Some(true),
    metadata: Some(false),
    request: Some(RequestType::Http),
    ..Default::default()
};

spider.links(url, Some(crawler_params), true, "application/json").await.expect("Failed to retrieve links from URL");
```

--------------------------------

### Crawl a website with crawl_url and RequestParams

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Crawls a website with a limit of 200 and the Smart request type. Uses the spider instance and returns JSON data.

```rust
let url = "https://example.com";
let crawl_params = RequestParams {
    limit: Some(200),
    request: Some(RequestType::Smart),
    ..Default::default()
};
let crawl_result = spider.crawl_url(url, Some(crawl_params), false, "application/json", None::<fn(serde_json::Value)>).await.expect("Failed to crawl the URL");
```

--------------------------------

### Basic Search with app.search

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/search.md

Performs a basic search query and prints the structured search results. By default, the /search endpoint returns metadata such as titles, URLs, and snippets without fetching page content.

```python
from spider import Spider

app = Spider()

query = "site:spider.cloud web scraping"
results = app.search(query)

print(results)
```

--------------------------------

### Retrieve links from a URL with app.links

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Extract all links from a specified URL.

```python
url = 'https://example.com'
links = app.links(url)
```

--------------------------------

### Search for websites with app.search

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Perform a search for websites to crawl or gather search results with specified parameters.

```python
query = 'a sports website'
crawl_params = {
    'request': 'smart_mode',
    'search_limit': 5,
    'limit': 5,
    'fetch_page_content': True
}
crawl_result = app.search(query, params=crawl_params)
```

--------------------------------

### Screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Takes a screenshot of a URL and returns the result. It accepts a context, target URL, and optional parameters.

```APIDOC
## Screenshot

### Description
Takes a screenshot of a URL and returns the result.

### Method
Screenshot

### Parameters
- **ctx** (context.Context) - Required - The context for the request.
- **url** (string) - Required - The URL to screenshot.
- **params** (*spider.SpiderParams) - Optional - Parameters for the screenshot.

### Request Example
go
pages, err := client.Screenshot(ctx, "https://example.com", nil)


### Response
Returns a list of pages, each containing URL and Content fields.

```

--------------------------------

### AI Studio Commands

Source: https://github.com/spider-rs/spider-clients/blob/main/cli/README.md

AI Studio commands grouped under 'ai'. These require an active AI Studio subscription, billed separately from credits. See spider.cloud/ai/pricing for details.

```sh
spider-cloud-cli ai scrape --url http://example.com --prompt "Extract the product name and price"
spider-cloud-cli ai crawl --url http://example.com --prompt "Summarize each blog post" --limit 10
spider-cloud-cli ai search --prompt "Find the best Rust web crawlers"
spider-cloud-cli ai browser --url http://example.com --prompt "Click the pricing tab and read the plans"
spider-cloud-cli ai links --url http://example.com --prompt "Only the documentation links"
```

--------------------------------

### Import Spider class from @spider-cloud/spider-client

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/getting-started.md

Imports the Spider class from the JavaScript SDK. This is the first step to use the SDK in your project.

```javascript
import { Spider } from "@spider-cloud/spider-client";
```

--------------------------------

### Extract links from a URL with spider.links

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Calls spider.links on a URL to retrieve all links. Expects the spider instance to be already created.

```rust
let url = "https://example.com";
let links = spider.links(url, None, false, "application/json").await.expect("Failed to retrieve links from URL");
```

--------------------------------

### Crawl a URL with Limit

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/cli/getting-started.md

Crawl a URL with an optional limit on the number of pages.

```sh
spider-cloud-cli crawl --url http://example.com --limit 10
```

--------------------------------

### Search for websites with spider.search

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Performs a search using the spider instance with a query and custom RequestParams including search_limit and fetch_page_content. Returns JSON data.

```rust
let query = "a sports website";
let crawl_params = RequestParams {
    request: Some(RequestType::Smart),
    search_limit: Some(5),
    limit: Some(5),
    fetch_page_content: Some(true),
    ..Default::default()
};
let crawl_result = spider.search(query, Some(crawl_params), false, "application/json").await.expect("Failed to perform search");
```

--------------------------------

### Crawl URL with Streaming

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Crawls a URL with streaming, processing each page as it arrives via a callback that prints the page URL.

```go
err := client.CrawlURLStream(ctx, "https://example.com", &spider.SpiderParams{
    Limit:        100,
    ReturnFormat: spider.FormatMarkdown,
}, func(page spider.SpiderResponse) {
    fmt.Printf("Received: %s\n", page.URL)
})
```

--------------------------------

### Basic search with Spider

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/search.md

Performs a basic search using the Spider client. The query is passed directly to the search method, and the results are logged to the console.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();

const query = "site:spider.cloud web scraping";
const results = await app.search(query);

console.log(results);
```

--------------------------------

### Crawl a website with crawl_url

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Crawl a website with a limit of 200 pages and smart mode request.

```python
url = 'https://example.com'
crawl_params = {
    'limit': 200,
    'request': 'smart_mode'
}
crawl_result = app.crawl_url(url, params=crawl_params)
```

--------------------------------

### crawl_url with different parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/crawl.md

Crawl a website with different parameters using the Spider SDK.

```APIDOC
## `crawl_url` with different parameters

### Description
Crawl a website with additional parameters such as `request_timeout` and `stealth`.

### Method
`crawl_url`

### Parameters
- **url** (str) - Required - The URL of the website to crawl.
- **params** (dict) - Optional - A dictionary of parameters. Recommended parameters include:
  - **limit** (int) - Optional - The maximum amount of pages allowed to crawl per website. Set to `0` or remove to crawl all pages.
  - **request_timeout** (int) - Optional - The maximum amount of time to wait for a response from the website.
  - **stealth** (bool) - Optional - Whether to use stealth mode. Default is `False` on chrome.

### Request Example
```python
from spider import Spider

app = Spider()
url = "https://spider.cloud"
crawled_data = app.crawl_url(
    url, params={"limit": 10, "request_timeout": 10, "stealth": True}
)

print(crawled_data)
```

### Response
Returns the crawled content in markdown format by default.
```

--------------------------------

### Crawl with custom parameters

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/crawl.md

Crawls a website with additional parameters such as return_format set to "raw". The limit parameter is still set to 10.

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const url = "https://spider.cloud";
const scrapedData = await app.crawlUrl(url, {
  limit: 10,
  return_format: "raw",
});
console.log(scrapedData);
```

--------------------------------

### Scrape with RequestParamsDict

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/scrape.md

Scrape a website using a separate `params` variable typed as `RequestParamsDict` from the `spider` package. This improves readability and provides intellisense for available options.

```python
from spider import Spider, spider_types

params: spider_types.RequestParamsDict = {
    "request_timeout": 10,
    # Easier to read and intellisense will help you with the available options
}

app = Spider()
url = "https://spider.cloud"
scraped_data = app.scrape_url(url, params)

print(scraped_data)
```

--------------------------------

### Scrape with return_format parameter

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/javascript/scrape.md

Scrape a website with the `return_format` parameter set to `"raw"`. The `scrapeUrl` method accepts optional parameters in a dictionary, such as `request` (default "http") and `return_format` (default "markdown").

```javascript
import { Spider } from "@spider-cloud/spider-client";

const app = new Spider();
const url = "https://spider.cloud";
const scrapedData = await app.scrapeUrl(url, {
  return_format: "raw",
});
console.log(scrapedData);
```

--------------------------------

### Unlimited Scrape with app.unlimited_scrape

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Scrape a single URL on the Unlimited plan. Requires an active Unlimited subscription.

```python
url = 'https://example.com'
scraped_data = app.unlimited_scrape(url)
```

--------------------------------

### GetCredits

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Retrieves the current credit balance for the API key. It accepts a context and returns a credits object.

```APIDOC
## GetCredits

### Description
Retrieves the current credit balance for the API key.

### Method
GetCredits

### Parameters
- **ctx** (context.Context) - Required - The context for the request.

### Request Example
go
credits, err := client.GetCredits(ctx)


### Response
Returns a credits object with a Credits field containing the number of credits.

```

--------------------------------

### Add external domains with with_external_domains

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Add external domains to include with the website crawl.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_external_domains(["https://www.myotherdomain.com"])

asyncio.run(main())
```

--------------------------------

### Wait for idle network with with_wait_for_idle_network

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Wait for the network to become idle when using Chrome, helping load client-side scripts. The first parameter enables the feature, the second is the max timeout in milliseconds.

```python
import asyncio
from spider_rs import Website

async def main():
    website = Website("https://choosealicense.com").with_wait_for_idle_network(True, 12000)

asyncio.run(main())
```

--------------------------------

### Search Web

Source: https://github.com/spider-rs/spider-clients/blob/main/go/README.md

Searches for a query with a limit of 5 results and markdown format.

```go
pages, err := client.Search(ctx, "web scraping APIs", &spider.SearchParams{
    SearchLimit: 5,
    SpiderParams: spider.SpiderParams{
        ReturnFormat: spider.FormatMarkdown,
    },
})
```

--------------------------------

### Scrape a single URL with scrape_url

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Calls spider.scrape_url on a URL with default parameters, returning JSON data. Expects the Spider instance to be already created.

```rust
let url = "https://example.com";
let scraped_data = spider.scrape_url(url, None, false, "application/json").await.expect("Failed to scrape the URL");
```

--------------------------------

### crawl_url

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/python/crawl.md

Crawl a website and return the content using the Spider SDK.

```APIDOC
## `crawl_url`

### Description
Crawls a website and returns the content in markdown format by default. The `limit` parameter controls the maximum number of pages to crawl.

### Method
`crawl_url`

### Parameters
- **url** (str) - Required - The URL of the website to crawl.
- **params** (dict) - Optional - A dictionary of parameters. Recommended parameters include:
  - **limit** (int) - Optional - The maximum amount of pages allowed to crawl per website. Set to `0` or remove to crawl all pages.
  - **request_timeout** (int) - Optional - The maximum amount of time to wait for a response from the website.
  - **stealth** (bool) - Optional - Whether to use stealth mode. Default is `False` on chrome.

### Request Example
```python
from spider import Spider

app = Spider()
url = "https://spider.cloud"
crawled_data = app.crawl_url(url, params={"limit": 10})
print(crawled_data)
```

### Response
Returns the crawled content in markdown format by default.
```

--------------------------------

### Take screenshots with with_screenshot

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/website.md

Take a screenshot of pages during a headless Chrome crawl. The configuration includes full-page capture, background omission, and saving options.

```python
import asyncio
from spider_rs import Website

async def main():
    website = (
        Website("https://choosealicense.com", False)
        .with_screenshot({
            "params": {
                "cdp_params": None,
                "full_page": True,
                "omit_background": False
            },
            "bytes": False,
            "save": True,
            "output_dir": None
        })
    )

asyncio.run(main())
```

--------------------------------

### Add spider-client dependency to Cargo.toml

Source: https://github.com/spider-rs/spider-clients/blob/main/book/src/rust/getting-started.md

Adds the spider-client crate version 0.1 to the project's dependencies.

```toml
[dependencies]
spider-client = "0.1"
```

--------------------------------

### Unlimited Crawl with app.unlimited_crawl

Source: https://github.com/spider-rs/spider-clients/blob/main/python/README.md

Crawl a website on the Unlimited plan. Requires an active Unlimited subscription. No queueing: when all purchased seats are in flight, the API returns an immediate 429 with a Retry-After header; retry with backoff. Every response carries X-Concurrency-Limit and X-Concurrency-Active headers. AI/LLM extraction params are rejected with a 400.

```python
url = 'https://example.com'
crawl_params = {
    'limit': 200,
    'request': 'smart_mode'
}
crawl_result = app.unlimited_crawl(url, params=crawl_params)
```

--------------------------------

### Unlimited Scrape and Crawl with Spider Rust Client

Source: https://github.com/spider-rs/spider-clients/blob/main/rust/README.md

Use unlimited_scrape and unlimited_crawl to run on the Unlimited plan, a flat monthly rate billed by concurrency seats. No queueing: when all seats are in flight, the API returns an immediate 429 with a Retry-After header; the SDK retries with exponential backoff automatically. AI/LLM extraction params are rejected with a 400.

```rust
let url = "https://example.com";
let scraped = spider.unlimited_scrape(url, None, "application/json").await.expect("Failed to scrape the URL");

let crawl_result = spider.unlimited_crawl(url, None, false, "application/json", None::<fn(serde_json::Value)>).await.expect("Failed to crawl the URL");
```
Tokens:
Raw