Spider

https://github.com/spider-rs/spider
Admin
Spider is a concurrent, streaming web crawler and scraper designed for data curation workloads,...
Tokens:25,156
Snippets:309
Trust Score:7.8
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Run Basic Crawl Example

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Execute the basic crawl example. Ensure you have cloned the repository and are in the project directory.

```bash
cargo run --example example
```

--------------------------------

### Run agent examples

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Commands to execute various agent examples using cargo with specific feature flags.

```bash
# Basic search
SERPER_API_KEY=xxx cargo run --example basic_search --features search_serper

# Extract data
OPENAI_API_KEY=xxx cargo run --example extract --features openai

# Research
OPENAI_API_KEY=xxx SERPER_API_KEY=xxx cargo run --example research --features "openai search_serper"

# Concurrent execution
OPENAI_API_KEY=xxx SERPER_API_KEY=xxx cargo run --example concurrent --features "openai search_serper"

# Embedded scripting (Python + JS, no API key required)
cargo run --example scripting --features scripting
```

--------------------------------

### Install Spider Packages

Source: https://github.com/spider-rs/spider/blob/main/README.md

Commands to install Spider for various environments.

```shell
cargo add spider
```

```shell
cargo install spider_cli
```

```shell
npm i @spider-rs/spider-rs
```

```shell
pip install spider_rs
```

```shell
cargo install spider_mcp
```

--------------------------------

### Run spider-rs examples

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Commands to clone the repository and execute various crawler examples with different feature flags.

```bash
git clone https://github.com/spider-rs/spider.git && cd spider

# Basic crawl
cargo run --example example

# With Chrome
cargo run --example chrome --features chrome

# Smart mode
cargo run --example smart --features smart

# Spider Cloud (browser)
SPIDER_CLOUD_API_KEY=sk-... cargo run --example spider_browser_cloud --features "chrome spider_cloud"

# AI automation (OpenAI vision)
OPENAI_API_KEY=sk-... cargo run --example openai --features "chrome openai"

# Remote multimodal (any LLM)
cargo run --example remote_multimodal --features "chrome openai"

# Advanced configuration (reusable config across sites)
cargo run --example advanced_configuration

# Anti-bot / stealth
cargo run --example anti_bots --features "chrome chrome_stealth"

# Cache + Chrome hybrid
cargo run --example cache_chrome_hybrid --features "cache_chrome_hybrid chrome"

# See all examples (48+ files)
ls examples/
```

--------------------------------

### Run Remote Multimodal Scraping Examples

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Commands to execute specific multimodal scraping examples. Requires an OPEN_ROUTER API key and specific feature flags.

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_scrape --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_multi --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual_automation --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_quotes --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_listing --features "spider/sync spider/chrome spider/agent_chrome"
```

--------------------------------

### Install Spider CLI

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Installs the Spider CLI. Use the 'smart' feature flag for HTTP first, browser fallback mode.

```bash
# default install (includes chrome support)
cargo install spider_cli
```

```bash
# optional smart mode (HTTP first, browser fallback)
cargo install -F smart spider_cli
```

--------------------------------

### Configure agent with AgentBuilder

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example of setting up an agent with various configurations and tool integrations.

```rust
Agent::builder()
    .with_config(config)
    .with_system_prompt("You are a helpful assistant")
    .with_max_concurrent_llm_calls(10)
    .with_openai(api_key, model)
    .with_spider_cloud("spider-cloud-api-key")
    .with_search_serper(api_key)
    .build()
```

--------------------------------

### Install spider_mcp via Cargo

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Standard installation command for the spider_mcp binary.

```bash
cargo install spider_mcp
```

--------------------------------

### Run end-to-end release example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Command to execute a single-prompt pipeline for data extraction.

```bash
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_end_to_end \
  -- "Find top travel books on https://books.toscrape.com and return structured product fields"
```

--------------------------------

### Install Spider CLI on Ubuntu

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Installs the pkg-config utility on Ubuntu, which is required for OpenSSL to be recognized by cargo during installation.

```bash
# On Ubuntu:
apt install pkg-config
```

--------------------------------

### Install minimal spider_mcp

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Install the package without default features like Chrome rendering.

```bash
cargo install spider_mcp --no-default-features
```

--------------------------------

### Quick Start with Spider Agent

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Initializes an agent with OpenAI and Serper credentials to perform search and extraction tasks.

```rust
use spider_agent::{Agent, AgentConfig};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let agent = Arc::new(Agent::builder()
        .with_openai("sk-...", "gpt-4o-mini")
        .with_search_serper("serper-key")
        .build()?);

    // Search
    let results = agent.search("rust web frameworks").await?;
    println!("Found {} results", results.len());

    // Extract from first result
    let html = agent.fetch(&results.results[0].url).await?.html;
    let data = agent.extract(&html, "Extract framework name and features").await?;
    println!("{}", serde_json::to_string_pretty(&data)?);

    Ok(())
}
```

--------------------------------

### Verify installation

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Commands to check the workspace and run tests for the spider_agent crate.

```bash
cargo check --workspace
cargo test -p spider_agent
cargo test -p spider_agent --features "openai search_serper"
RUN_LIVE_TESTS=1 cargo test -p spider_agent --features "openai search_serper" --test live_env_smoke -- --nocapture
```

--------------------------------

### Live Handle Index Mutation Example

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Illustrates live index mutation handling during a crawl.

```bash
cargo run --example callback
```

--------------------------------

### Enable Debug Logging

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Run the debug example to enable and view log output during the crawl.

```bash
cargo run --example debug
```

--------------------------------

### Quick Start HTML Cleaning

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_html/README.md

Demonstrates basic HTML cleaning using predefined profiles: base, slim, and smart. Ensure the HTML string is properly formatted.

```rust
use spider_agent_html::{clean_html_base, clean_html_slim, smart_clean_html};

let html = r###"<html><head><script>tracker();</script><style>body{}</style></head>
<body><h1>Hello</h1><p>World</p><svg>...</svg></body></html>"###;

// Base: remove scripts, styles, ads, tracking
let clean = clean_html_base(html);

// Slim: also remove SVG, canvas, video, base64
let slim = clean_html_slim(html);

// Smart: auto-select the optimal profile based on content analysis
let smart = smart_clean_html(html);
```

--------------------------------

### Install spider_agent_types dependency

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/README.md

Add the crate to your Cargo.toml file.

```toml
[dependencies]
spider_agent_types = "0.1"
```

--------------------------------

### Run Spider Agent Examples via CLI

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Execute various agent pipelines and flows using cargo run with the required API key.

```bash
# End-to-end pipeline
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_end_to_end \
  -- "Find top travel books on https://books.toscrape.com"

# Prompt-driven flows (crawl, scrape, search, transform, unblocker)
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_prompt_flows \
  -- "run all flows for https://books.toscrape.com/"

# Jobs pipeline
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_jobs_pipeline \
  -- "rust engineer remote" "https://remoteok.com/remote-rust-jobs"

# Browser cloud
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_browser_cloud
```

--------------------------------

### Configure Spider Website Instance

Source: https://github.com/spider-rs/spider/blob/main/README.md

Example of initializing and configuring a website crawler in Rust with various options like concurrency limits, depth, and stealth mode.

```rust
let mut website = Website::new("https://example.com")
    .with_limit(50)                    // concurrent requests
    .with_depth(10)                    // how deep to follow links
    .with_delay(500)                   // pause between requests (ms)
    .with_respect_robots_txt(true)
    .with_subdomains(true)
    .with_user_agent(Some("MyBot/1.0"))
    .with_stealth(true)
    .build()
    .unwrap();
```

--------------------------------

### Crawl a website

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Example for crawling a website with specific depth and limit constraints.

```json
{
  "url": "https://example.com",
  "limit": 5,
  "depth": 2,
  "return_format": "markdown"
}
```

--------------------------------

### Scrape a page

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Examples for fetching page content using spider_scrape.

```json
{
  "url": "https://example.com",
  "return_format": "markdown"
}
```

```json
{
  "url": "https://example.com",
  "headless": true,
  "wait_for": "#content",
  "return_format": "text"
}
```

--------------------------------

### Clone and Build Spider Project

Source: https://github.com/spider-rs/spider/blob/main/CONTRIBUTING.md

Clone the Spider repository and build the project locally. Ensure you have Rust and Git installed.

```bash
git clone https://github.com/spider-rs/spider.git
cd spider
cargo build
```

--------------------------------

### Configure custom Spider Cloud endpoint

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example of pointing the agent to a custom gateway endpoint.

```rust
let spider_cloud = SpiderCloudToolConfig::new("provider-key")
    .with_api_url("https://my-gateway.example.com/v1")
    .with_tool_name_prefix("web_api"); // tools become web_api_search, web_api_scrape, etc.
```

--------------------------------

### Subscribe to Realtime Changes

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates subscribing to realtime changes during a crawl. This example can be run with both 'subscribe' and 'subscribe_multiple'.

```bash
cargo run --example subscribe
```

```bash
cargo run --example subscribe_multiple
```

--------------------------------

### Crawl with Specific Path and Page Limits

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Configure a crawl budget to limit pages matching a specific path and set an overall page limit. This example allows 10 pages matching '/blog/' and a total of 100 pages.

```sh
spider --url https://choosealicense.com --budget "*,100,/blog/,10" crawl -o
```

--------------------------------

### CLI Commands for Spider

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Common CLI commands for installing, crawling, scraping, and downloading content with Spider.

```bash
cargo install spider_cli

# Basic crawl
spider crawl --url https://example.com

# With Spider Cloud
spider authenticate sk-...
spider crawl --url https://example.com --spider-cloud-mode smart

# Browser cloud
spider crawl --url https://example.com --spider-cloud-browser

# Scrape
spider scrape --url https://example.com

# Download
spider download --url https://example.com
```

--------------------------------

### Smart HTML Cleaning Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_html/README.md

Utilizes `smart_clean_html` to automatically select the most appropriate cleaning profile based on the HTML content's characteristics for optimal token reduction.

```rust
use spider_agent_html::smart_clean_html;

// Automatically picks Slim for SVG-heavy pages, Base for simple pages, etc.
let cleaned = smart_clean_html(large_html);
```

--------------------------------

### Remote Cache Skip Browser

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

An end-to-end example of remote cache warm-up and skip-browser return path using `HYBRID_CACHE_ENDPOINT`. Requires specific feature flags.

```bash
HYBRID_CACHE_ENDPOINT=http://127.0.0.1:8080 cargo run --example cache_remote_skip_browser --features="spider/sync spider/chrome spider/chrome_remote_cache" -- https://example.com/
```

--------------------------------

### Transform HTML

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Example for converting raw HTML to markdown using spider_transform.

```json
{
  "html": "<h1>Hello</h1><p>World</p>",
  "return_format": "markdown"
}
```

--------------------------------

### Generate JSON Schema

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Creates a JSON schema from provided data examples for structured extraction.

```rust
use spider_agent::{generate_schema, SchemaGenerationRequest};

let request = SchemaGenerationRequest {
    examples: vec![
        json!({"name": "Product A", "price": 19.99}),
        json!({"name": "Product B", "price": 29.99}),
    ],
    description: Some("Product listing data".to_string()),
    strict: false,
    name: Some("products".to_string()),
};

let schema = generate_schema(&request);
// Use schema.to_extraction_schema() for structured extraction
```

--------------------------------

### Install Spider Agent via Cargo

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Add the spider_agent dependency to your Cargo.toml file with the desired features enabled.

```toml
[dependencies]
spider_agent = { version = "0.1", features = ["openai", "search_serper"] }
```

--------------------------------

### Crawl with Budget and Domain Limit

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Set a crawl budget to limit the number of pages crawled within a specific domain. This example allows crawling only one page from the specified domain.

```sh
spider --url https://choosealicense.com --budget "*,1" crawl -o
```

--------------------------------

### Execute Python scripts with ScriptEngine

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Demonstrates initializing the ScriptEngine and executing a Python script within a provided context.

```rust
use spider_agent::scripting::{ScriptConfig, ScriptContext, ScriptEngine};
use std::time::Duration;

let engine = ScriptEngine::new(ScriptConfig {
    enabled: true,
    allow_network: true,
    default_timeout: Duration::from_secs(5),
    ..ScriptConfig::default()
});

let result = engine.run_python(
    r#"
import json, re
nums = [int(n) for n in re.findall(r"\d+", agent.html)]
print(json.dumps({"url": agent.url, "found": nums}))
"#.to_string(),
    ScriptContext {
        url: Some("https://shop.example.com".into()),
        html: Some("<p>price 1999, stock 42</p>".into()),
        ..ScriptContext::default()
    },
    None,
).await;
println!("{}", result.stdout);
```

--------------------------------

### Extract links

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Example for extracting links from a page using spider_links.

```json
{
  "url": "https://example.com",
  "subdomains": true
}
```

--------------------------------

### Configure Basic Dependencies

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Standard dependency configuration for basic crawling.

```toml
[dependencies]
spider = { version = "2.0.12" }
```

--------------------------------

### generate_schema

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Auto-generates JSON schemas from provided examples for structured data extraction.

```APIDOC
## generate_schema

### Description
Generates a JSON schema based on provided example data.

### Parameters
- `request` (SchemaGenerationRequest): The request object containing examples, description, strict mode flag, and schema name.

### Returns
- Returns a schema object that can be converted to an extraction schema using `to_extraction_schema()`.
```

--------------------------------

### Build spider_mcp from source

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Compile the project from source code with release optimizations.

```bash
cargo build -p spider_mcp --release
```

--------------------------------

### Reusable Configuration for Multiple Sites

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Demonstrates how to define a shared configuration and apply it to multiple website instances.

```rust
use spider::{configuration::Configuration, website::Website};

let config = Configuration::new()
    .with_user_agent(Some("MyBot/1.0"))
    .with_respect_robots_txt(true)
    .with_subdomains(false)
    .build();

for url in ["https://a.com", "https://b.com", "https://c.com"] {
    match Website::new(url).with_config(config.to_owned()).build() {
        Ok(mut site) => {
            site.crawl().await;
            let links = site.get_all_links_visited().await;
            println!("{url}: {} pages", links.len());
        }
        Err(e) => println!("Invalid URL: {:?}", e.get_url()),
    }
}
```

--------------------------------

### Schema Generation API

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/SKILLS.md

APIs for generating and refining JSON schemas from example data.

```APIDOC
## Schema Generation (`schema_gen`)

### Description
Provides functionalities to generate and manage JSON schemas based on example data.

### Data Structures
- **`SchemaGenerationRequest`** (struct): Request to generate schema from example data.
- **`GeneratedSchema`** (struct): Result with JSON schema, field descriptions, confidence.
- **`SchemaCache`** (struct): LRU cache of generated schemas with usage tracking.

### Functions
- **`generate_schema()`** (fn): Generate schema from request.
- **`infer_schema()`** (fn): Infer schema from a single JSON value.
- **`infer_schema_from_examples()`** (fn): Infer schema from multiple examples (merges types).
- **`refine_schema()`** (fn): Refine existing schema with new examples.
- **`build_schema_generation_prompt()`** (fn): Build LLM prompt for schema generation.
```

--------------------------------

### Run real-world intelligence pipelines

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Commands for e-commerce and job market intelligence pipelines.

```bash
# E-commerce competitor intelligence
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_ecommerce_competitor \
  -- "https://books.toscrape.com/" "travel books"

# Job market intelligence pipeline
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_jobs_pipeline \
  -- "rust engineer remote" "https://remoteok.com/remote-rust-jobs"
```

--------------------------------

### Basic async web crawling

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Initialize a website crawler with cloud configuration and subscribe to page results.

```rust
use spider::configuration::{SpiderCloudConfig, SpiderCloudMode, SpiderCloudReturnFormat};
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    // Get your API key free at https://spider.cloud
    let config = SpiderCloudConfig::new("YOUR_API_KEY")
        .with_mode(SpiderCloudMode::Smart)
        .with_return_format(SpiderCloudReturnFormat::Markdown);

    let mut website = Website::new("https://example.com")
        .with_limit(10)
        .with_spider_cloud_config(config)
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            let url = page.get_url();
            let markdown = page.get_content();
            let status = page.status_code;

            println!("[{status}] {url}\n---\n{markdown}\n");
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Configure Smart Mode Dependencies

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Enable smart mode to intelligently switch between HTTP and JS rendering.

```toml
[dependencies]
spider = { version = "2.0.12", features = ["smart"] }
```

--------------------------------

### Initialize automation and analyze content

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/README.md

Configure automation settings and perform HTML content analysis using the provided types.

```rust
use spider_agent_types::{
    ActionType, AutomationConfig, RemoteMultimodalConfig,
    ToolCallingMode, HtmlDiffMode, ContentAnalysis,
};

// Create automation config
let config = AutomationConfig::new("Extract product data");

// Analyze HTML content
let analysis = ContentAnalysis::analyze("<html>...</html>");
println!("Needs screenshot: {}", analysis.needs_screenshot);
println!("Text ratio: {:.1}%", analysis.text_ratio * 100.0);

// Parse LLM responses
use spider_agent_types::{extract_last_json_object, extract_assistant_content};
let json = extract_last_json_object(r#"Here is the result: {"name": "test"}"#);
```

--------------------------------

### Initialize Spider Cloud crawler

Source: https://github.com/spider-rs/spider/blob/main/README.md

Configure the website instance with SpiderCloudConfig to enable proxy and unblocking services.

```rust
use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};
use spider::website::Website;

let cloud = SpiderCloudConfig::new("sk-...")
    .with_mode(SpiderCloudMode::Smart); // proxy by default, auto-unblock when blocked

let mut website = Website::new("https://example.com")
    .with_spider_cloud_config(cloud)
    .build()?;
```

--------------------------------

### Configure Spider Agent with Cloud Tools

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Initialize the agent using Spider Cloud configurations, including options for AI-enabled routes.

```rust
use spider_agent::{Agent, SpiderCloudToolConfig};

// Full config
let agent = Agent::builder()
    .with_spider_cloud_config(
        SpiderCloudToolConfig::new("sk-...")
            .with_enable_ai_routes(true)  // paid plan — enables /ai/* routes
    )
    .build()?;

// Or shorthand (defaults)
let agent = Agent::builder()
    .with_spider_cloud("sk-...")
    .build()?;

// Available tools: spider_cloud_crawl, spider_cloud_scrape,
// spider_cloud_search, spider_cloud_links, spider_cloud_transform,
// spider_cloud_unblocker
// AI tools (paid): spider_cloud_ai_crawl, spider_cloud_ai_scrape,
// spider_cloud_ai_search, spider_cloud_ai_browser, spider_cloud_ai_links
```

--------------------------------

### CLI Options

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Display available command-line options for the spider-mcp executable.

```text
spider-mcp [OPTIONS]

Options:
  --log-level <LEVEL>  Log level: error, warn, info, debug, trace (default: warn)
```

--------------------------------

### Execute live integration tests

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Commands to run integration tests requiring network access and API keys.

```bash
# Core crawler tests against crawler-test.com
RUN_LIVE_TESTS=1 cargo test -p spider --test crawler_test_com

# Spider Cloud integration
SPIDER_CLOUD_API_KEY=sk-... RUN_LIVE_TESTS=1 cargo test -p spider_agent --test live_spider_cloud

# Spider Browser integration
SPIDER_CLOUD_API_KEY=sk-... RUN_LIVE_TESTS=1 cargo test -p spider_agent --test live_spider_browser

# Agent cloud integration
SPIDER_CLOUD_API_KEY=sk-... RUN_LIVE_TESTS=1 cargo test -p spider --test live_spider_agent_cloud
```

--------------------------------

### Crawl and Get Full Resources

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

This command crawls a website and gathers all associated resources, such as CSS and JavaScript files, in addition to the main HTML content.

```sh
spider --url https://choosealicense.com --full-resources crawl -o
```

--------------------------------

### Configure OpenAI Dependencies

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Enable OpenAI integration for dynamic browser scripting.

```toml
[dependencies]
spider = { version = "2.0.12", features = ["openai"] }
```

--------------------------------

### Run local crawler with streaming

Source: https://github.com/spider-rs/spider/blob/main/README.md

Use tokio to run the crawler locally and subscribe to the stream of crawled pages.

```rust
use spider::{tokio, website::Website};

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}  {}", page.status_code, page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Crawl Links with Chrome Rendering

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Crawls links using Chrome for rendering. Requires the `chrome` feature flag. This example can be run in headless or headed mode.

```bash
cargo run --example chrome --features chrome
```

```bash
cargo run --example chrome --features chrome_headed
```

--------------------------------

### Configure Claude Desktop

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Add the spider_mcp server to the Claude Desktop configuration.

```json
{
  "mcpServers": {
    "spider": {
      "command": "spider-mcp",
      "args": []
    }
  }
}
```

--------------------------------

### Initialize crawler with builder pattern

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Use the builder pattern to configure crawler settings fluently.

```rust
let mut website = Website::new("https://choosealicense.com");

website
   .with_respect_robots_txt(true)
   .with_subdomains(true)
   .with_tld(false)
   .with_delay(0)
   .with_request_timeout(None)
   .with_http2_prior_knowledge(false)
   .with_user_agent(Some("myapp/version".into()))
   .with_budget(Some(spider::hashbrown::HashMap::from([("*", 300), ("/licenses", 10)])))
   .with_limit(300)
   .with_caching(false)
   .with_external_domains(Some(Vec::from(["https://creativecommons.org/licenses/by/3.0/"].map( |d| d.to_string())).into_iter()))
   .with_headers(None)
   .with_blacklist_url(Some(Vec::from(["https://choosealicense.com/licenses/".into()])))
   .with_cron("1/5 * * * * *", Default::Default())
   .with_proxies(None);
```

--------------------------------

### Crawl with OpenAI Configuration

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Pass GPTConfigs to the website builder to drive browser actions.

```rust
extern crate spider;

use spider::{tokio, website::Website, configuration::GPTConfigs};

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://google.com")
        .with_openai(Some(GPTConfigs::new("gpt-4-1106-preview", "Search for Movies", 256)))
        .with_limit(1)
        .build()
        .unwrap();

    website.crawl().await;
}
```

--------------------------------

### Configure and execute a multi-site crawl in Rust

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Demonstrates how to define a shared configuration and apply it across multiple website instances using tokio tasks.

```rust
extern crate spider;

use spider::configuration::Configuration;
use spider::{tokio, website::Website};
use std::io::Error;
use std::time::Instant;

const CAPACITY: usize = 5;
const CRAWL_LIST: [&str; CAPACITY] = [
    "https://spider.cloud",
    "https://choosealicense.com",
    "https://jeffmendez.com",
    "https://spider-rs.github.io/spider-nodejs/",
    "https://spider-rs.github.io/spider-py/",
];

#[tokio::main]
async fn main() -> Result<(), Error> {
    let config = Configuration::new()
        .with_user_agent(Some("SpiderBot"))
        .with_blacklist_url(Some(Vec::from(["https://spider.cloud/resume".into()])))
        .with_subdomains(false)
        .with_tld(false)
        .with_redirect_limit(3)
        .with_retry(1)
        .with_respect_robots_txt(true)
        .with_external_domains(Some(
            Vec::from(["http://loto.rsseau.fr/"].map(|d| d.to_string())).into_iter(),
        ))
        .build();

    let mut handles = Vec::with_capacity(CAPACITY);

    for website_url in CRAWL_LIST {
        match Website::new(website_url)
            .with_config(config.to_owned())
            .build()
        {
            Ok(mut website) => {
                let handle = tokio::spawn(async move {
                    println!("Starting Crawl - {:?}", website.get_url().inner());

                    let start = Instant::now();
                    website.crawl().await;
                    let duration = start.elapsed();

                    let links = website.get_links();

                    for link in links {
                        println!("- {:?}", link.as_ref());
                    }

                    println!(
                        "{:?} - Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
                        website.get_url().inner(),
                        duration,
                        links.len()
                    );
                });

                handles.push(handle);
            }
            Err(e) => println!("{:?}", e)
        }
    }

    for handle in handles {
        let _ = handle.await;
    }

    Ok(())
}
```

--------------------------------

### Configure Spider Agent with Browser Tools

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Initialize the agent using Browser Cloud configurations, supporting stealth mode and geo-targeting.

```rust
use spider_agent::{Agent, SpiderBrowserToolConfig};

// Full config
let agent = Agent::builder()
    .with_spider_browser_config(
        SpiderBrowserToolConfig::new("sk-...")
            .with_stealth(true)
            .with_country("us")
    )
    .build()?;

// Or shorthand
let agent = Agent::builder()
    .with_spider_browser("sk-...")
    .build()?;

// Available tools: spider_browser_navigate, spider_browser_html,
// spider_browser_screenshot, spider_browser_evaluate,
// spider_browser_click, spider_browser_fill, spider_browser_wait
```

--------------------------------

### Planning Module

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/SKILLS.md

Manages multi-step execution plans, checkpoints, and replanning logic for web automation.

```APIDOC
## ExecutionPlan

### Description
Handles the creation and execution of multi-step plans generated by an LLM.

### Components
- **PlannedStep**: Defines a step with ID, action, dependencies, and confidence.
- **Checkpoint**: Verification point using URL, element, text, or JS condition.
- **ReplanContext**: Context provided to the LLM when a failure occurs.
- **PlanningModeConfig**: Configuration for max steps and auto-execute thresholds.
```

--------------------------------

### Use Different Page Encodings

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates handling pages with different character encodings. Requires the `encoding` feature flag.

```bash
cargo run --example encoding --features encoding
```

--------------------------------

### Download Files in Subscription

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates downloading files as part of a subscription during a crawl.

```bash
cargo run --example subscribe_download
```

--------------------------------

### Run prompt-driven route orchestration

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Command to execute prompt-driven flows using Spider Cloud.

```bash
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_prompt_flows \
  -- "run all flows for https://books.toscrape.com/ including search scrape crawl links transform unblocker"
```

--------------------------------

### Load Skills from S3 in Rust

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/SKILLS.md

Demonstrates loading custom skills from S3-compatible storage using `S3SkillSource` and `load_from_s3`. Configure bucket, path, region, and endpoint URL as needed. Skills loaded from S3 will override built-in skills with the same name.

```rust
use spider_agent::automation::skills::{S3SkillSource, load_from_s3, builtin_web_challenges};

let mut registry = builtin_web_challenges();
let source = S3SkillSource::new("my-bucket", "skills/")
    .with_region("us-east-1")
    .with_endpoint_url("https://r2.example.com"); // optional, for R2/MinIO

let loaded = load_from_s3(&mut registry, &source).await?;
// S3 skills override built-in skills with the same name
```

--------------------------------

### Release Script Execution

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Command to run the release script for publishing crates in dependency order.

```bash
# Release script publishes in dependency order (--no-verify for cli/utils/worker)
./release.sh
```

--------------------------------

### Configure Chrome Rendering

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Point the server to a running Chrome instance using the SPIDER_CHROME_URL environment variable.

```json
{
  "mcpServers": {
    "spider": {
      "command": "spider-mcp",
      "env": {
        "SPIDER_CHROME_URL": "ws://localhost:9222"
      }
    }
  }
}
```

--------------------------------

### Run Benchmarks

Source: https://github.com/spider-rs/spider/blob/main/CONTRIBUTING.md

Execute the performance benchmarks for the Spider project using Criterion. This helps in performance optimization.

```bash
cargo bench
```

--------------------------------

### Configure Caching Dependencies

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Enable disk or memory caching via feature flags.

```toml
[dependencies]
spider = { version = "2.0.12", features = ["cache"] }
```

--------------------------------

### Configure RemoteMultimodalConfig

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Demonstrates using preset configurations for performance or manual fine-grained control over agent behavior.

```rust
use spider_agent::RemoteMultimodalConfig;

// Fast mode: All performance-positive features enabled
// - Tool calling (Auto), HTML diff (Auto), Confidence retries, Concurrent execution
let config = RemoteMultimodalConfig::fast();

// Fast with planning: Adds multi-step planning and self-healing
// Best for complex multi-step automations
let config = RemoteMultimodalConfig::fast_with_planning();

// Manual configuration for fine-grained control:
use spider_agent::{
    ToolCallingMode, HtmlDiffMode, ReasoningEffort,
    PlanningModeConfig, SelfHealingConfig, ConfidenceRetryStrategy,
};

let config = RemoteMultimodalConfig::default()
    .with_tool_calling_mode(ToolCallingMode::Auto)
    .with_html_diff_mode(HtmlDiffMode::Auto)
    .with_reasoning_effort(Some(ReasoningEffort::Medium))
    .with_planning_mode(PlanningModeConfig::default())
    .with_self_healing(SelfHealingConfig::default())
    .with_confidence_strategy(ConfidenceRetryStrategy::default())
    .with_concurrent_execution(true);
```

--------------------------------

### Enable Chrome Rendering

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Configure headless browser rendering for JavaScript-heavy sites. Requires the 'chrome' feature.

```rust
// features = ["chrome"]
use spider::configuration::RequestInterceptConfiguration;

let mut website = Website::new("https://spa-app.com")
    .with_chrome_intercept(RequestInterceptConfiguration::new(true))  // block ads/analytics/stylesheets
    .with_stealth(true)
    .with_wait_for_idle_network(Some(WaitForIdleNetwork::new(Some(Duration::from_secs(30)))))
    .build()
    .unwrap();
website.crawl().await;
```

```rust
.with_chrome_connection(Some("ws://localhost:9222".into()))
```

--------------------------------

### Configure Browser Cloud

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Enable remote headless Chrome via CDP for browser-based crawling.

```rust
use spider::configuration::SpiderBrowserConfig;

let browser = SpiderBrowserConfig::new("sk-...")
    .with_stealth(true)          // anti-fingerprinting
    .with_country("us");         // geo-targeting

let mut website = Website::new("https://example.com")
    .with_limit(10)
    .with_spider_browser_config(browser)
    .build()
    .unwrap();

website.crawl().await;
```

--------------------------------

### Configuration and Model Management

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/SKILLS.md

Configuration structures for tasks, models, and cleaning profiles.

```APIDOC
## Configuration

### Description
Task-level and model-level configuration settings.

### Data Structures
- **RemoteMultimodalConfig** (struct) - Per-task config: model, API key, vision, HTML diff, tool calling, planning, self-healing.
- **AutomationConfig** (struct) - Task-level settings: goal, recovery strategy, retry policy, cleaning profile.
- **ModelProfile** (struct) - Model metadata: name, provider, context window, pricing.
- **HtmlCleaningProfile** (enum) - Cleaning levels: Raw, Minimal, Default, Slim, Aggressive.

### Functions
- **model_profile(name: &str)** - Look up a model's profile by name.
- **supports_vision(model: &str)** - Check if model supports vision.
- **arena_rank(model: &str)** - Get model's arena ranking.
```

--------------------------------

### Configure local crawler dependencies

Source: https://github.com/spider-rs/spider/blob/main/README.md

Add the standard spider crate to your Cargo.toml for local crawling.

```toml
[dependencies]
spider = "2"
```

--------------------------------

### Use Smart Mode

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Automatically upgrade from HTTP to Chrome rendering when JavaScript is detected. Requires the 'smart' feature.

```rust
// features = ["smart"]
let mut website = Website::new("https://example.com")
    .build()
    .unwrap();
website.crawl_smart().await;
```

--------------------------------

### Run Spider tests

Source: https://github.com/spider-rs/spider/blob/main/README.md

Commands to execute unit tests and live network tests for the Spider project.

```bash
cargo test -p spider                  # unit tests
RUN_LIVE_TESTS=1 cargo test           # live network tests
```

--------------------------------

### Enable HTTP Caching

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Set with_caching to true to persist crawl data to disk.

```rust
extern crate spider;

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://spider.cloud")
        .with_caching(true)
        .build()
        .unwrap();

    website.crawl().await;

    println!("Links found {:?}", website.get_links().len());
    /// next run to website.crawl().await; will be faster since content is stored on disk.
}
```

--------------------------------

### Execute unit tests

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Commands to run unit tests for the spider and spider_agent packages.

```bash
cargo test -p spider
cargo test -p spider_agent
```

--------------------------------

### Linux Benchmark Results

Source: https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md

Benchmark results on a Linux (ubuntu-latest) machine with a 2-core CPU, 7 GB RAM, and 14 GB SSD, testing crawl speed for 10 samples on https://rsseau.fr/.

```sh
----------------------
linux ubuntu-latest
2-core CPU
7 GB of RAM memory
14 GB of SSD disk space
-----------------------

Test url: `https://rsseau.fr/`

185 pages
```

```text
|                                          | `libraries` |
| :--------------------------------------- | :---------- |
| **`Rust[spider]: crawl 10 samples`**     | `50ms`      |
| **`Go[colly]: crawl 10 samples`**       | `30s`       |
| **`Node.js[crawler]: crawl 10 samples`** | `3.4s`      |
| **`C[wget]: crawl 10 samples`**          | `60s`       |
```

--------------------------------

### Configure Chrome Dependencies

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Add the chrome and chrome_intercept features to your Cargo.toml.

```toml
[dependencies]
spider = { version = "2", features = ["chrome", "chrome_intercept"] }
```

--------------------------------

### OpenAI Integration

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Configure GPT-based content extraction during a crawl with chrome interception.

```rust
// features = ["chrome", "openai"]
use spider::configuration::GPTConfigs;

let gpt_config = GPTConfigs::new("gpt-4o", "Extract the main content", 512);

let mut website = Website::new("https://example.com")
    .with_chrome_intercept(RequestInterceptConfiguration::new(true))
    .with_wait_for_idle_network(Some(WaitForIdleNetwork::new(Some(Duration::from_secs(30)))))
    .with_screenshot(Some(screenshot_config))
    .with_openai(Some(gpt_config))
    .build()
    .unwrap();
```

--------------------------------

### WARC Output Configuration

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Enable WARC file generation for archiving crawled data.

```rust
// features = ["warc"]
use spider::utils::warc::WarcConfig;

let mut config = Configuration::new();
config.warc = Some(WarcConfig {
    path: "output.warc.gz".to_string(),
    ..Default::default()
});
```

--------------------------------

### Register Spider Cloud tools

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Registering Spider Cloud routes as custom tools using the builder.

```rust
use spider_agent::Agent;

let agent = Agent::builder()
    .with_spider_cloud("spider-cloud-api-key")
    .build()?;

// Available tools:
// - spider_cloud_crawl
// - spider_cloud_scrape
// - spider_cloud_search
// - spider_cloud_links
// - spider_cloud_transform
// - spider_cloud_unblocker
```

--------------------------------

### Execute feature-specific tests

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Commands to run tests for specific features like smart mode, parallel backends, or io_uring.

```bash
# Smart vs Chrome comparison
cargo test -p spider --test smart_vs_chrome --features "smart chrome"

# Parallel backends
cargo test -p spider --test parallel_backends --features parallel_backends

# io_uring
cargo test -p spider uring_fs --features io_uring
```

--------------------------------

### Download HTML to React Components

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Scrapes a webpage, converts the HTML to React components, and stores them in the file system.

```bash
cargo run --example download_to_react
```

--------------------------------

### Screenshot Configuration

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Enable screenshots during a crawl by providing a ScreenShotConfig.

```rust
// features = ["chrome", "chrome_screenshot"]
use spider::configuration::{ScreenShotConfig, ScreenshotParams};

let screenshot_params = ScreenshotParams::new(Default::default(), Some(true), Some(true));
// ScreenShotConfig::new(params, bytes, save, output_dir)
let screenshot_config = ScreenShotConfig::new(screenshot_params, true, true, None);

let mut website = Website::new("https://example.com")
    .with_screenshot(Some(screenshot_config))
    .build()
    .unwrap();
```

--------------------------------

### Budget Limiting

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Configure crawl limits for specific paths or the entire site using a HashMap.

```rust
use std::collections::HashMap;

let mut website = Website::new("https://example.com")
    .with_budget(Some(HashMap::from([
        ("/blog", 50),   // max 50 pages under /blog
        ("*", 200),      // max 200 pages total
    ])))
    .build()
    .unwrap();
```

--------------------------------

### Architecture Overview

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_html/SKILLS.md

Highlights the key architectural components and design principles of the HTML cleaning utilities.

```APIDOC
## Architecture

- **Streaming Rewriter**: Utilizes `lol_html::rewrite_str()` with element content handlers for efficient, single-pass processing without requiring a full DOM tree allocation. This results in O(n) time complexity.
- **Smart Analysis**: The `smart_clean_html()` function employs `ContentAnalysis::analyze()` (from `spider_agent_types`) to evaluate text ratio and byte counts of elements like SVG and scripts. Based on this analysis, it selects the most suitable cleaning profile.
- **Intent-Aware Cleaning**: The `clean_html_with_profile_and_intent()` function allows for cleaning adjustments based on the intended use of the processed HTML, such as extraction, observation, or mapping.
- **Composability**: All cleaning functions are pure functions with a `&str -> String` signature, making them easy to chain together, test independently, and benchmark.
```

--------------------------------

### Configure Spider Cloud tools with SpiderCloudToolConfig

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Advanced configuration for Spider Cloud tools, including custom API URLs and tool prefixes.

```rust
use spider_agent::{Agent, SpiderCloudToolConfig};

let spider_cloud = SpiderCloudToolConfig::new("spider-cloud-api-key")
    .with_api_url("https://api.spider.cloud")
    .with_tool_name_prefix("spider_cloud")
    .with_enable_ai_routes(true); // Only enable if your plan includes /ai/* routes

let agent = Agent::builder()
    .with_spider_cloud_config(spider_cloud)
    .build()?;
```

--------------------------------

### Configure Spider Cloud dependencies

Source: https://github.com/spider-rs/spider/blob/main/README.md

Add the spider crate with the spider_cloud feature enabled to your Cargo.toml.

```toml
[dependencies]
spider = { version = "2", features = ["spider_cloud"] }
```

--------------------------------

### Agent::builder().with_spider_browser()

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Configures the agent to use Spider Browser tools for direct browser automation, navigation, and interaction.

```APIDOC
## Agent::builder().with_spider_browser()

### Description
Initializes the agent with Spider Browser capabilities for granular control over browser sessions.

### Configuration
- **api_key** (String) - Required - The Spider Cloud API key.
- **stealth** (bool) - Optional - Enables stealth mode to avoid detection.
- **country** (String) - Optional - Sets the proxy location (e.g., "us").

### Available Tools
- spider_browser_navigate
- spider_browser_html
- spider_browser_screenshot
- spider_browser_evaluate
- spider_browser_click
- spider_browser_fill
- spider_browser_wait
```

--------------------------------

### Load Built-in and S3 Skills Combined in Rust

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/SKILLS.md

Shows how to combine built-in skills with S3-loaded skills in a single operation using `with_builtin_and_s3`. This is a convenient way to manage skill sources.

```rust
use spider_agent::automation::skills::{S3SkillSource, with_builtin_and_s3};

let source = S3SkillSource::new("my-bucket", "skills/");
let registry = with_builtin_and_s3(&source).await?;
```

--------------------------------

### Configure Global Allocator

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Add the desired allocator to your Cargo.toml dependencies to improve throughput under high concurrency.

```toml
[dependencies]
mimalloc = { version = "0.1", default-features = false }
# or: tikv-jemallocator = "0.6"
```

--------------------------------

### Advanced Configuration and Caching

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Showcases advanced configuration reuse, including Chrome hybrid caching. Requires specific feature flags.

```bash
cargo run --example cache_chrome_hybrid --features="spider/sync spider/chrome spider/cache_chrome_hybrid"
```

```bash
cargo run --example advanced_configuration
```

--------------------------------

### Execute Smart Crawl

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Use crawl_smart to automatically determine the best extraction method.

```rust
extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com");
    website.crawl_smart().await;

    for link in website.get_links() {
        println!("- {:?}", link.as_ref());
    }
}
```

--------------------------------

### Agent::builder().with_spider_cloud()

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Configures the agent to use Spider Cloud tools for tasks like crawling, scraping, searching, and AI-driven browser interactions.

```APIDOC
## Agent::builder().with_spider_cloud()

### Description
Initializes the agent with Spider Cloud capabilities, enabling access to web crawling, scraping, and AI-powered browser tools.

### Configuration
- **api_key** (String) - Required - The Spider Cloud API key.
- **enable_ai_routes** (bool) - Optional - Enables access to paid AI-specific routes (e.g., /ai/*).

### Available Tools
- spider_cloud_crawl
- spider_cloud_scrape
- spider_cloud_search
- spider_cloud_links
- spider_cloud_transform
- spider_cloud_unblocker
- spider_cloud_ai_crawl (Paid)
- spider_cloud_ai_scrape (Paid)
- spider_cloud_ai_search (Paid)
- spider_cloud_ai_browser (Paid)
- spider_cloud_ai_links (Paid)
```

--------------------------------

### Crawl and Output Links to File

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

This command crawls a website and outputs all visited links to a JSON file. Ensure the output file path is correctly specified.

```sh
spider --url https://choosealicense.com crawl -o > spider_choosealicense.json
```

--------------------------------

### Configure Spider Cloud

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Set up advanced cloud crawling modes and return formats using SpiderCloudConfig.

```rust
use spider::configuration::{SpiderCloudConfig, SpiderCloudMode};

// Full control via SpiderCloudConfig
let config = SpiderCloudConfig::new("sk-...")
    .with_mode(SpiderCloudMode::Smart)       // proxy + auto unblocker fallback
    .with_return_format("raw");              // "raw" = original HTML

let mut website = Website::new("https://example.com")
    .with_spider_cloud_config(config)
    .build()
    .unwrap();

// Or shorthand — defaults to Proxy mode
let mut website = Website::new("https://example.com");
website.with_spider_cloud("sk-...");
```
Tokens:
Raw