### Run Basic Crawl Example

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Execute the basic crawl example. Ensure you have cloned the repository and are in the project directory.

```bash
cargo run --example example
```

--------------------------------

### Quick Start: Crawl with Spider Cloud

Source: https://github.com/spider-rs/spider/blob/main/README.md

This example demonstrates how to use Spider to crawl a website and extract data using Spider Cloud. Ensure you have an API key from spider.cloud. The crawler is configured for smart mode and returns data in Markdown format.

```rust
use spider::{
    configuration::{SpiderCloudConfig, SpiderCloudMode, SpiderCloudReturnFormat},
    tokio, // re-export
    website::Website,
};

#[tokio::main]
async fn main() {
    // Get your API key free at https://spider.cloud
    let config = SpiderCloudConfig::new("YOUR_API_KEY")
        .with_mode(SpiderCloudMode::Smart)
        .with_return_format(SpiderCloudReturnFormat::Markdown);

    let mut website = Website::new("https://example.com")
        .with_limit(10)
        .with_spider_cloud_config(config)
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            let url = page.get_url();
            let markdown = page.get_content();
            let status = page.status_code;

            println!("[{status}] {url}\n---\n{markdown}\n");
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Run Continuous Crawls with Cron Jobs

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This example shows how to set up continuous crawls using cron jobs. It requires the `sync` and `cron` features. The `cron_str` field defines the schedule, and `run_cron` starts the continuous crawling process.

```toml
[dependencies]
spider = { version = "2", features = ["sync", "cron"] }
```

```rust
extern crate spider;

use spider::website::{Website, run_cron};
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com");
    // set the cron to run or use the builder pattern `website.with_cron`.
    website.cron_str = "1/5 * * * * *".into();

    let mut rx2 = website.subscribe(16);

    let join_handle = tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            println!("{:?}", res.get_url());
        }
    });

    // take ownership of the website. You can also use website.run_cron, except you need to perform abort manually on handles created.
    let mut runner = run_cron(website).await;

    println!("Starting the Runner for 10 seconds");
    tokio::time::sleep(tokio::time::Duration::from_secs(10)).await;
    let _ = tokio::join!(runner.stop(), join_handle);
}
```

--------------------------------

### Install Spider-NodeJS

Source: https://github.com/spider-rs/spider/blob/main/README.md

Install the Spider Node.js package using npm.

```bash
npm i @spider-rs/spider-rs
```

--------------------------------

### Basic Search Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example command to run a basic search using the Spider Agent, requiring SERPER_API_KEY and enabling the 'search_serper' feature.

```bash
# Basic search
SERPER_API_KEY=xxx cargo run --example basic_search --features search_serper
```

--------------------------------

### Run Spider Cloud End-to-End Release Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Execute a command for an end-to-end release example, demonstrating a single-prompt pipeline to find travel books and return structured product fields.

```bash
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_end_to_end \
  -- "Find top travel books on https://books.toscrape.com/ and return structured product fields"
```

--------------------------------

### Spider Agent CLI Examples

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Illustrates command-line execution of Spider Agent examples for various tasks like end-to-end pipelines, prompt-driven flows, and jobs pipelines.

```bash
# End-to-end pipeline
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_end_to_end \
  -- "Find top travel books on https://books.toscrape.com"

# Prompt-driven flows (crawl, scrape, search, transform, unblocker)
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_prompt_flows \
  -- "run all flows for https://books.toscrape.com/"

# Jobs pipeline
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_cloud_jobs_pipeline \
  -- "rust engineer remote" "https://remoteok.com/remote-rust-jobs"

# Browser cloud
SPIDER_CLOUD_API_KEY=sk-... cargo run -p spider_agent --example spider_browser_cloud
```

--------------------------------

### Run Remote Multimodal Scraping Examples

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Commands to execute specific multimodal scraping examples. Requires an OPEN_ROUTER API key and specific feature flags.

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_scrape --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_multi --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_dual_automation --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_quotes --features "spider/sync spider/chrome spider/agent_chrome"
```

```bash
OPEN_ROUTER=replace_me_with_key cargo run --example remote_multimodal_listing --features "spider/sync spider/chrome spider/agent_chrome"
```

--------------------------------

### Running Spider Examples

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Provides commands for running various Spider examples, including basic crawls, Chrome integration, smart mode, and AI automation features.

```bash
git clone https://github.com/spider-rs/spider.git && cd spider

# Basic crawl
cargo run --example example

# With Chrome
cargo run --example chrome --features chrome

# Smart mode
cargo run --example smart --features smart

# Spider Cloud (browser)
SPIDER_CLOUD_API_KEY=sk-... cargo run --example spider_browser_cloud --features "chrome spider_cloud"

# AI automation (OpenAI vision)
OPENAI_API_KEY=sk-... cargo run --example openai --features "chrome openai"

# Remote multimodal (any LLM)
cargo run --example remote_multimodal --features "chrome openai"

# Advanced configuration (reusable config across sites)
cargo run --example advanced_configuration

# Anti-bot / stealth
cargo run --example anti_bots --features "chrome chrome_stealth"

# Cache + Chrome hybrid
cargo run --example cache_chrome_hybrid --features "cache_chrome_hybrid chrome"

# See all examples (48+ files)
ls examples/
```

--------------------------------

### Install Spider CLI

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Installs the Spider CLI. Use the 'smart' feature flag for HTTP first, browser fallback mode.

```bash
# default install (includes chrome support)
cargo install spider_cli
```

```bash
# optional smart mode (HTTP first, browser fallback)
cargo install -F smart spider_cli
```

--------------------------------

### Install Spider-Py

Source: https://github.com/spider-rs/spider/blob/main/README.md

Install the Spider Python package using pip.

```bash
pip install spider_rs
```

--------------------------------

### Install Spider CLI

Source: https://github.com/spider-rs/spider/blob/main/README.md

Install the Spider command-line interface tool using Cargo.

```bash
cargo install spider_cli
```

--------------------------------

### Install spider_mcp with Minimal Features

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Install spider_mcp with only HTTP capabilities, excluding Chrome rendering.

```bash
cargo install spider_mcp --no-default-features
```

--------------------------------

### Install spider_mcp with Cargo

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Install the spider_mcp server using the cargo package manager.

```bash
cargo install spider_mcp
```

--------------------------------

### Install Spider CLI on Ubuntu

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Installs the pkg-config utility on Ubuntu, which is required for OpenSSL to be recognized by cargo during installation.

```bash
# On Ubuntu:
apt install pkg-config
```

--------------------------------

### Spider Cloud Integration Example

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Integrate Spider Cloud for enhanced crawling by providing an API key. This example shows how to subscribe to crawl events and print page information.

```rust
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://example.com");
    website.with_spider_cloud("YOUR_API_KEY");

    let mut rx = website.subscribe(16);
    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{} - {}", page.get_url(), page.get_content());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Run Spider Cloud E-commerce Competitor Intelligence Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Execute a command for e-commerce competitor intelligence, specifying a URL and a search query for travel books.

```bash
# E-commerce competitor intelligence
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_ecommerce_competitor \
  -- "https://books.toscrape.com/" "travel books"
```

--------------------------------

### Concurrent Execution Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example command to run concurrent searches with the Spider Agent, requiring API keys and specific features.

```bash
# Concurrent execution
OPENAI_API_KEY=xxx SERPER_API_KEY=xxx cargo run --example concurrent --features "openai search_serper"
```

--------------------------------

### Quick Start HTML Cleaning

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_html/README.md

Demonstrates basic HTML cleaning using predefined profiles: base, slim, and smart. Ensure the HTML string is properly formatted.

```rust
use spider_agent_html::{clean_html_base, clean_html_slim, smart_clean_html};

let html = r###"<html><head><script>tracker();</script><style>body{}</style></head>
<body><h1>Hello</h1><p>World</p><svg>...</svg></body></html>"###;

// Base: remove scripts, styles, ads, tracking
let clean = clean_html_base(html);

// Slim: also remove SVG, canvas, video, base64
let slim = clean_html_slim(html);

// Smart: auto-select the optimal profile based on content analysis
let smart = smart_clean_html(html);
```

--------------------------------

### Research Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example command to perform research with synthesis using the Spider Agent, requiring both OPENAI_API_KEY and SERPER_API_KEY, and enabling 'openai' and 'search_serper' features.

```bash
# Research
OPENAI_API_KEY=xxx SERPER_API_KEY=xxx cargo run --example research --features "openai search_serper"
```

--------------------------------

### Start Rust Project with SPIDER_WORKER

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

To start a Rust project with the SPIDER_WORKER environment variable, use `cargo run --example example --features decentralized`. The SPIDER_WORKER variable accepts a comma-separated list of URLs for worker nodes.

```bash
SPIDER_WORKER=http://127.0.0.1:3030 cargo run --example example --features decentralized
```

--------------------------------

### Live Handle Index Mutation Example

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Illustrates live index mutation handling during a crawl.

```bash
cargo run --example callback
```

--------------------------------

### Run Spider Cloud Job Market Intelligence Pipeline Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Execute a command for a job market intelligence pipeline, specifying a job search query and a target URL.

```bash
# Job market intelligence pipeline
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_jobs_pipeline \
  -- "rust engineer remote" "https://remoteok.com/remote-rust-jobs"
```

--------------------------------

### Extract Data Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Example command to run data extraction using the Spider Agent, requiring OPENAI_API_KEY and enabling the 'openai' feature.

```bash
# Extract data
OPENAI_API_KEY=xxx cargo run --example extract --features openai
```

--------------------------------

### Enable Debug Logging

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Run the debug example to enable and view log output during the crawl.

```bash
cargo run --example debug
```

--------------------------------

### Run Spider Cloud Prompt Flows Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Execute a command to run prompt-driven route orchestration for a given URL, including search, scrape, crawl, links, transform, and unblocker tools.

```bash
SPIDER_CLOUD_API_KEY=your-key cargo run -p spider_agent --example spider_cloud_prompt_flows \
  -- "run all flows for https://books.toscrape.com/ including search scrape crawl links transform unblocker"
```

--------------------------------

### Basic Async Crawling with Spider Cloud

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

A fundamental asynchronous example demonstrating how to crawl a website using Spider Cloud configuration. Ensure you have a Spider Cloud API key.

```rust
use spider::configuration::{SpiderCloudConfig, SpiderCloudMode, SpiderCloudReturnFormat};
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    // Get your API key free at https://spider.cloud
    let config = SpiderCloudConfig::new("YOUR_API_KEY")
        .with_mode(SpiderCloudMode::Smart)
        .with_return_format(SpiderCloudReturnFormat::Markdown);

    let mut website = Website::new("https://example.com")
        .with_limit(10)
        .with_spider_cloud_config(config)
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            let url = page.get_url();
            let markdown = page.get_content();
            let status = page.status_code;

            println!("[{status}] {url}\n---\n{markdown}\n");
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Spider Agent Quick Start

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Initialize the Spider Agent with OpenAI and Serper search providers. Perform a search, fetch the first result's HTML, and extract structured data using a natural language prompt.

```rust
use spider_agent::{Agent, AgentConfig};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let agent = Arc::new(Agent::builder()
        .with_openai("sk-...", "gpt-4o-mini")
        .with_search_serper("serper-key")
        .build()?);

    // Search
    let results = agent.search("rust web frameworks").await?;
    println!("Found {} results", results.len());

    // Extract from first result
    let html = agent.fetch(&results.results[0].url).await?.html;
    let data = agent.extract(&html, "Extract framework name and features").await?;
    println!("{}", serde_json::to_string_pretty(&data)?);

    Ok(())
}
```

--------------------------------

### Install spider_agent_types dependency

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/README.md

Add the crate to your Cargo.toml file.

```toml
[dependencies]
spider_agent_types = "0.1"
```

--------------------------------

### Clone and Build Spider Project

Source: https://github.com/spider-rs/spider/blob/main/CONTRIBUTING.md

Clone the Spider repository and build the project locally. Ensure you have Rust and Git installed.

```bash
git clone https://github.com/spider-rs/spider.git
cd spider
cargo build
```

--------------------------------

### Subscribe to Realtime Changes

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates subscribing to realtime changes during a crawl. This example can be run with both 'subscribe' and 'subscribe_multiple'.

```bash
cargo run --example subscribe
```

```bash
cargo run --example subscribe_multiple
```

--------------------------------

### Install and Run Spider Worker for Decentralization

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

Install the spider_worker using cargo and run it to enable decentralized processing. Set the RUST_LOG and SPIDER_WORKER_PORT environment variables as needed.

```sh
# install the worker
car go install spider_worker
# start the worker [set the worker on another machine in prod]
RUST_LOG=info SPIDER_WORKER_PORT=3030 spider_worker
```

--------------------------------

### Basic CLI Crawling and Scraping

Source: https://context7.com/spider-rs/spider/llms.txt

Install and use the spider CLI for basic crawling, scraping, and downloading web content. Supports various options for controlling the crawl.

```bash
# Install CLI
cargo install spider_cli

# Basic crawl
spider crawl --url https://example.com

# Crawl with options
spider crawl --url https://example.com \
  --respect-robots-txt \
  --subdomains \
  --depth 5 \
  --limit 100 \
  --delay 500 \
  --agent "MyBot/1.0" \
  --verbose

# Scrape (collect HTML content)
spider scrape --url https://example.com

# Download pages
spider download --url https://example.com

# Chrome headless mode
spider crawl --url https://spa-app.com --headless

# With wait conditions
spider crawl --url https://example.com \
  --headless \
  --wait-for-idle-network 30000 \
  --wait-for-selector "div.loaded" \
  --stealth

# Output as markdown
spider crawl --url https://example.com --return-format markdown

# Write to WARC archive
spider crawl --url https://example.com --warc output.warc.gz
```

--------------------------------

### Subscribe to Website Crawl Events

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This example shows how to subscribe to crawl events using the `subscribe` method. It requires the `sync` feature. A broadcast channel is returned, allowing you to receive crawl results asynchronously.

```toml
[dependencies]
spider = { version = "2", features = ["sync"] }
```

```rust
extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com");
    let mut rx2 = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            println!("{:?}", res.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
}
```

--------------------------------

### Control Crawler: Pause, Resume, and Shutdown

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This example shows how to control the crawler's lifecycle using pause, resume, and shutdown functions. It requires the `control` feature. Ensure you have `tokio::time::sleep` imported if using it.

```toml
[dependencies]
spider = { version = "2", features = ["control"] }
```

```rust
extern crate spider;

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    use spider::utils::{pause, resume, shutdown};
    let url = "https://choosealicense.com/";
    let mut website: Website = Website::new(&url);

    tokio::spawn(async move {
        pause(url).await;
        sleep(tokio::time::Duration::from_millis(5000)).await;
        resume(url).await;
        // perform shutdown if crawl takes longer than 15s
        sleep(tokio::time::Duration::from_millis(15000)).await;
        // you could also abort the task to shutdown crawls if using website.crawl in another thread.
        shutdown(url).await;
    });

    website.crawl().await;
}
```

--------------------------------

### Smart HTML Cleaning Example

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_html/README.md

Utilizes `smart_clean_html` to automatically select the most appropriate cleaning profile based on the HTML content's characteristics for optimal token reduction.

```rust
use spider_agent_html::smart_clean_html;

// Automatically picks Slim for SVG-heavy pages, Base for simple pages, etc.
let cleaned = smart_clean_html(large_html);
```

--------------------------------

### Remote Cache Skip Browser

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

An end-to-end example of remote cache warm-up and skip-browser return path using `HYBRID_CACHE_ENDPOINT`. Requires specific feature flags.

```bash
HYBRID_CACHE_ENDPOINT=http://127.0.0.1:8080 cargo run --example cache_remote_skip_browser --features="spider/sync spider/chrome spider/chrome_remote_cache" -- https://example.com/
```

--------------------------------

### Regex Blacklisting for URLs

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This example demonstrates how to blacklist URLs using regular expressions. It requires the `regex` feature. URLs matching the provided regex will not be crawled.

```toml
[dependencies]
spider = { version = "2", features = ["regex"] }
```

```rust
extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com");
    website.configuration.blacklist_url.push("/licenses/".into());
    website.crawl().await;

    for link in website.get_links() {
        println!("- {:?}", link.as_ref());
    }
}
```

--------------------------------

### Crawl with Specific Path and Page Limits

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Configure a crawl budget to limit pages matching a specific path and set an overall page limit. This example allows 10 pages matching '/blog/' and a total of 100 pages.

```sh
spider --url https://choosealicense.com --budget "*,100,/blog/,10" crawl -o
```

--------------------------------

### Concurrent Multi-Website Crawling with Tokio

Source: https://context7.com/spider-rs/spider/llms.txt

Crawl multiple websites concurrently using Rust's Tokio async runtime. This example demonstrates setting up individual website crawls and joining them.

```rust
use spider::features::chrome_common::RequestInterceptConfiguration;
use spider::tokio;
use spider::website::Website;
use std::io::Result;

async fn crawl_website(url: &str) -> Result<()> {
    let mut website: Website = Website::new(url)
        .with_limit(10)
        .with_chrome_intercept(RequestInterceptConfiguration::new(true))
        .with_stealth(true)
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    let handle = tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{:?}", page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
    let _ = handle.await;

    let links = website.get_all_links_visited().await;
    println!("{}: {} pages", url, links.len());

    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    // Crawl multiple sites concurrently
    let _ = tokio::join!(
        crawl_website("https://choosealicense.com"),
        crawl_website("https://jeffmendez.com"),
        crawl_website("https://github.com/spider-rs"),
    );

    Ok(())
}
```

--------------------------------

### Register Spider Cloud Routes as Custom Tools

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Register Spider Cloud routes as custom tools directly from the builder. This example shows how to build an agent with Spider Cloud integration, listing available tools.

```rust
use spider_agent::Agent;

let agent = Agent::builder()
    .with_spider_cloud("spider-cloud-api-key")
    .build()?;

// Available tools:
// - spider_cloud_crawl
// - spider_cloud_scrape
// - spider_cloud_search
// - spider_cloud_links
// - spider_cloud_transform
// - spider_cloud_unblocker
```

--------------------------------

### Scrape and Gather HTML Content

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This example demonstrates how to scrape HTML content from a website. The `scrape` method fetches the page content, and `get_pages` retrieves the collected pages. The output includes the final URL, HTML content, and separators.

```rust
extern crate spider;

use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    use std::io::{Write, stdout};

    let url = "https://choosealicense.com/";
    let mut website = Website::new(&url);

    website.scrape().await;

    let mut lock = stdout().lock();

    let separator = "-".repeat(url.len());

    for page in website.get_pages().unwrap().iter() {
        writeln!(
            lock,
            "{}
{}

{}

{}",
            separator,
            page.get_url_final(),
            page.get_html(),
            separator
        )
        .unwrap();
    }
}
```

--------------------------------

### Sitemap Crawling with Spider

Source: https://context7.com/spider-rs/spider/llms.txt

Crawl URLs from a sitemap before or alongside regular crawling. This example configures custom sitemap paths and demonstrates crawling the sitemap first, persisting links, and then performing a regular crawl.

```rust
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://choosealicense.com");

    website
        .configuration
        .with_respect_robots_txt(true)
        .with_user_agent(Some("SpiderBot"))
        .with_ignore_sitemap(true) // Don't auto-crawl sitemap with regular crawl
        .with_sitemap(Some("/sitemap/sitemap-0.xml"));

    // Crawl sitemap first
    website.crawl_sitemap().await;

    // Persist links to extend the next crawl
    website.persist_links();

    // Continue with regular crawl including sitemap links
    website.crawl().await;

    let links = website.get_all_links_visited().await;
    println!("Found {} total pages", links.len());
}
```

--------------------------------

### HTTP Caching with Spider

Source: https://context7.com/spider-rs/spider/llms.txt

Enable disk-based HTTP caching to avoid re-fetching unchanged pages across crawl sessions. This example shows how to enable caching, subscribe to page events, perform an initial crawl to populate the cache, and then a subsequent crawl that utilizes cached responses.

```rust
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    // Enable caching with the `cache` feature
    let mut website: Website = Website::new("https://choosealicense.com")
        .with_caching(true)
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("Page: {}", page.get_url());
        }
    });

    // First crawl populates cache
    website.crawl().await;

    // Subsequent crawls use cached responses
    let mut website2 = website.clone();
    website2.crawl().await;

    website.unsubscribe();
}
```

--------------------------------

### Crawl with Smart Mode (HTTP + Chrome)

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Utilize smart mode for crawling, which starts with standard HTTP requests and dynamically upgrades to Chrome rendering only when JavaScript execution is detected. Requires the `smart` feature.

```rust
// features = ["smart"]
let mut website = Website::new("https://example.com")
    .build()
    .unwrap();
website.crawl_smart().await;
```

--------------------------------

### Perform a Smart Crawl

Source: https://github.com/spider-rs/spider/blob/main/spider/README.md

This Rust code demonstrates how to initiate a smart crawl using Spider. It requires a Chrome connection or a locally installed browser. The `crawl_smart()` method intelligently decides whether to use HTTP or JavaScript rendering.

```rust
extern crate spider;

use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com");
    website.crawl_smart().await;

    for link in website.get_links() {
        println!("- {:?}", link.as_ref());
    }
}
```

--------------------------------

### Build spider_mcp from Source

Source: https://github.com/spider-rs/spider/blob/main/spider_mcp/README.md

Build the spider_mcp server from source code using cargo.

```bash
cargo build -p spider_mcp --release
```

--------------------------------

### Publishing via release script

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Executes the release script in dependency order.

```bash
# Release script publishes in dependency order (--no-verify for cli/utils/worker)
./release.sh
```

--------------------------------

### Schema Generation API

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/SKILLS.md

APIs for generating and refining JSON schemas from example data.

```APIDOC
## Schema Generation (`schema_gen`)

### Description
Provides functionalities to generate and manage JSON schemas based on example data.

### Data Structures
- **`SchemaGenerationRequest`** (struct): Request to generate schema from example data.
- **`GeneratedSchema`** (struct): Result with JSON schema, field descriptions, confidence.
- **`SchemaCache`** (struct): LRU cache of generated schemas with usage tracking.

### Functions
- **`generate_schema()`** (fn): Generate schema from request.
- **`infer_schema()`** (fn): Infer schema from a single JSON value.
- **`infer_schema_from_examples()`** (fn): Infer schema from multiple examples (merges types).
- **`refine_schema()`** (fn): Refine existing schema with new examples.
- **`build_schema_generation_prompt()`** (fn): Build LLM prompt for schema generation.
```

--------------------------------

### Generate JSON Schemas in Rust

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Auto-generate JSON schemas from provided data examples using SchemaGenerationRequest.

```rust
use spider_agent::{generate_schema, SchemaGenerationRequest};

let request = SchemaGenerationRequest {
    examples: vec![
        json!({"name": "Product A", "price": 19.99}),
        json!({"name": "Product B", "price": 29.99}),
    ],
    description: Some("Product listing data".to_string()),
    strict: false,
    name: Some("products".to_string()),
};

let schema = generate_schema(&request);
// Use schema.to_extraction_schema() for structured extraction
```

--------------------------------

### Spider CLI Commands

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Demonstrates basic usage of the Spider CLI for crawling, scraping, and downloading, including options for Spider Cloud and browser modes.

```bash
cargo install spider_cli

# Basic crawl
spider crawl --url https://example.com

# With Spider Cloud
spider authenticate sk-...
spider crawl --url https://example.com --spider-cloud-mode smart

# Browser cloud
spider crawl --url https://example.com --spider-cloud-browser

# Scrape
spider scrape --url https://example.com

# Download
spider download --url https://example.com
```

--------------------------------

### Reusable configuration across multiple sites

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Demonstrates sharing a single Configuration instance across multiple Website instances.

```rust
use spider::{configuration::Configuration, website::Website};

let config = Configuration::new()
    .with_user_agent(Some("MyBot/1.0"))
    .with_respect_robots_txt(true)
    .with_subdomains(false)
    .build();

for url in ["https://a.com", "https://b.com", "https://c.com"] {
    match Website::new(url).with_config(config.to_owned()).build() {
        Ok(mut site) => {
            site.crawl().await;
            let links = site.get_all_links_visited().await;
            println!("{url}: {} pages", links.len());
        }
        Err(e) => println!("Invalid URL: {:?}", e.get_url()),
    }
}
```

--------------------------------

### Agent - Memory API

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Provides methods to manage the agent's session memory, including getting, setting, and clearing memory.

```APIDOC
## Agent Memory Management

### Description
APIs for managing the agent's session memory.

### Methods

#### GET /api/agent/memory

##### Description
Retrieves the current session memory.

##### Response
- **memory** (object) - The current memory state.

#### POST /api/agent/memory

##### Description
Sets or updates the session memory.

##### Parameters
- **memory** (object) - Required - The memory object to set.

#### DELETE /api/agent/memory

##### Description
Clears the entire session memory.

### Response Example (GET)
{
  "memory": {
    "last_query": "rust web frameworks"
  }
}

### Request Example (POST)
{
  "memory": {
    "last_query": "async rust"
  }
}
```

--------------------------------

### Initialize automation and analyze content

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/README.md

Configure automation settings and perform HTML content analysis using the provided types.

```rust
use spider_agent_types::{
    ActionType, AutomationConfig, RemoteMultimodalConfig,
    ToolCallingMode, HtmlDiffMode, ContentAnalysis,
};

// Create automation config
let config = AutomationConfig::new("Extract product data");

// Analyze HTML content
let analysis = ContentAnalysis::analyze("<html>...</html>");
println!("Needs screenshot: {}", analysis.needs_screenshot);
println!("Text ratio: {:.1}%", analysis.text_ratio * 100.0);

// Parse LLM responses
use spider_agent_types::{extract_last_json_object, extract_assistant_content};
let json = extract_last_json_object(r#"Here is the result: {"name": "test"}"#);
```

--------------------------------

### Crawl and Get Full Resources

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

This command crawls a website and gathers all associated resources, such as CSS and JavaScript files, in addition to the main HTML content.

```sh
spider --url https://choosealicense.com --full-resources crawl -o
```

--------------------------------

### Configure Spider Cloud Agent

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Demonstrates how to configure the Spider Agent with Spider Cloud tools, including enabling AI routes with a paid plan. Supports both full and shorthand configurations.

```rust
use spider_agent::{Agent, SpiderCloudToolConfig};

// Full config
let agent = Agent::builder()
    .with_spider_cloud_config(
        SpiderCloudToolConfig::new("sk-...")
            .with_enable_ai_routes(true)  // paid plan — enables /ai/* routes
    )
    .build()?;

// Or shorthand (defaults)
let agent = Agent::builder()
    .with_spider_cloud("sk-...")
    .build()?;

// Available tools: spider_cloud_crawl, spider_cloud_scrape,
// spider_cloud_search, spider_cloud_links, spider_cloud_transform,
// spider_cloud_unblocker
// AI tools (paid): spider_cloud_ai_crawl, spider_cloud_ai_scrape,
// spider_cloud_ai_search, spider_cloud_ai_browser, spider_cloud_ai_links
```

--------------------------------

### Crawl Links with Chrome Rendering

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Crawls links using Chrome for rendering. Requires the `chrome` feature flag. This example can be run in headless or headed mode.

```bash
cargo run --example chrome --features chrome
```

```bash
cargo run --example chrome --features chrome_headed
```

--------------------------------

### Perform basic website crawling

Source: https://context7.com/spider-rs/spider/llms.txt

Initializes a Website crawler and configures crawling parameters such as robots.txt compliance and URL blacklisting before executing the crawl.

```rust
use spider::tokio;
use spider::website::Website;
use std::time::Instant;

#[tokio::main]
async fn main() {
    // Create a website crawler with basic configuration
    let mut website: Website = Website::new("https://choosealicense.com");

    // Configure directly on the configuration struct
    website.configuration.respect_robots_txt = true;
    website.configuration.subdomains = false;
    website.configuration.delay = 0; // milliseconds between requests
    website.configuration.user_agent = Some(Box::new("SpiderBot".into()));

    // Add URLs to blacklist
    website
        .configuration
        .blacklist_url
        .insert(Default::default())
        .push("https://choosealicense.com/non-software".into());

    let start = Instant::now();
    website.crawl().await;
    let duration = start.elapsed();

    // Get all visited links
    let links = website.get_all_links_visited().await;

    for link in links.iter() {
        println!("- {:?}", link.as_ref());
    }

    println!("Crawled {} pages in {:?}", links.len(), duration);
}
```

--------------------------------

### Build Spider Agent with Configuration

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/README.md

Configure and build agents with various settings like system prompts, concurrent LLM calls, and API integrations for OpenAI and Spider Cloud.

```rust
Agent::builder()
    .with_config(config)
    .with_system_prompt("You are a helpful assistant")
    .with_max_concurrent_llm_calls(10)
    .with_openai(api_key, model)
    .with_spider_cloud("spider-cloud-api-key")
    .with_search_serper(api_key)
    .build()
```

--------------------------------

### Local Crawl Without API Key

Source: https://github.com/spider-rs/spider/blob/main/CLAUDE.md

Perform a local crawl without needing an API key. This example demonstrates basic crawling and retrieving links from the crawled pages.

```rust
let mut website = Website::new("https://example.com");
website.crawl().await;
for link in website.get_links() {
    println!("{link}");
}
```

--------------------------------

### Crawl with Budget and Domain Limit

Source: https://github.com/spider-rs/spider/blob/main/spider_cli/README.md

Set a crawl budget to limit the number of pages crawled within a specific domain. This example allows crawling only one page from the specified domain.

```sh
spider --url https://choosealicense.com --budget "*,1" crawl -o
```

--------------------------------

### Chrome Headless Rendering with Spider

Source: https://context7.com/spider-rs/spider/llms.txt

Enable Chrome/Chromium headless rendering for JavaScript-heavy websites. This example configures request interception to block ads/analytics and enables stealth mode for anti-fingerprinting. Requires the `chrome` feature.

```rust
use spider::features::chrome_common::RequestInterceptConfiguration;
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    // Requires `chrome` feature
    let mut website: Website = Website::new("https://spa-app.com")
        .with_limit(10)
        .with_chrome_intercept(RequestInterceptConfiguration::new(true)) // Block ads/analytics
        .with_stealth(true) // Anti-fingerprinting
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    let handle = tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{:?}", page.get_url());
        }
    });

    website.crawl().await;
    website.unsubscribe();
    let _ = handle.await;
}
```

--------------------------------

### Configure Spider Agent with Spider Cloud Tools

Source: https://context7.com/spider-rs/spider/llms.txt

Set up the spider agent to utilize Spider Cloud's managed crawling and AI extraction tools. Requires an API key.

```rust
use spider_agent::{Agent, SpiderCloudToolConfig, SpiderBrowserToolConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Agent with Spider Cloud HTTP tools
    let agent = Agent::builder()
        .with_spider_cloud_config(
            SpiderCloudToolConfig::new("sk-...")
                .with_enable_ai_routes(true) // Enable AI endpoints (paid plan)
        )
        .build()?;

    // Available tools: spider_cloud_crawl, spider_cloud_scrape, spider_cloud_search,
    // spider_cloud_links, spider_cloud_transform, spider_cloud_unblocker
    // AI tools: spider_cloud_ai_crawl, spider_cloud_ai_scrape, spider_cloud_ai_search

    // Agent with Spider Browser Cloud tools (CDP)
    let browser_agent = Agent::builder()
        .with_spider_browser_config(
            SpiderBrowserToolConfig::new("sk-...")
                .with_stealth(true)
                .with_country("us")
        )
        .build()?;

    // Available tools: spider_browser_navigate, spider_browser_html,
    // spider_browser_screenshot, spider_browser_evaluate,
    // spider_browser_click, spider_browser_fill, spider_browser_wait

    Ok(())
}
```

--------------------------------

### Planning Module

Source: https://github.com/spider-rs/spider/blob/main/spider_agent_types/SKILLS.md

Manages multi-step execution plans, checkpoints, and replanning logic for web automation.

```APIDOC
## ExecutionPlan

### Description
Handles the creation and execution of multi-step plans generated by an LLM.

### Components
- **PlannedStep**: Defines a step with ID, action, dependencies, and confidence.
- **Checkpoint**: Verification point using URL, element, text, or JS condition.
- **ReplanContext**: Context provided to the LLM when a failure occurs.
- **PlanningModeConfig**: Configuration for max steps and auto-execute thresholds.
```

--------------------------------

### Use Different Page Encodings

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates handling pages with different character encodings. Requires the `encoding` feature flag.

```bash
cargo run --example encoding --features encoding
```

--------------------------------

### Budget-Based Crawl Limits with Spider

Source: https://context7.com/spider-rs/spider/llms.txt

Control the scope of a crawl using path-based budgets to limit the number of pages crawled per URL pattern. This example sets a total limit and specific limits for sub-paths.

```rust
use spider::tokio;
use spider::website::Website;
use spider::hashbrown::HashMap;

#[tokio::main]
async fn main() {
    let mut website = Website::new("https://choosealicense.com")
        .with_budget(Some(HashMap::from([
            ("*", 15),    // max 15 pages total
            ("en", 11),   // max 11 pages under /en/
            ("fr", 3),    // max 3 pages under /fr/
        ])))
        .with_limit(15) // equivalent to ("*", 15)
        .build()
        .unwrap();

    website.crawl().await;

    let links = website.get_all_links_visited().await;
    println!("Crawled {} pages (budget-limited)", links.len());
}
```

--------------------------------

### OpenAI-Powered Browser Automation

Source: https://context7.com/spider-rs/spider/llms.txt

Utilize OpenAI GPT models for dynamic browser automation with per-URL prompt configuration. Requires the `openai` feature and `OPENAI_API_KEY` environment variable. Supports screenshots and request interception.

```rust
use spider::configuration::{GPTConfigs, WaitForIdleNetwork, ScreenShotConfig, ScreenshotParams};
use spider::features::chrome_common::RequestInterceptConfiguration;
use spider::hashbrown::HashMap;
use spider::website::Website;
use spider::{tokio, CaseInsensitiveString};
use std::time::Duration;

#[tokio::main]
async fn main() {
    // Requires `openai` feature and OPENAI_API_KEY env var
    let _ = tokio::fs::create_dir_all("./storage/").await;

    let screenshot_params = ScreenshotParams::new(Default::default(), Some(true), Some(false));
    let screenshot_config = ScreenShotConfig::new(screenshot_params, true, true, None);

    let website_url = "https://www.google.com";

    // Configure per-URL prompts
    let mut gpt_config = GPTConfigs::default();
    let prompt_url_map = HashMap::from([
        (
            CaseInsensitiveString::new(website_url),
            Box::new(GPTConfigs::new("gpt-4o-2024-05-13", "Search for Movies", 500)),
        ),
    ]);
    gpt_config.prompt_url_map = Some(Box::new(prompt_url_map));

    let mut website: Website = Website::new(website_url)
        .with_chrome_intercept(RequestInterceptConfiguration::new(true))
        .with_wait_for_idle_network(Some(WaitForIdleNetwork::new(Some(Duration::from_secs(30)))))
        .with_screenshot(Some(screenshot_config))
        .with_limit(2)
        .with_stealth(true)
        .with_openai(Some(gpt_config))
        .build()
        .unwrap();

    let mut rx = website.subscribe(16);

    tokio::spawn(async move {
        while let Ok(page) = rx.recv().await {
            println!("{}\n{}", page.get_url(), page.get_html());
        }
    });

    website.crawl().await;
}
```

--------------------------------

### Download Files in Subscription

Source: https://github.com/spider-rs/spider/blob/main/examples/README.md

Demonstrates downloading files as part of a subscription during a crawl.

```bash
cargo run --example subscribe_download
```

--------------------------------

### Load Skills from S3 in Rust

Source: https://github.com/spider-rs/spider/blob/main/spider_agent/SKILLS.md

Demonstrates loading custom skills from S3-compatible storage using `S3SkillSource` and `load_from_s3`. Configure bucket, path, region, and endpoint URL as needed. Skills loaded from S3 will override built-in skills with the same name.

```rust
use spider_agent::automation::skills::{S3SkillSource, load_from_s3, builtin_web_challenges};

let mut registry = builtin_web_challenges();
let source = S3SkillSource::new("my-bucket", "skills/")
    .with_region("us-east-1")
    .with_endpoint_url("https://r2.example.com"); // optional, for R2/MinIO

let loaded = load_from_s3(&mut registry, &source).await?;
// S3 skills override built-in skills with the same name
```

--------------------------------

### Apply Reusable Configuration Across Multiple Sites

Source: https://context7.com/spider-rs/spider/llms.txt

Create a shared configuration object and apply it to multiple website crawls. This is useful for setting common options like user agent, robots.txt parsing, subdomains, and delays for a batch of URLs.

```rust
use spider::configuration::Configuration;
use spider::website::Website;
use spider::tokio;

#[tokio::main]
async fn main() {
    // Create reusable configuration
    let config = Configuration::new()
        .with_user_agent(Some("MyBot/1.0"))
        .with_respect_robots_txt(true)
        .with_subdomains(false)
        .with_delay(100)
        .build();

    let urls = ["https://a.com", "https://b.com", "https://c.com"];

    for url in urls {
        match Website::new(url).with_config(config.to_owned()).build() {
            Ok(mut site) => {
                site.crawl().await;
                let links = site.get_all_links_visited().await;
                println!("{}: {} pages", url, links.len());
            }
            Err(e) => println!("Invalid URL: {:?}", e.get_url()),
        }
    }
}
```

--------------------------------

### Run Benchmarks

Source: https://github.com/spider-rs/spider/blob/main/CONTRIBUTING.md

Execute the performance benchmarks for the Spider project using Criterion. This helps in performance optimization.

```bash
cargo bench
```