Spider

https://docs.rs/spider/latest/spider
Admin
Spider is a multi-threaded web crawling library that rapidly crawls website pages to gather links...
Tokens:282,997
Snippets:3,607
Trust Score:9.7
Update:2 months ago
Show doc for...
Tokens:
Raw
### Trim Start Matches Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates removing prefixes that match a specific pattern.

```rust
assert_eq!("11foo1bar11".trim_start_matches('1'), "foo1bar11");
assert_eq!("123foo1bar123".trim_start_matches(char::is_numeric), "foo1bar123");
```

--------------------------------

### Trim Start Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates trimming leading whitespace from a string.

```rust
let s = "  עברית  ";
assert!(Some('ע') == s.trim_start().chars().next());
```

--------------------------------

### Initialize Proxy instances

Source: https://docs.rs/spider/latest/spider/client/struct.Proxy.html

Examples of creating different types of proxy configurations.

```rust
let proxy = reqwest::Proxy::http("https://secure.example")?;
```

```rust
let proxy = reqwest::Proxy::http("socks5://192.168.1.1:9000")?;
```

--------------------------------

### Box::from_raw Examples

Source: https://docs.rs/spider/latest/spider/features/automation/type.PageFactory.html

Examples demonstrating how to reconstruct a Box from a raw pointer.

```rust
let x = Box::new(5);
let ptr = Box::into_raw(x);
let x = unsafe { Box::from_raw(ptr) };
```

```rust
use std::alloc::{alloc, Layout};

unsafe {
    let ptr = alloc(Layout::new::<i32>()) as *mut i32;
    // In general .write is required to avoid attempting to destruct
    // the (uninitialized) previous contents of `ptr`, though for this
    // simple example `*ptr = 5` would have worked as well.
    ptr.write(5);
    let x = Box::from_raw(ptr);
}
```

--------------------------------

### Trim Start Matches Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates trimming characters from the start of a string slice based on a pattern.

```rust
let x: &[_] = &['1', '2'];
assert_eq!("12foo1bar12".trim_start_matches(x), "foo1bar12");
```

--------------------------------

### Initialize with Proxy Configuration

Source: https://docs.rs/spider/latest/spider/features/automation/struct.RemoteMultimodalConfigs.html

Example showing how to initialize the configuration and chain a proxy setting.

```rust
use spider_agent::automation::RemoteMultimodalConfigs;

let mm = RemoteMultimodalConfigs::new(
    "http://localhost:11434/v1/chat/completions",
    "qwen2.5-vl",
)
.with_proxies(Some(vec!["http://localhost:8080".to_string()]));
```

--------------------------------

### Box::from_non_null Examples

Source: https://docs.rs/spider/latest/spider/features/automation/type.PageFactory.html

Examples demonstrating how to reconstruct a Box from a NonNull pointer using experimental APIs.

```rust
#![feature(box_vec_non_null)]

let x = Box::new(5);
let non_null = Box::into_non_null(x);
let x = unsafe { Box::from_non_null(non_null) };
```

```rust
#![feature(box_vec_non_null)]

use std::alloc::{alloc, Layout};
use std::ptr::NonNull;

unsafe {
    let non_null = NonNull::new(alloc(Layout::new::<i32>()).cast::<i32>())
        .expect("allocation failed");
    // In general .write is required to avoid attempting to destruct
    // the (uninitialized) previous contents of `non_null`.
    non_null.write(5);
    let x = Box::from_non_null(non_null);
}
```

--------------------------------

### Configure Crawler from Prompt Example

Source: https://docs.rs/spider/latest/spider/features/automation/fn.configure_crawler_from_prompt.html

Demonstrates how to generate a configuration from a prompt and apply it to a website instance.

```rust
let config = configure_crawler_from_prompt(
    "http://localhost:11434/v1/chat/completions",
    "llama3",
    None,
    "Crawl only blog posts, max 50 pages, respect robots.txt"
).await?;

website.apply_prompt_configuration(&config);
```

--------------------------------

### Basic Pinning Example

Source: https://docs.rs/spider/latest/spider/client/dns/type.Addrs.html

Demonstrates basic pinning of a boxed value.

```rust
let foo = Box::new(());
let bar = Pin::from(foo);
```

--------------------------------

### Configure Client with Proxy

Source: https://docs.rs/spider/latest/spider/client/struct.Proxy.html

Examples of applying proxy settings to a Client builder.

```rust
let client = reqwest::Client::builder()
    .proxy(reqwest::Proxy::http("https://my.prox")?)
    .build()?;
```

```rust
let client = reqwest::Client::builder()
    .proxy(reqwest::Proxy::https("https://example.prox:4545")?)
    .build()?;
```

```rust
let client = reqwest::Client::builder()
    .proxy(reqwest::Proxy::all("http://pro.xy")?)
    .build()?;
```

--------------------------------

### Configure Website Crawler

Source: https://docs.rs/spider/latest/spider/configuration/struct.Configuration.html

Example showing how to initialize a Website instance and modify its configuration fields directly.

```rust
use spider::website::Website;
let mut website: Website = Website::new("https://choosealicense.com");
website.configuration.blacklist_url.insert(Default::default()).push("https://choosealicense.com/licenses/".to_string().into());
website.configuration.respect_robots_txt = true;
website.configuration.subdomains = true;
website.configuration.tld = true;
```

--------------------------------

### String capacity Method Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Returns the capacity of the String in bytes.

```rust
let s = String::with_capacity(10);

assert!(s.capacity() >= 10);
```

--------------------------------

### Example Usage of run_spawn_pages_with_factory

Source: https://docs.rs/spider/latest/spider/features/automation/fn.run_spawn_pages_with_factory.html

Demonstrates how to define a page factory and execute the function with custom options.

```rust
use std::sync::Arc;

// Create a page factory from the browser
let browser = Arc::clone(&browser);
let page_factory: PageFactory<String> = Box::new(move |url| {
    let browser = browser.clone();
    Box::pin(async move {
        browser.new_page(&url).await.map_err(|e| e.to_string())
    })
});

let options = SpawnPageOptions::new()
    .with_extraction("Extract page content")
    .with_page_setup(Arc::new(|page| {
        Box::pin(async move {
            // Setup event tracking propagated from main page
        })
    }));

let spawn_results = run_spawn_pages_with_factory(
    Arc::new(page_factory),
    result.spawn_pages,
    &config,
    options,
).await;
```

--------------------------------

### Usage Example for run_spawn_pages_concurrent

Source: https://docs.rs/spider/latest/spider/features/automation/fn.run_spawn_pages_concurrent.html

Demonstrates how to invoke the function after checking for spawn pages in an automation result.

```rust
let result = run_remote_multimodal_with_page(&config, &page, url).await?;
if result.has_spawn_pages() {
    let spawn_results = run_spawn_pages_concurrent(
        &browser,
        result.spawn_pages,
        &config,
    ).await;
    for spawn_result in spawn_results {
        println!("{}: {:?}", spawn_result.url, spawn_result.result);
    }
}
```

--------------------------------

### Initialize and crawl a website

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Basic usage example for creating a new Website instance and initiating a crawl process.

```Rust
use spider::website::Website;
let mut website = Website::new("http://example.com");
website.crawl();
// `Website` will be filled with links or pages when crawled. If you need pages with the resource
// call the `website.scrape` method with `website.get_pages` instead.
for link in website.get_links() {
    // do something
}
```

--------------------------------

### Client::request

Source: https://docs.rs/spider/latest/spider/client/type.Client.html

Start building a Request with a specific HTTP method and URL.

```APIDOC
### pub fn request<U>(&self, method: Method, url: U) -> RequestBuilder

Start building a Request with the specified Method and URL. Returns a RequestBuilder to set headers and body before sending.

#### Parameters
- **method** (Method) - Required - The HTTP method to use.
- **url** (U: IntoUrl) - Required - The URL to request.
```

--------------------------------

### Trim Matches Examples

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates removing prefixes and suffixes that match a specific pattern.

```rust
assert_eq!("11foo1bar11".trim_matches('1'), "foo1bar");
assert_eq!("123foo1bar123".trim_matches(char::is_numeric), "foo1bar");

let x: &[_] = &['1', '2'];
assert_eq!("12foo1bar12".trim_matches(x), "foo1bar");
```

```rust
assert_eq!("1foo1barXX".trim_matches(|c| c == '1' || c == 'X'), "foo1bar");
```

--------------------------------

### Arc::weak_count

Source: https://docs.rs/spider/latest/spider/website/type.OnLinkFindCallback.html

Gets the number of Weak pointers to this allocation.

```APIDOC
## pub fn weak_count(this: &Arc<T, A>) -> usize

### Description
Gets the number of `Weak` pointers to this allocation. Note that another thread can change the weak count at any time.
```

--------------------------------

### Usage Example

Source: https://docs.rs/spider/latest/spider/features/automation/fn.process_spawn_pages_with_config.html

Demonstrates how to invoke the function using URLs extracted from a page response and custom spawn options.

```rust
if let Some(spawn_urls) = page_response.spawn_pages.take() {
    let options = SpawnPageOptions::new()
        .with_extraction("Extract page content");

    let results = process_spawn_pages_with_config(
        &browser,
        &browser_context_id,
        spawn_urls,
        &mm_config,
        &configuration,
        options,
    ).await;
}
```

--------------------------------

### Configure page setup callback

Source: https://docs.rs/spider/latest/spider/features/automation/struct.SpawnPageOptions.html

Sets a callback to be executed on each spawned page for tasks like event listener registration.

```rust
use std::sync::Arc;

let options = SpawnPageOptions::new()
    .with_page_setup(Arc::new(|page: &Page| {
        Box::pin(async move {
            // Setup event tracking on this page
            // e.g., setup_chrome_events(page, tracker).await;
        })
    }));
```

--------------------------------

### starts_with<P>(pat: P)

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_EXTRA_PROMPT.html

Checks if the string starts with the given pattern.

```APIDOC
## pub fn starts_with<P>(&self, pat: P) -> bool

### Description
Returns `true` if the given pattern matches a prefix of this string slice.
```

--------------------------------

### Arc::strong_count

Source: https://docs.rs/spider/latest/spider/website/type.OnLinkFindCallback.html

Gets the number of strong (Arc) pointers to this allocation.

```APIDOC
## pub fn strong_count(this: &Arc<T, A>) -> usize

### Description
Gets the number of strong (`Arc`) pointers to this allocation. Note that another thread can change the strong count at any time.
```

--------------------------------

### pub async fn crawl(&mut self)

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Starts a standard crawl operation.

```APIDOC
## pub async fn crawl(&mut self)

### Description
Start a standard crawl.
```

--------------------------------

### pub fn prepare_url(u: &str) -> String

Source: https://docs.rs/spider/latest/spider/utils/fn.prepare_url.html

Prepares a URL string for parsing by ensuring it starts with a valid protocol (http or https).

```APIDOC
## prepare_url

### Description
Prepares the url for parsing if it fails. Use this method if the url does not start with http or https.

### Signature
`pub fn prepare_url(u: &str) -> String`

### Parameters
- **u** (&str) - The input URL string to be prepared.

### Returns
- **String** - The formatted URL string.
```

--------------------------------

### Trim End Examples

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates removing trailing whitespace and handling text directionality.

```rust
let s = "\n Hello\tworld\t\n";
assert_eq!("\n Hello\tworld", s.trim_end());
```

```rust
let s = "  English  ";
assert!(Some('h') == s.trim_end().chars().rev().next());

let s = "  עברית  ";
assert!(Some('ת') == s.trim_end().chars().rev().next());
```

--------------------------------

### Example output for concurrent cache access

Source: https://docs.rs/spider/latest/spider/features/openai_common/type.AICache.html

Displays the expected console output when multiple tasks attempt to initialize the same cache key concurrently.

```text
Task 1 started.
Task 0 started.
Task 2 started.
Task 3 started.
get_html() called by task 2.
Task 2 got the value. (len: 19419)
Task 1 got the value. (len: 19419)
Task 0 got the value. (len: 19419)
Task 3 got the value. (len: 19419)
```

--------------------------------

### pub async fn run_cron(website: Website) -> Runner

Source: https://docs.rs/spider/latest/spider/website/fn.run_cron.html

Starts a cron job taking ownership of the website.

```APIDOC
## run_cron

### Description
Starts a cron job taking ownership of the website. This function is available on the `cron` feature flag.

### Signature
`pub async fn run_cron(website: Website) -> Runner`

### Parameters
- **website** (Website) - Required - The website instance to take ownership of for the cron job.

### Returns
- **Runner** - The runner instance managing the cron job.
```

--------------------------------

### Trim Right Examples (Deprecated)

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates the deprecated trim_right method and its behavior with different languages.

```rust
let s = " Hello\tworld\t";

assert_eq!(" Hello\tworld", s.trim_right());
```

```rust
let s = "English  ";
assert!(Some('h') == s.trim_right().chars().rev().next());

let s = "עברית  ";
assert!(Some('ת') == s.trim_right().chars().rev().next());
```

--------------------------------

### Website::with_remote_fetcher

Source: https://docs.rs/spider/latest/spider/fetcher/index.html

Method to install a custom fetcher onto a Website instance.

```APIDOC
## Method: Website::with_remote_fetcher

### Description
Installs a custom `RemoteFetcher` implementation onto a `Website` instance. This reroutes the per-URL network round-trip through the user-provided implementation.

### Parameters
- **fetcher** (T: RemoteFetcher) - The custom fetcher implementation to use for network requests.
```

--------------------------------

### Initialize and use SerperProvider

Source: https://docs.rs/spider/latest/spider/agent/struct.SerperProvider.html

Demonstrates how to instantiate the provider with an API key and execute a search query using a reqwest client.

```rust
use spider_agent::search::SerperProvider;
use spider_agent::config::SearchOptions;

let provider = SerperProvider::new("your-api-key");
let client = reqwest::Client::new();
let results = provider.search("rust web crawler", &SearchOptions::default(), &client).await?;
```

--------------------------------

### fn should_run(&self) -> bool

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Decides whether or not to start running the job.

```APIDOC
## fn should_run(&self) -> bool

### Description
Logic to decide whether the job should be initiated.
```

--------------------------------

### pub async fn crawl_smart(&mut self)

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Starts a website crawl using smart concurrency, utilizing HTTP first and JavaScript rendering as needed. Requires the 'smart' and 'decentralized' features.

```APIDOC
## pub async fn crawl_smart(&mut self)

### Description
Start to crawl website with async concurrency smart. Use HTTP first and JavaScript Rendering as needed. This has no effect without the `smart` flag enabled.

### Requirements
- Crate features: `decentralized`, `smart`
```

--------------------------------

### pub async fn configure_from_prompt(&mut self, api_url: &str, model_name: &str, api_key: Option<&str>, prompt: &str) -> Result<&mut Self, EngineError>

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Configures the website from a natural language prompt using an LLM.

```APIDOC
## pub async fn configure_from_prompt(&mut self, api_url: &str, model_name: &str, api_key: Option<&str>, prompt: &str) -> Result<&mut Self, EngineError>

### Description
Configure the website from a natural language prompt using an LLM. This is a convenience method that calls the LLM to generate configuration and applies it to the website in one step.

### Arguments
- **api_url** (str) - OpenAI-compatible chat completions endpoint
- **model_name** (str) - Model identifier (e.g., “gpt-4”, “llama3”, “qwen2.5”)
- **api_key** (Option<&str>) - Optional API key for authenticated endpoints
- **prompt** (str) - Natural language description of crawling requirements

### Requirements
- Crate features: `agent` and `serde`
```

--------------------------------

### Get a reference to the first value

Source: https://docs.rs/spider/latest/spider/client/header/struct.OccupiedEntry.html

Use get() to access the first value in the entry. This method panics if no values are associated with the entry.

```rust
let mut map = HeaderMap::new();
map.insert(HOST, "hello.world".parse().unwrap());

if let Entry::Occupied(mut e) = map.entry("host") {
    assert_eq!(e.get(), &"hello.world");

    e.append("hello.earth".parse().unwrap());

    assert_eq!(e.get(), &"hello.world");
}
```

--------------------------------

### Initialize and Use BraveProvider

Source: https://docs.rs/spider/latest/spider/features/search_providers/struct.BraveProvider.html

Demonstrates how to instantiate the BraveProvider with an API key and execute a search query.

```rust
use spider::features::search_providers::BraveProvider;
use spider::features::search::{SearchOptions, SearchProvider};

let provider = BraveProvider::new("your-api-key");
let results = provider.search("rust web crawler", &SearchOptions::default(), None).await?;
```

--------------------------------

### Initialize and use an Agent

Source: https://docs.rs/spider/latest/spider/agent/struct.Agent.html

Demonstrates creating an Agent with OpenAI and Serper configurations, then spawning a concurrent search task.

```rust
use spider_agent::{Agent, AgentConfig};
use std::sync::Arc;

let agent = Arc::new(Agent::builder()
    .with_openai("sk-...", "gpt-4o")
    .with_search_serper("serper-key")
    .build()?);

// Spawn concurrent tasks
let agent_clone = agent.clone();
tokio::spawn(async move {
    agent_clone.search("rust web frameworks").await
});
```

--------------------------------

### Get length of HeaderValue

Source: https://docs.rs/spider/latest/spider/client/header/struct.HeaderValue.html

Returns the length of the HeaderValue in bytes.

```rust
let val = HeaderValue::from_static("hello");
assert_eq!(val.len(), 5);
```

--------------------------------

### Trim Left Examples (Deprecated)

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Demonstrates the deprecated trim_left method and its behavior with different languages.

```rust
let s = " Hello\tworld\t";

assert_eq!("Hello\tworld\t", s.trim_left());
```

```rust
let s = "  English";
assert!(Some('E') == s.trim_left().chars().next());

let s = "  עברית";
assert!(Some('ע') == s.trim_left().chars().next());
```

--------------------------------

### Initialize and use BraveProvider

Source: https://docs.rs/spider/latest/spider/agent/struct.BraveProvider.html

Demonstrates how to instantiate the provider with an API key and execute a search query using a reqwest client.

```rust
use spider_agent::search::BraveProvider;
use spider_agent::config::SearchOptions;

let provider = BraveProvider::new("your-api-key");
let client = reqwest::Client::new();
let results = provider.search("rust web crawler", &SearchOptions::default(), &client).await?;
```

--------------------------------

### Agent::prompt

Source: https://docs.rs/spider/latest/spider/agent/struct.Agent.html

Send a prompt to the LLM and get a response.

```APIDOC
## pub async fn prompt(&self, messages: Vec<Message>) -> Result<String, AgentError>

### Description
Send a prompt to the LLM and get a response.

### Parameters
- **messages** (Vec<Message>) - Required - A list of messages to send to the LLM.
```

--------------------------------

### Performing a Search with BingProvider

Source: https://docs.rs/spider/latest/spider/agent/struct.BingProvider.html

Example usage of the BingProvider to execute a search query. Requires an API key and a reqwest client.

```rust
use spider_agent::search::BingProvider;
use spider_agent::config::SearchOptions;

let provider = BingProvider::new("your-api-key");
let client = reqwest::Client::new();
let results = provider.search("rust web crawler", &SearchOptions::default(), &client).await?;
```

--------------------------------

### Initialize and use SerperProvider

Source: https://docs.rs/spider/latest/spider/features/search_providers/struct.SerperProvider.html

Demonstrates how to instantiate the provider with an API key and execute a search query. Requires the 'search_serper' or 'search' feature flags.

```rust
use spider::features::search_providers::SerperProvider;
use spider::features::search::{SearchOptions, SearchProvider};

let provider = SerperProvider::new("your-api-key");
let results = provider.search("rust web crawler", &SearchOptions::default(), None).await?;
```

--------------------------------

### pub async fn run_cron(&self) -> Runner

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Starts a cron job for the website. Requires the 'cron' feature.

```APIDOC
## pub async fn run_cron(&self) -> Runner

### Description
Starts a cron job for the website. If you use subscribe on another thread, you need to abort the handle in conjunction with runner.stop.

### Requirements
- Crate feature: `cron`
```

--------------------------------

### String is_empty Method Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Checks if the String has a length of zero.

```rust
let mut v = String::new();
assert!(v.is_empty());

v.push('a');
assert!(!v.is_empty());
```

--------------------------------

### Website::new

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Initializes a new Website instance for a given domain.

```APIDOC
## Website::new(domain: &str)

### Description
Creates a new instance of the Website struct configured for the specified domain.

### Signature
`pub fn new(domain: &str) -> Website`

### Parameters
- **domain** (&str) - Required - The URL or domain to crawl.
```

--------------------------------

### Configure website from prompt

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Convenience method to generate and apply configuration from a natural language prompt in one step.

```rust
let mut website = Website::new("https://example.com");
website.configure_from_prompt(
    "http://localhost:11434/v1/chat/completions",
    "llama3",
    None,
    "Only crawl product pages, use 100ms delay, max depth 5, respect robots.txt"
).await?;

website.crawl().await;
```

--------------------------------

### String len Method Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Returns the length of the String in bytes.

```rust
let a = String::from("foo");
assert_eq!(a.len(), 3);

let fancy_f = String::from("ƒoo");
assert_eq!(fancy_f.len(), 4);
assert_eq!(fancy_f.chars().count(), 3);
```

--------------------------------

### Initialize and use OpenAIProvider

Source: https://docs.rs/spider/latest/spider/agent/struct.OpenAIProvider.html

Demonstrates creating an instance of OpenAIProvider and executing a completion request using the reqwest client.

```rust
use spider_agent::llm::{OpenAIProvider, Message, CompletionOptions};

let provider = OpenAIProvider::new("sk-...", "gpt-4o");
let client = reqwest::Client::new();

let messages = vec![
    Message::system("You are a helpful assistant."),
    Message::user("What is Rust?"),
];

let response = provider.complete(messages, &CompletionOptions::default(), &client).await?;
println!("{}", response.content);
```

--------------------------------

### Convert Box to raw pointer

Source: https://docs.rs/spider/latest/spider/client/dns/type.Addrs.html

Examples demonstrating how to consume a Box into a raw pointer and handle memory cleanup.

```rust
let x = Box::new(String::from("Hello"));
let ptr = Box::into_raw(x);
let x = unsafe { Box::from_raw(ptr) };
```

```rust
use std::alloc::{dealloc, Layout};
use std::ptr;

let x = Box::new(String::from("Hello"));
let ptr = Box::into_raw(x);
unsafe {
    ptr::drop_in_place(ptr);
    dealloc(ptr as *mut u8, Layout::new::<String>());
}
```

```rust
let x = Box::new(String::from("Hello"));
let ptr = Box::into_raw(x);
unsafe {
    drop(Box::from_raw(ptr));
}
```

--------------------------------

### Result::is_ok and is_err Implementation

Source: https://docs.rs/spider/latest/spider/agent/type.AgentResult.html

Examples of checking the status of a Result.

```rust
let x: Result<i32, &str> = Ok(-3);
assert_eq!(x.is_ok(), true);

let x: Result<i32, &str> = Err("Some error message");
assert_eq!(x.is_ok(), false);
```

```rust
let x: Result<i32, &str> = Ok(-3);
assert_eq!(x.is_err(), false);

let x: Result<i32, &str> = Err("Some error message");
assert_eq!(x.is_err(), true);
```

--------------------------------

### pub async fn scrape(&mut self)

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Starts a website scrape or download operation with async concurrency.

```APIDOC
## pub async fn scrape(&mut self)

### Description
Start to scrape/download website with async concurrency.
```

--------------------------------

### subscribe_guard

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Setup subscription counter to track concurrent operation completions. Requires 'sync' feature.

```APIDOC
## pub fn subscribe_guard(&mut self) -> Option<ChannelGuard>

### Description
Setup subscription counter to track concurrent operation completions. This helps keep a chrome instance active until all operations are completed. This method is only available when the 'sync' feature is enabled.
```

--------------------------------

### Initialize and use TavilyProvider

Source: https://docs.rs/spider/latest/spider/agent/struct.TavilyProvider.html

Demonstrates how to instantiate the provider with an API key and execute a search query using a reqwest client.

```rust
use spider_agent::search::TavilyProvider;
use spider_agent::config::SearchOptions;

let provider = TavilyProvider::new("your-api-key");
let client = reqwest::Client::new();
let results = provider.search("rust web crawler", &SearchOptions::default(), &client).await?;
```

--------------------------------

### Result::flatten Implementation

Source: https://docs.rs/spider/latest/spider/agent/type.AgentResult.html

Examples of flattening nested Result types.

```rust
let x: Result<Result<&'static str, u32>, u32> = Ok(Ok("hello"));
assert_eq!(Ok("hello"), x.flatten());

let x: Result<Result<&'static str, u32>, u32> = Ok(Err(6));
assert_eq!(Err(6), x.flatten());

let x: Result<Result<&'static str, u32>, u32> = Err(6);
assert_eq!(Err(6), x.flatten());
```

```rust
let x: Result<Result<Result<&'static str, u32>, u32>, u32> = Ok(Ok(Ok("hello")));
assert_eq!(Ok(Ok("hello")), x.flatten());
assert_eq!(Ok("hello"), x.flatten().flatten());
```

--------------------------------

### Configure and execute run_spawn_pages_with_options

Source: https://docs.rs/spider/latest/spider/features/automation/fn.run_spawn_pages_with_options.html

Demonstrates initializing SpawnPageOptions with extraction and screenshot settings, executing the function, and iterating over the resulting spawned pages.

```rust
let options = SpawnPageOptions::new()
    .with_extraction("Extract the main content, title, and any links")
    .with_screenshot(true)
    .with_max_rounds(2);

let spawn_results = run_spawn_pages_with_options(
    &browser,
    result.spawn_pages,
    &config,
    options,
).await;

for spawn_result in &spawn_results {
    if let Some(data) = spawn_result.extracted() {
        println!("Extracted from {}: {}", spawn_result.url, data);
    }
    if let Some(screenshot) = spawn_result.screenshot() {
        // Save screenshot (base64 encoded PNG)
        println!("Got screenshot from {} ({} bytes)", spawn_result.url, screenshot.len());
    }
}
```

--------------------------------

### Client::get

Source: https://docs.rs/spider/latest/spider/client/type.Client.html

Convenience method to make a GET request to a URL.

```APIDOC
### pub fn get<U>(&self, url: U) -> RequestBuilder

Convenience method to make a GET request to a URL. Returns a RequestBuilder for further configuration.

#### Parameters
- **url** (U: IntoUrl) - Required - The URL to request.
```

--------------------------------

### queue

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Get a sender for queueing extra links mid crawl. Requires 'sync' feature.

```APIDOC
## pub fn queue(&mut self, capacity: usize) -> Option<Sender<String>>

### Description
Get a sender for queueing extra links mid crawl. This method is only available when the 'sync' feature is enabled.

### Parameters
- **capacity** (usize) - Required - The capacity of the queue.
```

--------------------------------

### Apply prompt configuration to website

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Configures a website instance using settings generated from an LLM prompt.

```rust
use spider::features::automation::{RemoteMultimodalEngine, configure_crawler_from_prompt};

let config = configure_crawler_from_prompt(
    "http://localhost:11434/v1/chat/completions",
    "llama3",
    None,
    "Crawl blog posts only, respect robots.txt, max 100 pages, 200ms delay"
).await?;

let mut website = Website::new("https://example.com");
website.apply_prompt_configuration(&config);
```

--------------------------------

### Consume Box with Allocator

Source: https://docs.rs/spider/latest/spider/client/dns/type.Addrs.html

Demonstrates consuming a Box into a raw pointer and allocator, with examples for automatic and manual cleanup.

```rust
#![feature(allocator_api)]

use std::alloc::System;

let x = Box::new_in(String::from("Hello"), System);
let (ptr, alloc) = Box::into_raw_with_allocator(x);
let x = unsafe { Box::from_raw_in(ptr, alloc) };
```

```rust
#![feature(allocator_api)]

use std::alloc::{Allocator, Layout, System};
use std::ptr::{self, NonNull};

let x = Box::new_in(String::from("Hello"), System);
let (ptr, alloc) = Box::into_raw_with_allocator(x);
unsafe {
    ptr::drop_in_place(ptr);
    let non_null = NonNull::new_unchecked(ptr);
    alloc.deallocate(non_null.cast(), Layout::new::<String>());
}
```

--------------------------------

### Strip Circumfix Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Removes both a prefix and a suffix exactly once if both exist and do not overlap.

```rust
assert_eq!("bar:hello:foo".strip_circumfix("bar:", ":foo"), Some("hello"));
assert_eq!("bar:foo".strip_circumfix("foo", "foo"), None);
assert_eq!("foo:bar;".strip_circumfix("foo:", ';'), Some("bar"));
assert_eq!("foo:bar:baz".strip_circumfix("foo:bar:", ":bar:baz"), None);
```

--------------------------------

### Initialize and use BingProvider

Source: https://docs.rs/spider/latest/spider/features/search_providers/struct.BingProvider.html

Demonstrates how to instantiate the BingProvider with an API key and execute a search query. Requires the 'search_bing' or 'search' feature flags.

```rust
use spider::features::search_providers::BingProvider;
use spider::features::search::{SearchOptions, SearchProvider};

let provider = BingProvider::new("your-api-key");
let results = provider.search("rust web crawler", &SearchOptions::default(), None).await?;
```

--------------------------------

### Strip Suffix Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Removes a suffix exactly once if it exists, returning an Option.

```rust
assert_eq!("bar:foo".strip_suffix(":foo"), Some("bar"));
assert_eq!("bar:foo".strip_suffix("bar"), None);
assert_eq!("foofoo".strip_suffix("foo"), Some("foo"));
```

--------------------------------

### Strip Prefix Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Removes a prefix exactly once if it exists, returning an Option.

```rust
assert_eq!("foo:bar".strip_prefix("foo:"), Some("bar"));
assert_eq!("foo:bar".strip_prefix("bar"), None);
assert_eq!("foofoo".strip_prefix("foo"), Some("foo"));
```

--------------------------------

### String as_bytes Method Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Returns the byte slice representation of the String.

```rust
let s = String::from("hello");

assert_eq!(&[104, 101, 108, 108, 111], s.as_bytes());
```

--------------------------------

### String as_str Method Example

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Extracts a string slice from the underlying String.

```rust
let s = String::from("foo");

assert_eq!("foo", s.as_str());
```

--------------------------------

### Crawler Implementation for Website

Source: https://docs.rs/spider/latest/spider/traits/trait.Crawler.html

Example implementation of the Crawler trait for the Website struct.

```rust
impl Crawler for Website
    type Page = Page
```

--------------------------------

### Capture screenshots during crawl

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Example of using tokio to spawn a task that processes crawled pages and captures screenshots.

```rust
    tokio::spawn(async move {
        while let Ok(page) = rx2.recv().await {
            println!("📸 - {:?}", page.get_url());
            page
                .screenshot(
                    true,
                    true,
                    spider::configuration::CaptureScreenshotFormat::Png,
                    Some(75),
                    None::<std::path::PathBuf>,
                    None,
                )
                .await;
            rxg.inc();
        }
    });
    website.crawl().await;
}
```

--------------------------------

### Extracting Result Data

Source: https://docs.rs/spider/latest/spider/features/automation/struct.RemoteMultimodalEngine.html

Example of accessing the extracted field from a result object.

```rust
println!("Extracted: {:?}", result.extracted);
```

--------------------------------

### Initialize and Use AutomationMemory

Source: https://docs.rs/spider/latest/spider/features/automation/struct.AutomationMemory.html

Demonstrates initializing the memory, setting key-value pairs, and generating a context string for LLM consumption.

```rust
use spider_agent_types::AutomationMemory;

let mut memory = AutomationMemory::default();
memory.set("user_logged_in", serde_json::json!(true));
memory.set("cart_items", serde_json::json!(["item1", "item2"]));

// Memory is serialized and included in LLM context each round
let context = memory.to_context_string();
```

--------------------------------

### fn handle(&mut self) -> Pin<Box<dyn Future<Output = ()> + Send>>

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Defines the action that occurs when the cron starts the job.

```APIDOC
## fn handle(&mut self) -> Pin<Box<dyn Future<Output = ()> + Send>>

### Description
This is where your job's logic is defined. This method is invoked when the cron triggers the job.
```

--------------------------------

### Initialize and use TavilyProvider

Source: https://docs.rs/spider/latest/spider/features/search_providers/struct.TavilyProvider.html

Demonstrates how to instantiate the provider with an API key and execute a search query. Requires the 'search_tavily' or 'search' feature flags.

```rust
use spider::features::search_providers::TavilyProvider;
use spider::features::search::{SearchOptions, SearchProvider};

let provider = TavilyProvider::new("your-api-key");
let results = provider.search("rust web crawler", &SearchOptions::default(), None).await?;
```

--------------------------------

### Result::is_ok_and Implementation

Source: https://docs.rs/spider/latest/spider/agent/type.AgentResult.html

Example of checking if a Result is Ok and matches a predicate.

```rust
let x: Result<u32, &str> = Ok(2);
assert_eq!(x.is_ok_and(|x| x > 1), true);

let x: Result<u32, &str> = Ok(0);
assert_eq!(x.is_ok_and(|x| x > 1), false);

let x: Result<u32, &str> = Err("hey");
assert_eq!(x.is_ok_and(|x| x > 1), false);

let x: Result<String, &str> = Ok("ownership".to_string());
assert_eq!(x.as_ref().is_ok_and(|x| x.len() > 1), true);
println!("still alive {:?}", x);
```

--------------------------------

### Result::transpose Implementation

Source: https://docs.rs/spider/latest/spider/agent/type.AgentResult.html

Example of transposing a Result of an Option into an Option of a Result.

```rust
#[derive(Debug, Eq, PartialEq)]
struct SomeErr;

let x: Result<Option<i32>, SomeErr> = Ok(Some(5));
let y: Option<Result<i32, SomeErr>> = Some(Ok(5));
assert_eq!(x.transpose(), y);
```

--------------------------------

### BraveProvider::new

Source: https://docs.rs/spider/latest/spider/agent/struct.BraveProvider.html

Creates a new instance of the BraveProvider with the specified API key.

```APIDOC
## pub fn new(api_key: impl Into<String>) -> BraveProvider

### Description
Creates a new Brave provider with the given API key.

### Parameters
- **api_key** (impl Into<String>) - Required - The API key for authenticating with the Brave Search API.
```

--------------------------------

### Check string prefix

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Returns true if the pattern matches the start of the string.

```rust
let bananas = "bananas";

assert!(bananas.starts_with("bana"));
assert!(!bananas.starts_with("nana"));
```

```rust
let bananas = "bananas";

// Note that both of these assert successfully.
assert!(bananas.starts_with(&['b', 'a', 'n', 'a']));
assert!(bananas.starts_with(&['a', 'b', 'c', 'd']));
```

--------------------------------

### SmartCache::start_cleanup_task

Source: https://docs.rs/spider/latest/spider/features/automation/struct.SmartCache.html

Starts a background task to periodically remove expired entries.

```APIDOC
## pub fn start_cleanup_task(self: Arc<SmartCache<V>>, interval: Duration) -> JoinHandle<()>

### Description
Start a background cleanup task. Runs cleanup every `interval` duration.

### Parameters
- **interval** (Duration) - The frequency of cleanup runs.
```

--------------------------------

### Initialize subscription guard

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Demonstrates setting up a subscription guard to track concurrent operations. Requires the 'sync' feature and careful management of the guard to prevent infinite crawl loops.

```rust
use spider::tokio;
use spider::website::Website;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("http://example.com");
    let mut rx2 = website.subscribe(18);
    let mut rxg = website.subscribe_guard().unwrap();
```

--------------------------------

### Parsing robots.txt with RobotFileParser

Source: https://docs.rs/spider/latest/spider/packages/robotparser/parser/index.html

Demonstrates initializing the parser and checking fetch permissions for a specific URL.

```rust
extern crate spider;

use spider::packages::robotparser::RobotFileParser;
use reqwest::blocking::Client;

fn main() {
    let parser = RobotFileParser::new();
    let client = Client::new();
    parser.read(&client, &"http://www.python.org/robots.txt");
    assert!(parser.can_fetch("*", "http://www.python.org/robots.txt"));
}
```

--------------------------------

### Get database path signature

Source: https://docs.rs/spider/latest/spider/features/disk/fn.get_db_path.html

Function signature for retrieving the database path.

```rust
pub fn get_db_path(crawl_id: &Option<String>) -> String
```

--------------------------------

### Get String Representation

Source: https://docs.rs/spider/latest/spider/client/struct.StatusCode.html

Returns the numerical string representation of the status code.

```rust
let status = http::StatusCode::OK;
assert_eq!(status.as_str(), "200");
```

--------------------------------

### setup_browser_configuration

Source: https://docs.rs/spider/latest/spider/features/chrome/fn.setup_browser_configuration.html

Initializes the browser configuration for the spider crate. This function is available only when the 'chrome' feature is enabled.

```APIDOC
## setup_browser_configuration

### Description
Setup the browser configuration. Returns an Option containing a tuple of (Browser, Handler, Option<String>), where the third element is the connected endpoint URL if one was used.

### Signature
`pub async fn setup_browser_configuration(config: &Configuration) -> Option<(Browser, Handler, Option<String>)>`

### Parameters
- **config** (&Configuration) - Required - The configuration object used to set up the browser.

### Returns
- **Option<(Browser, Handler, Option<String>)>** - Returns `Some` with the browser instance, handler, and optional endpoint URL, or `None` if configuration fails. The URL is `Some(..)` for multi-endpoint failover and single-remote paths, and `None` for the local-launch path.
```

--------------------------------

### Get u16 from StatusCode

Source: https://docs.rs/spider/latest/spider/client/struct.StatusCode.html

Retrieves the underlying u16 value from a StatusCode instance.

```rust
let status = http::StatusCode::OK;
assert_eq!(status.as_u16(), 200);
```

--------------------------------

### Demonstrate concurrent cache initialization with optionally_get_with

Source: https://docs.rs/spider/latest/spider/features/openai_common/type.AICache.html

Shows how multiple concurrent tasks can safely request the same key, ensuring the initialization future is only executed once.

```rust
// Cargo.toml
//
// [dependencies]
// moka = { version = "0.12", features = ["future"] }
// futures-util = "0.3"
// reqwest = "0.11"
// tokio = { version = "1", features = ["rt-multi-thread", "macros" ] }
use moka::future::Cache;

// This async function tries to get HTML from the given URI.
async fn get_html(task_id: u8, uri: &str) -> Option<String> {
    println!("get_html() called by task {task_id}.");
    reqwest::get(uri).await.ok()?.text().await.ok()
}

#[tokio::main]
async fn main() {
    let cache = Cache::new(100);

    // Spawn four async tasks.
    let tasks: Vec<_> = (0..4_u8)
        .map(|task_id| {
            let my_cache = cache.clone();
            tokio::spawn(async move {
                println!("Task {task_id} started.");

                // Try to insert and get the value for key1. Although
                // all four async tasks will call `try_get_with`
                // at the same time, get_html() must be called only once.
                let value = my_cache
                    .optionally_get_with(
                        "key1",
                        get_html(task_id, "https://www.rust-lang.org"),
                    ).await;

                // Ensure the value exists now.
                assert!(value.is_some());
                assert!(my_cache.get(&"key1").await.is_some());

                println!(
                    "Task {task_id} got the value. (len: {})",
                    value.unwrap().len()
                );
            })
        })
        .collect();

    // Run all tasks concurrently and wait for them to complete.
    futures_util::future::join_all(tasks).await;
}
```

--------------------------------

### pub async fn create_dir_all(path: String) -> Result<()>

Source: https://docs.rs/spider/latest/spider/utils/uring_fs/fn.create_dir_all.html

Recursively creates directories at the specified path. This function is available on Linux and requires the 'io_uring' feature flag.

```APIDOC
## Function: create_dir_all

### Description
Recursively creates directories at the provided `path`. This operation is performed asynchronously.

### Signature
`pub async fn create_dir_all(path: String) -> Result<()>`

### Parameters
- **path** (String) - The target directory path to create.

### Requirements
- **Platform**: Linux
- **Feature Flag**: `io_uring`
```

--------------------------------

### Configure Remote Multimodal Automation

Source: https://docs.rs/spider/latest/spider/website/struct.Website.html

Demonstrates how to set up remote multimodal configurations to drive browser automation using an OpenAI-compatible endpoint.

```rust
use spider::website::Website;
use spider::configuration::Configuration;
use spider::features::automation::{RemoteMultimodalConfigs, RemoteMultimodalConfig};

// Build the engine configs (similar to GPTConfigs::new(...))
let mm_cfgs = RemoteMultimodalConfigs::new(
    "http://localhost:11434/v1/chat/completions",
    "qwen2.5-vl", // any OpenAI-compatible model id your endpoint understands
)
// .with_api_key("your-api-key-if-needed")
.with_system_prompt_extra("Never log in. Prefer closing popups and continuing.")
.with_user_message_extra("Goal: reach the pricing page, then stop.")
.with_cfg(RemoteMultimodalConfig {
    // keep HTML smaller if you want lower token usage
    include_html: true,
    html_max_bytes: 24_000,
    include_url: true,
    include_title: true,
    // loop controls
    max_rounds: 6,
    post_plan_wait_ms: 400,
    ..Default::default()
})
.with_concurrency_limit(8);

// Attach to the crawler configuration
let mut cfg = Configuration::new();
cfg.with_remote_multimodal(Some(mm_cfgs));

// Use the configuration in a Website (example)
let mut site = Website::new("https://example.com");
site.with_config(cfg);

// Start crawling/scraping as you normally would...
// site.crawl().await?;

Ok(())
```

--------------------------------

### PromptUrlGate::new

Source: https://docs.rs/spider/latest/spider/features/automation/struct.PromptUrlGate.html

Creates a new, empty instance of PromptUrlGate.

```APIDOC
## pub fn new() -> PromptUrlGate

### Description
Creates a new empty prompt URL gate.

### Returns
- **PromptUrlGate** - A new instance with no URL overrides.
```

--------------------------------

### Rust match_indices method

Source: https://docs.rs/spider/latest/spider/features/gemini/struct.BROWSER_ACTIONS_SYSTEM_PROMPT.html

Returns an iterator over disjoint matches of a pattern and their starting indices.

```rust
let v: Vec<_> = "abcXXXabcYYYabc".match_indices("abc").collect();
assert_eq!(v, [(0, "abc"), (6, "abc"), (12, "abc")]);

let v: Vec<_> = "1abcabc2".match_indices("abc").collect();
assert_eq!(v, [(1, "abc"), (4, "abc")]);

let v: Vec<_> = "ababa".match_indices("aba").collect();
assert_eq!(v, [(0, "aba")]); // only the first `aba`
```

--------------------------------

### PrefetchManager::prefetch

Source: https://docs.rs/spider/latest/spider/features/automation/struct.PrefetchManager.html

Starts prefetching a specific URL using the provided fetcher function.

```APIDOC
### pub async fn prefetch<F, Fut>(&self, url: String, fetcher: F)

Starts prefetching a URL. The fetcher is a closure that takes a String and returns a Future resolving to an Option<String>.
```

--------------------------------

### SerperProvider::new

Source: https://docs.rs/spider/latest/spider/agent/struct.SerperProvider.html

Creates a new instance of the SerperProvider with the specified API key.

```APIDOC
## pub fn new(api_key: impl Into<String>) -> SerperProvider

### Description
Initializes a new SerperProvider instance using the provided API key for authentication.

### Parameters
- **api_key** (impl Into<String>) - Required - The API key for the Serper.dev service.
```

--------------------------------

### EngineMethod Enum Definition

Source: https://docs.rs/spider/latest/spider/fetch_engine/enum.EngineMethod.html

The definition of the EngineMethod enum, which currently supports only the GET variant.

```rust
pub enum EngineMethod {
    Get,
}
```

--------------------------------

### entry

Source: https://docs.rs/spider/latest/spider/client/header/struct.HeaderMap.html

Gets the given key’s corresponding entry in the map for in-place manipulation.

```APIDOC
## pub fn entry<K>(&mut self, key: K) -> Entry<'_, T>

### Description
Gets the given key’s corresponding entry in the map for in-place manipulation. Panics if capacity exceeds max HeaderMap capacity.
```

--------------------------------

### OpenAIProvider::new

Source: https://docs.rs/spider/latest/spider/agent/struct.OpenAIProvider.html

Creates a new instance of the OpenAIProvider with the specified API key and model.

```APIDOC
## pub fn new(api_key: impl Into<String>, model: impl Into<String>) -> OpenAIProvider

### Description
Creates a new OpenAI provider instance. Defaults to 'Auto' mode, which automatically selects the appropriate API based on the host.
```

--------------------------------

### pub async fn prefetch(domains: &[&str], client: &Client, ttl: Duration)

Source: https://docs.rs/spider/latest/spider/utils/robots_cache/fn.prefetch.html

Batch prefetch robots.txt for multiple domains concurrently. Failures are silently ignored, and subsequent calls to get_or_fetch will retry.

```APIDOC
## prefetch

### Description
Batch prefetch robots.txt for multiple domains concurrently. Fetches all domains in parallel using a `JoinSet`. Failures are silently ignored — the next call to `get_or_fetch` will retry.

### Signature
`pub async fn prefetch(domains: &[&str], client: &Client, ttl: Duration)`

### Parameters
- **domains** (&[&str]) - A slice of domain strings to prefetch.
- **client** (&Client) - The HTTP client used to perform the requests.
- **ttl** (Duration) - The time-to-live duration for the cached robots.txt entries.

### Requirements
- Available on crate feature `robots_cache` only.
```

--------------------------------

### PromptConfiguration Struct Definition

Source: https://docs.rs/spider/latest/spider/features/automation/struct.PromptConfiguration.html

Defines the available configuration fields for prompt-based crawler setup.

```rust
pub struct PromptConfiguration {Show 31 fields
    pub respect_robots_txt: Option<bool>,
    pub subdomains: Option<bool>,
    pub tld: Option<bool>,
    pub depth: Option<usize>,
    pub delay: Option<u64>,
    pub request_timeout_ms: Option<u64>,
    pub crawl_timeout_ms: Option<u64>,
    pub blacklist_url: Option<Vec<String>>,
    pub whitelist_url: Option<Vec<String>>,
    pub external_domains: Option<Vec<String>>,
    pub user_agent: Option<String>,
    pub headers: Option<HashMap<String, String>>,
    pub http2_prior_knowledge: Option<bool>,
    pub accept_invalid_certs: Option<bool>,
    pub proxies: Option<Vec<String>>,
    pub redirect_limit: Option<usize>,
    pub budget: Option<HashMap<String, u32>>,
    pub max_page_bytes: Option<f64>,
    pub full_resources: Option<bool>,
    pub only_html: Option<bool>,
    pub return_page_links: Option<bool>,
    pub use_chrome: Option<bool>,
    pub stealth_mode: Option<String>,
    pub viewport_width: Option<u32>,
    pub viewport_height: Option<u32>,
    pub wait_for_idle_network: Option<bool>,
    pub wait_for_delay_ms: Option<u64>,
    pub wait_for_selector: Option<String>,
    pub evaluate_on_new_document: Option<String>,
    pub shared_queue: Option<bool>,
    pub retry: Option<u8>,
}
```
Tokens:
Raw