Gottem

https://github.com/spider-rs/gottem
Admin
Universal scraper that reliably gets data by talking to every major scraping vendor and local...
Tokens:9,608
Snippets:113
Trust Score:7.8
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install gottem-cli

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Installs the gottem binary to your system's PATH.

```bash
cargo install gottem-cli
```

--------------------------------

### Install Gottem CLI from Source

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Installs the gottem CLI binary by building it from the source code. Use this for unreleased changes or development.

```bash
git clone https://github.com/spider-rs/gottem.git
cd gottem
cargo install --path crates/gottem-cli
```

--------------------------------

### Install gottem CLI

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Installs the gottem command-line binary using Cargo.

```sh
cargo install gottem-cli      # provides the `gottem` binary
```

--------------------------------

### Gottem Route Configuration Example

Source: https://github.com/spider-rs/gottem/blob/main/llms.txt

Example TOML configuration for a 'firecrawl.scrape' route using the 'http_json' adapter. Specifies endpoint, tier, cost, authentication, request body template, and response parsing.

```toml
[[route]]
id          = "firecrawl.scrape"
adapter     = "http_json"
endpoint    = "https://api.firecrawl.dev/v1/scrape"
method      = "POST"
tier        = 4
cost        = 10                # milli-cents
timeout_ms  = 30000

[route.auth]
kind = "bearer"
env  = "FIRECRAWL_API_KEY"

[route.body]
kind     = "json"
template = '''{"url":"{{url}}","formats":["markdown"]}'''

[route.parse]
kind = "json_path"
path = "$.data.markdown"

[[route.validate]]
kind = "min_bytes"
n    = 500
```

--------------------------------

### Basic Crawl Command

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Initiates a web crawl starting from a given URL with specified parameters like wait time and browser mode.

```bash
gottem crawl https://example.com --param waitFor=2000 --param mode=chrome
```

--------------------------------

### Gottem CLI Route Inspection Commands

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Provides examples of using the gottem CLI to list, show details for, and validate loaded routes.

```bash
gottem routes list           # tabular view of every loaded route
gottem routes show <id>      # full detail for one route
gottem routes validate       # check that every route's env var is set
```

--------------------------------

### Example TOML Route Configuration

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-adapters-browseruse/README.md

This TOML snippet shows how to configure a route's body template for the Browser Use Cloud adapter. It defines a JSON body with a task to visit a URL and return content as markdown, and enables proxy usage.

```toml
```toml
[route.body]
kind     = "json"
template = '''{"task":"Visit {{url}} and return the main content as markdown.","use_proxy":true}'''
```
```

--------------------------------

### Hedge fetching with a delay

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Initiates a scrape using a low-tier route and starts a staggered backup after a specified delay to ensure content retrieval.

```bash
gottem fetch https://example.com --mode hedge --hedge-delay-ms 2000
```

--------------------------------

### Register ChromeCdpAdapter

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-adapters-chrome/README.md

Demonstrates how to register the ChromeCdpAdapter with the AdapterRegistry. This is typically done at the start of an application that utilizes Gottem's Chrome capabilities.

```rust
use gottem_core::AdapterRegistry;
use gottem_adapters_chrome::ChromeCdpAdapter;

let mut registry = AdapterRegistry::new();
registry.register(ChromeCdpAdapter::arc());

```

--------------------------------

### Gottem Hosted API - Compare Endpoint

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Example using curl to send a POST request to the hosted gottem.dev API's /v1/compare endpoint for route comparison.

```bash
curl -X POST https://api.gottem.dev/v1/compare \
  -H "Authorization: Bearer gtm_your_key_here" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com"}'
```

--------------------------------

### Gottem Hosted API - Scrape Endpoint

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Example using curl to send a POST request to the hosted gottem.dev API's scrape endpoint.

```bash
curl -X POST https://api.gottem.dev/scrape \
  -H "Authorization: Bearer gtm_your_key_here" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com"}'
```

--------------------------------

### Gottem Library Crawl Builder

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Demonstrates using the gottem_core library to build and run a crawl with custom page processing logic. Requires crawl adapters to be installed.

```rust
use std::sync::Arc;
use gottem_core::{CancelToken, ControlFlow, CrawlRequest, Orchestrator};
use url::Url;

let orch: Arc<Orchestrator> = /* built with crawl adapters installed */;
orch.crawl_builder(
        CrawlRequest::new(Url::parse("https://example.com")?)
            .with_limit(50)
            .with_depth(2),
    )
    .on_page(|page| async move {
        save_to_db(page).await;
        ControlFlow::Continue
    })
    .run(CancelToken::new())
    .await?;
```

--------------------------------

### Firecrawl Route Configuration

Source: https://github.com/spider-rs/gottem/blob/main/README.md

An example TOML configuration for the Firecrawl route, specifying the adapter, endpoint, authentication, request body, and parsing rules.

```toml
[[route]]
id          = "firecrawl.scrape"
adapter     = "http_json"
endpoint    = "https://api.firecrawl.dev/v1/scrape"
method      = "POST"
tier        = 4
cost        = 10
timeout_ms  = 30000

[route.auth]
kind = "bearer"
env  = "FIRECRAWL_API_KEY"

[route.body]
kind     = "json"
template = '''{"url":"{{url}}","formats":["markdown"]}'''

[route.parse]
kind = "json_path"
path = "$.data.markdown"

[[route.validate]]
kind = "min_bytes"
n    = 500
```

--------------------------------

### Crawl URL with custom parameters

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Initiates a crawl from a given URL, streaming results as NDJSON. This example shows how to pass custom parameters to the scraping routes.

```sh
gottem crawl <URL> [--depth N] [--limit M] [--engine auto|spider-cloud|local]
                   [--subdomains] [--tld]
                   [--allow PAT --allow PAT ...]
                   [--deny PAT --deny PAT ...]
                   [--respect-robots]
                   [--concurrency N]
                   [--param k=v --param k=v ...]
```

--------------------------------

### Local BFS Crawl

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Initiates a Breadth-First Search crawl starting from a given URL, using the local scrape ladder for each page. Configurable with depth and limit.

```bash
gottem crawl https://example.com --depth 2 --limit 50
```

--------------------------------

### Initialize and Run Gottem Scraper

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-core/README.md

Demonstrates setting up the RouteCatalog, AdapterRegistry, and Orchestrator to perform a web scrape. Includes basic budget and cancellation handling.

```rust
use std::sync::Arc;
use gottem_core::{
    AdapterRegistry,
    Budget,
    CancelToken,
    Capabilities,
    LadderStrategy,
    Orchestrator,
    RouteCatalogBuilder,
    ScrapeRequest,
    Tier,
};
use url::Url;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let catalog = Arc::new(
        gottem_routes_builtin::register_all(RouteCatalogBuilder::new())?.build()
    );

    let mut registry = AdapterRegistry::new();
    gottem_adapters_http::register_all(&mut registry, None);
    registry.register(gottem_adapters_spider::SpiderAdapter::arc());

    let orch = Orchestrator::new(
        catalog.clone(),
        Arc::new(registry),
        Arc::new(Budget::new(1_000)), // $0.10 ceiling, in milli-cents
    );

    let strategy = Arc::new(LadderStrategy::new(
        catalog.clone(), Tier::T0, Tier::T9, Capabilities::default(), 5,
    ));

    let resp = orch.fetch(
        ScrapeRequest::get(Url::parse("https://example.com")?),
        strategy,
        CancelToken::new(),
    ).await?;

    println!("{}", resp.content.unwrap_or_default());
    Ok(())
}

```

--------------------------------

### Fetch content with meta information

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content from a URL using the lowest-cost route first, displaying metadata. Requires setting vendor API keys.

```bash
export FIRECRAWL_API_KEY=fc-...
gottem fetch https://example.com --show-meta
```

--------------------------------

### Drive Hosted Gottem API from CLI

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Run fetches on the hosted Gottem API by setting the API key and using the --remote flag. Standard CLI flags are supported.

```sh
export GOTTEM_API_KEY=gtm_...
gottem fetch --remote <URL>                 # runs on api.gottem.dev
gottem fetch --remote --mode race <URL>     # the usual flags carry over
```

--------------------------------

### Using Gottem as a Library in Rust

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Demonstrates how to initialize and use the Gottem orchestrator within a Rust application to fetch content from a URL.

```rust
use std::sync::Arc;
use gottem_core::{Budget, CancelToken, LadderStrategy, Orchestrator,
                  RouteCatalogBuilder, ScrapeRequest, Tier, AdapterRegistry, Capabilities};
use url::Url;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let catalog = Arc::new(
        gottem_routes_builtin::register_all(RouteCatalogBuilder::new())?.build()
    );

    let mut registry = AdapterRegistry::new();
    gottem_adapters_http::register_all(&mut registry, None);
    registry.register(gottem_adapters_spider::SpiderAdapter::arc());

    let orch = Arc::new(Orchestrator::new(
        catalog.clone(),
        Arc::new(registry),
        Arc::new(Budget::new(1_000)),  // $0.10 ceiling
    ));

    let strategy = Arc::new(LadderStrategy::new(
        catalog.clone(), Tier::T0, Tier::T9, Capabilities::default(), 5,
    ));

    let resp = orch.fetch(
        ScrapeRequest::get(Url::parse("https://example.com")?),
        strategy,
        CancelToken::new(),
    ).await?;

    println!("{}", resp.content.unwrap_or_default());
    Ok(())
}
```

--------------------------------

### Rust library: Crawl with page callback

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Demonstrates using the gottem core library to build a crawler. It sets up a callback function to process each fetched page and continues the crawl.

```rust
use std::sync::Arc;
use gottem_core::{CancelToken, ControlFlow, CrawlRequest, Orchestrator};
use url::Url;

let orch: Arc<Orchestrator> = /* built with crawl adapters installed */;
orch.crawl_builder(
        CrawlRequest::new(Url::parse("https://example.com")?)
            .with_limit(50)
            .with_depth(2),
    )
    .on_page(|page| async move {
        save(page).await;
        ControlFlow::Continue
    })
    .run(CancelToken::new())
    .await?;
```

--------------------------------

### Fetch URL with metadata

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Fetches a URL and displays metadata about the route and cost.

```sh
gottem fetch https://example.com --show-meta
```

--------------------------------

### Registering SpiderAdapter

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-adapters-spider/README.md

Demonstrates how to register the SpiderAdapter with the AdapterRegistry. You can either register a default adapter or one pre-configured with a custom spider::Client.

```rust
use gottem_core::AdapterRegistry;
use gottem_adapters_spider::SpiderAdapter;

let mut registry = AdapterRegistry::new();
registry.register(SpiderAdapter::arc());
// or plug in a pre-configured spider::Client (proxies, custom UA):
// registry.register(SpiderAdapter::arc_with_client(my_client));
```

--------------------------------

### Fetch with custom routes configuration

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content from a URL using a custom routing configuration defined in a TOML file.

```bash
gottem --config routes.toml fetch https://example.com
```

--------------------------------

### Fetch remotely with race mode and meta

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content remotely using the 'race' mode and displays metadata. Leverages the hosted gottem API.

```bash
gottem fetch --remote --mode race --show-meta https://example.com
```

--------------------------------

### Gottem Fetch Modes

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Illustrates various command-line modes for the gottem fetch command, including parallel racing, staggered backups, budget capping, tier filtering, JavaScript rendering, and structured output.

```bash
gottem fetch URL                                  # ladder, default
```

```bash
gottem fetch URL --mode race --routes a,b,c       # race A B C in parallel
```

```bash
gottem fetch URL --mode hedge --hedge-count 2     # primary + 2 staggered backups
```

```bash
gottem fetch URL --budget-mc 100                  # cap at $0.01 per fetch
```

```bash
gottem fetch URL --tier-min 4 --tier-max 7        # skip local; cap below T8
```

```bash
gottem fetch URL --require-js                     # only routes that render JS
```

```bash
gottem fetch URL --format json                    # structured output with metadata
```

--------------------------------

### Gottem CLI Fetch Command

Source: https://github.com/spider-rs/gottem/blob/main/llms.txt

Use the 'fetch' command to retrieve content from a URL. Supports various modes like ladder, race, and hedge, with options for budget, tier limits, JavaScript rendering, route selection, output format, and metadata display.

```sh
gottem fetch URL [--mode ladder|race|hedge] [--budget-mc N] [--tier-min N --tier-max M] [--require-js] [--routes a,b,c] [--format content|json] [--show-meta]
gottem --config routes.toml fetch URL    # layer custom routes on top of builtin
```

--------------------------------

### Fetch remotely with API key

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content from a URL using the hosted gottem API. Requires setting the GOTTEM_API_KEY environment variable.

```bash
export GOTTEM_API_KEY=gtm_your_key_here
gottem fetch --remote https://example.com
```

--------------------------------

### Register All Built-in Routes

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-routes-builtin/README.md

Demonstrates how to register all available built-in vendor routes into a RouteCatalogBuilder. This is the primary way to include all routes provided by the crate.

```rust
use gottem_core::RouteCatalogBuilder;
use gottem_routes_builtin::register_all;

let catalog = register_all(RouteCatalogBuilder::new())
    .expect("builtin routes load")
    .build();
// Spider (4 routes) + Firecrawl (2) are enabled by default.
```

--------------------------------

### Show details for a specific route

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

View full details for a particular scraping route, such as 'spider.smart'.

```bash
gottem routes show spider.smart
```

--------------------------------

### List available routes

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Inspect available scraping routes without needing API keys.

```bash
gottem routes list
```

--------------------------------

### Gottem CLI Remote Fetch Command

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Shows how to use the gottem CLI to perform fetches against the hosted gottem.dev API using a remote key.

```bash
export GOTTEM_API_KEY=gtm_your_key_here

# Runs on api.gottem.dev — no local vendor keys, no browser.
gottem fetch --remote https://example.com

# All the usual flags carry over to the hosted run.
gottem fetch --remote --mode race --show-meta https://example.com
gottem fetch --remote --format json https://example.com

# Or pass the key explicitly instead of the env var.
gottem fetch --remote --api-key gtm_your_key_here https://example.com
```

--------------------------------

### Gottem CLI Routes Commands

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Commands to list, show, and validate vendor catalog routes. Use 'gottem --config routes.toml fetch URL' to layer custom routes.

```sh
gottem routes list                    # tabular catalog (27 builtin routes, 13 vendors + local crawl)
gottem routes show <route-id>          # full detail for one route
gottem routes validate                # check env vars are set for each route's auth
gottem --config routes.toml fetch URL  # layer custom vendor routes on top of builtin
```

--------------------------------

### Gottem CLI Probe Command

Source: https://github.com/spider-rs/gottem/blob/main/llms.txt

Use the 'probe' command to check a URL's accessibility and basic properties. Allows specifying tier limits and minimum byte size for the response.

```sh
gottem probe URL [--tier-min N --tier-max M] [--min-bytes 500]
```

--------------------------------

### Gottem CLI Routes Commands

Source: https://github.com/spider-rs/gottem/blob/main/llms.txt

Manage routes using the 'routes' subcommand. List available routes, show details for a specific route, or validate environment variables required for routes.

```sh
gottem routes list                       # tabular catalog
gottem routes show <route-id>            # full route detail
gottem routes validate                   # check env vars per route's auth
```

--------------------------------

### Registering HTTP Adapters

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-adapters-http/README.md

Registers all available HTTP adapters with the AdapterRegistry. You can provide a custom reqwest::Client or use the default.

```rust
use std::sync::Arc;
use gottem_core::AdapterRegistry;
use gottem_adapters_http::register_all;

let mut reg = AdapterRegistry::new();
register_all(&mut reg, None); // None = default client; Some(client) to plug your own
let reg = Arc::new(reg);
```

--------------------------------

### Fetch URL using race mode with specific routes

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Fetches a URL using 'race' mode, which tries specified routes in parallel and returns the first successful result.

```sh
gottem fetch https://x.com --mode race --routes spider.smart,firecrawl.scrape
```

--------------------------------

### Gottem Library Raw Stream Processing

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Shows how to consume crawl results as a raw stream using the gottem_core library. Pages are processed and dropped as they flow through, preventing memory buildup.

```rust
let mut stream = orch.crawl(req, CancelToken::new()).await?;
while let Some(page) = futures_util::StreamExt::next(&mut stream).await {
    /* ... */
}
```

--------------------------------

### Advanced Crawl Configuration

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Configures crawling with subdomain discovery, specific allow/deny patterns for URLs, respect for robots.txt, and control over concurrency.

```bash
gottem crawl https://example.com \
    --subdomains \
    --allow /blog --allow /docs \
    --deny /admin --deny '\.pdf$' \
    --respect-robots \
    --concurrency 8
```

--------------------------------

### Fetch URL with JavaScript rendering and JSON format

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

Fetches a URL, requiring JavaScript rendering, and outputs the result in JSON format.

```sh
gottem fetch https://hard-site.com --require-js --format json
```

--------------------------------

### Race routes for fastest response

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content by racing specified routes in parallel, returning the fastest valid response. Useful for performance-critical scraping.

```bash
gottem fetch https://example.com --mode race --routes firecrawl.scrape,spider.http,zenrows.basic
```

--------------------------------

### Cap cost and pin tier band

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content while adhering to a maximum cost budget and restricting the scraping to a specific tier range.

```bash
gottem fetch https://example.com --budget-mc 100 --tier-min 4 --tier-max 7
```

--------------------------------

### Set Vendor API Keys

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Sets environment variables for API keys required by scraping vendors like Firecrawl and Spider. Ensure you replace 'fc-...' and 'sk-...' with your actual keys.

```bash
export FIRECRAWL_API_KEY=fc-...
export SPIDER_API_KEY=sk-...
```

--------------------------------

### Fetch remotely with explicit API key

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content remotely by passing the API key directly as an argument, bypassing environment variables.

```bash
gottem fetch --remote --api-key gtm_your_key_here https://example.com
```

--------------------------------

### Probe tiers on a target URL

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Probes every available tier on a given URL to determine which ones yield content, useful for identifying scraping challenges.

```bash
gottem probe https://hard-to-scrape.test
```

--------------------------------

### Fetch remotely with JSON format

Source: https://github.com/spider-rs/gottem/blob/main/crates/gottem-cli/README.md

Fetches content remotely and formats the output as JSON. Useful for programmatic consumption of scraped data.

```bash
gottem fetch --remote --format json https://example.com
```

--------------------------------

### Spider Cloud Endpoint Crawl

Source: https://github.com/spider-rs/gottem/blob/main/README.md

Forces the crawl to use the Spider /crawl endpoint, enabling native JSONL streaming. Supports depth and limit configuration.

```bash
gottem crawl https://example.com --engine spider-cloud --depth 3 --limit 100
```

--------------------------------

### Gottem Hosted API Endpoints

Source: https://github.com/spider-rs/gottem/blob/main/SKILL.md

The hosted Gottem API provides endpoints for scraping URLs, probing reachability, and comparing content quality across multiple providers.

```APIDOC
## POST /scrape

### Description
Fetches a URL using Gottem's routing engine. This is the hosted equivalent of the `gottem fetch` command.

### Method
POST

### Endpoint
https://api.gottem.dev/scrape

### Request Body
- **url** (string) - Required - The URL to fetch.
- **options** (object) - Optional - Additional options for the scrape request.
  - **mode** (string) - Optional - The scraping mode (e.g., 'race').

### Request Example
```json
{
  "url": "https://example.com",
  "options": {
    "mode": "race"
  }
}
```

### Response
#### Success Response (200)
- **provider** (string) - The provider used for the fetch.
- **content** (string) - The fetched content.
- **cost** (number) - The cost of the fetch.
- **sha256** (string) - The SHA-256 hash of the content.

#### Response Example
```json
{
  "provider": "some_provider",
  "content": "<html>...</html>",
  "cost": 0.001,
  "sha256": "a1b2c3d4e5f6..."
}
```

## POST /probe

### Description
Checks the reachability of a URL.

### Method
POST

### Endpoint
https://api.gottem.dev/probe

### Request Body
- **url** (string) - Required - The URL to probe.

### Request Example
```json
{
  "url": "https://example.com"
}
```

### Response
#### Success Response (200)
- **reachable** (boolean) - Indicates if the URL is reachable.

#### Response Example
```json
{
  "reachable": true
}
```

## POST /v1/compare

### Description
Runs a URL through several providers simultaneously and compares content quality, cost, and SHA-256 hashes of the results. Useful for quality validation.

### Method
POST

### Endpoint
https://api.gottem.dev/v1/compare

### Request Body
- **url** (string) - Required - The URL to compare.

### Request Example
```json
{
  "url": "https://example.com"
}
```

### Response
#### Success Response (200)
- **results** (array) - An array of comparison results for each provider.
  - **provider** (string) - The provider name.
  - **content** (string) - The fetched content.
  - **cost** (number) - The cost of the fetch.
  - **sha256** (string) - The SHA-256 hash of the content.
  - **quality** (number) - A score indicating content quality.

#### Response Example
```json
{
  "results": [
    {
      "provider": "provider_a",
      "content": "<html>...</html>",
      "cost": 0.001,
      "sha256": "hash_a",
      "quality": 0.9
    },
    {
      "provider": "provider_b",
      "content": "<html>...</html>",
      "cost": 0.0015,
      "sha256": "hash_b",
      "quality": 0.85
    }
  ]
}
```
```
Tokens:
Raw