Crawlberg

https://github.com/xberg-io/crawlberg
Admin
Crawlberg is a high-performance web crawling and scraping engine that extracts structured data,...
Tokens:1,469,883
Snippets:8,784
Trust Score:8
License:MIT
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install dependencies with task setup

Source: https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md

Installs all dependencies and pre-commit hooks via poly. Run after installing Git, Rust stable via rustup, and Task.

```bash
task setup
```

--------------------------------

### Quick Start C example

Source: https://github.com/xberg-io/crawlberg/blob/main/crates/crawlberg-ffi/README.md

Minimal C program that includes the crawlberg header and returns 0. The comment directs to the main repository for usage examples.

```c
#include "crawlberg.h"

int main(void) {
    // See https://github.com/xberg-io/crawlberg for usage examples.
    return 0;
}
```

--------------------------------

### crawlberg serve usage examples

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/cli/usage.md

Two example invocations of the serve command: one that starts the server on the default host and port, and another that binds to 127.0.0.1 on port 8080. The server prints a startup message to stderr and runs until interrupted.

```bash
# Start on default port
crawlberg serve

# Start on custom host and port
crawlberg serve --host 127.0.0.1 --port 8080
```

--------------------------------

### Crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-go.md

Crawls a website starting from url, following links up to the configured depth. The example shows calling Crawl with an empty CrawlEngineHandle and a placeholder URL, with error handling.

```go
func Crawl(engine CrawlEngineHandle, url string) (*CrawlResult, error)
```

```go
result, err := Crawl(CrawlEngineHandle{}, "value")
if err != nil {
    return err
}
```

--------------------------------

### createEngine() example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-java.md

Example call to createEngine with a new CrawlConfig. Returns a CrawlEngineHandle.

```java
var result = createEngine(new CrawlConfig());
```

--------------------------------

### crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-kotlin-android.md

Signature and example for the crawl function. Crawls a website starting from url, following links up to the configured depth. Throws CrawlError.

```kotlin
@Throws(CrawlError::class)
fun crawl(engine: CrawlEngineHandle, url: String): CrawlResult
```

```kotlin
val result = crawl(CrawlEngineHandle(), "value")
```

--------------------------------

### Source library path setup script

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/ci/README.md

Source the library path setup script and initialize all library paths before running a CI script that requires them. This is used for library path setup scripts.

```bash
source ./scripts/lib/library-paths.sh
setup_all_library_paths
./scripts/ci/python/run-tests.sh true
```

--------------------------------

### create_engine example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-rust.md

Example call to create_engine with a Some(CrawlConfig::default()) argument.

```rust
let result = create_engine(Some(CrawlConfig::default()))?;
```

--------------------------------

### create_engine() example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-python.md

Example call to create_engine() with a CrawlConfig instance.

```python
result = create_engine(config=CrawlConfig())
```

--------------------------------

### Start Docker Daemon

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/USAGE.md

Start Docker daemon on macOS or Linux and verify it is running.

```bash
# macOS
open -a Docker

# Linux
sudo systemctl start docker

# Check status
docker ps
```

--------------------------------

### CLI quickstart: scrape, crawl, and map commands

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/features.md

Shows basic usage of the crawlberg CLI for scraping a single page, crawling with depth and page limits, and mapping URLs via sitemap and link extraction. Output formats include json and markdown.

```bash
# Scrape a single page
crawlberg scrape https://example.com

# Crawl with depth limiting
crawlberg crawl https://example.com --depth 2 --max-pages 50 --format markdown

# Discover URLs via sitemap and link extraction
crawlberg map https://example.com --respect-robots-txt
```

--------------------------------

### Install Go package with go get

Source: https://github.com/xberg-io/crawlberg/blob/main/README.md

Install the Go package for Crawlberg. Requires Go and the go command.

```sh
go get github.com/xberg-io/crawlberg/packages/go
```

--------------------------------

### Install flutter_rust_bridge codegen tool

Source: https://github.com/xberg-io/crawlberg/blob/main/packages/dart/BUILDING.md

One-time setup to install the codegen tool required for generating Dart bridge files.

```sh
cargo install flutter_rust_bridge_codegen
```

--------------------------------

### Go: Create Engine and Map URLs with Defaults

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/engine/engine-map-basic.md

Creates a CrawlEngine with default settings using pkg.CreateEngine(nil), then maps URLs from a given URL using pkg.MapUrls. Panics on creation or mapping errors and prints the number of discovered URLs.

```go
package main

import (
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	engine, createErr := pkg.CreateEngine(nil)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.MapUrls(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", len(result.Urls))
}
```

--------------------------------

### Go: Configure and run concurrent crawl with max_pages limit

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/concurrent/concurrent-max-pages-exact.md

Sets up a crawl engine with max_concurrent=5, max_depth=1, and max_pages=3, then crawls example.com and prints the number of pages. The output should be 3, confirming the limit is not exceeded.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"max_concurrent":5,"max_depth":1,"max_pages":3}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Crawl(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", len(result.Pages))
}
```

--------------------------------

### crawl/2 - Signature and Example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-elixir.md

Crawls a website starting from a URL, following links up to the configured depth. The example shows a call with an empty map and a string URL.

```elixir
@spec crawl(engine, url) :: {:ok, term()} | {:error, term()}
def crawl(engine, url)
```

```elixir
{:ok, result} = crawl(%{}, "value")
```

--------------------------------

### YAML Configuration Example

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/USAGE.md

Example configuration in YAML format. The container parses this YAML file correctly.

```yaml
max_depth: 1
max_pages: 5
respect_robots_txt: true
user_agent: "crawlberg-config-test"

content:
  output_format: "markdown"
```

--------------------------------

### crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-dart.md

The signature and example for the crawl function. It crawls a website starting from the URL, following links up to the configured depth. Throws CrawlError on failure.

```dart
Future<CrawlResult> crawl(CrawlEngineHandle engine, String url)
```

```dart
final result = await crawl(CrawlEngineHandle(), "value");
```

--------------------------------

### get_version example response

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/guides/mcp-server.md

Example JSON response from the get_version tool, returning the library version.

```json
{
  "version": "0.3.0"
}
```

--------------------------------

### JSON Configuration Example

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/USAGE.md

Example configuration in JSON format. The container parses this JSON file correctly.

```json
{
  "max_depth": 1,
  "max_pages": 5,
  "respect_robots_txt": true,
  "user_agent": "crawlberg-config-test",
  "content": {
    "output_format": "markdown"
  }
}
```

--------------------------------

### Configure proxy URL and crawl (C)

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/c/proxy/proxy-basic-success.md

Sets the proxy URL to http://127.0.0.1:8888 and disables robots.txt respect, then creates an engine and crawls https://example.com. The result handle is freed if non-null, and the engine handle is freed before returning EXIT_SUCCESS.

```c
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "crawlberg.h"

int main(void) {
    CBERGAlefHandle config_handle = cberg_crawl_config_from_json("{\"proxy\":{\"url\":\"http://127.0.0.1:8888\"},\"respect_robots_txt\":false}");
    CBERGAlefHandle engine = cberg_create_engine(config_handle);
    cberg_crawl_config_free(config_handle);
    char url[2048];
    snprintf(url, sizeof(url), "%s", "https://example.com");
    CBERGAlefHandle result = cberg_crawl(engine, url);
    if (result != 0) cberg_crawl_result_free(result);
    cberg_crawl_engine_handle_free(engine);
    return EXIT_SUCCESS;
}
```

--------------------------------

### crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-zig.md

Crawls a website starting from url, following links up to the configured depth. Throws CrawlError.

```zig
pub fn crawl(engine: ?[]const u8, url: []const u8) CrawlError![]u8
```

```zig
const result = try crawl(.{}, "value");
```

--------------------------------

### crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-swift.md

Crawls a website starting from url, following links up to the configured depth. Throws CrawlError.

```swift
public static func crawl(engine: CrawlEngineHandle, url: String) throws -> CrawlResult
```

```swift
let result = try crawl(CrawlEngineHandle(), "value")
```

--------------------------------

### crawl() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-php.md

Crawls a website starting from `url`, following links up to the configured depth. Throws `CrawlError` on failure.

```php
public static function crawl(CrawlEngineHandle $engine, string $url): CrawlResult
```

```php
$result = crawl(new CrawlEngineHandle(), "value");
```

--------------------------------

### Go: Configure max_asset_size and scrape

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/scrape/scrape-asset-max-size.md

Sets download_assets to true and max_asset_size to 150 in the CrawlConfig, then creates an engine and scrapes example.com. Prints the status code and the number of assets to show that assets exceeding the limit are skipped.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"download_assets":true,"max_asset_size":150}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Scrape(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.StatusCode)
	fmt.Printf("%+v\n", len(result.Assets))
}
```

--------------------------------

### C: Adaptive Strategy Crawl Example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/c/strategy/strategy-adaptive-window.md

Initializes a crawl configuration with the 'best_first' strategy, creates an engine, and crawls a sample URL. The example frees the configuration and result handles to avoid memory leaks.

```c
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "crawlberg.h"

int main(void) {
    CBERGAlefHandle config_handle = cberg_crawl_config_from_json("{\"crawl_strategy\":\"best_first\",\"max_concurrent\":1,\"max_depth\":1,\"respect_robots_txt\":false}");
    CBERGAlefHandle engine = cberg_create_engine(config_handle);
    cberg_crawl_config_free(config_handle);
    char url[2048];
    snprintf(url, sizeof(url), "%s", "https://example.com");
    CBERGAlefHandle result = cberg_crawl(engine, url);
    if (result != 0) cberg_crawl_result_free(result);
    cberg_crawl_engine_handle_free(engine);
    return EXIT_SUCCESS;
}
```

--------------------------------

### Scrape and get word count with @xberg-io/crawlberg-wasm

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/wasm/metadata/metadata-word-count.md

Uses WasmCrawlConfig.default() and WasmSsrfPolicy.default() to configure the engine, then scrape() returns a result with metadata.wordCount. The example logs statusCode and wordCount.

```typescript
import { WasmCrawlConfig, WasmSsrfPolicy, createEngine, scrape } from "@xberg-io/crawlberg-wasm";
async function main() {
  const engineConfig = WasmCrawlConfig.default();
  engineConfig.ssrf = (() => { const _u0 = WasmSsrfPolicy.default(); _u0.denyPrivate = false; return _u0; })();
  const engine = createEngine(engineConfig);
  const url = "https://example.com";
  const result = await scrape(engine, url);
  console.log(result.statusCode);
  console.log(result.metadata.wordCount);
}

void main();
```

--------------------------------

### Start and poll a crawl job via HTTP

Source: https://github.com/xberg-io/crawlberg/blob/main/plugin/skills/serving-the-api/SKILL.md

Starts an asynchronous crawl job and then polls its status. The POST returns a job id immediately; polling the {id} endpoint returns { status, total, completed, data, error }. The example extracts the id with jq and then queries the job status.

```bash
id=$(curl -s http://127.0.0.1:3000/v1/crawl \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","maxDepth":2,"maxPages":100}' | jq -r '.id')

curl -s "http://127.0.0.1:3000/v1/crawl/$id" | jq '{status,completed,total}'
```

--------------------------------

### TOML Configuration Example

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/USAGE.md

Example configuration in TOML format. The container parses this TOML file correctly.

```toml
max_depth = 1
max_pages = 5
respect_robots_txt = true
user_agent = "crawlberg-config-test"

[content]
output_format = "markdown"
```

--------------------------------

### Crawl a Website with CrawlConfig (Rust, Python, TypeScript, CLI)

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/getting-started/quickstart.mdx

Crawl a website starting from a URL with configurable depth and page limits. The Rust, Python, and TypeScript examples use the crawlberg API to create an engine with CrawlConfig and crawl, printing page details. The CLI example shows equivalent commands with flags like --depth and --max-pages.

```rust
use crawlberg::{CrawlConfig, create_engine, crawl};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine = create_engine(Some(CrawlConfig {
        max_depth: Some(2),
        max_pages: Some(50),
        stay_on_domain: true,
        respect_robots_txt: true,
        ..Default::default()
    }))?;

    let result = crawl(&engine, "https://example.com").await?;

    println!("Crawled {} pages (final URL: {})", result.pages.len(), result.final_url);

    for page in &result.pages {
        let title = page.metadata.title.as_deref().unwrap_or("(no title)");
        let md_len = page.markdown.as_ref().map_or(0, |m| m.content.len());
        println!("  [depth={}] {} - {} ({} chars markdown)", page.depth, page.url, title, md_len);
    }

    Ok(())
}

```

```python
import asyncio
from crawlberg import CrawlConfig, create_engine, crawl

async def main() -> None:
    engine = create_engine(CrawlConfig(
        max_depth=2,
        max_pages=50,
        stay_on_domain=True,
        respect_robots_txt=True,
    ))
    result = await crawl(engine, "https://example.com")

    print(f"Crawled {len(result.pages)} pages (final URL: {result.final_url})")

    for page in result.pages:
        title = page.metadata.title or "(no title)"
        md_len = len(page.markdown.content) if page.markdown else 0
        print(f"  [depth={page.depth}] {page.url} - {title} ({md_len} chars markdown)")

asyncio.run(main())

```

```typescript
import { createEngine, crawl } from "@xberg-io/crawlberg";

const engine = createEngine({
    maxDepth: 2,
    maxPages: 50,
    stayOnDomain: true,
    respectRobotsTxt: true,
});

const result = await crawl(engine, "https://example.com");

console.log(`Crawled ${result.pages.length} pages (final URL: ${result.finalUrl})`);

for (const page of result.pages) {
    const title = page.metadata.title ?? "(no title)";
    const mdLen = page.markdown?.content.length ?? 0;
    console.log(`  [depth=${page.depth}] ${page.url} - ${title} (${mdLen} chars markdown)`);
}

```

```bash
# Crawl with depth and page limits
crawlberg crawl https://example.com --depth 2 --max-pages 50 --respect-robots-txt

# Markdown output
crawlberg crawl https://example.com --depth 2 --format markdown

```

--------------------------------

### Configure SSRF allowlist for loopback and scrape

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/c/validation/validation-ssrf-allowlisted-loopback.md

Creates a crawl configuration with an SSRF allowlist for the 127.0.0.0/8 CIDR, deny_private set to true, and max_redirects of 5. Then creates an engine, scrapes example.com, and frees resources. The allowlist permits loopback addresses while private-network protection remains enabled.

```c
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "crawlberg.h"

int main(void) {
    CBERGAlefHandle config_handle = cberg_crawl_config_from_json("{\"respect_robots_txt\":false,\"ssrf\":{\"allowlist\":[{\"type\":\"cidr\",\"value\":\"127.0.0.0/8\"}],\"deny_private\":true,\"max_redirects\":5},\"ssrf_deny_private_explicit\":true}");
    CBERGAlefHandle engine = cberg_create_engine(config_handle);
    cberg_crawl_config_free(config_handle);
    char url[2048];
    snprintf(url, sizeof(url), "%s", "https://example.com");
    CBERGAlefHandle result = cberg_scrape(engine, url);
    if (result != 0) cberg_scrape_result_free(result);
    cberg_crawl_engine_handle_free(engine);
    return EXIT_SUCCESS;
}
```

--------------------------------

### Extract filename from Content-Disposition header in C

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/c/download/download-filename-extraction.md

Initializes a crawl engine with download_documents enabled and robots.txt disabled, then scrapes https://example.com and frees resources. The example demonstrates the basic setup for extracting filenames from Content-Disposition headers.

```c
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "crawlberg.h"

int main(void) {
    CBERGAlefHandle config_handle = cberg_crawl_config_from_json("{\"download_documents\":true,\"respect_robots_txt\":false}");
    CBERGAlefHandle engine = cberg_create_engine(config_handle);
    cberg_crawl_config_free(config_handle);
    char url[2048];
    snprintf(url, sizeof(url), "%s", "https://example.com");
    CBERGAlefHandle result = cberg_scrape(engine, url);
    if (result != 0) cberg_scrape_result_free(result);
    cberg_crawl_engine_handle_free(engine);
    return EXIT_SUCCESS;
}
```

--------------------------------

### Zig crawl with no outgoing links

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/zig/crawl/crawl-empty-page-no-links.md

Calls crawlberg.crawl with a JSON config specifying max_concurrent=1 and max_depth=2, starting at https://example.com. The result is printed to stderr. This example is used to verify that crawling completes when a child page has no outgoing links.

```zig
const std = @import("std");
const crawlberg = @import("crawlberg");

pub fn main() !void {
    const _result_json = try crawlberg.crawl("{\"max_concurrent\":1,\"max_depth\":2}", "https://example.com");
    defer std.heap.c_allocator.free(_result_json);
    std.debug.print("{s}\n", .{_result_json});

}
```

--------------------------------

### Go: Scrape with default preset using pkg.Scrape

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/content/content-default-preset.md

Uses the default preprocessing preset, which keeps the footer that the aggressive preset drops. It creates an engine with a config that disables robots.txt respect, scrapes https://example.com, and prints the Markdown content.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"respect_robots_txt":false}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Scrape(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.Markdown.Content)
}
```

--------------------------------

### Example test output

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/README.md

Sample output from the test script showing configuration info, test results, and summary.

```text
╔════════════════════════════════════════════════════════╗
║ Docker Configuration Volume Mount Test Suite           ║
╚════════════════════════════════════════════════════════╝

[INFO] Configuration:
[INFO]   Variant:         all
[INFO]   Verbose:         false
[INFO]   Keep Containers: false
[INFO]   Port Range:      18100-18199

[INFO] Docker is available

Test 01: Volume mount to /etc/crawlberg/crawlberg.toml (variant: core)
[PASS] Test passed

Test 02: Volume mount to /app/.config/crawlberg/config.toml (variant: core)
[PASS] Test passed

...

╔════════════════════════════════════════════════════════╗
║ Test Summary                                           ║
╚════════════════════════════════════════════════════════╝

Total Tests:   16
Passed Tests:  16
Failed Tests:  0
Pass Rate:     100%

Tested Variants:
  - crawlberg:core
  - crawlberg:full
```

--------------------------------

### Crawl with stay_on_domain=True to ignore external links

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/python/crawl/crawl-external-links-ignored.md

Configure a crawl engine with stay_on_domain=True to discover external links but not follow them. The example prints the number of pages crawled and the stayed_on_domain flag, which indicates whether the crawl remained on the starting domain.

```python
import asyncio
import os
from crawlberg import crawl, create_engine, CrawlConfig

async def main() -> None:
    engine_config = CrawlConfig(max_concurrent=1, max_depth=1, stay_on_domain=True)
    engine = create_engine(engine_config)
    url = "https://example.com"
    result = await crawl(engine, url)
    print(len(result.pages))
    print(result.stayed_on_domain)

asyncio.run(main())
```

--------------------------------

### Swift: Discover sitemap via robots.txt Sitemap directive

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/swift/sitemap/sitemap-from-robots-txt.md

Configure Crawlberg to respect robots.txt and map URLs from a site. The example sets respect_robots_txt to true, creates an engine, and calls mapUrls to get a result, then prints the number of URLs found.

```swift
import Crawlberg

let engineConfig = try crawlConfigFromJson("{\"respect_robots_txt\":true}")
let engineObj = try createEngine(config: engineConfig)
let url = "https://example.com"
let result = try await Crawlberg.mapUrls(engine: engineObj, url: url)
debugPrint(result.urls.count)
```

--------------------------------

### Go: Configure and run concurrent crawl with max_depth

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/crawl/crawl-concurrent-depth.md

Creates a crawl engine with max_concurrent=3 and max_depth=1, then crawls example.com and prints the page count and domain-stay status. Requires the crawlberg Go package and a valid config JSON.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"max_concurrent":3,"max_depth":1,"respect_robots_txt":false}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Crawl(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", len(result.Pages))
	fmt.Printf("%+v\n", result.StayedOnDomain)
}
```

--------------------------------

### Crawl multiple redirects in traversal with Crawlberg

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/dart/crawl/crawl-multiple-redirects-in-traversal.md

Initializes the Rust bridge, creates a crawler with a JSON config limiting concurrency and depth, crawls a starting URL, and prints the number of pages crawled. This example covers handling nested links with redirects during traversal.

```dart
```dart title="Dart"
import 'dart:io';
import 'package:crawlberg/crawlberg.dart';
import 'package:crawlberg/src/crawlberg_bridge_generated/frb_generated.dart' show RustLib;
Future<void> main() async {
  await RustLib.init();
  try {
    final engineConfig = await createCrawlConfigFromJson(json: r'{"max_concurrent":1,"max_depth":2}');
    final engine = await CrawlbergBridge.createEngine(config: engineConfig);
    final url = 'https://example.com';
    final result = await CrawlbergBridge.crawl(engine, url);
    stdout.writeln(result.pages.length);
  } finally {
    RustLib.dispose();
  }
}
```
```

--------------------------------

### Click a button element with pkg.Interact

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/interaction/interact-click-element.md

Creates a CrawlConfig with browser mode "always", builds an engine, and sends a click action on selector "#submit" to https://example.com. Prints the success flag, action type, and action index from the first ActionResult. Requires the crawlberg Go package.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"browser":{"mode":"always"}}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	var actionsRaw []json.RawMessage
	if err := json.Unmarshal([]byte(`[{"selector":"#submit","type":"click"}]`), &actionsRaw); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	var actions []pkg.PageAction
	for _, raw := range actionsRaw {
		elem, err := pkg.UnmarshalPageAction(raw)
		if err != nil {
			panic(fmt.Sprintf("unmarshal PageAction failed: %v", err))
		}
		actions = append(actions, elem)
	}
	result, err := pkg.Interact(engine, url, actions)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.ActionResults[0].Success)
	fmt.Printf("%+v\n", result.ActionResults[0].ActionType)
	fmt.Printf("%+v\n", result.ActionResults[0].ActionIndex)
}
```

--------------------------------

### Scrape URL with Crawlberg in Kotlin (Android)

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/kotlin-android/markdown/citations-balanced-parens.md

Use this snippet to scrape a web page with Crawlberg in a Kotlin (Android) environment. It creates an engine, scrapes the given URL, and prints the HTTP status code and markdown content. The example is safe to run and requires no additional setup.

```kotlin
import io.xberg.crawlberg.android.*

fun main() = kotlinx.coroutines.runBlocking {
    val engine = Crawlberg.createEngine(null)
    val url = "https://example.com"
    val result = Crawlberg.scrape(engine, url)
    println(result.statusCode)
    println(result.markdown?.content)
}
```

--------------------------------

### create_engine() example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-ruby.md

Example call to create_engine with a CrawlConfig instance. Returns a CrawlEngineHandle. Raises CrawlError on invalid configuration.

```ruby
result = create_engine(config: CrawlConfig.new)

```

--------------------------------

### BM25 Filter with Zero Threshold Passes All Pages (C)

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/c/filter/filter-bm25-threshold-zero.md

Configures a Crawlberg engine with a BM25 content filter and a zero threshold, then crawls a URL. The zero threshold causes all pages to pass the filter. The example includes necessary includes, configuration setup, engine creation, crawling, and cleanup.

```c
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "crawlberg.h"

int main(void) {
    CBERGAlefHandle config_handle = cberg_crawl_config_from_json("{\"bm25_query\":\"unrelated query\",\"bm25_threshold\":0.0,\"content_filter\":\"bm25\",\"max_depth\":1}");
    CBERGAlefHandle engine = cberg_create_engine(config_handle);
    cberg_crawl_config_free(config_handle);
    char url[2048];
    snprintf(url, sizeof(url), "%s", "https://example.com");
    CBERGAlefHandle result = cberg_crawl(engine, url);
    if (result != 0) cberg_crawl_result_free(result);
    cberg_crawl_engine_handle_free(engine);
    return EXIT_SUCCESS;
}
```

--------------------------------

### Scrape URL with crawlberg Go package

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/markdown/citations-balanced-parens.md

Creates a crawlberg engine, scrapes the example.com URL, and prints the status code and markdown content. The example verifies that citations handle links inside parentheses with balanced rendering. Panics on engine creation or scrape errors.

```go
package main

import (
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	engine, createErr := pkg.CreateEngine(nil)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Scrape(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.StatusCode)
	fmt.Printf("%+v\n", result.Markdown.Content)
}

```

--------------------------------

### Include subdomains in URL map discovery with Crawlberg

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/java/map/map-include-subdomains.md

Configures a Crawlberg engine with allow_subdomains set to true and max_depth 0, then maps URLs from a starting page. The example prints the number of discovered URLs, which includes subdomain links. Use when you need to include subdomains in URL map discovery.

```java
import io.xberg.crawlberg.*;
import com.fasterxml.jackson.databind.ObjectMapper;

public final class Example {
    public static void main(String[] args) throws Exception {
        var MAPPER = new ObjectMapper();
        var engineConfig = MAPPER.readValue("{\"allow_subdomains\":true,\"max_depth\":0,\"respect_robots_txt\":false}", CrawlConfig.class);
        var engine = io.xberg.crawlberg.Crawlberg.createEngine(engineConfig);
        String url = "https://example.com";
        var result = Crawlberg.mapUrls(engine, url);
        System.out.println(result.urls().size());
    }
}
```

--------------------------------

### Configure and run DFS crawl in C#

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/csharp/strategy/strategy-dfs-depth-first.md

Sets up a CrawlConfig with crawl_strategy 'dfs', max_concurrent 1, and max_depth 2, then creates an engine and crawls 'https://example.com'. Outputs the page count and the URLs of pages at indices 1 through 4.

```csharp
using System;
using System.Text.Json;
using Crawlberg;

var ConfigOptions = new JsonSerializerOptions { PropertyNameCaseInsensitive = true };
var engineConfig = JsonSerializer.Deserialize<CrawlConfig>("{\"crawl_strategy\":\"dfs\",\"max_concurrent\":1,\"max_depth\":2}", ConfigOptions)!;
var engine = CrawlbergConverter.CreateEngine(engineConfig);
var url = "https://example.com";
var result = await CrawlbergConverter.CrawlAsync(engine, url);
Console.WriteLine(result.Pages.Count);
Console.WriteLine(result.Pages[1].Url);
Console.WriteLine(result.Pages[2].Url);
Console.WriteLine(result.Pages[3].Url);
Console.WriteLine(result.Pages[4].Url);
```

--------------------------------

### createEngine() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-kotlin-android.md

Signature and example for the createEngine function. If config is null, uses CrawlConfig.default(). Returns an error if the configuration is invalid. Throws CrawlError.

```kotlin
@Throws(CrawlError::class)
fun createEngine(config: CrawlConfig? = null): CrawlEngineHandle
```

```kotlin
val result = createEngine(CrawlConfig())
```

--------------------------------

### Install CLI via cargo install

Source: https://github.com/xberg-io/crawlberg/blob/main/readme_templates/root.md

Install the Crawlberg CLI using cargo install.

```sh
cargo install crawlberg-cli
```

--------------------------------

### Install Crawlberg Ruby Gem via Bundler or gem install

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Install the Crawlberg Ruby gem. Add the gem to your Gemfile and run bundle install, or install directly with gem install. Requires Ruby 3.2+. The gem includes a native extension built with Magnus and rb_sys.

```ruby
gem "crawlberg", "~> 0.3"
```

```bash
bundle install
```

```bash
gem install crawlberg
```

--------------------------------

### createEngine() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-dart.md

The signature and example for the createEngine function. If config is null, it uses CrawlConfig.default(). Throws CrawlError if the configuration is invalid.

```dart
Future<CrawlEngineHandle> createEngine({CrawlConfig? config})
```

```dart
final result = createEngine(CrawlConfig());
```

--------------------------------

### generateCitations() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-php.md

Converts markdown links to numbered citations. Example: `[Example](https://example.com)` becomes `Example[1]` with `[1]: <https://example.com>` in the reference list. Images `!alt` are preserved unchanged.

```php
public static function generateCitations(string $markdown): CitationResult
```

```php
$result = generateCitations("value");
```

--------------------------------

### CreateEngine() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-go.md

Creates a new crawl engine with the given configuration. If config is nil, uses CrawlConfig.default(). Returns an error if the configuration is invalid. The example shows error handling.

```go
func CreateEngine(config CrawlConfig) (*CrawlEngineHandle, error)
```

```go
result, err := CreateEngine(CrawlConfig{})
if err != nil {
    return err
}
```

--------------------------------

### Install Crawlberg plugin in Gemini CLI

Source: https://github.com/xberg-io/crawlberg/blob/main/README.md

Install the Crawlberg plugin for Gemini CLI using the gemini extensions install command.

```text
gemini extensions install https://github.com/xberg-io/crawlberg
```

--------------------------------

### Press a keyboard key (Enter) and verify success in Go

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/interaction/interact-press-key.md

Uses the crawlberg Go package to create an engine with browser mode 'always', then performs a click, type, and press (Enter) action sequence on example.com. Prints the success, action type, and action index of the third action result.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"browser":{"mode":"always"}}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	var actionsRaw []json.RawMessage
	if err := json.Unmarshal([]byte(`[{"selector":"#searchbox","type":"click"},{"selector":"#searchbox","text":"test","type":"type"},{"key":"enter","type":"press"}]`), &actionsRaw); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	var actions []pkg.PageAction
	for _, raw := range actionsRaw {
		elem, err := pkg.UnmarshalPageAction(raw)
		if err != nil {
			panic(fmt.Sprintf("unmarshal PageAction failed: %v", err))
		}
		actions = append(actions, elem)
	}
	result, err := pkg.Interact(engine, url, actions)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.ActionResults[2].Success)
	fmt.Printf("%+v\n", result.ActionResults[2].ActionType)
	fmt.Printf("%+v\n", result.ActionResults[2].ActionIndex)
}
```

--------------------------------

### Install Crawlberg plugin in Factory Droid

Source: https://github.com/xberg-io/crawlberg/blob/main/README.md

Install the Crawlberg plugin for Factory Droid. First add the plugin marketplace, then install the crawlberg plugin.

```text
droid plugin marketplace add https://github.com/xberg-io/crawlberg
droid plugin install crawlberg@crawlberg
```

--------------------------------

### generateCitations() signature and example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-kotlin-android.md

Signature and example for the generateCitations function. Converts markdown links to numbered citations, e.g., [Example](https://example.com) becomes Example[1] with [1]: <https://example.com> in the reference list. Images !alt are preserved unchanged.

```kotlin
fun generateCitations(markdown: String): CitationResult
```

```kotlin
val result = generateCitations("value")
```

--------------------------------

### Install @xberg-io/crawlberg-wasm

Source: https://github.com/xberg-io/crawlberg/blob/main/crates/crawlberg-wasm/README.md

Installs the WASM package for the Crawlberg crawler.

```bash
pnpm add @xberg-io/crawlberg-wasm
```

--------------------------------

### Go: Rotate user agents with CrawlConfig and pkg.Scrape

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/stealth/stealth-ua-rotation-config.md

Configures a CrawlConfig with a user_agents array and creates an engine to scrape example.com, printing the HTTP status code. The JSON config must parse successfully and the engine creation must succeed; otherwise the program panics with a descriptive message.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"user_agents":["Mozilla/5.0 (Windows NT 10.0)","Chrome/120.0.0.0"]}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Scrape(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", result.StatusCode)
}
```

--------------------------------

### generate_citations example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-rust.md

Example call to generate_citations with a string literal.

```rust
let result = generate_citations("value");
```

--------------------------------

### cberg_generate_citations() example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-c.md

Example call to cberg_generate_citations() with a markdown string.

```c
CBERGAlefHandle result = cberg_generate_citations("value");
```

--------------------------------

### Get help for test-docker-config-local.sh

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/test/USAGE.md

Displays help information for the test-docker-config-local.sh script.

```bash
./scripts/test/test-docker-config-local.sh --help
```

--------------------------------

### Before: Free up disk space (inline commands)

Source: https://github.com/xberg-io/crawlberg/blob/main/scripts/ci/README.md

Shows the original inline commands for freeing up disk space in a CI workflow, including removing unnecessary packages. This is the 'before' example for the script replacement.

```yaml
- name: Free up disk space
  run: |
    echo "=== Initial disk space ==="
    df -h /
    echo "=== Removing unnecessary packages ==="
    sudo rm -rf /usr/share/dotnet
    # ... 30+ lines of commands ...
```

--------------------------------

### generate_citations() example

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/content/docs/reference/api-python.md

Example call to generate_citations() with a markdown string.

```python
result = generate_citations("value")
```

--------------------------------

### Go: Configure user-agent rotation with pkg.CreateEngine and pkg.Crawl

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/go/stealth/stealth-ua-rotation-round-robin.md

Configures a crawl engine with two user agents and a max depth of 1 and max pages of 3, then crawls a single URL and prints the number of pages in the result. The user-agent rotation cycles through the provided agents across requests.

```go
package main

import (
	"encoding/json"
	"fmt"
	pkg "github.com/xberg-io/crawlberg/packages/go"
)

func main() {
	var engineConfig pkg.CrawlConfig
	if err := json.Unmarshal([]byte(`{"max_depth":1,"max_pages":3,"user_agents":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) TestAgent-1","Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) TestAgent-2"]}`), &engineConfig); err != nil {
		panic(fmt.Sprintf("config parse failed: %v", err))
	}
	engine, createErr := pkg.CreateEngine(&engineConfig)
	if createErr != nil {
		panic(fmt.Sprintf("create handle failed: %v", createErr))
	}
	url := `https://example.com`
	result, err := pkg.Crawl(engine, url)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%+v\n", len(result.Pages))
}
```

--------------------------------

### Install Crawlberg plugin in GitHub Copilot CLI

Source: https://github.com/xberg-io/crawlberg/blob/main/README.md

Install the Crawlberg plugin for GitHub Copilot CLI. First add the plugin marketplace, then install the crawlberg plugin.

```text
copilot plugin marketplace add https://github.com/xberg-io/crawlberg
copilot plugin install crawlberg@crawlberg
```

--------------------------------

### Install @xberg-io/crawlberg with pnpm

Source: https://github.com/xberg-io/crawlberg/blob/main/crates/crawlberg-node/README.md

Install the @xberg-io/crawlberg package using pnpm.

```bash
pnpm add @xberg-io/crawlberg
```

--------------------------------

### Create default CrawlEngine and map URLs

Source: https://github.com/xberg-io/crawlberg/blob/main/docs-site/src/snippets/generated/ruby/engine/engine-map-basic.md

Creates a CrawlEngine with default settings by passing nil to Crawlberg.create_engine, then maps URLs from a single example.com URL using Crawlberg.map_urls. Prints the number of discovered URLs with inspect.

```ruby
require "crawlberg"
engine = Crawlberg.create_engine(nil)
url = 'https://example.com'
result = Crawlberg.map_urls(engine, url)
puts result.urls.length.inspect
```
Tokens:
Raw