### Basic Crawl Setup and Execution
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/crawling
Initializes the CrawlEngine with basic configuration and performs a crawl starting from a seed URL. It then iterates through the results to print information about each crawled page.
```rust
use kreuzcrawl::{CrawlEngine, CrawlConfig};
let engine = CrawlEngine::builder()
.config(CrawlConfig {
max_depth: Some(2),
max_pages: Some(50),
..Default::default()
})
.build()?;
let result = engine.crawl("https://example.com").await?;
for page in &result.pages {
println!("{} (depth {}): {} bytes", page.url, page.depth, page.body_size);
}
```
--------------------------------
### Version Endpoint Response
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Example JSON response from the `GET /version` endpoint.
```json
{ "version": "0.1.0-rc.1" }
```
--------------------------------
### Start API Server via CLI
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Use the `kreuzcrawl serve` command to start the API server. Specify the host and port to bind to.
```bash
kreuzcrawl serve --host 0.0.0.0 --port 3000
```
--------------------------------
### Install and Configure Pre-commit Hooks
Source: https://docs.kreuzcrawl.kreuzberg.dev/contributing
Install the pre-commit tool and configure it for pre-commit and commit-msg hook types. This automates checks before committing.
```bash
pip install pre-commit
pre-commit install --hook-type pre-commit --hook-type commit-msg
```
--------------------------------
### Sitemap Index File Example
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/url-discovery
An example of a sitemap index file, which references multiple other sitemap files (including compressed ones) that should be processed recursively.
```xml
https://example.com/sitemap-posts.xml
https://example.com/sitemap-pages.xml.gz
```
--------------------------------
### Start REST API Server
Source: https://docs.kreuzcrawl.kreuzberg.dev/cli/usage
Starts the kreuzcrawl REST API server. Requires the 'api' feature to be enabled.
```bash
kreuzcrawl serve [OPTIONS]
```
```bash
# Start on default port
kreuzcrawl serve
```
```bash
# Start on custom host and port
kreuzcrawl serve --host 127.0.0.1 --port 8080
```
--------------------------------
### Start API Server Programmatically
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Start the API server programmatically using the `serve_api` function. This requires specifying the host, port, and an optional `CrawlConfig`.
```rust
use kreuzcrawl::{CrawlConfig, serve_api};
serve_api("0.0.0.0", 3000, CrawlConfig::default()).await?;
```
```rust
use kreuzcrawl::{CrawlConfig, serve_api};
serve_api("127.0.0.1", 8080, CrawlConfig {
max_concurrent: Some(20),
respect_robots_txt: true,
..Default::default()
}).await?;
```
--------------------------------
### Install Kreuzcrawl Gem
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl gem directly using the 'gem install' command.
```bash
gem install kreuzcrawl
```
--------------------------------
### Start MCP Server with Custom Configuration
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/mcp-server
Configure the MCP server with custom crawl settings before starting. This allows for fine-tuning behavior like respecting robots.txt and staying on the same domain.
```rust
use kreuzcrawl::mcp::start_mcp_server_with_config;
use kreuzcrawl::CrawlConfig;
let config = CrawlConfig {
respect_robots_txt: true,
stay_on_domain: true,
..Default::default()
};
start_mcp_server_with_config(config).await?;
```
--------------------------------
### DiskCache Initialization with Default Location
Source: https://docs.kreuzcrawl.kreuzberg.dev/concepts/plugin-system
Example of creating a `DiskCache` using the default cache location.
```rust
let cache = DiskCache::default_location()?;
```
--------------------------------
### Start MCP Server
Source: https://docs.kreuzcrawl.kreuzberg.dev/cli/usage
Starts the MCP server using stdio transport. Requires the 'mcp' feature. The server communicates via stdin/stdout using the MCP protocol. Startup messages are printed to stderr.
```bash
kreuzcrawl mcp
```
--------------------------------
### Install Kreuzcrawl CLI with Cargo and Features
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl CLI with specific features enabled, such as 'api' and 'mcp', using Cargo.
```bash
cargo install kreuzcrawl-cli --features "api,mcp"
```
--------------------------------
### Crawl a website
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-go
Crawl a website starting from a given URL, following links up to the configured depth. Requires a crawl engine handle and the starting URL.
```go
func Crawl(engine CrawlEngineHandle, url string) (CrawlResult, error)
```
--------------------------------
### Get Kreuzcrawl Go Package
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Fetch the Kreuzcrawl Go package using the 'go get' command. Requires Go 1.21+.
```bash
go get github.com/kreuzberg-dev/kreuzcrawl/packages/go
```
--------------------------------
### Install Kreuzcrawl CLI with Homebrew
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Use Homebrew to install the Kreuzcrawl CLI on macOS or Linux. This method does not require a Rust toolchain.
```bash
brew install kreuzberg-dev/tap/kreuzcrawl
```
--------------------------------
### Bm25Filter Initialization
Source: https://docs.kreuzcrawl.kreuzberg.dev/concepts/plugin-system
Example of initializing a `Bm25Filter` with a search query and a scoring threshold.
```rust
let filter = Bm25Filter::new("rust async programming", 0.3);
```
--------------------------------
### Standard XML Sitemap Example
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/url-discovery
An example of a standard XML sitemap structure, defining a single URL with its location, last modification date, change frequency, and priority.
```xml
https://example.com/page1
2024-01-15
weekly
0.8
```
--------------------------------
### Create Default CrawlConfig
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-python
Instantiates a `CrawlConfig` object with default settings. Use this as a starting point for custom configurations.
```python
from crawler.config import CrawlConfig
config = CrawlConfig.default()
```
--------------------------------
### Create Default CrawlConfig
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-csharp
Instantiates a CrawlConfig object with default settings. Use this as a starting point for custom configurations.
```csharp
public CrawlConfig CreateDefault()
{
}
```
--------------------------------
### Basic LlmExtractor Setup and Usage
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/llm-extraction
Demonstrates setting up the LlmExtractor with a JSON schema and instruction, integrating it into a CrawlEngine, and processing extracted data.
```rust
use kreuzcrawl::{CrawlEngine, CrawlConfig, LlmExtractor};
use serde_json::json;
let schema = json!({
"type": "object",
"properties": {
"title": { "type": "string" },
"summary": { "type": "string" },
"topics": {
"type": "array",
"items": { "type": "string" }
}
},
"required": ["title", "summary"]
});
let extractor = LlmExtractor::new(
"your-api-key", // API key
"openai/gpt-4o-mini", // Model identifier
Some(schema), // JSON schema (optional)
Some("Extract the article title, a brief summary, and main topics.".into()), // Instruction (optional)
None, // Custom prompt template (optional)
)?;
let engine = CrawlEngine::builder()
.config(CrawlConfig {
max_depth: Some(1),
max_pages: Some(10),
..Default::default()
})
.content_filter(extractor)
.build()?;
let result = engine.crawl("https://example.com/blog").await?;
for page in &result.pages {
if let Some(ref data) = page.extracted_data {
println!("{}: {}", page.url, serde_json::to_string_pretty(data)?);
}
}
```
--------------------------------
### DiskCache Initialization with Custom Configuration
Source: https://docs.kreuzcrawl.kreuzberg.dev/concepts/plugin-system
Example of creating a `DiskCache` with a specified cache directory, TTL, and maximum number of entries.
```rust
let cache = DiskCache::new("./cache", 3600, 10000)?;
```
--------------------------------
### Health Check Endpoint Response
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Example JSON response from the `GET /health` endpoint.
```json
{ "status": "ok", "version": "0.1.0-rc.1" }
```
--------------------------------
### Crawl Endpoint Request
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Example JSON payload for the `POST /v1/crawl` endpoint to start an asynchronous crawl job.
```json
{
"url": "https://example.com",
"maxDepth": 2,
"maxPages": 100,
"includePaths": ["/docs/.*"],
"excludePaths": ["/blog/.*"],
"onlyMainContent": true
}
```
--------------------------------
### Get Default CrawlConfig
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-wasm
Creates a new CrawlConfig instance with default values. Use this as a starting point for custom configurations.
```typescript
const config = CrawlConfig.default();
```
--------------------------------
### Crawl Status Endpoint Response (Completed)
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Example JSON response from the `GET /v1/crawl/{id}` endpoint when the crawl job has completed.
```json
{
"status": "completed",
"total": 25,
"completed": 25,
"data": [
{ "url": "https://example.com/", "status_code": 200, "depth": 0, ... },
{ "url": "https://example.com/about", "status_code": 200, "depth": 1, ... }
]
}
```
--------------------------------
### Register Plugins via Builder
Source: https://docs.kreuzcrawl.kreuzberg.dev/concepts/plugin-system
Example of configuring and building a `CrawlEngine` by registering various plugins like frontier, rate limiter, store, event emitter, strategy, content filter, and cache.
```rust
let engine = CrawlEngine::builder()
.config(config)
.frontier(my_redis_frontier)
.rate_limiter(my_distributed_limiter)
.store(PostgresStore::new(pool))
.event_emitter(WebhookEmitter::new(webhook_url))
.strategy(AdaptiveStrategy::new(10, 0.05))
.content_filter(Bm25Filter::new("search query", 0.3))
.cache(DiskCache::default_location()?)
.build()?;
```
--------------------------------
### Crawl Status Endpoint Response (In Progress)
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/api-server
Example JSON response from the `GET /v1/crawl/{id}` endpoint when the crawl job is still in progress.
```json
{
"status": "in_progress",
"total": 0,
"completed": 12
}
```
--------------------------------
### Crawl Tool Parameters
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/mcp-server
Example JSON payload for the `crawl` tool. Define the starting URL and control crawling depth, page limits, and domain restrictions.
```json
{
"url": "https://docs.rs/tokio",
"max_depth": 2,
"max_pages": 50,
"stay_on_domain": true
}
```
--------------------------------
### Verify Kreuzcrawl CLI Installation
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Check the installed version of the Kreuzcrawl CLI to confirm the installation was successful.
```bash
kreuzcrawl --version
```
--------------------------------
### Login Flow Example
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/interaction
A sequence of actions demonstrating a typical login flow, including typing credentials and clicking the login button, followed by a wait for dashboard content and scraping the page.
```json
[
{ "type": "type", "selector": "#username", "text": "admin" },
{ "type": "type", "selector": "#password", "text": "s3cret" },
{ "type": "click", "selector": "#login-btn" },
{ "type": "wait", "selector": ".dashboard" },
{ "type": "scrape" }
]
```
--------------------------------
### Get API version with GET /version
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/rest-api
Retrieve the current version of the API.
```json
{
"version": "0.1.0"
}
```
--------------------------------
### Initialize and Run ResearchAgent
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/research
Sets up the ResearchAgent with a query, crawl configuration, and initiates the research process. The synthesis result is printed to the console. Requires `ai` feature.
```rust
use kreuzcrawl::research::{ResearchAgent, ResearchConfig};
use kreuzcrawl::CrawlConfig;
let config = ResearchConfig {
query: "Rust async runtime comparison".into(),
max_steps: 10,
max_pages_per_step: 5,
max_depth: 3,
seed_urls: vec![
"https://tokio.rs".into(),
"https://async-std.rs".into(),
],
};
let agent = ResearchAgent::new(config)
.with_crawl_config(CrawlConfig {
stay_on_domain: true,
..Default::default()
});
let result = agent.research().await?;
println!("{}", result.synthesis);
```
--------------------------------
### Install Kreuzcrawl Node.js Package
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl npm package for Node.js environments.
```bash
npm install @kreuzberg/kreuzcrawl
```
--------------------------------
### Install Kreuzcrawl WASM Package
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl WASM package for browser or Node.js environments.
```bash
npm install @kreuzberg/kreuzcrawl-wasm
```
--------------------------------
### Pull Request Preparation Steps
Source: https://docs.kreuzcrawl.kreuzberg.dev/contributing
Lists the steps to perform before opening a pull request, including formatting, linting, and testing.
```bash
cargo fmt --all
cargo clippy --workspace --all-features --all-targets -- -D warnings
cargo test --workspace
```
--------------------------------
### CrawlConfig JSON Example
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/configuration
Example of a CrawlConfig object serialized to JSON. Duration fields are represented in milliseconds.
```json
{
"max_depth": 3,
"max_pages": 100,
"max_concurrent": 5,
"stay_on_domain": true,
"respect_robots_txt": true,
"request_timeout": 30000,
"max_redirects": 10,
"retry_count": 2,
"retry_codes": [429, 503],
"include_paths": ["^/docs/"],
"exclude_paths": ["/admin/"],
"main_content_only": false,
"cookies_enabled": false,
"download_assets": false,
"download_documents": true,
"document_max_size": 52428800,
"capture_screenshot": false,
"save_browser_profile": false,
"browser": {
"mode": "auto",
"timeout": 30000,
"wait": "network_idle"
}
}
```
--------------------------------
### Install Kreuzcrawl Node.js Package with pnpm
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl npm package using the pnpm package manager.
```bash
pnpm add @kreuzberg/kreuzcrawl
```
--------------------------------
### Create Crawl Engine
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-python
Initializes a new crawl engine. If no configuration is provided, it defaults to `CrawlConfig.default()`. An error is raised if the configuration is invalid.
```python
def create_engine(config: CrawlConfig = None) -> CrawlEngineHandle
```
--------------------------------
### Basic CLI Command Structure
Source: https://docs.kreuzcrawl.kreuzberg.dev/cli/usage
Illustrates the general syntax for invoking kreuzcrawl commands and options.
```bash
kreuzcrawl [OPTIONS]
```
--------------------------------
### Install Kreuzcrawl Python Package
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl Python package from PyPI using pip. Requires Python 3.10+.
```bash
pip install kreuzcrawl
```
--------------------------------
### Install Kreuzcrawl PHP Extension via Composer
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl PHP extension using Composer. Requires PHP 8.2+.
```bash
composer require kreuzberg-dev/kreuzcrawl
```
--------------------------------
### Proxy Configuration with Authentication
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/configuration
Configure proxy settings including URL, username, and password.
```rust
ProxyConfig {
url: "http://proxy:8080".to_string(), // or "socks5://proxy:1080"
username: Some("user".to_string()),
password: Some("pass".to_string()),
}
```
--------------------------------
### Start MCP Server Programmatically
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/mcp-server
Initiate the MCP server within your Rust application. Ensure you have the `tokio` runtime available.
```rust
use kreuzcrawl::mcp::start_mcp_server;
#[tokio::main]
async fn main() -> Result<(), Box> {
start_mcp_server().await?;
Ok(())
}
```
--------------------------------
### Initialize and Use WarcWriter API
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/warc
Demonstrates the low-level API for creating WARC files. Ensure to call `write_warcinfo` once before writing response records and `finish` to finalize the file. The `write_response` method returns the WARC-Record-ID.
```rust
use std::path::Path;
use chrono::Utc;
use kreuzcrawl::WarcWriter;
let mut writer = WarcWriter::new(Path::new("output.warc"))?;
// Write the warcinfo record (call once, before any response records).
writer.write_warcinfo("kreuzcrawl/0.1.0", "my-host.example.com")?;
// Write a response record for each crawled page.
let record_id = writer.write_response(
"https://example.com/page",
200,
&[("Content-Type", "text/html")],
b"Hello",
Utc::now(),
)?;
// Flush and finalize.
writer.finish()?;
```
--------------------------------
### Create Default BrowserConfig
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-python
Instantiates BrowserConfig with default settings. Use this for basic browser fallback configurations.
```python
BrowserConfig.default()
```
--------------------------------
### Create Crawl Engine C API
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-c
Use this function to initialize a new crawl engine. Pass NULL for config to use default settings. Returns NULL if the configuration is invalid.
```c
KcrawlCrawlEngineHandle* kcrawl_create_engine(KcrawlCrawlConfig config);
```
--------------------------------
### Start Asynchronous Crawl Job - Response Body
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/rest-api
Response upon successfully starting a crawl job, including the job ID.
```json
{
"success": true,
"id": "550e8400-e29b-41d4-a716-446655440000"
}
```
--------------------------------
### LlmExtractor Setup with JSON Schema
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/llm-extraction
Configures the LlmExtractor with a specific JSON schema to enforce structured output for product information extraction.
```rust
let schema = json!({
"type": "object",
"properties": {
"product_name": { "type": "string" },
"price": { "type": "number" },
"currency": { "type": "string" },
"in_stock": { "type": "boolean" }
},
"required": ["product_name", "price"]
});
let extractor = LlmExtractor::new(
api_key,
"openai/gpt-4o-mini",
Some(schema),
Some("Extract product information from this page.".into()),
None,
)?;
```
--------------------------------
### createEngine()
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-java
Creates a new crawl engine with the given configuration. If no configuration is provided, it uses the default configuration. An error is returned if the configuration is invalid.
```APIDOC
## createEngine()
### Description
Create a new crawl engine with the given configuration. If `config` is `null`, uses `CrawlConfig.default()`. Returns an error if the configuration is invalid.
### Signature
```java
public static CrawlEngineHandle createEngine(CrawlConfig config) throws CrawlError
```
### Parameters
#### Path Parameters
- **config** (Optional) - Optional - The configuration options
### Errors
Throws `CrawlErrorException`.
```
--------------------------------
### Install Kreuzcrawl CLI with Cargo
Source: https://docs.kreuzcrawl.kreuzberg.dev/getting-started/installation
Install the Kreuzcrawl CLI using Cargo, Rust's package manager. This requires a Rust toolchain.
```bash
cargo install kreuzcrawl-cli
```
--------------------------------
### crawl()
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-elixir
Crawls a website starting from a given URL, following links up to the configured depth. Requires a crawl engine handle and the starting URL.
```APIDOC
## crawl()
### Description
Crawl a website starting from `url`, following links up to the configured depth.
### Signature
```elixir
@spec crawl(engine, url) :: {:ok, term()} | {:error, term()}
def crawl(engine, url)
```
### Parameters
#### Path Parameters
- **engine** (CrawlEngineHandle) - Required - The crawl engine handle
- **url** (String.t()) - Required - The URL to fetch
### Returns
`CrawlResult`
### Errors
Returns `{:error, reason}`
```
--------------------------------
### Discover URLs via Sitemap and Crawling using CLI
Source: https://docs.kreuzcrawl.kreuzberg.dev/features
Use the `map` command to discover URLs, respecting robots.txt.
```bash
kreuzcrawl map https://example.com --respect-robots-txt
```
--------------------------------
### createEngine()
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-php
Creates a new crawl engine with the given configuration. If no configuration is provided, it uses the default configuration. Returns an error if the configuration is invalid.
```APIDOC
## createEngine()
### Description
Create a new crawl engine with the given configuration. If `config` is `null`, uses `CrawlConfig::default()`. Returns an error if the configuration is invalid.
### Signature
```php
public static function createEngine(?CrawlConfig $config = null): CrawlEngineHandle
```
### Parameters
#### Path Parameters
- **config** (`?CrawlConfig`) - Optional - The configuration options
### Returns
`CrawlEngineHandle`
### Errors
Throws `CrawlError`.
```
--------------------------------
### Create Crawl Engine
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-php
Creates a new crawl engine with optional configuration. Uses default configuration if none is provided. Throws CrawlError for invalid configurations.
```php
public static function createEngine(?CrawlConfig $config = null): CrawlEngineHandle
```
--------------------------------
### Crawl Website
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-java
Crawls a website starting from a given URL, following links up to a configured depth. Requires a crawl engine handle and the starting URL.
```java
public static CrawlResult crawl(CrawlEngineHandle engine, String url) throws CrawlError
```
--------------------------------
### CreateEngine()
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-csharp
Creates a new crawl engine with the given configuration. If `config` is `null`, it uses `CrawlConfig.default()`. Returns an error if the configuration is invalid.
```APIDOC
## CreateEngine()
### Description
Create a new crawl engine with the given configuration. If `config` is `null`, uses `CrawlConfig.default()`. Returns an error if the configuration is invalid.
### Signature
```csharp
public static CrawlEngineHandle CreateEngine(CrawlConfig? config = null)
```
### Parameters
#### Path Parameters
- **config** (CrawlConfig?) - Optional - The configuration options
### Returns
`CrawlEngineHandle`
### Errors
Throws `CrawlError`.
```
--------------------------------
### Initialize BrowserPool
Source: https://docs.kreuzcrawl.kreuzberg.dev/guides/browser
Set up a BrowserPool to manage a single Chrome instance and a limited number of concurrent tabs. Chrome is launched lazily on the first acquire or warm call.
```rust
use std::sync::Arc;
use kreuzcrawl::BrowserPoolConfig;
use kreuzcrawl::BrowserPool;
let pool = BrowserPool::new(BrowserPoolConfig {
max_pages: 8, // up to 8 concurrent tabs
browser_endpoint: None, // launch a local Chrome
chrome_args: Vec::new(),
launch_timeout: std::time::Duration::from_secs(30),
});
// Optionally warm the pool so the first request doesn't pay startup cost.
pool.warm().await?;
// Acquire a page, use it, then close.
let page = pool.acquire_page().await?;
// ... use page.page() for CDP operations ...
page.close().await;
```
--------------------------------
### Crawl Website
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-rust
Crawls a website starting from a given URL, following links up to the configured depth. Requires a crawl engine handle and the starting URL.
```rust
pub async fn crawl(engine: CrawlEngineHandle, url: &str) -> Result
```
--------------------------------
### Create Crawl Engine
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-java
Creates a new crawl engine with the specified configuration. If the configuration is null, default settings are used. An error is returned for invalid configurations.
```java
public static CrawlEngineHandle createEngine(CrawlConfig config) throws CrawlError
```
--------------------------------
### Crawl Website
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-typescript
Crawls a website starting from a given URL, following links up to a configured depth. Requires an initialized crawl engine handle and the starting URL.
```typescript
function crawl(engine: CrawlEngineHandle, url: string): Promise;
```
--------------------------------
### Implement PostgresStore for CrawlStore
Source: https://docs.kreuzcrawl.kreuzberg.dev/concepts/plugin-system
Example implementation of the `CrawlStore` trait using PostgreSQL via `sqlx` to persist page data.
```rust
use async_trait::async_trait;
use kreuzcrawl::traits::{CrawlStore, CrawlStats};
use kreuzcrawl::types::{CrawlPageResult, ScrapeResult};
use kreuzcrawl::error::CrawlError;
pub struct PostgresStore {
pool: sqlx::PgPool,
}
#[async_trait]
impl CrawlStore for PostgresStore {
async fn store_page(&self, url: &str, result: &ScrapeResult) -> Result<(), CrawlError> {
sqlx::query("INSERT INTO pages (url, title, html) VALUES ($1, $2, $3)")
.bind(url)
.bind(&result.metadata.title)
.bind(&result.html)
.execute(&self.pool)
.await
.map_err(|e| CrawlError::Other(e.to_string()))?;
Ok(())
}
async fn store_crawl_page(
&self, url: &str, result: &CrawlPageResult,
) -> Result<(), CrawlError> {
// persist crawl page result
Ok(())
}
async fn store_error(
&self, url: &str, error: &CrawlError,
) -> Result<(), CrawlError> {
// log or persist error
Ok(())
}
async fn on_complete(&self, stats: &CrawlStats) -> Result<(), CrawlError> {
// finalize crawl session
Ok(())
}
}
```
--------------------------------
### Crawl Website (Ruby)
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-ruby
Crawls a website starting from a given URL, following links up to the configured depth. Requires a crawl engine handle and the starting URL.
```ruby
def self.crawl(engine, url)
```
```
--------------------------------
### Create Crawl Engine C#
Source: https://docs.kreuzcrawl.kreuzberg.dev/reference/api-csharp
Creates a new crawl engine with optional configuration. If no configuration is provided, it defaults to CrawlConfig.default(). Throws CrawlError for invalid configurations.
```csharp
public static CrawlEngineHandle CreateEngine(CrawlConfig? config = null)
```