### Install Feedstock Dependencies

Source: https://www.usefeedstock.com/llms.txt

Setup commands for the Feedstock library and Playwright browsers.

```bash
bun add feedstock
bunx playwright install chromium
```

--------------------------------

### Installation

Source: https://www.usefeedstock.com/llms.txt

Steps to install the feedstock package and its dependencies.

```APIDOC
## Installation

### Steps

1.  **Install the package**
    ```bash
    bun add feedstock
    ```

2.  **Install Playwright browsers**
    ```bash
    bunx playwright install chromium
    ```
```

--------------------------------

### Deep Crawl - Quick Start

Source: https://www.usefeedstock.com/llms.txt

Basic example of initiating a deep crawl and processing results.

```APIDOC
## POST /deep-crawling

### Description
Initiates a deep crawl starting from a given URL and returns a list of crawl results.

### Method
POST

### Endpoint
/deep-crawling

### Parameters
#### Query Parameters
- **url** (string) - Required - The starting URL for the crawl.
- **options** (object) - Optional - Configuration for the crawl.
  - **cacheMode** (string) - Optional - Cache behavior (e.g., 'Bypass').
  - **maxDepth** (number) - Optional - Maximum depth of links to follow (default: 3).
  - **maxPages** (number) - Optional - Maximum number of pages to crawl (default: 100).

### Request Example
```json
{
  "url": "https://example.com",
  "options": {
    "cacheMode": "Bypass",
    "maxDepth": 2,
    "maxPages": 50
  }
}
```

### Response
#### Success Response (200)
- **results** (array) - An array of crawl results.
  - **url** (string) - The URL of the crawled page.
  - **success** (boolean) - Indicates if the crawl for this page was successful.

#### Response Example
```json
{
  "results": [
    {
      "url": "https://example.com",
      "success": true
    },
    {
      "url": "https://example.com/page1",
      "success": true
    }
  ]
}
```
```

--------------------------------

### Your First Crawl

Source: https://www.usefeedstock.com/llms.txt

A basic example demonstrating how to crawl a single URL and access its metadata and content.

```APIDOC
## Your First Crawl

### Description

This example shows how to initialize a `WebCrawler`, crawl a single URL, and process the results.

### Request Example

```typescript
import { WebCrawler } from "feedstock";

const crawler = new WebCrawler();

const result = await crawler.crawl("https://example.com");

if (result.success) {
  console.log("Title:", result.metadata?.title);
  console.log("Markdown:", result.markdown?.rawMarkdown);
  console.log("Links found:", result.links.internal.length);
}

await crawler.close();
```

### Response Example

```json
{
  "success": true,
  "metadata": {
    "title": "Example Domain"
  },
  "markdown": {
    "rawMarkdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents. You may use this\ncontent freely."
  },
  "links": {
    "internal": [],
    "external": []
  }
}
```
```

--------------------------------

### Example robots.txt configuration

Source: https://www.usefeedstock.com/llms.txt

A sample robots.txt file demonstrating user-agent matching, allow/disallow rules, crawl delays, and sitemaps.

```text
User-agent: *
Disallow: /private/
Disallow: /admin
Allow: /admin/public
Crawl-delay: 2

User-agent: feedstock
Disallow: /secret/
Allow: /secret/public/
Crawl-delay: 1

Sitemap: https://example.com/sitemap.xml
```

--------------------------------

### Crawler Lifecycle Management

Source: https://www.usefeedstock.com/llms.txt

Explains how to start and close the crawler to manage browser and cache resources.

```APIDOC
## Crawler Lifecycle Management

### Description
Manage the lifecycle of the `WebCrawler`. The crawler must be explicitly started before use and closed when finished to release resources.

### Method
- `start()`: Launches the browser and opens the cache.
- `close()`: Closes the browser and cache.

### Endpoint
N/A (Instance methods)

### Request Example
```typescript
await crawler.start();   // launches browser, opens cache
// ... crawl pages ...
await crawler.close();   // closes browser, closes cache
```

<Callout type="info">
  If you call `crawl()` without calling `start()` first, the crawler will auto-start.
</Callout>
```

--------------------------------

### Deep Crawl - With Filters and Rate Limiting

Source: https://www.usefeedstock.com/llms.txt

Example showing how to configure filters (e.g., domain, content type) and rate limiting for a deep crawl.

```APIDOC
## POST /deep-crawling (with Filters and Rate Limiting)

### Description
Initiates a deep crawl with advanced configurations including URL filtering and rate limiting to control the crawling process.

### Method
POST

### Endpoint
/deep-crawling

### Parameters
#### Query Parameters
- **url** (string) - Required - The starting URL for the crawl.
- **options** (object) - Optional - Configuration for the crawl.
  - **maxDepth** (number) - Optional - Maximum depth of links to follow.
  - **maxPages** (number) - Optional - Maximum number of pages to crawl.
  - **filterChain** (object) - Optional - A chain of filters to apply to URLs.
    - **allowed** (array of strings) - Optional - List of allowed domains.
    - **contentType** (string) - Optional - Filter by content type.
  - **rateLimiter** (object) - Optional - Rate limiter configuration.
    - **baseDelay** (number) - Optional - Base delay in milliseconds between requests.
  - **robotsParser** (object) - Optional - Configuration for `robots.txt` compliance.

### Request Example
```json
{
  "url": "https://example.com",
  "options": {
    "maxDepth": 2,
    "maxPages": 100,
    "filterChain": {
      "allowed": ["example.com"],
      "contentType": "text/html"
    },
    "rateLimiter": {
      "baseDelay": 500
    },
    "robotsParser": {}
  }
}
```

### Response
#### Success Response (200)
- **results** (array) - An array of crawl results.

#### Response Example
```json
{
  "results": [
    {
      "url": "https://example.com/page1",
      "success": true
    }
  ]
}
```
```

--------------------------------

### Feedstock Crawler Setup and Imports

Source: https://www.usefeedstock.com/llms.txt

Imports necessary components from the Feedstock library for building a web crawler, including filters, rate limiters, and scorers.

```typescript
import {
  WebCrawler,
  CacheMode,
  FilterChain,
  DomainFilter,
  ContentTypeFilter,
  URLPatternFilter,
  RateLimiter,
  RobotsParser,
  CompositeScorer,
  KeywordRelevanceScorer,
  PathDepthScorer,
} from "feedstock";
```

--------------------------------

### Integrate CrawlerMonitor with a stream

Source: https://www.usefeedstock.com/llms.txt

Example of using the monitor within a crawler stream loop.

```typescript
const monitor = new CrawlerMonitor();
monitor.start();

for await (const result of crawler.deepCrawlStream(startUrl, {}, config)) {
  monitor.recordPageComplete({
    success: result.success,
    fromCache: result.cacheStatus === "hit",
    responseTimeMs: 0,
    bytesDownloaded: result.html.length,
  });

  if (monitor.getStats().pagesTotal % 10 === 0) {
    console.log(monitor.formatStats());
  }
}
```

--------------------------------

### Apply FilterChain with Reasons

Source: https://www.usefeedstock.com/llms.txt

Instantiate and use a `FilterChain` to apply multiple filters sequentially. This example demonstrates how to add filters, apply them to URLs, and retrieve all recorded denials and denials grouped by filter.

```typescript
const chain = new FilterChain()
  .add(new DomainFilter({ allowed: ["example.com"] }))
  .add(new URLPatternFilter({ exclude: [/\/admin/] }))
  .add(new ContentTypeFilter());

// Crawl with this chain...
await chain.apply("https://other.com/page");       // denied: domain
await chain.apply("https://example.com/admin");     // denied: pattern
await chain.apply("https://example.com/file.pdf");  // denied: content-type
await chain.apply("https://example.com/docs");      // allowed

// Get all denials
const denials = chain.getDenials();
// [
//   { url: "https://other.com/page", reason: 'Domain "other.com" is not in allowed list', filter: "domain" },
//   { url: "https://example.com/admin", reason: "Matched exclude pattern: \/admin", filter: "url-pattern" },
//   { url: "https://example.com/file.pdf", reason: 'File extension ".pdf" is blocked', filter: "content-type" },
// ]

// Group by filter
const byFilter = chain.getDenialsByFilter();
// { "domain": [...], "url-pattern": [...], "content-type": [...] }
```

--------------------------------

### Deep Crawl - BestFirst Strategy with Scorer

Source: https://www.usefeedstock.com/llms.txt

Example demonstrating how to use a custom scorer to implement the BestFirst traversal strategy, prioritizing URLs based on defined criteria.

```APIDOC
## POST /deep-crawling (with BestFirst strategy)

### Description
Initiates a deep crawl using the BestFirst strategy, which prioritizes URLs based on a provided scorer.

### Method
POST

### Endpoint
/deep-crawling

### Parameters
#### Query Parameters
- **url** (string) - Required - The starting URL for the crawl.
- **options** (object) - Optional - Configuration for the crawl.
  - **maxDepth** (number) - Optional - Maximum depth of links to follow.
  - **maxPages** (number) - Optional - Maximum number of pages to crawl.
  - **scorer** (object) - Required for BestFirst - A scorer object (e.g., `CompositeScorer`) that defines prioritization logic.

### Request Example
```json
{
  "url": "https://example.com",
  "options": {
    "maxDepth": 3,
    "maxPages": 50,
    "scorer": {
      "type": "CompositeScorer",
      "scorers": [
        {
          "type": "KeywordRelevanceScorer",
          "keywords": ["docs", "api"],
          "weight": 2.0
        },
        {
          "type": "PathDepthScorer",
          "maxDepth": 10,
          "weight": 1.0
        }
      ]
    }
  }
}
```

### Response
#### Success Response (200)
- **results** (array) - An array of crawl results, ordered by the scorer's prioritization.

#### Response Example
```json
{
  "results": [
    {
      "url": "https://example.com/docs/api",
      "success": true
    },
    {
      "url": "https://example.com/about",
      "success": true
    }
  ]
}
```
```

--------------------------------

### Process Raw HTML

Source: https://www.usefeedstock.com/llms.txt

An example of processing raw HTML content directly without launching a browser.

```APIDOC
## Process Raw HTML

### Description

This example demonstrates how to use `processHtml` to convert raw HTML strings into markdown without the need for a full browser instance.

### Request Example

```typescript
const html = "<html><body><h1>Hello</h1><p>World</p></body></html>";
const result = await crawler.processHtml(html);

console.log(result.markdown?.rawMarkdown);
// # Hello\n\nWorld
```

### Response Example

```json
{
  "markdown": {
    "rawMarkdown": "# Hello\n\nWorld"
  }
}
```
```

--------------------------------

### Manage Crawler Lifecycle

Source: https://www.usefeedstock.com/llms.txt

Explicitly start and close the crawler to manage browser and cache resources.

```typescript
await crawler.start();   // launches browser, opens cache
// ... crawl pages ...
await crawler.close();   // closes browser, closes cache
```

--------------------------------

### Apply Domain Filter with Reason

Source: https://www.usefeedstock.com/llms.txt

Use `applyWithReason` to get detailed denial information from a single filter. This is useful for understanding why a specific URL was rejected by a particular filter.

```typescript
const filter = new DomainFilter({ allowed: ["example.com"] });
const result = await filter.applyWithReason("https://other.com/page");

console.log(result);
// {
//   allowed: false,
//   reason: 'Domain "other.com" is not in allowed list',
//   filter: "domain"
// }
```

--------------------------------

### Integrate Table Extraction with Crawler

Source: https://www.usefeedstock.com/llms.txt

Examples of using the strategy within a crawler or directly via processHtml.

```typescript
const result = await crawler.crawl("https://example.com/data", {
  extractionStrategy: {
    type: "css",  // or use TableExtractionStrategy directly via processHtml
    params: { ... },
  },
});
```

```typescript
const strategy = new TableExtractionStrategy({ minRows: 2 });
const tables = await strategy.extract(result.url, result.cleanedHtml!);
```

--------------------------------

### Combine URLSeeder with Deep Crawl

Source: https://www.usefeedstock.com/llms.txt

Use discovered URLs as starting points for deep crawling.

```typescript
const { urls } = await seeder.seed("docs.example.com");

for (const startUrl of urls.filter(u => u.endsWith("/index.html"))) {
  const results = await crawler.deepCrawl(startUrl, {}, { maxDepth: 1 });
  // Process results...
}
```

--------------------------------

### GET /denial-reasons

Source: https://www.usefeedstock.com/llms.txt

Endpoints and methods for retrieving and managing URL denial reasons during filter execution.

```APIDOC
## GET /denial-reasons

### Description
Retrieves information regarding why URLs were rejected by filters. This includes per-filter reasoning and aggregated denial logs.

### Method
GET

### Endpoint
/denial-reasons

### Response
#### Success Response (200)
- **allowed** (boolean) - Whether the URL passed the filter.
- **reason** (string) - The specific explanation for the rejection.
- **filter** (string) - The identifier of the filter that caused the rejection.

#### Response Example
{
  "allowed": false,
  "reason": "Domain \"other.com\" is not in allowed list",
  "filter": "domain"
}
```

--------------------------------

### Basic Deep Crawl

Source: https://www.usefeedstock.com/llms.txt

Initiates a deep crawl from a starting URL with a specified cache mode and crawl depth/page limits. Processes results after the crawl completes.

```typescript
const results = await crawler.deepCrawl(
  "https://example.com",
  { cacheMode: CacheMode.Bypass },
  { maxDepth: 2, maxPages: 50 },
);

for (const result of results) {
  console.log(`${result.url}: ${result.success}`);
}
```

--------------------------------

### Deep Crawl - Streaming Results

Source: https://www.usefeedstock.com/llms.txt

Example of using streaming to process crawl results as they become available, suitable for large crawls.

```APIDOC
## POST /deep-crawling/stream

### Description
Initiates a deep crawl and streams results as they are processed, allowing for real-time handling of crawled data.

### Method
POST

### Endpoint
/deep-crawling/stream

### Parameters
#### Query Parameters
- **url** (string) - Required - The starting URL for the crawl.
- **options** (object) - Optional - Configuration for the crawl.
  - **cacheMode** (string) - Optional - Cache behavior (e.g., 'Bypass').
  - **maxDepth** (number) - Optional - Maximum depth of links to follow (default: 3).
  - **maxPages** (number) - Optional - Maximum number of pages to crawl (default: 100).

### Request Example
```json
{
  "url": "https://example.com",
  "options": {
    "cacheMode": "Bypass",
    "maxDepth": 3,
    "maxPages": 100
  }
}
```

### Response
#### Success Response (200)
- **stream** (stream) - A stream of crawl result objects.
  - **url** (string) - The URL of the crawled page.
  - **success** (boolean) - Indicates if the crawl for this page was successful.

#### Response Example
(Streaming response, each result object is emitted as it's available)
```json
{
  "url": "https://example.com/page1",
  "success": true
}
{
  "url": "https://example.com/page2",
  "success": false
}
```
```

--------------------------------

### Define Custom Chunking Strategy

Source: https://www.usefeedstock.com/llms.txt

Create a custom chunking strategy by extending the ChunkingStrategy class and implementing the chunk method. This example splits text by sentence-ending punctuation.

```typescript
import { ChunkingStrategy } from "feedstock";

class SentenceChunking extends ChunkingStrategy {
  chunk(text: string): string[] {
    return text.split(/(?<=[.!?])\s+/);
  }
}
```

--------------------------------

### Implement Custom JSON-LD Extraction Strategy

Source: https://www.usefeedstock.com/llms.txt

Extend the `ExtractionStrategy` base class to create a custom extractor. This example demonstrates parsing embedded JSON-LD scripts from HTML.

```typescript
import { ExtractionStrategy, type ExtractedItem } from "feedstock";

class JsonApiExtractor extends ExtractionStrategy {
  async extract(url: string, html: string): Promise<ExtractedItem[]> {
    // Parse embedded JSON-LD, microdata, etc.
    const scripts = html.match(/<script type="application\/ld\+json">(.*?)<\/script>/gs);
    return (scripts ?? []).map((s, i) => ({
      index: i,
      content: s.replace(/<\/?script[^>]*>/g, "").trim(),
    }));
  }
}
```

--------------------------------

### Basic Deep Crawl with Feedstock

Source: https://www.usefeedstock.com/llms.txt

Performs a basic deep crawl starting from a given URL, with options to control cache mode and crawl depth/page limits. Logs the status code and URL of crawled pages.

```typescript
const crawler = new WebCrawler();

const results = await crawler.deepCrawl(
  "https://docs.example.com",
  { cacheMode: CacheMode.Enabled },
  { maxDepth: 3, maxPages: 200 },
);

console.log(`Crawled ${results.length} pages`);
for (const r of results) {
  console.log(`${r.statusCode} ${r.url}`);
}

await crawler.close();
```

--------------------------------

### Initialize and use RobotsParser

Source: https://www.usefeedstock.com/llms.txt

Demonstrates fetching robots.txt, checking URL permissions, accessing crawl delays, and discovering sitemaps.

```typescript
import { RobotsParser } from "feedstock";

const parser = new RobotsParser("feedstock"); // user-agent name

// Fetch and parse robots.txt for a URL's origin
const directives = await parser.fetch("https://example.com/page");

// Check if a specific URL is allowed
if (parser.isAllowed("https://example.com/admin", directives)) {
  // OK to crawl
}

// Access crawl delay
if (directives.crawlDelay) {
  rateLimiter.setDelay("https://example.com/", directives.crawlDelay * 1000);
}

// Discover sitemaps
console.log(directives.sitemaps);
// ["https://example.com/sitemap.xml"]
```

--------------------------------

### WebCrawler Initialization

Source: https://www.usefeedstock.com/llms.txt

Demonstrates how to create a new WebCrawler instance, both with default settings and with custom options.

```APIDOC
## WebCrawler Initialization

### Description
Instantiate the `WebCrawler` class. It can be initialized with default settings or custom configurations for browser behavior and caching.

### Method
`new WebCrawler(options?: WebCrawlerOptions)`

### Parameters
#### Request Body
- **options** (WebCrawlerOptions) - Optional - Configuration options for the crawler.
  - **config** (Partial<BrowserConfig>) - Optional - Browser specific configurations.
  - **crawlerStrategy** (CrawlerStrategy) - Optional - Strategy for fetching pages.
  - **scrapingStrategy** (ContentScrapingStrategy) - Optional - Strategy for cleaning HTML.
  - **markdownGenerator** (MarkdownGenerationStrategy) - Optional - Strategy for markdown output.
  - **logger** (Logger) - Optional - Custom logger instance.
  - **verbose** (boolean) - Optional - Enables verbose logging.

### Request Example
```typescript
import { WebCrawler } from "feedstock";

// Minimal — uses all defaults
const crawler = new WebCrawler();

// With options
const crawler = new WebCrawler({
  verbose: true,
  config: {
    browserType: "chromium",
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});
```
```

--------------------------------

### Initialize ProxyRotationStrategy

Source: https://www.usefeedstock.com/llms.txt

Configure proxy rotation with a list of servers and report results.

```typescript
import { ProxyRotationStrategy } from "feedstock";

const rotation = new ProxyRotationStrategy([
  { server: "http://proxy1:8080" },
  { server: "http://proxy2:8080", username: "user", password: "pass" },
  { server: "http://proxy3:8080" },
]);

// Get next proxy
const proxy = rotation.getProxy();
// Use it in your crawler config...

// Report result
rotation.reportResult(proxy, true);  // success
rotation.reportResult(proxy, false); // failure
```

--------------------------------

### Initialize and use RateLimiter

Source: https://www.usefeedstock.com/llms.txt

Configures the RateLimiter with backoff parameters and demonstrates manual request tracking.

```typescript
import { RateLimiter } from "feedstock";

const limiter = new RateLimiter({
  baseDelay: 200,      // 200ms between requests to same domain
  maxDelay: 30_000,    // Cap backoff at 30s
  backoffFactor: 2,    // Double delay on failure
  recoveryFactor: 0.75, // Reduce delay by 25% on success
  jitter: 0.1,         // 10% random jitter
});

// Before each request
await limiter.waitIfNeeded("https://example.com/page1");

// After each request
const shouldRetry = limiter.reportResult("https://example.com/page1", 200);
```

--------------------------------

### Get Data from CrawlCache

Source: https://www.usefeedstock.com/llms.txt

Retrieve a cached entry using its URL. If found, it logs the parsed result and the cached timestamp.

```typescript
// Get
const entry = cache.get("https://example.com");
if (entry) {
  console.log(JSON.parse(entry.result));
  console.log("Cached at:", entry.cachedAt);
}
```

--------------------------------

### Initialize WebCrawler

Source: https://www.usefeedstock.com/llms.txt

Instantiate the WebCrawler with default settings or custom browser configurations.

```typescript
import { WebCrawler } from "feedstock";

// Minimal — uses all defaults
const crawler = new WebCrawler();

// With options
const crawler = new WebCrawler({
  verbose: true,
  config: {
    browserType: "chromium",
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});
```

--------------------------------

### Implement Custom LanguageFilter

Source: https://www.usefeedstock.com/llms.txt

Extend `ContentFilterStrategy` to create custom filtering logic. This example `LanguageFilter` keeps only blocks that appear to be in English.

```typescript
import { ContentFilterStrategy } from "feedstock";

class LanguageFilter extends ContentFilterStrategy {
  filter(content: string, query?: string): string {
    // Keep only English-looking blocks
    return content.split("\n\n")
      .filter(block => /^[a-zA-Z\s.,!?]+$/.test(block))
      .join("\n\n");
  }
}
```

--------------------------------

### Create BrowserConfig

Source: https://www.usefeedstock.com/llms.txt

Configure browser instance settings like type, viewport, and proxy. Set once when creating the WebCrawler.

```typescript
import { createBrowserConfig } from "feedstock";

const config = createBrowserConfig({
  browserType: "chromium",   // "chromium" | "firefox" | "webkit"
  headless: true,
  viewport: { width: 1920, height: 1080 },
  userAgent: "my-bot/1.0",
  proxy: { server: "http://proxy:8080" },
  backend: { kind: "playwright" },
});
```

--------------------------------

### Configure Lightpanda Cloud Mode

Source: https://www.usefeedstock.com/llms.txt

Initializes the WebCrawler to connect to the Lightpanda cloud service.

```typescript
const crawler = new WebCrawler({
  config: {
    backend: {
      kind: "lightpanda",
      mode: "cloud",
      token: process.env.LIGHTPANDA_TOKEN!,
      endpoint: "wss://euwest.cloud.lightpanda.io/ws", // optional, default
    },
  },
});
```

--------------------------------

### Initialize CrawlCache

Source: https://www.usefeedstock.com/llms.txt

Initialize the CrawlCache. By default, it uses `~/.feedstock/cache.db`. A custom path can be provided.

```typescript
import { CrawlCache } from "feedstock";

const cache = new CrawlCache();         // default: ~/.feedstock/cache.db
const cache = new CrawlCache("/tmp/my-cache.db"); // custom path
```

--------------------------------

### Browser Backends Configuration

Source: https://www.usefeedstock.com/llms.txt

Configuration options for selecting and setting up browser backends, including Playwright and Lightpanda.

```APIDOC
# Browser Backends (/browser-backends)

Feedstock supports two browser backends, configured via the `backend` field in `BrowserConfig`.

## Playwright (Default)

Launches a local browser via Playwright. Supports Chromium, Firefox, and WebKit.

### Setup

```bash
bun add feedstock
bunx playwright install chromium
```

### Configuration

```typescript
const crawler = new WebCrawler({
  config: {
    backend: { kind: "playwright" },
    browserType: "chromium", // or "firefox", "webkit"
    headless: true,
  },
});
```

## Lightpanda

[Lightpanda](https://lightpanda.io) is an AI-native headless browser built for speed. It connects via Chrome DevTools Protocol (CDP) and works as a drop-in replacement.

### Local Mode

Launches a local Lightpanda process and connects via CDP.

#### Setup

```bash
bun add @lightpanda/browser
```

#### Configuration

```typescript
const crawler = new WebCrawler({
  config: {
    backend: {
      kind: "lightpanda",
      mode: "local",
      host: "127.0.0.1",  // optional, default
      port: 9222,          // optional, default
    },
  },
});
```

<Callout type="info">
  `@lightpanda/browser` is an optional dependency. It's only needed when using local Lightpanda mode.
</Callout>

### Cloud Mode

Connect to Lightpanda Cloud without installing anything locally.

#### Configuration

```typescript
const crawler = new WebCrawler({
  config: {
    backend: {
      kind: "lightpanda",
      mode: "cloud",
      token: process.env.LIGHTPANDA_TOKEN!,
      endpoint: "wss://euwest.cloud.lightpanda.io/ws", // optional, default
    },
  },
});
```

## How It Works

Both backends produce a standard Playwright `Browser` object. The rest of the pipeline (page navigation, content capture, scraping, extraction) works identically regardless of which backend you choose.

```
Playwright backend:  launcher.launch() → Browser
Lightpanda local:    lightpanda.serve() → chromium.connectOverCDP() → Browser
Lightpanda cloud:    chromium.connectOverCDP(wss://...) → Browser
```

All session management, page lifecycle, and cleanup is handled by the `BrowserManager`.
```

--------------------------------

### Initialize WebCrawler with Default Fetch-First Behavior

Source: https://www.usefeedstock.com/llms.txt

Instantiates a WebCrawler with the default fetch-first engine behavior enabled. This is suitable for most static pages.

```typescript
const crawler = new WebCrawler(); // fetch-first enabled
```

--------------------------------

### Basic Usage of Markdown Generation

Source: https://www.usefeedstock.com/llms.txt

Demonstrates how to crawl a URL and access the generated markdown content.

```APIDOC
## Usage: Basic Markdown Generation

### Description
This example shows how to perform a web crawl and access the different markdown outputs from the `MarkdownGenerationResult`.

### Method
`crawler.crawl(url: string): Promise<CrawlResult>

### Example Code
```typescript
const result = await crawler.crawl("https://example.com");

// Access raw markdown
console.log(result.markdown?.rawMarkdown);

// Access citation-style markdown
console.log(result.markdown?.markdownWithCitations);

// Access just the references list
console.log(result.markdown?.referencesMarkdown);
```
```

--------------------------------

### Initialize FreshnessScorer

Source: https://www.usefeedstock.com/llms.txt

Scores URLs based on date patterns found in the path.

```typescript
new FreshnessScorer(0.5)
```

--------------------------------

### Define a Custom Engine in Feedstock

Source: https://www.usefeedstock.com/llms.txt

Extends the base `Engine` class to create a custom fetching engine. Define its capabilities, quality score, and implementation for fetching, starting, and closing.

```typescript
import { Engine, type EngineCapabilities } from "feedstock";

class MyCustomEngine extends Engine {
  readonly name = "custom";
  readonly quality = 25; // between fetch and playwright
  readonly capabilities: EngineCapabilities = {
    javascript: true,
    screenshot: false,
    pdf: false,
    networkRequests: false,
    consoleMessages: false,
    waitConditions: false,
    customJs: false,
  };

  async start() { /* ... */ }
  async close() { /* ... */ }
  async fetch(url, config) { /* ... */ }
}
```

--------------------------------

### Initialize KeywordRelevanceScorer

Source: https://www.usefeedstock.com/llms.txt

Scores URLs based on keyword matches in the URL and anchor text.

```typescript
new KeywordRelevanceScorer(["product", "pricing"], 2.0)
```

--------------------------------

### Backward Compatibility: Apply vs. ApplyWithReason

Source: https://www.usefeedstock.com/llms.txt

Illustrates the difference between the old `apply()` method which returns a boolean, and the new `applyWithReason()` method which returns a more detailed object including the denial reason. The old API is still supported.

```typescript
// Old API — still works
const allowed = await filter.apply(url); // boolean

// New API — with reason
const result = await filter.applyWithReason(url); // { allowed, reason?, filter? }
```

--------------------------------

### Track crawl progress with CrawlerMonitor

Source: https://www.usefeedstock.com/llms.txt

Initialize the monitor, record page events, and retrieve performance statistics.

```typescript
import { CrawlerMonitor } from "feedstock";

const monitor = new CrawlerMonitor();
monitor.start();

// Track each page
monitor.recordPageStart("https://example.com/page1");
// ... crawl ...
monitor.recordPageComplete({
  success: true,
  fromCache: false,
  responseTimeMs: 150,
  bytesDownloaded: 45_000,
});

// Get stats
const stats = monitor.getStats();
console.log(stats.pagesTotal);     // 1
console.log(stats.pagesPerSecond); // ~6.67
console.log(stats.avgResponseTime); // 150

// Human-readable summary
console.log(monitor.formatStats());
// Pages: 1 (1 ok, 0 failed, 0 cached)
// Time: 0.2s | 6.7 pages/s | avg 150ms/page
// Downloaded: 0.04 MB
```

--------------------------------

### Clear FilterChain Denials

Source: https://www.usefeedstock.com/llms.txt

Use `clearDenials()` to reset the internal denial tracking of a `FilterChain`. This is useful when starting a new crawl or processing a new set of URLs and you want to ensure no previous denial data interferes.

```typescript
chain.clearDenials();
```

--------------------------------

### Putting It All Together: Advanced Site Crawler

Source: https://www.usefeedstock.com/llms.txt

Combines multiple configurations including verbose logging, markdown generation, exclusion of specific HTML tags, filters, rate limiting, robots.txt parsing, and prioritized crawling to build a comprehensive site crawler. It then saves the generated markdown to files.

```typescript
const crawler = new WebCrawler({ verbose: true });

const results = await crawler.deepCrawl(
  "https://docs.example.com",
  {
    cacheMode: CacheMode.Enabled,
    generateMarkdown: true,
    excludeTags: ["nav", "footer", "aside"],
  },
  {
    maxDepth: 3,
    maxPages: 500,
    concurrency: 3,
    filterChain,
    rateLimiter,
    robotsParser,
    scorer,
  },
);

// Save results
for (const result of results) {
  if (result.success && result.markdown) {
    const filename = result.url.replace(/[^a-z0-9]/gi, "_") + ".md";
    await Bun.write(`output/${filename}`, result.markdown.rawMarkdown);
  }
}

console.log(`Saved ${results.length} pages to output/`);
await crawler.close();
```

--------------------------------

### Configure Lightpanda Local Mode

Source: https://www.usefeedstock.com/llms.txt

Initializes the WebCrawler to connect to a local Lightpanda process.

```bash
bun add @lightpanda/browser
```

```typescript
const crawler = new WebCrawler({
  config: {
    backend: {
      kind: "lightpanda",
      mode: "local",
      host: "127.0.0.1",  // optional, default
      port: 9222,          // optional, default
    },
  },
});
```

--------------------------------

### Initialize SlidingWindowChunking

Source: https://www.usefeedstock.com/llms.txt

Initialize SlidingWindowChunking with specified words per window and overlap. This strategy splits text by word count with overlapping segments.

```typescript
import { SlidingWindowChunking } from "feedstock";

const chunker = new SlidingWindowChunking(
  500,  // words per window
  50,   // overlap words
);
const chunks = chunker.chunk(longText);
```

--------------------------------

### Custom Logger Implementation

Source: https://www.usefeedstock.com/llms.txt

Shows how to implement a custom logger for Feedstock by adhering to the `Logger` interface. This allows integration with existing logging libraries like pino or winston.

```typescript
import type { Logger } from "feedstock";

const myLogger: Logger = {
  debug(msg, data) { /* ... */ },
  info(msg, data)  { /* ... */ },
  warn(msg, data)  { /* ... */ },
  error(msg, data) { /* ... */ },
};

const crawler = new WebCrawler({ logger: myLogger });
```

--------------------------------

### Configure Playwright Backend

Source: https://www.usefeedstock.com/llms.txt

Initializes the WebCrawler with the default Playwright backend.

```typescript
const crawler = new WebCrawler({
  config: {
    backend: { kind: "playwright" },
    browserType: "chromium", // or "firefox", "webkit"
    headless: true,
  },
});
```

--------------------------------

### Create CrawlerRunConfig

Source: https://www.usefeedstock.com/llms.txt

Configure per-crawl behavior, including caching, waiting conditions, and content extraction. Pass to crawl() or crawlMany().

```typescript
import { createCrawlerRunConfig, CacheMode } from "feedstock";

const config = createCrawlerRunConfig({
  cacheMode: CacheMode.Bypass,
  waitFor: { kind: "selector", value: "#loaded" },
  screenshot: true,
  excludeTags: ["nav", "footer", "aside"],
  cssSelector: "article",
});
```

--------------------------------

### Basic Web Crawl with Feedstock

Source: https://www.usefeedstock.com/llms.txt

Demonstrates a basic web crawl using Feedstock. Imports the WebCrawler and specifies cache bypass. Logs raw markdown, internal links, and images from the crawled page. Ensure to close the crawler when done.

```typescript
import { WebCrawler, CacheMode } from "feedstock";

const crawler = new WebCrawler();

const result = await crawler.crawl("https://example.com", {
  cacheMode: CacheMode.Bypass,
});

console.log(result.markdown?.rawMarkdown);
console.log(result.links.internal);
console.log(result.media.images);

await crawler.close();
```

--------------------------------

### Adding Rate Limiting and Robots.txt Parsing

Source: https://www.usefeedstock.com/llms.txt

Sets up a RateLimiter to control crawl speed and a RobotsParser to respect site policies. This is crucial for being a good web citizen.

```typescript
const rateLimiter = new RateLimiter({ baseDelay: 500 });
const robotsParser = new RobotsParser("my-site-crawler");
```

--------------------------------

### BrowserConfig Configuration

Source: https://www.usefeedstock.com/llms.txt

Defines the settings for the browser instance, including engine type, viewport, and proxy settings.

```APIDOC
## BrowserConfig

### Description
Controls the browser instance settings. This configuration is set once when initializing the WebCrawler.

### Request Body
- **browserType** (string) - Optional - Browser engine: "chromium", "firefox", or "webkit". Default: "chromium".
- **headless** (boolean) - Optional - Run without UI. Default: true.
- **viewport** (object) - Optional - Page viewport size { width, height }. Default: 1920x1080.
- **userAgent** (string) - Optional - Custom user agent string.
- **proxy** (object) - Optional - Proxy server configuration.
- **ignoreHttpsErrors** (boolean) - Optional - Ignore SSL errors. Default: true.
- **javaEnabled** (boolean) - Optional - Enable JavaScript. Default: true.
- **extraArgs** (string[]) - Optional - Extra browser launch arguments.
- **textMode** (boolean) - Optional - Text-only mode. Default: false.
- **backend** (object) - Optional - Browser backend. Default: { kind: "playwright" }.
- **verbose** (boolean) - Optional - Enable verbose logging. Default: false.

### Request Example
{
  "browserType": "chromium",
  "headless": true,
  "viewport": { "width": 1920, "height": 1080 },
  "userAgent": "my-bot/1.0"
}
```

--------------------------------

### Configure DomainFilter

Source: https://www.usefeedstock.com/llms.txt

Whitelist or blacklist specific domains. Blocked domains take priority over allowed domains.

```typescript
// Only crawl these domains
new DomainFilter({ allowed: ["example.com", "docs.example.com"] })

// Block specific domains
new DomainFilter({ blocked: ["ads.example.com", "tracker.io"] })
```

--------------------------------

### Crawl and Extract Product Data

Source: https://www.usefeedstock.com/llms.txt

Initialize a WebCrawler and use it to fetch and extract product data from a given URL based on the defined CSS schema. Ensure content is loaded using `waitFor` for JS-rendered pages.

```typescript
import { WebCrawler, CacheMode } from "feedstock";

const crawler = new WebCrawler();

const result = await crawler.crawl("https://store.example.com/products", {
  cacheMode: CacheMode.Bypass,
  waitFor: { kind: "selector", value: ".product-card" },
  extractionStrategy: { type: "css", params: schema },
});

const products = JSON.parse(result.extractedContent!)
  .map((item: { content: string }) => JSON.parse(item.content));

console.log(products);
// [
//   { title: "Widget Pro", price: "$29.99", image: "/img/widget.jpg", tags: ["new", "featured"] },
//   ...
// ]

await crawler.close();
```

--------------------------------

### Handle Pagination for Product Listings

Source: https://www.usefeedstock.com/llms.txt

Iterate through multiple pages of a product listing to extract data from all available pages. Accumulate results from each page into a single array.

```typescript
const allProducts = [];

for (let page = 1; page <= 5; page++) {
  const result = await crawler.crawl(
    `https://store.example.com/products?page=${page}`,
    {
      cacheMode: CacheMode.Bypass,
      extractionStrategy: { type: "css", params: schema },
    },
  );

  if (result.extractedContent) {
    const items = JSON.parse(result.extractedContent)
      .map((item: { content: string }) => JSON.parse(item.content));
    allProducts.push(...items);
  }
}

console.log(`Extracted ${allProducts.length} products`);
```

--------------------------------

### Direct Usage of CssExtractionStrategy

Source: https://www.usefeedstock.com/llms.txt

Instantiate and use the `CssExtractionStrategy` directly to extract data from a given URL and HTML content.

```APIDOC
### Direct Usage

```typescript
import { CssExtractionStrategy } from "feedstock";

const strategy = new CssExtractionStrategy({
  name: "articles",
  baseSelector: "article",
  fields: [
    { name: "headline", selector: "h2", type: "text" },
    { name: "body", selector: ".content", type: "html" },
  ],
});

const items = await strategy.extract(url, html);
```

Each extracted item includes both `content` (JSON string) and `metadata` (parsed object) for convenience.
```

--------------------------------

### Simulate User Behavior

Source: https://www.usefeedstock.com/llms.txt

Adds mouse movements and scrolling to a page session.

```typescript
import { simulateUser } from "feedstock";

crawler.setHook("afterGoto", async (page) => {
  await simulateUser(page);
});
```

--------------------------------

### Use No-Op Extraction Strategy

Source: https://www.usefeedstock.com/llms.txt

Import and instantiate `NoExtractionStrategy` to return HTML content as-is. This is the default behavior when no strategy is explicitly configured.

```typescript
import { NoExtractionStrategy } from "feedstock";

const strategy = new NoExtractionStrategy();
const items = await strategy.extract(url, html);
```

--------------------------------

### Direct Strategy Usage

Source: https://www.usefeedstock.com/llms.txt

Instantiates the CssExtractionStrategy directly to extract data from a provided URL and HTML content.

```typescript
import { CssExtractionStrategy } from "feedstock";

const strategy = new CssExtractionStrategy({
  name: "articles",
  baseSelector: "article",
  fields: [
    { name: "headline", selector: "h2", type: "text" },
    { name: "body", selector: ".content", type: "html" },
  ],
});

const items = await strategy.extract(url, html);
```

--------------------------------

### Perform Product Scraping

Source: https://www.usefeedstock.com/llms.txt

Demonstrates using the crawler with a CSS extraction strategy to parse product data from a website.

```typescript
const result = await crawler.crawl("https://store.example.com", {
  extractionStrategy: {
    type: "css",
    params: {
      name: "products",
      baseSelector: ".product-card",
      fields: [
        { name: "title", selector: ".product-title", type: "text" },
        { name: "price", selector: ".price", type: "text" },
        { name: "url", selector: "a.product-link", type: "attribute", attribute: "href" },
        { name: "image", selector: "img", type: "attribute", attribute: "src" },
        { name: "tags", selector: ".tag", type: "list" },
        { name: "description", selector: ".desc", type: "html" },
      ],
    },
  },
});

const products = JSON.parse(result.extractedContent!).map(
  (item) => JSON.parse(item.content)
);
// [{ title: "Widget A", price: "$9.99", url: "/widget-a", tags: ["sale", "new"] }, ...]
```

--------------------------------

### Import IdentityChunking

Source: https://www.usefeedstock.com/llms.txt

Import the IdentityChunking class from the feedstock library. This strategy returns the entire text as a single chunk.

```typescript
import { IdentityChunking } from "feedstock";
```

--------------------------------

### Initialize DomainAuthorityScorer

Source: https://www.usefeedstock.com/llms.txt

Assigns scores based on domain preference.

```typescript
new DomainAuthorityScorer(["example.com"], 1.5)
```

--------------------------------

### Accessing Markdown Output

Source: https://www.usefeedstock.com/llms.txt

Demonstrates how to access different Markdown formats from the crawl result.

```typescript
const result = await crawler.crawl("https://example.com");

// Raw markdown
console.log(result.markdown?.rawMarkdown);

// Citation-style (links as numbered references)
console.log(result.markdown?.markdownWithCitations);

// Just the references list
console.log(result.markdown?.referencesMarkdown);
```

--------------------------------

### Import RegexChunking

Source: https://www.usefeedstock.com/llms.txt

Import the RegexChunking class from the feedstock library. This strategy splits text based on provided regular expressions.

```typescript
import { RegexChunking } from "feedstock";
```

--------------------------------

### Turndown Configuration Options

Source: https://www.usefeedstock.com/llms.txt

Details the default Turndown configuration options used by Feedstock's markdown generator.

```APIDOC
## Configuration: Turndown Options

### Description
The default markdown generator in Feedstock utilizes the following Turndown options for HTML-to-Markdown conversion:

### Default Options
- **`headingStyle`**: Set to `"atx"`. This means headings will be formatted using hash symbols (e.g., `# H1`, `## H2`).
- **`codeBlockStyle`**: Set to `"fenced"`. Code blocks will be enclosed in triple backticks (```).
- **`bulletListMarker`**: Set to `"-"`. Unordered list items will use dashes (`-`) as markers.
```

--------------------------------

### Implement Custom Scraping Strategy

Source: https://www.usefeedstock.com/llms.txt

Create a custom scraping strategy by extending `ContentScrapingStrategy` and implementing the `scrape` method. This allows for custom HTML cleaning, link extraction, media extraction, and metadata extraction logic.

```typescript
import { ContentScrapingStrategy, type ScrapingResult } from "feedstock";

class MyScrapingStrategy extends ContentScrapingStrategy {
  scrape(url: string, html: string, config: CrawlerRunConfig): ScrapingResult {
    // Your custom scraping logic
    return {
      cleanedHtml: "...",
      success: true,
      media: { images: [], videos: [], audios: [] },
      links: { internal: [], external: [] },
      metadata: {},
    };
  }
}

const crawler = new WebCrawler({
  scrapingStrategy: new MyScrapingStrategy(),
});
```

--------------------------------

### Initialize BM25ContentFilter

Source: https://www.usefeedstock.com/llms.txt

Employ `BM25ContentFilter` for relevance-based filtering using BM25 scoring. Adjust `k1`, `b`, and `threshold` to fine-tune relevance detection. The filter returns content blocks scoring above the threshold for a given query.

```typescript
import { BM25ContentFilter } from "feedstock";

const filter = new BM25ContentFilter({
  k1: 1.5,        // term frequency saturation
  b: 0.75,         // document length normalization
  threshold: 0.1,  // minimum relevance score (0-1)
});

const relevant = filter.filter(content, "TypeScript web crawler");
```

--------------------------------

### FilterChain Configuration

Source: https://www.usefeedstock.com/llms.txt

Configuring a FilterChain to manage multiple URL filters with short-circuit evaluation.

```APIDOC
## FilterChain Configuration

### Description
Compose multiple filters into a chain. If any filter rejects a URL, the chain short-circuits and subsequent filters are not executed.

### Usage
```typescript
const chain = new FilterChain()
  .add(new DomainFilter({ allowed: ["example.com"] }))
  .add(new URLPatternFilter({ exclude: [/\/admin/, /\/login/] }))
  .add(new ContentTypeFilter());

const results = await crawler.deepCrawl(url, {}, { filterChain: chain });
```
```

--------------------------------

### Crawl with Default Cache

Source: https://www.usefeedstock.com/llms.txt

Perform a crawl operation using the default cache settings. The first crawl will be a 'miss', and subsequent identical crawls will be a 'hit'.

```typescript
// Default: cache enabled
const result = await crawler.crawl("https://example.com");
console.log(result.cacheStatus); // "miss" on first, "hit" on second
```

--------------------------------

### Initialize PruningContentFilter

Source: https://www.usefeedstock.com/llms.txt

Use `PruningContentFilter` to remove short blocks and low-quality patterns. Configure `minWords` to set the minimum word count for blocks to be kept.

```typescript
import { PruningContentFilter } from "feedstock";

const filter = new PruningContentFilter({
  minWords: 5,  // blocks with fewer words are removed
});

const cleaned = filter.filter(rawContent);
```