### Environment Variable Setup Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Shows how to configure the crawler using environment variables.

```bash
export CRAWLER_URL="https://example.com"
export CRAWLER_OUTPUT_FILE_NAME="env-output.json"
```

--------------------------------

### Install Dependencies and Start API Server

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

Install project dependencies and start the Express JS API server. The server runs on port 3000 by default.

```bash
npm install
npm run start:server
```

--------------------------------

### Example Usage with All Options

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Demonstrates how to use all available command-line options to configure a specific crawl, including the starting URL, link matching pattern, CSS selector, maximum pages, and output file name.

```bash
# With all options
gpt-crawler --url "https://docs.example.com" \
  --match "https://docs.example.com/**" \
  --selector ".content" \
  --maxPagesToCrawl 100 \
  --outputFileName docs.json
```

--------------------------------

### CLI Usage Examples

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Examples demonstrating how to use the GPT Crawler via the command-line interface.

```bash
# Example: Basic CLI command
gpt-crawler --url https://example.com --output output.json
```

--------------------------------

### Install Dependencies

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

Install the necessary project dependencies using npm.

```sh
npm i
```

--------------------------------

### Interactive Prompts Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Illustrates the interactive prompts that appear when the CLI is run without specifying required arguments. Users are prompted for the starting URL, URL pattern, and CSS selector.

```bash
gpt-crawler
Will prompt:
? What is the first URL of the website you want to crawl? https://example.com
? What is the URL pattern you want to match? https://example.com/**
? What is the CSS selector you want to match? .content
```

--------------------------------

### Configuration Object Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

A detailed example of a configuration object used for setting up the crawler.

```typescript
const config = {
  url: 'https://example.com',
  outputFileName: 'output.json',
  maxPagesToCrawl: 50,
  selector: '#content',
  onVisitPage: async (page) => {
    // Custom logic for each visited page
    console.log(`Processed: ${page.url}`);
  }
};
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Installs all necessary project dependencies using npm.

```bash
npm install
```

--------------------------------

### Server Configuration Example (.env file)

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

Example of a .env file used to configure the API server's port, host, and node environment.

```env
API_PORT=3000
API_HOST=0.0.0.0
NODE_ENV=production
```

--------------------------------

### Basic Crawl with Interactive Prompts

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Starts the GPT Crawler using 'npm start', which will trigger interactive prompts for necessary configuration if not provided.

```bash
npm start
# Interactive prompts will appear
```

--------------------------------

### GPTCrawlerCore Configuration Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/types.md

Demonstrates how to configure and initialize the GPTCrawlerCore with a Config object. Ensure Playwright is installed.

```typescript
import GPTCrawlerCore from './core.js';
import { Config } from './config.js';

const config: Config = {
  url: 'https://example.com/docs',
  match: 'https://example.com/docs/**',
  exclude: 'https://example.com/docs/admin/**',
  selector: '.content',
  maxPagesToCrawl: 100,
  outputFileName: 'output.json',
  resourceExclusions: ['png', 'jpg', 'svg', 'css'],
  maxFileSize: 10,
  maxTokens: 500000
};

const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
const outputPath = await crawler.write();
```

--------------------------------

### Starting GPT Crawler Server

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

Commands to start the GPT Crawler API server in development or production mode.

```bash
npm run start:server          # Development mode
```

```bash
npm run start:server:prod     # Production mode (no swagger generation)
```

--------------------------------

### Run the Crawler

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

Execute the crawler using the npm start command after configuring the settings.

```sh
npm start
```

--------------------------------

### POST /crawl Request Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

This example demonstrates how to make a POST request to the /crawl endpoint with a JSON payload specifying crawl configuration.

```bash
curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d '{ \
    "url": "https://example.com/docs", \
    "match": "https://example.com/docs/**", \
    "selector": ".content", \
    "maxPagesToCrawl": 50, \
    "outputFileName": "example-docs.json", \
    "resourceExclusions": ["png", "jpg", "svg"], \
    "maxFileSize": 10, \
    "maxTokens": 1000000 \
  }'
```

--------------------------------

### CLI Configuration Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configure GPT Crawler using command-line flags. Required options will prompt interactively if omitted.

```bash
gpt-crawler \
  --url "https://example.com/docs" \
  --match "https://example.com/docs/**" \
  --selector ".content" \
  --maxPagesToCrawl 50 \
  --outputFileName output.json
```

--------------------------------

### Start Server Listener

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Starts the Express server to listen on the configured host and port, logging the listening URL to the console.

```typescript
app.listen(port, hostname, () => {
  console.log(`API server listening at http://${hostname}:${port}`);
});
```

--------------------------------

### Run the Containerized Crawler

Source: https://github.com/builderio/gpt-crawler/blob/main/containerapp/README.md

Execute the run script to build and start the Dockerized crawler. Ensure Docker is installed and running.

```bash
. ./run.sh
```

--------------------------------

### Generate Swagger Documentation

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Generates Swagger/OpenAPI documentation by running the start:server script. This process creates `swagger-output.json` before the server starts.

```bash
npm run start:server  # Generates swagger-output.json before starting
```

--------------------------------

### write() Standalone Function Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Example of using the standalone `write` function to aggregate and save output files.

```typescript
import { write } from 'gpt-crawler';

await write('output.json');
```

--------------------------------

### Crawl with All Options Provided Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Executes a crawl with all configuration options specified directly on the command line, including URL, match pattern, selector, max pages, and output file name.

```bash
gpt-crawler \
  --url "https://docs.example.com" \
  --match "https://docs.example.com/**" \
  --selector ".markdown" \
  --maxPagesToCrawl 50 \
  --outputFileName output.json
```

--------------------------------

### Config Type Definition Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Shows the structure of the `Config` type, detailing its 12 properties.

```typescript
interface Config {
  url: string;
  match?: string;
  exclude?: string[];
  selector?: string;
  maxPagesToCrawl?: number;
  outputFileName?: string;
  cookie?: string;
  onVisitPage?: (page: Page) => Promise<void>;
  waitForSelectorTimeout?: number;
  resourceExclusions?: string[];
  maxFileSize?: number;
  maxTokens?: number;
}
```

--------------------------------

### Source Code Reference Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Illustrates the format for referencing source code locations within the documentation.

```plaintext
src/core.ts:51-159
```

--------------------------------

### Start GPT Crawler API Server

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Launch the API server for the GPT Crawler using the npm run script.

```bash
npm run start:server
```

--------------------------------

### Cookie Configuration Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/types.md

Shows how to provide an array of cookies within the Config object. Useful for authentication or consent.

```typescript
const config: Config = {
  // ... other config ...
  cookie: [
    { name: 'gdpr_consent', value: 'accepted' },
    { name: 'session_id', value: 'abc123xyz' }
  ]
};
```

--------------------------------

### HTTP Request/Response Example (POST /crawl)

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Illustrates a typical HTTP POST request to the /crawl endpoint and its expected response structure.

```json
{
  "url": "https://example.com",
  "outputFileName": "crawl-results.json"
}
```

```json
{
  "status": "success",
  "message": "Crawl initiated successfully."
}
```

--------------------------------

### Example Usage with Short Flags

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Shows an alternative way to specify crawl options using the short flag equivalents for URL, match pattern, and output file name.

```bash
# With short flags
gpt-crawler -u "https://docs.example.com" \
  -m "https://docs.example.com/**" \
  -s ".content" \
  -o docs.json
```

--------------------------------

### crawl() Standalone Function Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Example of using the standalone `crawl` function, likely with the Crawlee library.

```typescript
import { crawl } from 'gpt-crawler';

await crawl({
  url: 'https://example.com',
  outputFileName: 'crawl-output.json'
});
```

--------------------------------

### Run GPT Crawler with CLI

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Execute the crawler using npm start or directly with command-line arguments for URL and matching patterns.

```bash
npm start
# or
gpt-crawler -u "https://example.com" -m "https://example.com/**" -s ".content"
```

--------------------------------

### POST /crawl Response Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

This is an example of a successful JSON response from the /crawl endpoint, containing an array of crawled page objects.

```json
[
  {
    "title": "Getting Started - Example Docs",
    "url": "https://example.com/docs/getting-started",
    "html": "# Getting Started\n\nWelcome to our documentation..."
  },
  {
    "title": "API Reference - Example Docs",
    "url": "https://example.com/docs/api",
    "html": "# API Reference\n\nOur API provides the following endpoints..."
  }
]
```

--------------------------------

### Default Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

The default configuration used when running `npm start`. This provides a baseline for common crawl tasks.

```typescript
export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
  maxTokens: 2000000,
};
```

--------------------------------

### waitForXPath() Function Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Shows how to use the `waitForXPath` utility function to wait for a specific element.

```typescript
import { waitForXPath } from 'gpt-crawler';

// Assuming 'page' is a Playwright Page object
await waitForXPath(page, '//div[@id="my-element"]');
```

--------------------------------

### configSchema Validation Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Shows the usage of the `configSchema` for validating configuration objects.

```typescript
import { configSchema } from 'gpt-crawler';
import { z } from 'zod';

const validConfig: z.infer<typeof configSchema> = {
  url: 'https://example.com'
};

const invalidConfig = { url: 123 }; // This would fail validation
```

--------------------------------

### API Configuration Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configure GPT Crawler by sending a JSON payload in the request body to the `/crawl` endpoint. The `onVisitPage` callback is not supported via the API.

```bash
curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "match": "https://docs.example.com/**",
    "selector": ".content",
    "maxPagesToCrawl": 50,
    "outputFileName": "output.json",
    "resourceExclusions": ["png", "jpg"],
    "maxFileSize": 5
  }'
```

--------------------------------

### Callback Function Usage (onVisitPage)

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

An example of using the `onVisitPage` callback to execute custom logic for each visited page.

```typescript
const config = {
  // ... other config options
  onVisitPage: async (page) => {
    console.log(`Visiting page: ${page.url}`);
    // Add custom logic here, e.g., data extraction
  }
};
```

--------------------------------

### handler() Function Example (CLI Argument Handler)

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Illustrates the use of the `handler` function, likely for processing CLI arguments.

```typescript
import { handler } from 'gpt-crawler';

// Example usage within a CLI script
const args = handler(process.argv.slice(2));
```

--------------------------------

### Crawl and Wait for Completion Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Runs a crawl using short flags and waits for the crawl and write operations to complete synchronously. The output file path is then displayed in the console.

```bash
gpt-crawler -u "https://example.com" -m "https://example.com/**" -s ".content"
# CLI will execute crawl() and write() synchronously
# Output file path is displayed in console
```

--------------------------------

### Programmatic Configuration Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configure GPT Crawler by creating a Config object in your TypeScript/JavaScript code. The `onVisitPage` callback allows custom processing per page.

```typescript
import GPTCrawlerCore from './core.js';
import { Config } from './config.js';

const config: Config = {
  url: 'https://docs.example.com',
  match: 'https://docs.example.com/**',
  exclude: 'https://docs.example.com/admin/**',
  selector: '.markdown-body',
  maxPagesToCrawl: 100,
  outputFileName: 'docs-output.json',
  waitForSelectorTimeout: 5000,
  resourceExclusions: ['png', 'jpg', 'svg', 'css', 'woff2'],
  maxFileSize: 10,
  maxTokens: 1000000,
  cookie: [
    { name: 'consent', value: 'accepted' }
  ],
  onVisitPage: async ({ page, pushData }) => {
    // Custom processing per page
    const meta = await page.evaluate(() => ({
      canonicalUrl: document.querySelector('link[rel="canonical"]')?.href
    }));
    await pushData({ ...meta, timestamp: new Date().toISOString() });
  }
};

const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
const outputPath = await crawler.write();
```

--------------------------------

### Programmatic GPT Crawler Initialization and Crawl

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Initializes GPTCrawlerCore with configuration and starts the crawling process programmatically. Saves output to a JSON file.

```typescript
import GPTCrawlerCore from './core.js';

const config = {
  url: 'https://example.com/docs',
  match: 'https://example.com/docs/**',
  selector: '.content',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};

const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
const outputPath = await crawler.write();
```

--------------------------------

### API Error Response Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Example of a JSON response when an error occurs during crawling. Includes a general error message and detailed error information.

```json
{
  "message": "Error occurred during crawling",
  "error": "Detailed error information"
}
```

--------------------------------

### Basic Configuration Usage

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/config.md

An example of how to define a basic configuration object using the inferred Config TypeScript type. It includes essential fields like URL, match pattern, max pages, and output file name.

```typescript
import { Config } from './config.js';

const myConfig: Config = {
  url: 'https://example.com',
  match: 'https://example.com/**',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};
```

--------------------------------

### GET /api-docs

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

Serves the Swagger/OpenAPI documentation for the API, providing an interactive interface to test the /crawl endpoint.

```APIDOC
## GET /api-docs

### Description
Serves the Swagger/OpenAPI documentation for the API.

### Method
GET

### Endpoint
/api-docs

### Response
#### Success Response (200 OK)
- **Body** (text/html) - Interactive Swagger UI documentation page

### Notes
- This endpoint is automatically generated from the Swagger documentation file.
- Provides an interactive interface to test the `/crawl` endpoint.
- Documentation is served from `swagger-output.json`.
```

--------------------------------

### getPageHtml() Function Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Demonstrates the usage of the `getPageHtml` utility function to extract page content.

```typescript
import { getPageHtml } from 'gpt-crawler';

const html = await getPageHtml('https://example.com');
```

--------------------------------

### Error Handling Example (500 Status)

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Demonstrates how the API server handles internal errors, returning a 500 status code.

```json
{
  "status": "error",
  "message": "Internal server error during crawl process."
}
```

--------------------------------

### Cookie Type Definition Example

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Defines the structure of the `Cookie` type.

```typescript
interface Cookie {
  name: string;
  value: string;
  // ... other potential fields like domain, path, expires
}
```

--------------------------------

### GPTCrawlerCore Constructor

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Initializes a new GPTCrawlerCore instance with the provided configuration. The configuration object dictates the crawling behavior, including the starting URL, matching patterns, and output file naming.

```APIDOC
## GPTCrawlerCore Constructor

### Description
Initializes a new GPTCrawlerCore instance with the provided configuration.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **config** (Config) - Required - Configuration object for the crawler

### Request Example
```typescript
import GPTCrawlerCore from './core.js';

const config = {
  url: 'https://example.com/docs',
  match: 'https://example.com/docs/**',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};

const crawler = new GPTCrawlerCore(config);
```

### Response
None

### Error Handling
None explicitly documented for constructor.
```

--------------------------------

### Config with Cookie Usage

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/config.md

Illustrates how to include cookie information within the configuration object. This example shows an array of Cookie objects being passed to the 'cookie' field.

```typescript
const config: Config = {
  // ...
  cookie: [
    { name: 'gdpr_consent', value: 'accepted' },
    { name: 'session_id', value: 'abc123' }
  ]
};
```

--------------------------------

### Clone GPT Crawler Repository

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

Clone the repository to your local machine. Ensure you have Node.js version 16 or higher installed.

```sh
git clone https://github.com/builderio/gpt-crawler
```

--------------------------------

### Sitemap-based Crawl Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configuration for crawling an entire site starting from a sitemap XML file, specifying limits for pages, file size, and tokens.

```typescript
const config: Config = {
  url: 'https://example.com/sitemap.xml',
  match: 'https://example.com/**',
  selector: '.content',
  maxPagesToCrawl: 1000,
  outputFileName: 'full-site.json',
  maxFileSize: 50,
  maxTokens: 5000000
};
```

--------------------------------

### Execute Web Crawl

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Initiate the crawling process using the crawl() method. This method starts from the configured URL, follows matching links, extracts content, and stores data. It returns a promise that resolves upon completion.

```typescript
const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
```

--------------------------------

### Configure Crawler Settings

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

Configure the crawler by editing the `config.ts` file. Specify the starting URL, matching pattern, content selector, and output file name.

```typescript
export const defaultConfig: Config = {
  url: "https://www.builder.io/c/docs/developers",
  match: "https://www.builder.io/c/docs/**",
  selector: ".docs-builder-container",
  maxPagesToCrawl: 50,
  outputFileName: "output.json",
};
```

--------------------------------

### Error Response Example for /crawl

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

This JSON structure represents an error response from the /crawl endpoint, including a general message and specific error details.

```json
{
  "message": "Error occurred during crawling",
  "error": "Error details here"
}
```

--------------------------------

### GPTCrawlerCore.crawl()

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Executes the web crawling process based on the configured parameters. It starts from the specified URL, follows matching links, extracts content, and stores the data.

```APIDOC
## GPTCrawlerCore.crawl()

### Description
Executes the web crawling process according to the configured parameters. The crawler will start from the configured URL, follow links matching the configured pattern, extract content using the configured selector, and store extracted data to the storage directory.

### Method
`async crawl(): Promise<void>`

### Parameters
None

### Request Example
```typescript
const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
```

### Response
* **Promise<void>** - Resolves when the crawl is complete.

### Error Handling
Throws validation errors if the configuration is invalid.
```

--------------------------------

### Program Configuration from package.json

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Configures the CLI program with version and description metadata loaded from the project's package.json file.

```typescript
const { version, description } = require("../../package.json");
program.version(version).description(description);
```

--------------------------------

### Displaying Version and Help

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Commands to display the GPT Crawler's version and help information, which are configured from package.json.

```bash
gpt-crawler --version
gpt-crawler --help
```

--------------------------------

### Run GPT Crawler with Options via CLI

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Executes the GPT Crawler from the command line with specified URL, match pattern, CSS selector, and maximum pages to crawl.

```bash
gpt-crawler --url "https://example.com/docs" \
  --match "https://example.com/docs/**" \
  --selector ".content" \
  --maxPagesToCrawl 50
```

--------------------------------

### Basic Documentation Site Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

A simple configuration for crawling a basic documentation site, specifying URL, match pattern, CSS selector, and page limits.

```typescript
const config: Config = {
  url: 'https://docs.example.com',
  match: 'https://docs.example.com/**',
  selector: '.main-content',
  maxPagesToCrawl: 100,
  outputFileName: 'docs.json'
};
```

--------------------------------

### TypeScript Type Annotations

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Demonstrates the use of TypeScript type annotations for configuration and other elements.

```typescript
import { Config } from 'gpt-crawler';

const config: Config = {
  url: 'https://example.com',
  match: '**/page/**',
  exclude: ['**/private/**'],
  selector: 'main',
  maxPagesToCrawl: 100,
  outputFileName: 'output.json',
  cookie: 'session=12345',
  onVisitPage: async (page) => {
    console.log(`Visiting ${page.url}`);
  },
  waitForSelectorTimeout: 5000,
  resourceExclusions: ['.css', '.js'],
  maxFileSize: 1024 * 1024, // 1MB
  maxTokens: 4096
};
```

--------------------------------

### Programmatic Initialization and Usage

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Illustrates how to initialize and use the GPT Crawler programmatically within a Node.js application.

```typescript
import { GPTCrawlerCore } from 'gpt-crawler';

const crawler = new GPTCrawlerCore({
  url: 'https://example.com',
  outputFileName: 'output.json'
});

await crawler.crawl();
await crawler.write();
```

--------------------------------

### Navigate to Container App Directory

Source: https://github.com/builderio/gpt-crawler/blob/main/containerapp/README.md

Change the current directory to the container app's location.

```bash
cd gpt-crawler/containerapp
```

--------------------------------

### Initialize Express Server

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Initializes the Express application, sets the port and hostname from environment variables.

```typescript
const app: Express = express();
const port = Number(process.env.API_PORT) || 3000;
const hostname = process.env.API_HOST || "localhost";
```

--------------------------------

### GPTCrawlerCore Class Initialization

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Demonstrates the initialization of the main GPTCrawlerCore class.

```typescript
import { GPTCrawlerCore } from 'gpt-crawler';

const crawler = new GPTCrawlerCore({
  url: 'https://example.com',
  outputFileName: 'output.json'
});
```

--------------------------------

### Basic Documentation Crawl Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Defines a basic configuration object for crawling documentation websites. Specifies the target URL, matching pattern, content selector, and output file name.

```typescript
const config = {
  url: 'https://docs.example.com',
  match: 'https://docs.example.com/**',
  selector: '.main-content',
  maxPagesToCrawl: 100,
  outputFileName: 'docs.json'
};
```

--------------------------------

### Large Documentation with Size Limits Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configuration for crawling large API documentation sites, including exclude patterns, size limits, token limits, and resource exclusions.

```typescript
const config: Config = {
  url: 'https://api.example.com/docs',
  match: 'https://api.example.com/docs/**',
  exclude: ['https://api.example.com/docs/internal/**'],
  selector: 'article',
  maxPagesToCrawl: 200,
  outputFileName: 'api-docs.json',
  maxFileSize: 10,
  maxTokens: 2000000,
  resourceExclusions: ['png', 'jpg', 'svg', 'css', 'woff']
};
```

--------------------------------

### CLI Command-Line Options

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

Specifies the available command-line flags for configuring the GPT Crawler, including URL, matching patterns, selectors, crawl limits, and output file names.

```bash
gpt-crawler --url <string> --match <string> --selector <string> --maxPagesToCrawl <number> --outputFileName <string>
```

--------------------------------

### Blog with Cookie Consent Configuration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/configuration.md

Configuration for crawling a blog, including specific post matching, selector, and handling cookie consent with a timeout.

```typescript
const config: Config = {
  url: 'https://blog.example.com',
  match: 'https://blog.example.com/posts/**',
  selector: '.post-content',
  maxPagesToCrawl: 50,
  outputFileName: 'blog.json',
  cookie: { name: 'gdpr_consent', value: 'accepted' },
  waitForSelectorTimeout: 3000
};
```

--------------------------------

### Configure Large Site Crawl with Limits

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Set up crawler configuration for large websites, specifying URL, matching patterns, content selector, page limits, output file name, file size limits, token limits, and resource exclusions.

```typescript
const config = {
  url: 'https://api.example.com/docs',
  match: 'https://api.example.com/docs/**',
  selector: 'article',
  maxPagesToCrawl: 200,
  outputFileName: 'api-docs.json',
  maxFileSize: 10,        // 10 MB per file
  maxTokens: 2000000,     // 2M tokens per file
  resourceExclusions: ['png', 'jpg', 'css', 'woff']
};
```

--------------------------------

### Extract Page HTML with Selector

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Extract text content from a Playwright page using either CSS selectors or XPath. If the selector starts with '/', it's treated as XPath; otherwise, it's a CSS selector. Defaults to 'body' if no selector is provided.

```typescript
import { getPageHtml } from './core.js';

// Using CSS selector
const content = await getPageHtml(page, '.documentation');

// Using XPath
const content = await getPageHtml(page, '//article[@class="main"]');
```

--------------------------------

### Initialize GPTCrawlerCore

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Instantiate the GPTCrawlerCore with a configuration object. Ensure the config object includes necessary parameters like url, match, maxPagesToCrawl, and outputFileName.

```typescript
import GPTCrawlerCore from './core.js';

const config = {
  url: 'https://example.com/docs',
  match: 'https://example.com/docs/**',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};

const crawler = new GPTCrawlerCore(config);
```

--------------------------------

### GPTCrawlerCore crawl() and write() Methods

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

Shows how to execute a crawl and write the results using the GPTCrawlerCore instance.

```typescript
await crawler.crawl();
await crawler.write();
```

--------------------------------

### write()

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Processes crawled data and writes it to output files, handling data aggregation, splitting, and file size/token limits.

```APIDOC
## write() API

### Description
Processes crawled data and writes it to output files. This function is used internally by GPTCrawlerCore.

### Method
`async function write(config: Config): Promise<PathLike>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **config** (`Config`) - Required - Configuration object specifying output options.

### Request Example
```typescript
import { write } from './core.js';

const config = {
  outputFileName: 'output.json',
  maxFileSize: 5,          // 5 MB limit
  maxTokens: 1000000       // Token limit
};

const outputPath = await write(config);
console.log(`Data written to: ${outputPath}`);
```

### Response
#### Success Response (PathLike)
Returns the file path of the last generated output file.

#### Response Example
`"path/to/output-1.json"`

### Behavior
- Reads all JSON files from `storage/datasets/default/`
- Combines data while respecting token and file size limits
- Splits into multiple files if limits are exceeded
- Outputs files in the format `{outputFileName}-{n}.json` if split
- Logs progress to console

### Token/Size Handling
- If `maxTokens` is configured, uses GPT tokenizer to count tokens
- If `maxFileSize` is configured, splits output by file size (in MB)
- Applies both limits independently
```

--------------------------------

### File Organization Structure

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Illustrates the directory structure of the GPT Crawler project, showing the location of key documentation files and module source code.

```tree
output/
├── README.md                 # Project overview
├── INDEX.md                  # This file
├── types.md                  # Type definitions
├── endpoints.md              # HTTP endpoint documentation
├── configuration.md          # Configuration guide
└── api-reference/
    ├── core.md              # Core module API
    ├── config.md            # Config module API
    ├── cli.md               # CLI module API
    └── server.md            # Server module API
```

--------------------------------

### Basic CLI Invocation

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

The CLI is invoked using the 'gpt-crawler' command. This is the primary entry point for running the crawler from the terminal.

```bash
gpt-crawler [options]
```

--------------------------------

### GPT Crawler Configuration Options

Source: https://github.com/builderio/gpt-crawler/blob/main/README.md

An overview of common configuration options for the GPT Crawler, including URL, matching patterns, selectors, crawl limits, and file naming.

```typescript
type Config = {
  /** URL to start the crawl, if sitemap is provided then it will be used instead and download all pages in the sitemap */
  url: string;
  /** Pattern to match against for links on a page to subsequently crawl */
  match: string;
  /** Selector to grab the inner text from */
  selector: string;
  /** Don't crawl more than this many pages */
  maxPagesToCrawl: number;
  /** File name for the finished data */
  outputFileName: string;
  /** Optional resources to exclude
   *
   * @example
   * ['png','jpg','jpeg','gif','svg','css','js','ico','woff','woff2','ttf','eot','otf','mp4','mp3','webm','ogg','wav','flac','aac','zip','tar','gz','rar','7z','exe','dmg','apk','csv','xls','xlsx','doc','docx','pdf','epub','iso','dmg','bin','ppt','pptx','odt','avi','mkv','xml','json','yml','yaml','rss','atom','swf','txt','dart','webp','bmp','tif','psd','ai','indd','eps','ps','zipx','srt','wasm','m4v','m4a','webp','weba','m4b','opus','ogv','ogm','oga','spx','ogx','flv','3gp','3g2','jxr','wdp','jng','hief','avif','apng','avifs','heif','heic','cur','ico','ani','jp2','jpm','jpx','mj2','wmv','wma','aac','tif','tiff','mpg','mpeg','mov','avi','wmv','flv','swf','mkv','m4v','m4p','m4b','m4r','m4a','mp3','wav','wma','ogg','oga','webm','3gp','3g2','flac','spx','amr','mid','midi','mka','dts','ac3','eac3','weba','m3u','m3u8','ts','wpl','pls','vob','ifo','bup','svcd','drc','dsm','dsv','dsa','dss','vivo','ivf','dvd','fli','flc','flic','flic','mng','asf','m2v','asx','ram','ra','rm','rpm','roq','smi','smil','wmf','wmz','wmd','wvx','wmx','movie','wri','ins','isp','acsm','djvu','fb2','xps','oxps','ps','eps','ai','prn','svg','dwg','dxf','ttf','fnt','fon','otf','cab']
   */
  resourceExclusions?: string[];
  /** Optional maximum file size in megabytes to include in the output file */
  maxFileSize?: number;
  /** Optional maximum number tokens to include in the output file */
  maxTokens?: number;
};
```

--------------------------------

### Configuration Validation with Zod

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/types.md

Demonstrates how to validate a user-provided configuration object using the Zod schema. This ensures that all configuration parameters meet the required criteria before being used by the crawler.

```typescript
import { configSchema } from './config.js';

try {
  const validatedConfig = configSchema.parse(userProvidedConfig);
  // Config is now validated
} catch (error) {
  console.error('Configuration validation failed:', error);
}
```

--------------------------------

### POST /crawl

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/endpoints.md

Executes a web crawl with the provided configuration and returns the aggregated crawled data. This endpoint performs the full crawl and data processing synchronously.

```APIDOC
## POST /crawl

### Description
Executes a web crawl with the provided configuration and returns the aggregated crawled data.

### Method
POST

### Endpoint
/crawl

### Parameters
#### Request Body
- **url** (string) - Yes - URL to start crawl or sitemap XML URL
- **match** (string | string[]) - Yes - URL pattern(s) to match against
- **exclude** (string | string[]) - No - URL pattern(s) to exclude from crawling
- **selector** (string) - No - CSS selector or XPath to extract content
- **maxPagesToCrawl** (number) - Yes - Maximum pages to crawl
- **outputFileName** (string) - Yes - Output file name
- **cookie** (object | object[]) - No - Cookie(s) to set during crawl
- **waitForSelectorTimeout** (number) - No - Selector timeout in ms
- **resourceExclusions** (string[]) - No - File extensions to exclude
- **maxFileSize** (number) - No - Max output file size in MB
- **maxTokens** (number) - No - Max token count for output

### Request Example
```bash
curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d 
    "{
    "url": "https://example.com/docs",
    "match": "https://example.com/docs/**",
    "selector": ".content",
    "maxPagesToCrawl": 50,
    "outputFileName": "example-docs.json",
    "resourceExclusions": ["png", "jpg", "svg"],
    "maxFileSize": 10,
    "maxTokens": 1000000
  }"
```

### Response
#### Success Response (200 OK)
- **Body** (Array of page data objects)
  - **title** (string) - The page title extracted from \<title\> tag
  - **url** (string) - The final loaded URL of the page
  - **html** (string) - The extracted text content based on the selector

#### Response Example
```json
[
  {
    "title": "Getting Started - Example Docs",
    "url": "https://example.com/docs/getting-started",
    "html": "# Getting Started\n\nWelcome to our documentation..."
  },
  {
    "title": "API Reference - Example Docs",
    "url": "https://example.com/docs/api",
    "html": "# API Reference\n\nOur API provides the following endpoints..."
  }
]
```

### Error Response
#### Status Code
500 Internal Server Error

#### Body
```json
{
  "message": "Error occurred during crawling",
  "error": "Error details here"
}
```

### Notes
- The response size is limited by the `maxFileSize` and `maxTokens` configuration.
- The endpoint does not support the `onVisitPage` callback function.
- CORS is enabled for all origins.
- Crawling may take several seconds to minutes depending on the site and page count.
```

--------------------------------

### Execute Web Crawl via API

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Send a POST request to the /crawl endpoint with a JSON configuration object to initiate a web crawl and retrieve aggregated data. The response includes crawled pages with title, URL, and extracted content.

```bash
curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d '{ \
    "url": "https://example.com", \
    "match": "https://example.com/**", \
    "selector": ".content", \
    "maxPagesToCrawl": 50, \
    "outputFileName": "output.json" \
  }'
```

--------------------------------

### GPTCrawlerCore Integration

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Demonstrates the integration of GPTCrawlerCore within the server's request handling, showing instantiation, crawling, and writing phases.

```typescript
const crawler = new GPTCrawlerCore(validatedConfig);
await crawler.crawl();
const outputFileName: PathLike = await crawler.write();
```

--------------------------------

### Execute Crawling Process

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Use this standalone function to initiate the web crawling process. It validates configuration, sets up a PlaywrightCrawler, and handles various crawling scenarios including sitemaps and cookie application. Ensure your configuration object is valid.

```typescript
async function crawl(config: Config): Promise<void>
```

```typescript
import { crawl } from './core.js';

const config = {
  url: 'https://example.com',
  match: 'https://example.com/**',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};

await crawl(config);
```

--------------------------------

### Make API Request to GPT Crawler

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Send a POST request to the crawler's API endpoint to initiate a crawl with specified parameters.

```bash
curl -X POST http://localhost:3000/crawl -H "Content-Type: application/json" -d '{"url":"...","match":"...","maxPagesToCrawl":50,"outputFileName":"output.json"}'
```

--------------------------------

### crawl()

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Executes the crawling process using Crawlee's PlaywrightCrawler. It validates configuration, sets up the crawler, and handles various crawling scenarios.

```APIDOC
## crawl() API

### Description
Executes the crawling process using Crawlee's PlaywrightCrawler. This function is used internally by GPTCrawlerCore.

### Method
`async function crawl(config: Config): Promise<void>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **config** (`Config`) - Required - Configuration object for the crawler.

### Request Example
```typescript
import { crawl } from './core.js';

const config = {
  url: 'https://example.com',
  match: 'https://example.com/**',
  maxPagesToCrawl: 50,
  outputFileName: 'output.json'
};

await crawl(config);
```

### Response
#### Success Response (void)
Resolves when crawling is complete.

#### Response Example
None (Promise<void>)

### Behavior
- Validates configuration using Zod schema
- Creates a PlaywrightCrawler with the provided settings
- Handles both regular URLs and XML sitemaps
- Applies cookies if configured
- Excludes specified resource types (images, stylesheets, etc.)
- Stores extracted pages to `storage/datasets/default/`

### Throws
Throws Zod validation errors if the configuration is invalid.
```

--------------------------------

### Configure Crawl with Authentication

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/README.md

Configure the crawler for sites requiring authentication, including URL, matching patterns, content selector, page limits, output file name, and authentication cookies.

```typescript
const config = {
  url: 'https://internal.example.com/docs',
  match: 'https://internal.example.com/docs/**',
  selector: '.content',
  maxPagesToCrawl: 50,
  outputFileName: 'internal-docs.json',
  cookie: { name: 'auth_token', value: 'secret123' }
};
```

--------------------------------

### Module Dependency Graph

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Visualizes the dependencies between different modules of the GPT Crawler project, including main entry points and external libraries used.

```graphviz
main.ts
  ↓
config.ts (defaultConfig) → core.ts
                           ↘
cli.ts → config.ts → core.ts
         ↘ inquirer
           commander

server.ts → config.ts → core.ts
          ↘ express
            cors
            swagger-ui-express
```

--------------------------------

### Express Application Instance

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/MANIFEST.md

References the `app` variable, which represents the Express application instance.

```typescript
import { app } from 'gpt-crawler/server';

// Example: Starting the server
const port = process.env.PORT || 3000;
app.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});
```

--------------------------------

### GPTCrawlerCore.write()

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Processes the crawled data and writes it to output file(s). It aggregates data, respects file size and token limits, and can split output into multiple files.

```APIDOC
## GPTCrawlerCore.write()

### Description
Processes crawled data and writes it to output file(s). This function aggregates JSON files from the storage directory, respects configured file size and token limits, and splits output into multiple files if limits are exceeded.

### Method
`async write(): Promise<PathLike>`

### Parameters
None

### Request Example
```typescript
const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
const outputPath = await crawler.write();
console.log(`Output written to: ${outputPath}`);
```

### Response
* **Promise<PathLike>** - The file path of the generated output file (or the last file if multiple were created).

### Error Handling
None explicitly documented.
```

--------------------------------

### POST /crawl Endpoint Handler

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Handles POST requests to the /crawl endpoint. It validates the configuration, initiates the crawl, writes the output, and returns the crawled data.

```typescript
app.post("/crawl", async (req, res) => {
  const config: Config = req.body;
  try {
    const validatedConfig = configSchema.parse(config);
    const crawler = new GPTCrawlerCore(validatedConfig);
    await crawler.crawl();
    const outputFileName: PathLike = await crawler.write();
    const outputFileContent = await readFile(outputFileName, "utf-8");
    res.contentType("application/json");
    return res.send(outputFileContent);
  } catch (error) {
    return res
      .status(500)
      .json({ message: "Error occurred during crawling", error });
  }
});
```

--------------------------------

### Process and Write Crawled Data

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/core.md

Use this standalone function to process data collected during crawling and write it to output files. It handles combining data, respecting token and file size limits, and splitting output into multiple files if necessary. Configure `outputFileName`, `maxFileSize`, and `maxTokens` as needed.

```typescript
async function write(config: Config): Promise<PathLike>
```

```typescript
import { write } from './core.js';

const config = {
  outputFileName: 'output.json',
  maxFileSize: 5,          // 5 MB limit
  maxTokens: 1000000       // Token limit
};

const outputPath = await write(config);
console.log(`Data written to: ${outputPath}`);
```

--------------------------------

### Load Environment Variables with Dotenv

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/config.md

Load environment variables from a .env file into process.env using configDotenv from the dotenv library. This is essential for configuring server settings and crawling behavior.

```typescript
import { configDotenv } from "dotenv";
configDotenv();
```

--------------------------------

### Export Express App Instance

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Exports the configured Express application instance as the default export.

```typescript
export default app;
```

--------------------------------

### Custom Page Visit Logic

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/config.md

Implement custom logic within the onVisitPage hook to access page content, extract custom data using page.evaluate, and push this data along with metadata like title, URL, and timestamp.

```typescript
const config: Config = {
  // ... other config
  onVisitPage: async ({ page, pushData }) => {
    // Access page content
    const title = await page.title();
    const url = page.url();
    
    // Extract custom data
    const customData = await page.evaluate(() => {
      return {
        wordCount: document.body.innerText.split(/\s+/).length,
        headingCount: document.querySelectorAll('h1, h2, h3').length
      };
    });
    
    // Save custom data
    await pushData({
      title,
      url,
      ...customData,
      timestamp: new Date().toISOString()
    });
  }
};
```

--------------------------------

### Configure Server Middleware

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/server.md

Configures essential middleware for the Express server, including CORS, JSON body parsing, and Swagger UI.

```typescript
app.use(cors());                  // Enable CORS for all origins
app.use(express.json());          // Parse JSON request bodies
app.use("/api-docs", swaggerUi.serve, swaggerUi.setup(swaggerDocument));
```

--------------------------------

### Use GPT Crawler as a Library

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/INDEX.md

Integrate GPT Crawler into your TypeScript project by instantiating GPTCrawlerCore and calling crawl and write methods.

```typescript
const crawler = new GPTCrawlerCore(config);
await crawler.crawl();
await crawler.write();
```

--------------------------------

### handler() Function Signature

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/cli.md

The internal handler function that processes CLI arguments and executes the crawl. It accepts a Config object containing parsed options.

```typescript
async function handler(options: Config): Promise<void>
```

--------------------------------

### onVisitPage Callback Type

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/api-reference/config.md

Defines the signature for the optional `onVisitPage` callback function. This function is executed for each page visited and receives a context object containing the Playwright page and a data pushing utility.

```typescript
type onVisitPage = (context: {
  page: Page;
  pushData: (data: any) => Promise<void>;
}) => Promise<void>;
```

--------------------------------

### Config Type Definition

Source: https://github.com/builderio/gpt-crawler/blob/main/_autodocs/types.md

Defines the main configuration object for the crawler. This type is inferred from the Zod schema.

```typescript
type Config = {
  url: string;
  match: string | string[];
  exclude?: string | string[];
  selector?: string;
  maxPagesToCrawl: number;
  outputFileName: string;
  cookie?: Cookie | Cookie[];
  onVisitPage?: (context: { page: Page; pushData: (data: any) => Promise<void> }) => Promise<void>;
  waitForSelectorTimeout?: number;
  resourceExclusions?: string[];
  maxFileSize?: number;
  maxTokens?: number;
}
```