fastCRW

https://github.com/us/crw
Admin
fastCRW is an open-source web scraper and crawler built for AI agents, offering Firecrawl-compatible...
Tokens:108,335
Snippets:1,330
Trust Score:9.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Install and Start LightPanda

Source: https://github.com/us/crw/blob/main/docs/docs/js-rendering.md

Use these commands to automatically install and start the LightPanda CDP backend for JS rendering.

```bash
# Auto-install
crw-server setup
```

```bash
# Manual start
lightpanda serve --host 127.0.0.1 --port 9222 &
```

--------------------------------

### Setup LightPanda and Start crw-server

Source: https://context7.com/us/crw/llms.txt

These commands set up LightPanda, the fastest CDP backend for fastCRW, and then start both LightPanda and the crw-server. LightPanda is downloaded to `~/.local/bin/lightpanda`.

```bash
# Setup LightPanda (fastest CDP backend)
crw-server setup   # downloads LightPanda to ~/.local/bin/lightpanda

# Start LightPanda + crw-server
lightpanda serve --host 127.0.0.1 --port 9222 &
crw-server
```

--------------------------------

### Install and Launch crw-browse

Source: https://github.com/us/crw/blob/main/docs/mcp/index.html

Install the crw-browse tool from crates.io or download a prebuilt binary. Launch it to start a browser automation server.

```bash
# From crates.io
car go install crw-browse
crw-browse

# Or grab a prebuilt binary from the v0.4.0 GitHub release
# https://github.com/us/crw/releases/tag/v0.4.0
```

--------------------------------

### Install and Run LightPanda

Source: https://github.com/us/crw/blob/main/docs/js-rendering/index.html

Use this command to automatically install LightPanda. Alternatively, start the LightPanda server manually.

```bash
crw-server setup
```

```bash
lightpanda serve --host 127.0.0.1 --port 9222 &
```

--------------------------------

### Quick Start Pattern - Search

Source: https://github.com/us/crw/blob/main/docs/agent-onboarding/index.html

Example of searching the web for relevant information.

```APIDOC
3.  **Search** when you need to find relevant pages across the web:

    ```bash
    curl -X POST https://fastcrw.com/api/v1/search \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"query": "your search query", "limit": 5}'
    ```
```

--------------------------------

### Quick Start Pattern - Map and Scrape

Source: https://github.com/us/crw/blob/main/docs/agent-onboarding/index.html

Example of discovering pages on a domain and then extracting content from a specific page.

```APIDOC
## Quick Start Pattern

For most agent workflows, start with this sequence:

1.  **Discover** what pages exist on a target domain:

    ```bash
    curl -X POST https://fastcrw.com/api/v1/map \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://example.com"}'
    ```

2.  **Extract** content from specific pages:

    ```bash
    curl -X POST https://fastcrw.com/api/v1/scrape \
      -H "Authorization: Bearer $API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"url": "https://example.com/page", "formats": ["markdown"]}'
    ```
```

--------------------------------

### One-Line Install Script with wget

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Alternative one-line installation script using wget.

```bash
wget -qO- https://raw.githubusercontent.com/us/crw/main/install.sh | sh
```

--------------------------------

### Install crw-cli

Source: https://github.com/us/crw/blob/main/docs/crates/index.html

Install the `crw-cli` as a standalone binary for scraping URLs directly to markdown, JSON, or plain text without a server. Usage examples are provided for different output formats and file output.

```bash
cargo install crw-cli
```

```bash
crw https://example.com              # markdown to stdout
crw https://example.com --format json
crw https://example.com -o page.md

```

--------------------------------

### Use crw-server as a library in Rust

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Example of how to integrate crw-server into a Rust application using tokio and axum. This snippet shows the basic setup for creating the application and starting the server.

```rust
use crw_server::app::create_app;
use crw_server::state::AppState;

#[tokio::main]
async fn main() {
    let config = crw_core::AppConfig::load().unwrap();
    let state = AppState::new(config).await;
    let app = create_app(state);
    let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await.unwrap();
    axum::serve(listener, app).await.unwrap();
}
```

--------------------------------

### One-Line Install Script

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Recommended one-line installation script that auto-detects OS and architecture. Uses curl.

```bash
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh
```

--------------------------------

### Install crw-server

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Install the crw-server binary using Cargo.

```bash
cargo install crw-server
```

--------------------------------

### Install fastCRW Binary

Source: https://context7.com/us/crw/llms.txt

Installs the fastCRW binary, including the server and MCP components. Supports OS auto-detection or specific binary installation. Homebrew, APT, and Cargo installation methods are also provided.

```bash
# Auto-detects OS and architecture, installs crw + crw-server + crw-mcp
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh
```

```bash
# Install specific binary
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh
```

```bash
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw-server sh
```

```bash
# Homebrew
brew install us/crw/crw
```

```bash
brew install us/crw/crw-server
```

```bash
# APT (Debian/Ubuntu)
curl -fsSL https://apt.fastcrw.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/crw.gpg
echo "deb [signed-by=/usr/share/keyrings/crw.gpg] https://apt.fastcrw.com stable main" \
  | sudo tee /etc/apt/sources.list.d/crw.list
sudo apt update && sudo apt install crw
```

```bash
# Cargo
cargo install crw-cli
```

```bash
cargo install crw-server
```

```bash
cargo install crw-mcp
```

```bash
# Python SDK (auto-downloads crw-mcp binary on first use)
pip install crw
```

```bash
# Docker
docker run -p 3000:3000 ghcr.io/us/crw:latest
```

```bash
# Docker Compose (includes LightPanda JS-rendering sidecar)
docker compose up -d
```

--------------------------------

### Install and Launch crw-browse

Source: https://github.com/us/crw/blob/main/docs/docs/mcp.md

Install the crw-browse tool using cargo or download a prebuilt binary. Launch it as a self-hosted binary for browser automation.

```bash
# From crates.io
carp install crw-browse
crw-browse

# Or grab a prebuilt binary from the v0.4.0 GitHub release
# https://github.com/us/crw/releases/tag/v0.4.0
```

--------------------------------

### Start crw-server

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Start the crw-server with default settings. This command also enables JS rendering by downloading LightPanda if not already present.

```bash
crw-server

crw-server setup
```

--------------------------------

### Install crw-cli

Source: https://github.com/us/crw/blob/main/crates/crw-cli/README.md

Installs the crw binary using Cargo. Ensure you have Rust and Cargo installed.

```bash
cargo install crw-cli
```

--------------------------------

### Install CRW MCP Server (npm)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW MCP server using npm for zero installation. This is the recommended method for AI agents.

```bash
npx crw-mcp                           # zero install (npm)
```

--------------------------------

### Start crw Server

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Starts the crw server after LightPanda has been initiated.

```bash
crw-server
```

--------------------------------

### Docker Compose Setup for CRW

Source: https://github.com/us/crw/blob/main/docs/docker/index.html

Clone the CRW repository and use Docker Compose to start the CRW API server and the Lightpanda service for JavaScript rendering.

```bash
git clone https://github.com/us/crw.git
cd crw
docker compose up
```

--------------------------------

### Verify MCP Installation with Initialize Command

Source: https://github.com/us/crw/blob/main/docs/docs/mcp.md

Use this command to verify that the MCP client is installed and responding correctly by sending an initialization request.

```bash
echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"clientInfo":{"name":"test"},"protocolVersion":"2024-11-05"}}' \
  | crw-mcp 2>/dev/null
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "protocolVersion": "2024-11-05",
    "capabilities": {"tools": {}},
    "serverInfo": {"name": "crw-mcp", "version": "<current>"}
  }
}
```

--------------------------------

### Fetch CRW Agent Onboarding Skill File

Source: https://github.com/us/crw/blob/main/README.md

Fetch the SKILL.md file using curl to configure CRW for AI agents. This file contains tool descriptions, usage examples, and authentication setup.

```bash
curl -s https://fastcrw.com/agent-onboarding/SKILL.md
```

--------------------------------

### Install CRW MCP Server (Python SDK)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW Python SDK. This method automatically downloads the necessary binary.

```bash
pip install crw                        # Python SDK (auto-downloads binary)
```

--------------------------------

### Start a Crawl and Poll with TypeScript

Source: https://github.com/us/crw/blob/main/docs/docs/sdk-examples.md

Initiate a crawl job and poll its status using raw HTTP requests in TypeScript. This example shows the POST request to start a crawl and a GET request to check its status.

```typescript
const start = await fetch("https://fastcrw.com/api/v1/crawl", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/docs", limit: 10 }),
});

const { id } = await start.json();

const status = await fetch(`https://fastcrw.com/api/v1/crawl/${id}`, {
  headers: { Authorization: "Bearer YOUR_API_KEY" },
});
```

--------------------------------

### Crawl Job Start Response

Source: https://github.com/us/crw/blob/main/docs/docs/crawling.md

Example of a successful response after starting a crawl job, containing the unique job ID.

```json
{
  "success": true,
  "id": "550e8400-e29b-41d4-a716-446655440000"
}
```

--------------------------------

### Quick Start Pattern - Discover

Source: https://github.com/us/crw/blob/main/docs/docs/agent-onboarding.md

Discover all reachable URLs on a domain using the `/v1/map` endpoint.

```APIDOC
## Quick Start Pattern

For most agent workflows, start with this sequence:

1. **Discover** what pages exist on a target domain:

```bash
curl -X POST https://fastcrw.com/api/v1/map \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

```

--------------------------------

### Install CRW API Server (One-line)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW API server with a single command. This script auto-detects the operating system and architecture.

```bash
# One-line install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw-server sh
```

--------------------------------

### crw-browse Configuration

Source: https://github.com/us/crw/blob/main/crates/crw-browse/README.md

Example configuration for crw-browse within an MCP server setup. This specifies the command and arguments to run.

```json
{
  "mcpServers": {
    "crw-browse": {
      "command": "crw-browse",
      "args": ["--ws-url", "ws://localhost:9222"]
    }
  }
}
```

--------------------------------

### Start LightPanda Server

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Starts the LightPanda server for JavaScript rendering on localhost port 9222.

```bash
lightpanda serve --host 127.0.0.1 --port 9222 &
```

--------------------------------

### Install crw-mcp

Source: https://github.com/us/crw/blob/main/docs/docs/mcp.md

Install the crw-mcp binary using curl, npm, pip, cargo, or Docker. Use npx for zero-install.

```bash
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh
```

```bash
npx crw-mcp
```

```bash
pip install crw
```

```bash
cargo install crw-mcp
```

```bash
docker run -i ghcr.io/us/crw crw-mcp
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/us/crw/blob/main/README.md

Installs pre-commit hooks for the project. These hooks run formatting, linting, and testing checks before commits.

```bash
make hooks
```

--------------------------------

### Scrape a URL with Python

Source: https://github.com/us/crw/blob/main/docs/docs/sdk-examples.md

This Python example demonstrates how to scrape a URL and get the content in markdown format using the 'requests' library. Replace 'YOUR_API_KEY' with your actual API key.

```python
import requests

res = requests.post(
    "https://fastcrw.com/api/v1/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"url": "https://example.com", "formats": ["markdown"]},
)

res.raise_for_status()
payload = res.json()
print(payload["data"]["markdown"])
```

--------------------------------

### Automatic JS Rendering Setup

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Automatically sets up JavaScript rendering by downloading LightPanda and creating a configuration file.

```bash
crw-server setup
```

--------------------------------

### Example JSON-RPC 'tree' Tool Call

Source: https://github.com/us/crw/blob/main/crates/crw-browse/README.md

Illustrates calling the 'tree' tool via JSON-RPC to fetch the page's accessibility tree. The example shows the request and a partial response indicating the tree structure.

```json
→ {"jsonrpc":"2.0","id":3,"method":"tools/call",
   "params":{"name":"tree","arguments":{"max_nodes":50}}}
← {... data.tree = "[1] WebArea: Example Domain\n  [2] heading: Example Domain\n  [3] link: More information\n..." }
```

--------------------------------

### Example JSON-RPC Initialization

Source: https://github.com/us/crw/blob/main/crates/crw-browse/README.md

Demonstrates the initial JSON-RPC request to initialize the connection with the crw-browse server and the expected response.

```json
→ {"jsonrpc":"2.0","id":1,"method":"initialize",...}
← {"jsonrpc":"2.0","id":1,"result":{"serverInfo":{"name":"crw-browse",...}}}
```

--------------------------------

### Quick Start Pattern - Search

Source: https://github.com/us/crw/blob/main/docs/docs/agent-onboarding.md

Search the web and return results with content using the `/v1/search` endpoint.

```APIDOC
3. **Search** when you need to find relevant pages across the web:

```bash
curl -X POST https://fastcrw.com/api/v1/search \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "your search query", "limit": 5}'
```

```

--------------------------------

### Install to Custom Directory

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Installs the script to a custom directory using the CRW_INSTALL_DIR environment variable.

```bash
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_INSTALL_DIR=~/.local/bin sh
```

--------------------------------

### Add crw-server Crate

Source: https://github.com/us/crw/blob/main/docs/crates/index.html

Add the `crw-server` crate to build an Axum-based HTTP API server. This example shows how to build the application and start the server.

```rust
cargo add crw-server
```

```rust
use crw_server::app;
use crw_core::AppConfig;

let config = AppConfig::load()?;
let app = app::build_app(config).await;

let listener = tokio::net::TcpListener::bind("0.0.0.0:3000").await?;
axum::serve(listener, app).await?;

```

--------------------------------

### Install crw-renderer with CDP support

Source: https://github.com/us/crw/blob/main/crates/crw-renderer/README.md

To enable headless browser rendering via CDP, include the 'cdp' feature flag during installation.

```bash
cargo add crw-renderer --features cdp
```

--------------------------------

### Install CRW CLI (One-line)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW CLI with a single command. This script auto-detects the operating system and architecture.

```bash
# One-line install (auto-detects OS & arch):
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh
```

--------------------------------

### Install OpenClaw CRW Plugin

Source: https://github.com/us/crw/blob/main/docs/docs/integrations.md

Install the CRW plugin for OpenClaw to enable CRW functionalities.

```bash
openclaw plugins install openclaw-plugin-crw
```

--------------------------------

### Search with Go

Source: https://github.com/us/crw/blob/main/docs/sdk-examples/index.html

A Go example for performing a search. Make sure to use your correct API key and modify the search query and limit.

```Go
body := `{"query":"web scraping tools 2026","limit":5}`
req, _ := http.NewRequest("POST", "https://fastcrw.com/api/v1/search",
    strings.NewReader(body))
req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
req.Header.Set("Content-Type", "application/json")

resp, err := http.DefaultClient.Do(req)
if err != nil {
    panic(err)
}
def resp.Body.Close()

```

--------------------------------

### Install CRW CLI (Homebrew)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW command-line interface using Homebrew on macOS.

```bash
brew install us/crw/crw
```

--------------------------------

### Install CrewAI and CRW

Source: https://github.com/us/crw/blob/main/docs/docs/integrations.md

Install the necessary packages for CrewAI integration.

```bash
pip install crewai crewai-crw
```

--------------------------------

### Example Markdown Output

Source: https://github.com/us/crw/blob/main/README.md

This is an example of the markdown output generated by CRW for the 'Example Domain' URL.

```markdown
# Example Domain

This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination.
```

--------------------------------

### crw-browse Installation and Integration

Source: https://context7.com/us/crw/llms.txt

Install crw-browse using cargo and integrate it with Claude Code by adding it as an MCP server.

```bash
# Install and run
cargo install crw-browse
crw-browse
```

```bash
# Wire into Claude Code
claude mcp add crw-browse -- crw-browse
```

--------------------------------

### Install LangChain CRW

Source: https://github.com/us/crw/blob/main/docs/docs/integrations.md

Install the Python package for LangChain integration with CRW.

```bash
pip install langchain-crw
```

--------------------------------

### Install crw-renderer

Source: https://github.com/us/crw/blob/main/crates/crw-renderer/README.md

Add the crw-renderer crate to your Cargo.toml file.

```bash
cargo add crw-renderer
```

--------------------------------

### CLI - Basic Scrape

Source: https://context7.com/us/crw/llms.txt

Example of using the CLI to scrape a URL and output markdown to stdout.

```APIDOC
## CLI — Scrape from the Terminal

# Basic scrape — markdown to stdout
crw example.com
```

--------------------------------

### Self-Hosting CRW with LightPanda for JS Rendering

Source: https://context7.com/us/crw/llms.txt

Steps to set up and run CRW with LightPanda for JavaScript rendering, involving serving LightPanda separately and then starting the CRW server.

```bash
# With JS rendering via LightPanda
crw-server setup
lightpanda serve --host 127.0.0.1 --port 9222 &
crw-server
```

--------------------------------

### Configure crw-server with TOML

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Example TOML configuration file for crw-server, showing settings for the server, renderer, crawler, auth, and LLM extraction.

```toml
[server]
host = "0.0.0.0"
port = 3000
request_timeout_secs = 120
rate_limit_rps = 10        # Max requests/second (global). 0 = unlimited.

[renderer]
mode = "auto"  # auto | lightpanda | playwright | chrome | none
# render_js_default = true   # alias: force_js = true — force JS for every request that omits `renderJs`

[crawler]
max_concurrency = 10
requests_per_second = 10.0
respect_robots_txt = true

[auth]
# api_keys = ["fc-key-1234"]

[extraction.llm]
provider = "anthropic"  # "anthropic" or "openai"
# api_key = "sk-..."    # or CRW_EXTRACTION__LLM__API_KEY env var
```

--------------------------------

### Verify MCP Installation

Source: https://github.com/us/crw/blob/main/docs/mcp/index.html

Use this command to verify that the crw-mcp client is installed and functioning correctly by sending an initialization request.

```bash
echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"clientInfo":{"name":"test"},"protocolVersion":"2024-11-05"}}' \
  | crw-mcp 2>/dev/null
```

--------------------------------

### Install CRW Skill for AI Agents

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW skill to all detected AI agents using the npx command. Ensure your agent is restarted after installation for the changes to take effect.

```bash
npx crw-mcp init --all
```

--------------------------------

### Install crw-mcp using Cargo

Source: https://github.com/us/crw/blob/main/crates/crw-mcp/README.md

Install the crw-mcp binary using the Rust package manager.

```bash
cargo install crw-mcp
```

--------------------------------

### Run crw-server with Docker

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Instructions for running the crw-server using a pre-built Docker image or with a Docker Compose setup for JS rendering.

```bash
# Pre-built image
docker run -p 3000:3000 ghcr.io/us/crw:latest

# With JS rendering sidecar
docker compose up
```

--------------------------------

### Install CRW MCP Server (Homebrew)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW MCP server using Homebrew on macOS.

```bash
brew install us/crw/crw-mcp            # Homebrew
```

--------------------------------

### Start CRW MCP Server

Source: https://github.com/us/crw/blob/main/python/README.md

Start the CRW MCP stdio server from the command line. This is used to integrate CRW with AI coding agents.

```bash
crw-mcp
```

--------------------------------

### Install CRW Python SDK

Source: https://github.com/us/crw/blob/main/docs/mcp/index.html

Installs the CRW Python SDK using pip. Note that for the MCP binary, npx or cargo are recommended.

```bash
pip install crw
```

--------------------------------

### Quick Start Pattern - Extract

Source: https://github.com/us/crw/blob/main/docs/docs/agent-onboarding.md

Extract content from specific pages using the `/v1/scrape` endpoint.

```APIDOC
2. **Extract** content from specific pages:

```bash
curl -X POST https://fastcrw.com/api/v1/scrape \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/page", "formats": ["markdown"]}'
```

```

--------------------------------

### Install CRW CLI

Source: https://github.com/us/crw/blob/main/README.md

Installs the CRW command-line interface using a curl script. CRW_BINARY can be set to customize the binary name.

```bash
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh
```

--------------------------------

### Install CRW CLI (APT)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW CLI on Debian/Ubuntu systems using the APT package manager. This involves adding the CRW repository and GPG key.

```bash
# APT (Debian/Ubuntu):
curl -fsSL https://apt.fastcrw.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/crw.gpg
echo "deb [signed-by=/usr/share/keyrings/crw.gpg] https://apt.fastcrw.com stable main" | sudo tee /etc/apt/sources.list.d/crw.list
sudo apt update && sudo apt install crw
```

--------------------------------

### Python SDK - Search

Source: https://context7.com/us/crw/llms.txt

Example of using the Python SDK for web search.

```APIDOC
# ── Search (cloud only) ────────────────────────────────────────────────────
results = client.search("LLM benchmark 2026", limit=10, sources=["web", "news"])
for r in results.get("web", []):
    print(r["title"], r["url"])

results = client.search(
    "rust web frameworks",
    limit=5,
    tbs="qdr:m",
    scrape_options={"formats": ["markdown"]},
)
```

--------------------------------

### Build CLI Tool from Source

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Builds the crw CLI tool from source using cargo. No server setup is needed.

```bash
git clone https://github.com/us/crw.git
cd crw

# CLI tool (no server, no setup)
cargo build --release --bin crw
```

--------------------------------

### Cloud API Usage

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Example of how to use the cloud API for scraping. Replace YOUR_API_KEY with your actual key.

```bash
curl -X POST https://fastcrw.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

--------------------------------

### Scrape a URL with Go

Source: https://github.com/us/crw/blob/main/docs/docs/sdk-examples.md

A Go example for scraping a URL and retrieving content. This code sets up an HTTP POST request with the necessary headers and JSON body.

```go
package main

import (
  "bytes"
  "fmt"
  "io"
  "net/http"
)

func main() {
  body := []byte(`{"url":"https://example.com","formats":["markdown"]}`)
  req, _ := http.NewRequest("POST", "https://fastcrw.com/api/v1/scrape", bytes.NewBuffer(body))
  req.Header.Set("Authorization", "Bearer YOUR_API_KEY")
  req.Header.Set("Content-Type", "application/json")

  res, err := http.DefaultClient.Do(req)
  if err != nil {
    panic(err)
  }
  defer res.Body.Close()

  b, _ := io.ReadAll(res.Body)
  fmt.Println(string(b))
```

--------------------------------

### Running CRW with Docker and Environment Variables

Source: https://context7.com/us/crw/llms.txt

Example of launching the CRW Docker container and configuring its settings via environment variables.

```bash
docker run -p 8080:8080 \
  -e CRW_SERVER__PORT=8080 \
  -e CRW_RENDERER__MODE=auto \
  -e CRW_EXTRACTION__LLM__API_KEY=sk-ant-... \
  ghcr.io/us/crw:latest
```

--------------------------------

### Install CRW MCP Server (Cargo)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW MCP server using Cargo, the Rust package manager.

```bash
cargo install crw-mcp                  # Cargo
```

--------------------------------

### Run Benchmark Script

Source: https://github.com/us/crw/blob/main/README.md

Install necessary Python packages and run the benchmark script to test CRW's performance.

```bash
pip install datasets aiohttp
python bench/run_bench.py
```

--------------------------------

### Example JSON-RPC 'goto' Tool Call

Source: https://github.com/us/crw/blob/main/crates/crw-browse/README.md

Shows how to call the 'goto' tool via JSON-RPC to navigate to a URL and wait for the page to load. Includes the request and a sample success response.

```json
→ {"jsonrpc":"2.0","id":2,"method":"tools/call",
   "params":{"name":"goto","arguments":{"url":"https://example.com"}}}
← {"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text",
   "text":"{\"ok\":true,\"session\":\"a3f1\",\"url\":\"https://example.com\",
   \"navigated\":true,\"elapsed_ms\":412,\"data\":{\"status\":200}}"}]}}
```

--------------------------------

### Install fastCRW Skill

Source: https://github.com/us/crw/blob/main/npm/crw-mcp/skills/SKILL.md

Installs the CRW skill and MCP server to detected AI agents. This command is executed in a bash environment.

```bash
crw fastcrw.com
```

--------------------------------

### Single-Page Scrape Example

Source: https://github.com/us/crw/blob/main/crates/crw-crawl/README.md

Fetches a URL, extracts content, and optionally performs LLM structured extraction. Ensure the necessary types and renderer are initialized.

```rust
use crw_crawl::single::scrape_url;
use crw_core::types::{ScrapeRequest, OutputFormat};
use crw_renderer::FallbackRenderer;
use crw_core::config::{RendererConfig, StealthConfig};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = RendererConfig::default();
    let stealth = StealthConfig::default();
    let renderer = Arc::new(FallbackRenderer::new(&config, "my-crawler/1.0", None, &stealth));

    let request = ScrapeRequest {
        url: "https://example.com".into(),
        formats: vec![OutputFormat::Markdown, OutputFormat::Links],
        ..Default::default()
    };

    let result = scrape_url(&request, &renderer, None, "my-crawler/1.0", false).await?;
    println!("Title: {:?}", result.metadata.title);
    println!("Markdown: {}", result.markdown.unwrap_or_default());
    Ok(())
}
```

--------------------------------

### Install CRW MCP Server (Docker)

Source: https://github.com/us/crw/blob/main/README.md

Run the CRW MCP server using a Docker container.

```bash
docker run -i ghcr.io/us/crw crw-mcp  # Docker
```

--------------------------------

### Install crw-extract

Source: https://github.com/us/crw/blob/main/crates/crw-extract/README.md

Add the crw-extract crate to your Cargo project.

```bash
cargo add crw-extract
```

--------------------------------

### Start a Crawl Job

Source: https://github.com/us/crw/blob/main/docs/docs/crawling.md

Initiates a new crawl job by sending a POST request to the /v1/crawl endpoint with the starting URL and crawl parameters. The response includes a unique ID for the crawl job.

```APIDOC
## POST /v1/crawl

### Description
Starts a new asynchronous crawl job.

### Method
POST

### Endpoint
/v1/crawl

### Request Body
- **url** (string) - Required - Starting URL for the crawl.
- **maxDepth** (number) - Optional - Maximum depth to crawl from the start URL. Defaults to 2.
- **maxPages** (number) - Optional - Maximum number of pages to crawl. Defaults to 100.
- **limit** (number) - Optional - Alias for `maxPages`.
- **max_pages** (number) - Optional - Snake_case alias for `maxPages`.
- **formats** (string[]) - Optional - Output formats for each page. Defaults to `["markdown"]`.
- **onlyMainContent** (boolean) - Optional - If true, removes boilerplate content before conversion. Defaults to true.
- **jsonSchema** (object) - Optional - Schema for structured extraction per page.
- **renderJs** (boolean or null) - Optional - Controls JavaScript rendering. `true` forces JS, `false` skips JS, `null` uses auto-detect. Defaults to null.
- **waitFor** (number) - Optional - Milliseconds to wait after JS rendering on each page.
- **renderer** (string) - Optional - Pin a specific renderer (`auto`, `lightpanda`, `chrome`, `playwright`). Defaults to `auto`.

### Request Example
```json
{
  "url": "https://docs.example.com",
  "maxDepth": 1,
  "maxPages": 5,
  "formats": ["markdown"],
  "onlyMainContent": true
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the request was successful.
- **id** (string) - The unique identifier for the crawl job.

#### Response Example
```json
{
  "success": true,
  "id": "550e8400-e29b-41d4-a716-446655440000"
}
```
```

--------------------------------

### Python SDK - Scrape

Source: https://context7.com/us/crw/llms.txt

Example of using the Python SDK to scrape web content.

```APIDOC
# ── Scrape ─────────────────────────────────────────────────────────────────
result = client.scrape("https://example.com")
print(result["markdown"])

result = client.scrape(
    "https://example.com",
    formats=["markdown", "links"]
)
print(result["links"])   # list of URLs
```

--------------------------------

### Initialize CrwClient

Source: https://github.com/us/crw/blob/main/python/README.md

Demonstrates how to initialize the CrwClient, both with zero-configuration (auto-downloading the binary) and by connecting to a remote server.

```APIDOC
## Initialize CrwClient

### Description
Initialize the CrwClient for interacting with the CRW service. This can be done with zero configuration, which automatically downloads the crw-mcp binary, or by specifying the `api_url` and `api_key` for a remote server.

### Method
```python
CrwClient(api_url: str = None, api_key: str = None)
```

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from crw import CrwClient

# Zero-config (downloads crw-mcp binary automatically):
client = CrwClient()

# Or connect to a remote server:
client = CrwClient(api_url="https://fastcrw.com/api", api_key="fc-...")
```

### Response
#### Success Response (200)
N/A (Initialization does not return a response in this context)

#### Response Example
N/A
```

--------------------------------

### Run CRW MCP Server with npm

Source: https://github.com/us/crw/blob/main/docs/mcp/index.html

Executes the CRW MCP server using npx, which allows for zero installation.

```bash
npx crw-mcp
```

--------------------------------

### Install CRW API Server (Homebrew)

Source: https://github.com/us/crw/blob/main/README.md

Install the CRW API server using Homebrew on macOS. This server is compatible with Firecrawl and suitable for serving multiple applications.

```bash
brew install us/crw/crw-server
```

--------------------------------

### Initialize MCP with CRW

Source: https://github.com/us/crw/blob/main/docs/docs/quick-start.md

Use this command to add CRW functionality to your MCP setup. This is the recommended path for AI tools needing live web access.

```bash
claude mcp add crw -- npx crw-mcp
```

--------------------------------

### Start a crawl with crw-server

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Initiate an asynchronous Breadth-First Search (BFS) crawl starting from a given URL, with specified depth and limit, using a curl POST request to the /v1/crawl endpoint.

```bash
curl -X POST http://localhost:3000/v1/crawl \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "maxDepth": 2, "limit": 50}'
```

--------------------------------

### Start async BFS crawl

Source: https://github.com/us/crw/blob/main/crates/crw-server/README.md

Initiates an asynchronous Breadth-First Search (BFS) crawl starting from a given URL. Returns a job ID for tracking.

```APIDOC
## POST /v1/crawl

### Description
Start async BFS crawl (returns job ID).

### Method
POST

### Endpoint
/v1/crawl

### Request Body
- **url** (string) - Required - The starting URL for the crawl.
- **maxDepth** (integer) - Optional - The maximum depth of the crawl.
- **limit** (integer) - Optional - The maximum number of pages to crawl.

### Request Example
```json
{
  "url": "https://example.com",
  "maxDepth": 2,
  "limit": 50
}
```

### Response
#### Success Response (200)
- **jobId** (string) - The ID of the initiated crawl job.

### Response Example
```json
{
  "jobId": "crawl-12345"
}
```
```

--------------------------------

### CLI Scraping with HTML Output

Source: https://github.com/us/crw/blob/main/README.md

Specify the desired output format using the --format flag. This example requests HTML output.

```bash
crw example.com --format html
```

--------------------------------

### Quick Start: Claude Code with Custom Config

Source: https://github.com/us/crw/blob/main/crates/crw-mcp/README.md

Configure crw-mcp with custom settings using environment variables when adding it to Claude Code.

```bash
# With custom config via env vars
claude mcp add \
  -e CRW_CRAWLER__MAX_CONCURRENCY=5 \
  -e CRW_RENDERER__LIGHTPANDA__WS_URL=ws://127.0.0.1:9222 \
  crw -- crw-mcp
```

--------------------------------

### Scrape URL with Python

Source: https://github.com/us/crw/blob/main/docs/sdk-examples/index.html

A Python example for scraping a URL. Remember to replace 'YOUR_API_KEY' with your valid API key.

```Python
import requests

res = requests.post(
    "https://fastcrw.com/api/v1/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={"url": "https://example.com", "formats": ["markdown"]},
)

res.raise_for_status()
payload = res.json()
print(payload["data"]["markdown"])

```

--------------------------------

### Start Crawl Job (cURL)

Source: https://github.com/us/crw/blob/main/docs/docs/crawling.md

Initiates a crawl job using cURL, specifying the URL, depth, page limit, and desired format.

```bash
curl -X POST https://fastcrw.com/api/v1/crawl \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 1,
    "maxPages": 5,
    "formats": ["markdown"]
  }'
```

--------------------------------

### Configure Self-Hosted Authentication

Source: https://context7.com/us/crw/llms.txt

Example TOML configuration to enable API key authentication for self-hosted fastCRW instances.

```toml
# config.local.toml — enable auth on self-hosted
[auth]
api_keys = ["fc-key-1234", "fc-key-5678"]
```

--------------------------------

### Connect crw-browse to Browser

Source: https://github.com/us/crw/blob/main/crates/crw-browse/README.md

Connect the crw-browse tool to a running browser instance using its WebSocket URL. This command starts the crw-browse server.

```bash
crw-browse --ws-url ws://localhost:9222
```

--------------------------------

### BFS Crawl Example

Source: https://github.com/us/crw/blob/main/crates/crw-crawl/README.md

Initiates a breadth-first crawl with concurrent workers. Requires setting up a crawl state channel and defining crawl options.

```rust
use crw_crawl::crawl::{CrawlOptions, run_crawl};
use crw_core::types::{CrawlRequest, CrawlState, CrawlStatus};
use crw_renderer::FallbackRenderer;
use crw_core::config::{RendererConfig, StealthConfig};
use std::sync::Arc;
use uuid::Uuid;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = RendererConfig::default();
    let stealth = StealthConfig::default();
    let renderer = Arc::new(FallbackRenderer::new(&config, "my-crawler/1.0", None, &stealth));

    let (state_tx, mut state_rx) = tokio::sync::watch::channel(CrawlState {
        status: CrawlStatus::Scraping,
        completed: 0,
        total: 0,
        data: vec![],
    });

    let options = CrawlOptions {
        id: Uuid::new_v4(),
        req: CrawlRequest {
            url: "https://example.com".into(),
            max_depth: Some(2),
            limit: Some(10),
            ..Default::default()
        },
        renderer,
        max_concurrency: 5,
        respect_robots: true,
        requests_per_second: 2.0,
        user_agent: "my-crawler/1.0",
        state_tx,
        llm_config: None,
        proxy: None,
        jitter_factor: 0.2,
    };

    run_crawl(options).await;
    let final_state = state_rx.borrow().clone();
    println!("Crawled {} pages", final_state.completed);
    Ok(())
}
```

--------------------------------

### Run Docker Compose with Sidecar

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Starts the crw service using docker-compose, which includes a LightPanda sidecar for JS rendering.

```bash
docker compose up
```

--------------------------------

### Docker Compose for CRW and LightPanda

Source: https://github.com/us/crw/blob/main/docs/docs/js-rendering.md

Example Docker Compose configuration to run CRW with a LightPanda sidecar for JS rendering.

```yaml
services:
  crw:
    image: ghcr.io/us/crw:latest
    ports:
      - "3000:3000"
    environment:
      - CRW_RENDERER__LIGHTPANDA__WS_URL=ws://lightpanda:9222
  lightpanda:
    image: lightpanda/lightpanda:latest
    command: ["serve", "--host", "0.0.0.0", "--port", "9222"]
```

--------------------------------

### Initialize CrwClient (Zero-Config)

Source: https://github.com/us/crw/blob/main/python/README.md

Initialize the CrwClient without any configuration. The client will automatically download the crw-mcp binary.

```python
from crw import CrwClient

# Zero-config (downloads crw-mcp binary automatically):
client = CrwClient()
result = client.scrape("https://example.com")
print(result["markdown"])
```

--------------------------------

### Search with Python

Source: https://github.com/us/crw/blob/main/docs/docs/sdk-examples.md

This Python example demonstrates how to perform a search query with the CRW API. It sends a JSON payload with the query and limit, and retrieves the results.

```python
resp = requests.post(
    "https://fastcrw.com/api/v1/search",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"query": "web scraping tools 2026", "limit": 5},
)

resp.raise_for_status()
data = resp.json()["data"]
```

--------------------------------

### Run Pre-built Docker Image

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Launches the crw service using a pre-built Docker image. Port 3000 is exposed.

```bash
docker run -p 3000:3000 ghcr.io/us/crw:latest
```

--------------------------------

### Install n8n CRW Node

Source: https://github.com/us/crw/blob/main/docs/docs/integrations.md

Install the n8n-nodes-crw community node for n8n workflows.

```bash
docker run -e EXTRA_COMMUNITY_PACKAGES=n8n-nodes-crw n8nio/n8n
```

--------------------------------

### CRW Configuration via Environment Variables

Source: https://github.com/us/crw/blob/main/docs/configuration/index.html

CRW supports configuration through environment variables using the `CRW_` prefix and `__` for nesting. This example shows how to set the server port.

```bash
export CRW_SERVER__PORT=3000
export CRW_SERVER__HOST=0.0.0.0
export CRW_RENDERER__MODE=auto
export CRW_RENDERER__RENDER_JS_DEFAULT=true
export CRW_CRAWLER__MAX_CONCURRENCY=10
export CRW_CRAWLER__REQUESTS_PER_SECOND=10.0
export CRW_SERVER__RATE_LIMIT_RPS=10
export CRW_CRAWLER__STEALTH=true
export CRW_CRAWLER__PROXY="http://proxy:8080"
export CRW_EXTRACTION__LLM__API_KEY=""
export CRW_EXTRACTION__LLM__PROVIDER=anthropic

```

--------------------------------

### Start and Poll Crawl Job (Node.js)

Source: https://github.com/us/crw/blob/main/docs/docs/crawling.md

Initiates a crawl job and polls its status using Node.js `fetch`. Includes a delay for polling.

```javascript
const start = await fetch("https://fastcrw.com/api/v1/crawl", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://docs.example.com",
    maxDepth: 1,
    maxPages: 5,
    formats: ["markdown"]
  })
});

const { id } = await start.json();
await new Promise((resolve) => setTimeout(resolve, 2000));

const status = await fetch(`https://fastcrw.com/api/v1/crawl/${id}`, {
  headers: { "Authorization": "Bearer YOUR_API_KEY" }
});

console.log((await status.json()).status);
```

--------------------------------

### Basic HTTP Fetching with FallbackRenderer

Source: https://github.com/us/crw/blob/main/crates/crw-renderer/README.md

Demonstrates basic HTTP fetching using FallbackRenderer. Initializes the renderer with default configurations and fetches a URL, printing the status code and HTML length.

```rust
use crw_core::config::{RendererConfig, StealthConfig};
use crw_renderer::FallbackRenderer;
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = RendererConfig::default();
    let stealth = StealthConfig::default();
    let renderer = FallbackRenderer::new(&config, "my-app/1.0", None, &stealth);

    let result = renderer.fetch("https://example.com", &HashMap::new(), None, None, None).await?;
    println!("Status: {}", result.status_code);
    println!("HTML length: {}", result.html.len());
    Ok(())
}
```

--------------------------------

### Docker Deployment Configuration

Source: https://github.com/us/crw/blob/main/docs/configuration/index.html

Configure CRW for Docker deployments using environment variables. This example sets the server port, renderer mode, and an API key.

```bash
docker run -p 3000:3000 \
  -e CRW_SERVER__PORT=3000 \
  -e CRW_RENDERER__MODE=lightpanda \
  -e CRW_EXTRACTION__LLM__API_KEY=sk- \
  ghcr.io/us/crw:latest
```

--------------------------------

### TOML Configuration Example

Source: https://github.com/us/crw/blob/main/README.md

Illustrates layered TOML configuration with environment variable overrides for server, renderer, crawler, and auth settings. Defaults are in config.default.toml, local overrides in config.local.toml, and environment variables with CRW_ prefix.

```toml
[server]
host = "0.0.0.0"
port = 3000
rate_limit_rps = 10

[renderer]
mode = "auto"  # auto | lightpanda | playwright | chrome | none

[crawler]
max_concurrency = 10
requests_per_second = 10.0
respect_robots_txt = true

[auth]
# api_keys = ["fc-key-1234"]

```

--------------------------------

### Map Site with Node.js SDK

Source: https://github.com/us/crw/blob/main/docs/docs/map.md

This Node.js example demonstrates how to map a site using the Map API. It sends a POST request with the API key and a JSON body containing the target URL and discovery parameters.

```javascript
const resp = await fetch("https://fastcrw.com/api/v1/map", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com",
    maxDepth: 1,
    useSitemap: true
  })
});

const body = await resp.json();
console.log(body.data.links);
```

--------------------------------

### Build MCP stdio binary from Source

Source: https://github.com/us/crw/blob/main/docs/docs/installation.md

Builds the crw-mcp stdio binary from source using cargo.

```bash
git clone https://github.com/us/crw.git
cd crw

# MCP stdio binary
cargo build --release --bin crw-mcp
```

--------------------------------

### Get Crawl Status and Results

Source: https://github.com/us/crw/blob/main/docs/rest-api/index.html

Poll this endpoint using the crawl ID obtained from a `POST /v1/crawl` request to get the status and completed data of an ongoing or finished crawl.

```APIDOC
## GET /v1/crawl/{id}

### Description
Retrieves the status and results of a specific crawl job.

### Method
GET

### Endpoint
`/v1/crawl/{id}`

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the crawl job.

### Response
#### Success Response (200)
- **status** (string) - The current status of the crawl (e.g., "running", "completed", "failed").
- **data** (object) - The crawled data, if available.

#### Response Example
```json
{
  "status": "completed",
  "data": {
    "https://example.com": "...content..."
  }
}
```
```

--------------------------------

### Build REST API Server from Source (No JS Rendering)

Source: https://github.com/us/crw/blob/main/docs/installation/index.html

Builds the CRW REST API server from source without JavaScript rendering capabilities. The binary is placed in `target/release/`.

```bash
cargo build --release --bin crw-server
```

--------------------------------

### Discover Site URLs with cURL

Source: https://context7.com/us/crw/llms.txt

Initiate a site mapping request to enumerate all reachable URLs from a given starting point. This is useful for validating crawl scope before starting a full crawl.

```bash
curl -X POST https://fastcrw.com/api/v1/map \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "maxDepth": 2,
    "useSitemap": true
  }'
```
Tokens:
Raw