### Install via Cargo Source: https://github.com/0xmassi/webclaw/blob/main/packages/create-webclaw/README.md Installs the webclaw-cli binary from the GitHub repository using Cargo. ```bash cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli ``` -------------------------------- ### Install via Homebrew Source: https://github.com/0xmassi/webclaw/blob/main/packages/create-webclaw/README.md Installs WebClaw using Homebrew after tapping the 0xMassi/webclaw repository. ```bash brew tap 0xMassi/webclaw && brew install webclaw ``` -------------------------------- ### Install Rust via rustup Source: https://github.com/0xmassi/webclaw/blob/main/CONTRIBUTING.md Use this command to install the Rust toolchain required for development. ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` -------------------------------- ### Run extraction benchmark for parse and content components Source: https://github.com/0xmassi/webclaw/blob/main/benchmarks/extraction.md Run the extraction_bench example directly for a single component. `parse` includes HTML parsing and destruction; `content` starts with an already parsed document and measures readability scoring and markdown conversion. These components are diagnostic and do not sum to full extraction. ```sh /tmp/extraction-before parse crates/webclaw-core/testdata/express_test.html \ https://example.com/page 100 /tmp/parse.json /tmp/extraction-before content crates/webclaw-core/testdata/express_test.html \ https://example.com/page 100 /tmp/content.json ``` -------------------------------- ### Install webclaw MCP server Source: https://github.com/0xmassi/webclaw/blob/main/examples/mcp-web-scraping/README.md Run this command to install webclaw as a local MCP server. The installer detects supported MCP clients and can write the configuration for you. ```bash npx create-webclaw ``` -------------------------------- ### MCP Server (AI Agent Integration) Source: https://github.com/0xmassi/webclaw/blob/main/examples/README.md Run the MCP server via `npx -y @webclaw/mcp` (stdio transport, no install needed). Configure in Claude Desktop with optional `WEBCLAW_API_KEY` for cloud fallback. Available tools: scrape, search, crawl, map, batch, extract, summarize, diff, brand, research, list_extractors, vertical_scrape. ```bash npx -y @webclaw/mcp ``` ```json { "mcpServers": { "webclaw": { "command": "npx", "args": ["-y", "@webclaw/mcp"], "env": { "WEBCLAW_API_KEY": "wc_your_key" // optional, enables cloud fallback } } } } ``` -------------------------------- ### Run full benchmark suite Source: https://github.com/0xmassi/webclaw/blob/main/benchmarks/README.md Builds webclaw, installs other tools, and runs all benchmarks. The first two arguments are a label for the results directory and the directory holding the webclaw and webclaw-mcp binaries (default: build into target/release). Requires uv and a Rust toolchain; tools are installed into pinned environments under benchmarks/.work/. ```bash ./benchmarks/run.sh # builds webclaw, installs the other tools, runs everything ./benchmarks/run.sh my-run ./path/to/bins 3 ``` -------------------------------- ### Install Webclaw SDKs Source: https://github.com/0xmassi/webclaw/blob/main/README.md Install the Webclaw SDK for npm, pip, or Go. Use the package manager appropriate for your project. ```bash npm install @webclaw/sdk ``` ```bash pip install webclaw ``` ```bash go get github.com/0xMassi/webclaw-go ``` -------------------------------- ### Install webclaw via Homebrew Source: https://github.com/0xmassi/webclaw/blob/main/README.md Install webclaw using Homebrew by tapping the 0xMassi/webclaw repository and then installing the webclaw package. ```bash brew tap 0xMassi/webclaw brew install webclaw ``` -------------------------------- ### Mark meta process start Source: https://github.com/0xmassi/webclaw/blob/main/crates/webclaw-core/testdata/express_test.html Marks the start of the meta process. The code is truncated in the source. ```javascript performance.mark('meta-start');const ``` -------------------------------- ### Build and compare extraction benchmark Source: https://github.com/0xmassi/webclaw/blob/main/benchmarks/extraction.md Build the extraction_bench example on the baseline and candidate, copy the binaries, generate fixtures, and run the comparison script. Requires identical example and scripts in both builds; keep input files, features, compiler flags, and toolchain unchanged. Run on an idle machine without concurrent builds or profilers. ```sh cargo build --locked --release -p webclaw-core --example extraction_bench cp target/release/examples/extraction_bench /tmp/extraction-before # Apply the candidate change, then rebuild. cargo build --locked --release -p webclaw-core --example extraction_bench cp target/release/examples/extraction_bench /tmp/extraction-after python3 benchmarks/scripts/extraction_fixtures.py /tmp/extraction-fixtures python3 benchmarks/scripts/compare_extraction.py \ /tmp/extraction-before /tmp/extraction-after \ /tmp/extraction-fixtures/manifest.json /tmp/extraction-results ``` -------------------------------- ### Get all external links with jq Source: https://github.com/0xmassi/webclaw/blob/main/examples/README.md Fetches a page as JSON and uses jq to filter .content.links for hrefs starting with 'http', printing each href. ```bash webclaw https://example.com -f json | jq -r '.content.links[] | select(.href | startswith("http")) | .href' ``` -------------------------------- ### Run Python package benchmark Source: https://github.com/0xmassi/webclaw/blob/main/benchmarks/README.md Runs the Python package benchmark. The first command runs the default benchmark against Scrapling's Fetcher and Crawl4AI's AsyncWebCrawler. The second runs a custom run with a specific wheel file. The script builds the wheel from python/ (or uses the provided one), installs each tool three times into fresh venvs with no network and empty uv cache, runs post-install steps (scrapling install, crawl4ai-setup), records download size, install time, package count, and disk size, then calls each tool via its Python API on static golden pages (cases.jsonl without spa and pdf). It scores facts, samples peak RSS, lists failed calls with errors, and writes results/