### Install Dependencies with npm Source: https://developers.cloudflare.com/browser-run/stagehand Install the necessary dependencies for the project using npm. ```bash npm ci ``` -------------------------------- ### Basic Usage: With a Prompt and JSON schema Source: https://developers.cloudflare.com/browser-run/quick-actions/json-endpoint This example demonstrates how to extract webpage data by providing both a prompt to guide the extraction and a JSON schema to define the structure of the output. ```APIDOC ## POST /json ### Description Extracts structured data from a webpage using AI, with the output formatted according to a provided JSON schema. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts//browser-rendering/json` ### Parameters #### Request Body - **url** (string) - Required - The URL of the webpage to process. - **html** (string) - Required - The HTML content of the webpage to process. - **prompt** (string) - Required - A natural language prompt guiding the AI on what data to extract. - **response_format** (object) - Required - Specifies the desired output format. - **type** (string) - Required - Must be `json_schema`. - **schema** (object) - Required - A JSON schema defining the structure of the extracted data. ### Request Example ```json { "url": "https://developers.cloudflare.com/", "prompt": "Get me the list of AI products", "response_format": { "type": "json_schema", "schema": { "type": "object", "properties": { "products": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "link": { "type": "string" } }, "required": [ "name" ] } } } } } } ``` ### Response #### Success Response (200) - **success** (boolean) - Indicates if the operation was successful. - **result** (object) - The extracted data. - **products** (array) - A list of AI products found on the page, each with a name and link. #### Response Example ```json { "success": true, "result": { "products": [ { "name": "Build a RAG app", "link": "https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/" }, { "name": "Workers AI", "link": "https://developers.cloudflare.com/workers-ai/" }, { "name": "Vectorize", "link": "https://developers.cloudflare.com/vectorize/" }, { "name": "AI Gateway", "link": "https://developers.cloudflare.com/ai-gateway/" }, { "name": "AI Playground", "link": "https://playground.ai.cloudflare.com/" } ] } } ``` ``` -------------------------------- ### Install @cloudflare/playwright with bun Source: https://developers.cloudflare.com/browser-run/playwright Install the Cloudflare Playwright package as a development dependency using bun. ```bash bun add -d @cloudflare/playwright ``` -------------------------------- ### robots.txt Example with Content Signals Source: https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint This example shows how to configure robots.txt to allow search indexing but disallow AI training. ```robots.txt User-Agent: * Content-Signal: search=yes, ai-train=no Allow: / ``` -------------------------------- ### Example with all optional parameters Source: https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint This example demonstrates how to use the crawl endpoint with a comprehensive set of optional parameters to customize the crawling process. ```APIDOC ## POST /accounts/{account_id}/browser-rendering/crawl ### Description Initiates a crawl of a given URL with various optional parameters to control the process. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl` ### Parameters #### Request Body - **url** (string) - Required - The URL to crawl. - **crawlPurposes** (array) - Optional - An array of purposes for the crawl (e.g., "search"). - **limit** (integer) - Optional - The maximum number of pages to crawl. - **depth** (integer) - Optional - The maximum depth of links to follow. - **formats** (array) - Optional - An array of desired output formats (e.g., "markdown", "json", "html"). - **render** (boolean) - Optional - Whether to render the page (default is true). - **maxAge** (integer) - Optional - The maximum age of cached results in seconds. - **modifiedSince** (integer) - Optional - A Unix timestamp to only fetch resources modified since this time. - **source** (string) - Optional - Specifies the source of the content (e.g., "all"). - **options** (object) - Optional - Advanced options for crawling. - **includeExternalLinks** (boolean) - Optional - Whether to include external links. - **includeSubdomains** (boolean) - Optional - Whether to include subdomains. - **includePatterns** (array) - Optional - An array of URL patterns to include. - **excludePatterns** (array) - Optional - An array of URL patterns to exclude. - **jsonOptions** (object) - Optional - Options specific to JSON output format. - **prompt** (string) - Required if format is "json" - The prompt for AI to extract data. - **response_format** (object) - Optional - Specifies the desired JSON response format. - **type** (string) - Required - The type of response format (e.g., "json_schema"). - **json_schema** (object) - Required if type is "json_schema" - The JSON schema definition. - **authenticate** (object) - Optional - Credentials for HTTP basic authentication. - **username** (string) - Required - The username. - **password** (string) - Required - The password. - **setExtraHTTPHeaders** (object) - Optional - Custom HTTP headers to set for the request. - **gotoOptions** (object) - Optional - Options to control the page navigation behavior. - **waitUntil** (string) - Optional - When to consider navigation successful (e.g., "networkidle2"). - **timeout** (integer) - Optional - The maximum time in milliseconds to wait for navigation. - **waitForSelector** (object) - Optional - Options to wait for a specific element to appear. - **selector** (string) - Required - The CSS selector for the element. - **timeout** (integer) - Optional - The maximum time in milliseconds to wait for the selector. - **visible** (boolean) - Optional - Whether the element must be visible. - **rejectResourceTypes** (array) - Optional - An array of resource types to reject (e.g., "image", "media"). Only available when `render` is true. ### Request Example ```json { "url": "https://www.exampledocs.com/docs/", "crawlPurposes": ["search"], "limit": 50, "depth": 2, "formats": ["markdown"], "render": false, "maxAge": 7200, "modifiedSince": 1704067200, "source": "all", "options": { "includeExternalLinks": true, "includeSubdomains": true, "includePatterns": [ "**/api/v1/*" ], "excludePatterns": [ "*/learning-paths/*" ] } } ``` ``` -------------------------------- ### Install Wrangler and Create Lab Session Source: https://developers.cloudflare.com/browser-run/features/webmcp Install the latest version of Wrangler and create a lab browser session with a 5-minute keep-alive. Lab sessions are experimental and not for production. ```bash # make sure you have the latest version of wrangler npm i -g wrangler@latest # create a lab browser session with 5 minute keep-alive wrangler browser create --lab --keepAlive 300 ``` -------------------------------- ### Example Local Test URL Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker This is an example URL to test your Worker locally after deployment. Replace `` with your local development server address. ```text /?url=https://example.com ``` -------------------------------- ### Install playwright-core with bun Source: https://developers.cloudflare.com/browser-run/cdp/playwright Install the playwright-core package, which is the version without bundled browsers. ```bash bun add playwright-core ``` -------------------------------- ### TypeScript SDK Example Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example shows how to use the Cloudflare TypeScript SDK to achieve the same result as the cURL example, providing a JSON schema for structured data extraction. ```APIDOC ## BrowserRendering.json.create (TypeScript SDK) ### Description Uses the Cloudflare TypeScript SDK to call the Browser Rendering API, specifying a JSON schema for the response format. ### Method `client.browserRendering.json.create(options)` ### Parameters - **options** (object) - Required - Configuration for the API call. - **account_id** (string) - Required - Your Cloudflare account ID. - **url** (string) - Required - The URL of the webpage to render. - **prompt** (string) - Optional - A prompt to guide the data extraction. - **response_format** (object) - Required - Specifies the format for the response. - **type** (string) - Required - Must be `json_schema`. - **schema** (object) - Required - The JSON schema defining the structure of the extracted data. ### Request Example ```typescript import Cloudflare from "cloudflare"; const client = new Cloudflare({ apiToken: process.env["CLOUDFLARE_API_TOKEN"], }); const json = await client.browserRendering.json.create({ account_id: process.env["CLOUDFLARE_ACCOUNT_ID"], url: "https://developers.cloudflare.com/", prompt: "Get me the list of AI products", response_format: { type: "json_schema", schema: { type: "object", properties: { products: { type: "array", items: { type: "object", properties: { name: { type: "string", }, link: { type: "string", }, }, required: ["name"], }, }, }, }, }, }); console.log(json); ``` ``` -------------------------------- ### Install Cloudflare's Puppeteer fork with bun Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker Install the necessary Puppeteer package as a development dependency in your 'browser-worker' directory. ```bash bun add -d @cloudflare/puppeteer ``` -------------------------------- ### Install @cloudflare/playwright with pnpm Source: https://developers.cloudflare.com/browser-run/playwright Install the Cloudflare Playwright package as a development dependency using pnpm. ```bash pnpm add -D @cloudflare/playwright ``` -------------------------------- ### Install Cloudflare's Puppeteer fork with pnpm Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker Install the necessary Puppeteer package as a development dependency in your 'browser-worker' directory. ```bash pnpm add -D @cloudflare/puppeteer ``` -------------------------------- ### Install Cloudflare Puppeteer and Robots Parser Source: https://developers.cloudflare.com/browser-run/llms-full.txt Install the necessary packages for Browser Run and robots-parser. These commands are available for npm, yarn, pnpm, and bun. ```bash npm i -D @cloudflare/puppeteer ``` ```bash yarn add -D @cloudflare/puppeteer ``` ```bash pnpm add -D @cloudflare/puppeteer ``` ```bash bun add -d @cloudflare/puppeteer ``` ```bash npm i robots-parser ``` ```bash yarn add robots-parser ``` ```bash pnpm add robots-parser ``` ```bash bun add robots-parser ``` -------------------------------- ### Example Deployed Worker URL Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker This is an example URL to access your deployed Worker. Replace `` and `` with your specific Cloudflare Worker details. ```text ..workers.dev/?url=https://example.com ``` -------------------------------- ### Install puppeteer-core with bun Source: https://developers.cloudflare.com/browser-run/cdp/puppeteer Install the `puppeteer-core` package, which is the version without a bundled Chrome. This is required to connect to Browser Run sessions. ```bash bun add puppeteer-core ``` -------------------------------- ### Install latest Wrangler Source: https://developers.cloudflare.com/browser-run/llms-full.txt Before starting a lab session for WebMCP testing, ensure you have the latest version of the Wrangler CLI installed globally. This command updates your local installation to the newest release. ```bash # make sure you have the latest version of wrangler npm i -g wrangler@latest ``` -------------------------------- ### Example response for Get all links Source: https://developers.cloudflare.com/browser-run/quick-actions/links-endpoint This is an example of the JSON response you can expect when successfully retrieving links from a webpage using the Cloudflare Browser Rendering Links API. The 'result' field contains an array of URLs found on the specified page. ```json { "success": true, "result": [ "https://developers.cloudflare.com/", "https://developers.cloudflare.com/products/", "https://developers.cloudflare.com/api/", "https://developers.cloudflare.com/fundamentals/api/reference/sdks/", "https://dash.cloudflare.com/", "https://developers.cloudflare.com/fundamentals/subscriptions-and-billing/", "https://developers.cloudflare.com/api/", "https://developers.cloudflare.com/changelog/", "https://developers.cloudflare.com/glossary/", "https://developers.cloudflare.com/reference-architecture/", "https://developers.cloudflare.com/web-analytics/", "https://developers.cloudflare.com/support/troubleshooting/http-status-codes/", "https://developers.cloudflare.com/registrar/", "https://developers.cloudflare.com/1.1.1.1/setup/", "https://developers.cloudflare.com/workers/", "https://developers.cloudflare.com/pages/", "https://developers.cloudflare.com/r2/", "https://developers.cloudflare.com/images/", "https://developers.cloudflare.com/stream/", "https://developers.cloudflare.com/products/?product-group=Developer+platform", "https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/", "https://developers.cloudflare.com/workers-ai/", "https://developers.cloudflare.com/vectorize/", "https://developers.cloudflare.com/ai-gateway/", "https://playground.ai.cloudflare.com/", "https://developers.cloudflare.com/products/?product-group=AI", "https://developers.cloudflare.com/cloudflare-one/access-controls/policies/", "https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/", "https://developers.cloudflare.com/cloudflare-one/traffic-policies/", "https://developers.cloudflare.com/cloudflare-one/remote-browser-isolation/", "https://developers.cloudflare.com/learning-paths/replace-vpn/concepts/", "https://developers.cloudflare.com/products/?product-group=Cloudflare+One", "https://workers.cloudflare.com/playground#LYVwNgLglgDghgJwgegGYHsHALQBM4RwDcABAEbogB2+CAngLzbPYZb6HbW5QDGU2AAwAmAIyiAzMIAsATlmi5ALhYs2wDnC40+AkeKlyFcgLAAoAMLoqEAKY3sAESgBnGOhdRo1pSXV4CYhIqOGBbBgAiKBpbAA8AOgArFwjSVCgwe1DwqJiE5IjzKxt7CGwAFToYW184GBgwPgIoa2REuAA3OBdeBFgIAGpgdFxwW3NzOPckElxbVDhwCBIAbzMSEm66Kl4-WwheAAsACgRbAEcQWxcIAEpV9Y2SXmsbkkOIYDASBhIAAwAPABCRwAeQs5QAmgAFACi70+YAAfI8NgCKLg6Cink8AYdREiABK2MBgdAkADqmDAuAByHx2JxJABMCR5UOrhIwEQAGsQDASAB3bokADm9lsCAItlw5DomxIFjJIFwqDAiFslMwPMl8TprNRzOQGKxfyIZkNZwgIAQVGCtkFJAAStd3FQXLZjh8vgAaB5M962OBzBAuXxrAMbCIvEoOCBVWwRXwROyxFDesBEI6ID0QBgAVXKADFsAAOCI+w0bAC+lZx1du5prlerRHMqmY6k02h4-CEYkkMnkilkRWsdgczjcHi8LSovn8mlIITCkTChE0qT8GSyq4iZDJZEKlnHpQqCdq9UavGarWS1gmZhWEW50QA+sNRpkk7k5vkUtW7Ydl2gQ9ro-YGEOxiyMwQA", "https://workers.cloudflare.com/playground#LYVwNgLglgDghgJwgegGYHsHALQBM4RwDcABAEbogB2+CAngLzbPYZb6HbW5QDGU2AAwB2AMwAWAKyCAjMICc8meIBcLFm2Ac4XGnwEiJ0uYuUBYAFABhdFQgBTO9gAiUAM4x0bqNFsqSmngExCRUcMD2DABEUDT2AB4AdABWblGkqFBgjuGRMXFJqVGWNnaOENgAKnQw9v5wMDBgfARQtsjJcABucG68CLAQANTA6Ljg9paWCZ5IJLj2qHDgECQA3hYkJL10VLwB9hC8ABYAFAj2AI4g9m4QAJTrm1skvLZ388EkDE8vL8f2MBgdD+KIAd0wYFwUQANM8tgBfIgWeEkC4QEAIKgkABKt08VDc9hSblsp2092RiLhSMs6mYmm0uh4-CEYiksgUSnEJVsDicrg8Xh8bSo-kC2lIYQi0QihG06QCWRyMqiZGBZGK1j55SqNTq20azV4rXaqVsUwsayiwDgsQA+qNxtkoip8gtCmkEXT6Yzgsz9GyjJzTOJmEA", "https://workers.cloudflare.com/playground#LYVwNgLglgDghgJwgegGYHsHALQBM4RwDcABAEbogB2+CAngLzbPYZb6HbW5QDGU2AAwBWABwBGAOyjRANgDMAFgCcygFwsWbYBzhcafASInS5S1QFgAUAGF0VCAFMH2ACJQAzjHQeo0e2ok2ngExCRUcMCODABEUDSOAB4AdABWHjGkqFBgzpHRcQkp6THWdg7OENgAKnQwjoFwMDBgfARQ9sipcABucB68CLAQANTA6LjgjtbWSd5IJLiOqHDgECQA3lYkJP10VLxBjhC8ABYAFAiOAI4gjh4QAJSb2zskyABUH69vHyQASo4WnBeI4SAADK7jJzgkgAdz8pxIEFOYNOPnWdEo8M8SIg6BIHmcuBIV1u9wgHmR6B+Ow+yFpvHsD1JjmhYIYJBipwgEBgHjUyGQSUiLUcySZwEyVlpVwgIAQVF2cLgfiOJwuUPQTgANKzyQ9HkRXgBfHVWE1EayaZjaXT6Hj8IRiKQyBQqZRlexOFzuLw+PwdKiBYK6UgRKKxKKEXSZII5PKRmJkMDoMilWzeyo1OoNXbNVq8dqddL2GZWDYxYCqqgAfXGk1yMTUhSWxQyJutNrtoQdhmdJjd5mUzCAA" ] } ``` -------------------------------- ### Complete Wrangler Configuration with Queue Bindings (TOML) Source: https://developers.cloudflare.com/browser-run/llms-full.txt This is a complete `wrangler.toml` example including queue configurations, KV namespaces, and browser bindings. Ensure `compatibility_date` is set to a recent date. ```toml "$schema" = "./node_modules/wrangler/config-schema.json" name = "web-crawler" main = "src/index.ts" # Set this to today's date compatibility_date = "2026-04-29" compatibility_flags = [ "nodejs_compat" ] [[kv_namespaces]] binding = "CRAWLER_SCREENSHOTS_KV" id = "" [[kv_namespaces]] binding = "CRAWLER_LINKS_KV" id = "" [browser] binding = "CRAWLER_BROWSER" [[queues.consumers]] queue = "queues-web-crawler" max_batch_timeout = 60 [[queues.producers]] queue = "queues-web-crawler" binding = "CRAWLER_QUEUE" ``` -------------------------------- ### Extract data with prompt and JSON schema Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example demonstrates how to use the Browser Rendering API to extract specific data from a webpage by providing both a prompt to guide the extraction and a JSON schema to define the output structure. ```APIDOC ## POST /accounts/CF_ACCOUNT_ID/browser-rendering/json ### Description Renders a webpage and extracts data based on a prompt and a JSON schema. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts/CF_ACCOUNT_ID/browser-rendering/json` ### Parameters #### Request Body - **url** (string) - Required - The URL of the webpage to render. - **prompt** (string) - Required - The prompt to guide data extraction. - **response_format** (object) - Required - Defines the structure of the output. - **type** (string) - Required - Must be "json_schema". - **schema** (object) - Required - A JSON schema defining the desired output structure. ### Request Example ```json { "url": "https://developers.cloudflare.com/", "prompt": "Get me the list of AI products", "response_format": { "type": "json_schema", "schema": { "type": "object", "properties": { "products": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "link": { "type": "string" } }, "required": [ "name" ] } } } } } } ``` ### Response #### Success Response (200) - **success** (boolean) - Indicates if the request was successful. - **result** (object) - The extracted data conforming to the JSON schema. #### Response Example ```json { "success": true, "result": { "products": [ { "name": "Build a RAG app", "link": "https://developers.cloudflare.com/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/" }, { "name": "Workers AI", "link": "https://developers.cloudflare.com/workers-ai/" }, { "name": "Vectorize", "link": "https://developers.cloudflare.com/vectorize/" }, { "name": "AI Gateway", "link": "https://developers.cloudflare.com/ai-gateway/" }, { "name": "AI Playground", "link": "https://playground.ai.cloudflare.com/" } ] } } ``` ``` -------------------------------- ### POST Request with JSON Schema Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example demonstrates how to make a POST request to the Browser Rendering API to get structured JSON output. You provide a JSON schema in the request body to define the desired output structure. ```APIDOC ## POST /accounts/:account_id/browser-rendering/json ### Description Submits a request to render a webpage and extract data according to a specified JSON schema. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts/CF_ACCOUNT_ID/browser-rendering/json` ### Parameters #### Request Body - **response_format** (object) - Required - Specifies the format for the response. - **type** (string) - Required - Must be `json_schema`. - **schema** (object) - Required - The JSON schema defining the structure of the extracted data. ### Request Example ```json { "response_format": { "type": "json_schema", "schema": { "type": "object", "properties": { "products": { "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string" }, "link": { "type": "string" } }, "required": [ "name" ] } } } } } } ``` ### Response #### Success Response (200) - **success** (boolean) - Indicates if the request was successful. - **result** (object) - The extracted data conforming to the provided JSON schema. #### Response Example ```json { "success": true, "result": { "products": [ { "name": "Workers", "link": "https://developers.cloudflare.com/workers/" }, { "name": "Pages", "link": "https://developers.cloudflare.com/pages/" } ] } } ``` ``` -------------------------------- ### Complete Wrangler Configuration with Queue Bindings (JSONC) Source: https://developers.cloudflare.com/browser-run/llms-full.txt This is a complete `wrangler.jsonc` example including queue configurations, KV namespaces, and browser bindings. Ensure `compatibility_date` is set to a recent date. ```json { "$schema": "./node_modules/wrangler/config-schema.json", "name": "web-crawler", "main": "src/index.ts", // Set this to today's date "compatibility_date": "2026-04-29", "compatibility_flags": ["nodejs_compat"], "kv_namespaces": [ { "binding": "CRAWLER_SCREENSHOTS_KV", "id": "", }, { "binding": "CRAWLER_LINKS_KV", "id": "", }, ], "browser": { "binding": "CRAWLER_BROWSER", }, "queues": { "consumers": [ { "queue": "queues-web-crawler", "max_batch_timeout": 60, }, ], "producers": [ { "queue": "queues-web-crawler", "binding": "CRAWLER_QUEUE", }, ], }, } ``` -------------------------------- ### Generate a Playwright trace file in a Worker Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example starts tracing, navigates to a page, adds to-do items, stops tracing, and saves the trace as a zip file. The trace includes screenshots and snapshots for debugging. Ensure the 'MYBROWSER' binding is configured in Wrangler. ```typescript import fs from "fs"; import { launch } from "@cloudflare/playwright"; export default { async fetch(request: Request, env: Env) { const browser = await launch(env.MYBROWSER); const page = await browser.newPage(); // Start tracing before navigating to the page await page.context().tracing.start({ screenshots: true, snapshots: true }); await page.goto("https://demo.playwright.dev/todomvc"); const TODO_ITEMS = [ "buy some cheese", "feed the cat", "book a doctors appointment", ]; const newTodo = page.getByPlaceholder("What needs to be done?"); for (const item of TODO_ITEMS) { await newTodo.fill(item); await newTodo.press("Enter"); } // Stop tracing and save the trace to a zip file await page.context().tracing.stop({ path: "trace.zip" }); await browser.close(); const file = await fs.promises.readFile("trace.zip"); return new Response(file, { status: 200, headers: { "Content-Type": "application/zip", }, }); }, }; ``` -------------------------------- ### Build the Project with npm Source: https://developers.cloudflare.com/browser-run/stagehand Build the project using the npm build script. ```bash npm run build ``` -------------------------------- ### Configure MCP Client for Session Recording Source: https://developers.cloudflare.com/browser-run/features/session-recording Example configuration for an MCP client to enable session recording by adding `recording=true` to the `--wsEndpoint` URL. ```json { "mcpServers": { "browser-rendering": { "command": "npx", "args": [ "-y", "chrome-devtools-mcp@latest", "--wsEndpoint=wss://api.cloudflare.com/client/v4/accounts//browser-rendering/devtools/browser?recording=true&keep_alive=600000", "--wsHeaders={"Authorization":"Bearer "}" ] } } } ``` -------------------------------- ### Crawl Endpoint Example Source: https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint Example of how to use the /crawl endpoint to crawl a website, specifying crawl purposes and desired formats. This example demonstrates how to handle Content Signals by declaring only the 'search' purpose. ```APIDOC ## POST /crawl ### Description Initiates a website crawl. Respects robots.txt directives, bot protection, and Content Signals. Allows customization of crawl sources and declared purposes. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl` ### Parameters #### Request Body - **url** (string) - Required - The starting URL for the crawl. - **crawlPurposes** (array of strings) - Optional - Declares the intended use of the crawled content (e.g., `["search", "ai-input", "ai-train"]`). Defaults to all three if not provided. Must be a subset of disallowed purposes by Content Signals. - **formats** (array of strings) - Optional - Specifies the desired output formats for the crawled content (e.g., `["markdown"]`). ### Request Example ```json { "url": "https://example.com", "crawlPurposes": ["search"], "formats": ["markdown"] } ``` ### Response #### Success Response (200) - **result** (object) - Contains details about the crawl job. - **id** (string) - The unique identifier for the crawl job. - **url** (string) - The URL that was crawled. - **status** (string) - The current status of the crawl job. #### Response Example ```json { "success": true, "errors": [], "messages": [], "result": { "id": "crawl-job-id-12345", "url": "https://example.com", "status": "processing" } } ``` ### Error Handling - **400 Bad Request**: Returned if Content Signals are disallowed for the declared `crawlPurposes`. ``` -------------------------------- ### Navigate to Application Directory Source: https://developers.cloudflare.com/browser-run/llms-full.txt After creating the application, change into the new project directory using the terminal. ```bash cd queues-web-crawler ``` -------------------------------- ### Create a new Worker project with npm Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker Use this command to initialize a new Worker project named 'browser-worker' with the 'Hello World' template and 'Worker only' configuration. ```bash npm create cloudflare@latest -- browser-worker ``` -------------------------------- ### Install playwright-core with pnpm Source: https://developers.cloudflare.com/browser-run/cdp/playwright Install the playwright-core package, which is the version without bundled browsers. ```bash pnpm add playwright-core ``` -------------------------------- ### Create a new Worker project with pnpm Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker Use this command to initialize a new Worker project named 'browser-worker' with the 'Hello World' template and 'Worker only' configuration. ```bash pnpm create cloudflare@latest browser-worker ``` -------------------------------- ### Install playwright-core with yarn Source: https://developers.cloudflare.com/browser-run/cdp/playwright Install the playwright-core package, which is the version without bundled browsers. ```bash yarn add playwright-core ``` -------------------------------- ### Install playwright-core with npm Source: https://developers.cloudflare.com/browser-run/cdp/playwright Install the playwright-core package, which is the version without bundled browsers. ```bash npm i playwright-core ``` -------------------------------- ### Crawl Endpoint with All Optional Parameters Source: https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint Use this example to crawl a URL with all available optional parameters, including crawl purposes, limits, depth, formats, rendering options, and advanced filtering with include/exclude patterns. ```bash curl -X POST 'https://api.cloudflare.com/client/v4/accounts/{account_id}/browser-rendering/crawl' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '{ \ "url": "https://www.exampledocs.com/docs/", \ "crawlPurposes": ["search"], \ "limit": 50, \ "depth": 2, \ "formats": ["markdown"], \ "render": false, \ "maxAge": 7200, \ "modifiedSince": 1704067200, \ "source": "all", \ "options": { \ "includeExternalLinks": true, \ "includeSubdomains": true, \ "includePatterns": [ \ "**/api/v1/*" \ ], \ "excludePatterns": [ \ "*/learning-paths/*" \ ] \ } \ }' ``` -------------------------------- ### Initialize Browser and Page Source: https://developers.cloudflare.com/browser-run/llms-full.txt Launches a Puppeteer browser instance and creates a new page. Ensure the BROWSER environment variable is correctly configured. ```typescript const browser = await puppeteer.launch(env.BROWSER); const page = await browser.newPage(); ``` -------------------------------- ### Install @cloudflare/playwright with yarn Source: https://developers.cloudflare.com/browser-run/playwright Install the Cloudflare Playwright package as a development dependency using yarn. ```bash yarn add -D @cloudflare/playwright ``` -------------------------------- ### Create Cloudflare Worker Project Source: https://developers.cloudflare.com/browser-run/how-to/pdf-generation Use the create-cloudflare CLI to scaffold a new Hello World Cloudflare Worker project. ```bash npm create cloudflare@latest -- browser-worker ``` ```bash yarn create cloudflare browser-worker ``` ```bash pnpm create cloudflare@latest browser-worker ``` -------------------------------- ### Install @cloudflare/playwright with npm Source: https://developers.cloudflare.com/browser-run/playwright Install the Cloudflare Playwright package as a development dependency using npm. ```bash npm i -D @cloudflare/playwright ``` -------------------------------- ### Browser Run Response Example Source: https://developers.cloudflare.com/browser-run/llms-full.txt Example of a successful JSON response from Browser Run when extracting structured data. ```json { "success": true, "result": { "h1": "Heading 1", "h2": "Heading 2" } } ``` -------------------------------- ### Create a snapshot from custom HTML Source: https://developers.cloudflare.com/browser-run/quick-actions/snapshot This example shows how to create a snapshot using custom HTML content. It includes advanced options like disabling JavaScript, setting full-page screenshots, customizing the viewport, and specifying load behavior. ```APIDOC ## POST /snapshot with custom HTML ### Description Captures a snapshot using provided HTML content, with advanced options for screenshot, viewport, and loading behavior. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts//browser-rendering/snapshot` ### Parameters #### Request Body - **html** (string) - Required - The custom HTML content to render. - **setJavaScriptEnabled** (boolean) - Optional - Whether to enable JavaScript execution (defaults to true). - **screenshotOptions** (object) - Optional - Options for customizing the screenshot. - **fullPage** (boolean) - Optional - Whether to capture a full-page screenshot. - **viewport** (object) - Optional - Defines the viewport size. - **width** (number) - Optional - The viewport width. - **height** (number) - Optional - The viewport height. - **gotoOptions** (object) - Optional - Options for controlling page navigation and loading. - **waitUntil** (string) - Optional - When to consider the page loaded (e.g., 'domcontentloaded', 'networkidle0'). - **timeout** (number) - Optional - Maximum time in milliseconds to wait for the page to load. ### Request Example ```json { "html": "Advanced Snapshot", "setJavaScriptEnabled": false, "screenshotOptions": { "fullPage": true }, "viewport": { "width": 1200, "height": 800 }, "gotoOptions": { "waitUntil": "domcontentloaded", "timeout": 30000 } } ``` ### Response #### Success Response (200) - **screenshot** (string) - Base64-encoded screenshot of the webpage. - **content** (string) - The rendered HTML content of the webpage. #### Response Example ```json { "success": true, "result": { "screenshot": "AdvancedBase64Screenshot", "content": "Advanced Snapshot" } } ``` ``` -------------------------------- ### Create a new Worker project with yarn Source: https://developers.cloudflare.com/browser-run/how-to/deploy-worker Use this command to initialize a new Worker project named 'browser-worker' with the 'Hello World' template and 'Worker only' configuration. ```bash yarn create cloudflare browser-worker ``` -------------------------------- ### Install Puppeteer Core Source: https://developers.cloudflare.com/browser-run/llms-full.txt Install the puppeteer-core package, which is the version without bundled Chrome. Use this for Node.js environments. ```bash npm i puppeteer-core ``` ```bash yarn add puppeteer-core ``` ```bash pnpm add puppeteer-core ``` ```bash bun add puppeteer-core ``` -------------------------------- ### Install Puppeteer Package Source: https://developers.cloudflare.com/browser-run/how-to/pdf-generation Install the @cloudflare/puppeteer package to enable control over Browser Run instances within your Worker. ```bash npm i -D @cloudflare/puppeteer ``` ```bash yarn add -D @cloudflare/puppeteer ``` ```bash pnpm add -D @cloudflare/puppeteer ``` ```bash bun add -d @cloudflare/puppeteer ``` -------------------------------- ### Initialize Crawler Worker with Puppeteer Source: https://developers.cloudflare.com/browser-run/llms-full.txt This is a skeleton for the crawler. It launches the Puppeteer browser and iterates through the Queue's received messages. It fetches the site's `robots.txt` and uses `robots-parser` to check that this site allows crawling. If crawling is not allowed, the message is `ack`'ed, removing it from the Queue. If crawling is allowed, you can continue to crawl the site. The `puppeteer.launch()` is wrapped in a `try...catch` to allow the whole batch to be retried if the browser launch fails. The browser launch may fail due to going over the limit for number of browsers per account. ```typescript import puppeteer from "@cloudflare/puppeteer"; import robotsParser from "robots-parser"; async queue(batch, env, ctx): Promise { let browser: puppeteer.Browser | null = null; try { browser = await puppeteer.launch(env.CRAWLER_BROWSER); } catch { batch.retryAll(); return; } for (const message of batch.messages) { const { url } = message.body; let isAllowed = true; try { const robotsTextPath = new URL(url).origin + "/robots.txt"; const response = await fetch(robotsTextPath); const robots = robotsParser(robotsTextPath, await response.text()); isAllowed = robots.isAllowed(url) ?? true; // respect robots.txt! } catch {} if (!isAllowed) { message.ack(); continue; } // TODO: crawl! message.ack(); } await browser.close(); } ``` -------------------------------- ### Fetch rendered HTML from a URL Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example demonstrates how to use `curl` to POST a request to the /content endpoint, providing a URL to fetch its rendered HTML. Replace `` and `` with your actual account ID and API token. ```APIDOC ## Fetch rendered HTML from a URL ### Description This endpoint captures the fully rendered HTML of a given URL after JavaScript execution. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts//browser-rendering/content` ### Parameters #### Request Body - **url** (string) - Required - The URL of the webpage to fetch. ### Request Example ```bash curl -X 'POST' 'https://api.cloudflare.com/client/v4/accounts//browser-rendering/content' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer ' \ -d '{"url": "https://developers.cloudflare.com/"}' ``` ### Response #### Success Response (200) - **html** (string) - The rendered HTML of the page. #### Response Example ```json { "html": "\n\n...\n...\n" } ``` ``` -------------------------------- ### Install Playwright MCP (bun) Source: https://developers.cloudflare.com/browser-run/playwright/playwright-mcp Install the Playwright MCP npm package as a development dependency using bun. ```bash bun add -d @cloudflare/playwright-mcp ``` -------------------------------- ### Take a screenshot from custom HTML Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example demonstrates how to take a screenshot from custom HTML content. It sets the HTML to 'Hello World!' and configures screenshot options to omit the background for transparency. ```APIDOC ## POST /accounts//browser-rendering/screenshot ### Description Capture a screenshot from custom HTML content. ### Method POST ### Endpoint https://api.cloudflare.com/client/v4/accounts//browser-rendering/screenshot ### Parameters #### Request Body - **html** (string) - Required - The HTML content of the page. - **screenshotOptions** (object) - Optional - Options to control the screenshot. - **omitBackground** (boolean) - Optional - Hides the default white background. ### Request Example ```json { "html": "Hello World!", "screenshotOptions": { "omitBackground": true } } ``` ### Response #### Success Response (200) - The response will be the screenshot image file. ``` -------------------------------- ### Install Playwright MCP (pnpm) Source: https://developers.cloudflare.com/browser-run/playwright/playwright-mcp Install the Playwright MCP npm package as a development dependency using pnpm. ```bash pnpm add -D @cloudflare/playwright-mcp ``` -------------------------------- ### Install Playwright MCP (yarn) Source: https://developers.cloudflare.com/browser-run/playwright/playwright-mcp Install the Playwright MCP npm package as a development dependency using yarn. ```bash yarn add -D @cloudflare/playwright-mcp ``` -------------------------------- ### Convert a URL to PDF Source: https://developers.cloudflare.com/browser-run/quick-actions/pdf-endpoint This example demonstrates how to convert a given URL into a PDF document. You can also inject custom CSS and external stylesheets. ```APIDOC ## POST /client/v4/accounts//browser-rendering/pdf ### Description Generates a PDF from a given URL. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts//browser-rendering/pdf` ### Parameters #### Request Body - **url** (string) - Required - The URL of the webpage to convert to PDF. - **addStyleTag** (array) - Optional - An array of objects to inject custom CSS. - **content** (string) - The CSS content. - **url** (string) - The URL of an external CSS stylesheet. ### Request Example ```json { "url": "https://example.com/", "addStyleTag": [ { "content": "body { font-family: Arial; }" } ] } ``` ### Response #### Success Response (200) Returns the PDF content. #### Response Example (Binary PDF content) ``` -------------------------------- ### Install Playwright MCP (npm) Source: https://developers.cloudflare.com/browser-run/playwright/playwright-mcp Install the Playwright MCP npm package as a development dependency using npm. ```bash npm i -D @cloudflare/playwright-mcp ``` -------------------------------- ### Convert a URL to PDF Source: https://developers.cloudflare.com/browser-run/llms-full.txt This example shows how to convert a given URL into a PDF document. You can also inject custom CSS using the `addStyleTag` parameter. ```APIDOC ## POST /accounts//browser-rendering/pdf ### Description Instructs the browser to generate a PDF of a webpage. ### Method POST ### Endpoint `https://api.cloudflare.com/client/v4/accounts//browser-rendering/pdf` ### Parameters #### Request Body - **url** (string) - Required - The URL of the webpage to convert to PDF. - **addStyleTag** (array) - Optional - An array of objects to inject custom CSS into the page. - **content** (string) - The CSS content. - **url** (string) - The URL of an external CSS stylesheet. ### Request Example ```json { "url": "https://example.com/", "addStyleTag": [ { "content": "body { font-family: Arial; }" } ] } ``` ### Response #### Success Response (200) Returns the generated PDF file. #### Response Example (Binary PDF data) ``` -------------------------------- ### Install Playwright npm package Source: https://developers.cloudflare.com/browser-run/llms-full.txt Install the Playwright npm package as a development dependency using your preferred package manager. ```bash npm i -D @cloudflare/playwright ``` ```bash yarn add -D @cloudflare/playwright ``` ```bash pnpm add -D @cloudflare/playwright ``` ```bash bun add -d @cloudflare/playwright ```