Scrapfly

https://scrapfly.io/docs
Admin
Scrapfly provides a suite of web scraping APIs, including tools for bypassing anti-bot protection,...
Tokens:1,252,084
Snippets:14,781
Trust Score:5.6
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install Scrapfly Go SDK

Source: https://scrapfly.io/docs/onboarding/golang?view=markdown

Use 'go get' to install the Scrapfly Go SDK. This command fetches and installs the SDK package.

```bash
go get github.com/scrapfly/go-scrapfly
```

--------------------------------

### Python Virtual Environment Setup

Source: https://scrapfly.io/docs/mcp/integrations/langchain?iframe=1

Demonstrates setting up a Python virtual environment and installing the Scrapfly MCP adapters package.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install langchain-mcp-adapters
```

--------------------------------

### Start MCP Server

Source: https://scrapfly.io/docs/mcp/faq?iframe=1

Run the MCP server after installing dependencies and configuring your API key. This command starts the local server instance.

```bash
npm start
```

--------------------------------

### Dockerfile Example for Debian-slim

Source: https://scrapfly.io/docs/proxy-saver/certificates?view=markdown

Installs ca-certificates and curl, then downloads and installs the Scrapfly CA certificate for Debian-slim Docker images.

```dockerfile
FROM debian:bullseye-slim

# Install ca-certificates and curl
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl && \
rm -rf /var/lib/apt/lists/*

# Download and install the Proxy Saver certificate
RUN curl -fsSL -o /usr/local/share/ca-certificates/scrapfly-ca.crt https://scrapfly.io/ca.crt && \
update-ca-certificates

```

--------------------------------

### Quick Start Cloud Browser Initialization

Source: https://scrapfly.io/docs/cloud-browser-api/captcha-solver?view=markdown

Basic setup for connecting to the Cloud Browser without automatic captcha arming.

```python
from scrapfly import ScrapflyClient
from playwright.sync_api import sync_playwright

client = ScrapflyClient(key="")
ws_url = client.cloud_browser(proxy_pool="residential", country="us")
```

--------------------------------

### Install Scrapfly CLI

Source: https://scrapfly.io/docs/cli

Installs the Scrapfly CLI using a curl command. This is the quickest way to get started.

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh
```

--------------------------------

### Pin Scrapfly CLI Version During Installation

Source: https://scrapfly.io/docs/cli/github-actions

This example shows how to install a specific version of the Scrapfly CLI using the installer script with the `--version` flag. This ensures reproducible workflows by avoiding the latest version.

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --version v1.2.3
```

--------------------------------

### Configure VNC Session Access

Source: https://scrapfly.io/docs/cli/commands

Example of starting a browser session with VNC enabled and retrieving the required connection credentials.

```bash
scrapfly browser --session demo --enable-vnc --vnc-password hunter2 start &
# [session demo] vnc vnc://{run_id}@browser.scrapfly.io:5901
# [session demo] vnc password {project_salt}-hunter2
```

--------------------------------

### Configure CLI Installation with Flags

Source: https://scrapfly.io/docs/cli/install

Customizes the installation process by specifying a version or a custom installation directory.

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --version v0.1.0
```

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --prefix "$HOME/.local/bin"
```

--------------------------------

### Full RAG Workflow with ScrapflyReader

Source: https://scrapfly.io/docs/integration/llamaindex?__tab%5Burl-source-tab%5D=source-remote

A complete example showing the setup of ScrapflyReader, document loading, index creation, and querying with OpenAI.

```python
import os
from llama_index.core import VectorStoreIndex
from llama_index.llms.openai import OpenAI
from llama_index.readers.web import ScrapflyReader

# 1. Add OpenAI API key
os.environ["OPENAI_API_KEY"] = "YOUR OPEN API KEY"

# 2. Set up ScrapflyReader with your ScrapFly API key
scrapfly_reader = ScrapflyReader(
    api_key="YOUR SCRAPFLY KEY",
)

# 3. scrape web pages as markdown
documents = scrapfly_reader.load_data(
    urls=["https://web-scraping.dev/product/1"],
    scrape_config={"render_js": True},  # note here you can configure scrape options
    scrape_format="markdown",
)

# 4. Create index and query engine
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(
    llm=OpenAI(model="gpt-3.5-turbo-0125"),
)
# 5. Define prompt template
prompt_template = "Given the data fetched from the specified product URLs, \
                   find the following product fields {fields} in the \
                   provided markdown and return a JSON".format

# 6. run query
response = query_engine.query(prompt_template(
    fields=["price", "title"],
))
print(response)
# prints:
# {
#     "price": "$9.99 from $12.99",
#     "title": "Box of Chocolate Candy"
# }
```

--------------------------------

### Install OpenAI SDK dependencies

Source: https://scrapfly.io/docs/mcp/integrations/openai?__tab%5Burl-source-tab%5D=source-list

Install the necessary packages for your environment.

```bash
pip install openai requests
```

```bash
npm install openai axios
```

--------------------------------

### Setup Virtual Environment

Source: https://scrapfly.io/docs/mcp/integrations/crewai?__tab%5Burl-source-tab%5D=source-remote

Recommended commands to create and activate a Python virtual environment for dependency management.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install crewai
```

--------------------------------

### Perform a Basic Scrape

Source: https://scrapfly.io/docs/onboarding/golang

Example of initializing the client and scraping a product page with CSS selector parsing.

```go
package main

import (
  "fmt"
  "log"
  "github.com/scrapfly/go-scrapfly"
)

func main(){
  client, err := scrapfly.New("{{ YOUR_API_KEY }}")
  if err != nil { log.Fatal(err) }
  res, err := client.Scrape(&scrapfly.ScrapeConfig{
    URL:      "https://web-scraping.dev/product/1",
    RenderJS: true,
  })
  if err != nil { log.Fatal(err) }
  // HTML content
  fmt.Println(len(res.Result.Content))
  // Parse HTML with CSS selectors
  sel, _ := res.Selector()
  fmt.Println(sel.Find("h3").First().Text())
}
```

--------------------------------

### GET Request Example

Source: https://scrapfly.io/docs/scrape-api/getting-started

Example of a GET request to scrape a URL with JavaScript rendering enabled and specifying a country.

```APIDOC
## GET /scrape

### Description
Scrapes a target URL using the Scraping API. Supports JavaScript rendering and various proxy configurations.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **url** (string) - Required - Target URL to scrape. Must be URL encoded.
- **key** (string) - Required - API Key for authentication.
- **country** (string) - Optional - Proxy country (ISO 3166-1 alpha-2). Defaults to random. Supports exclusions and weighted distribution.
- **render_js** (boolean) - Optional - Whether to render JavaScript on the page. Defaults to false.

### Request Example
```bash
curl -X GET "https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key="
```

### Response
#### Success Response (200)
- **result.content** (string) - The scraped content of the page.
- **config** (object) - The scrape configuration used.
- **context** (object) - Information about activated features.

#### Response Example
```json
{
  "result": {
    "content": "<html>...</html>"
  },
  "config": {
    "url": "https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this",
    "country": "us",
    "render_js": true
  },
  "context": {
    "status": "success"
  }
}
```
```

--------------------------------

### Full Authentication Script Setup

Source: https://scrapfly.io/docs/cloud-browser-api/session-resume?view=markdown

Complete setup script including imports, configuration variables, and the authentication flow.

```python
from playwright.sync_api import sync_playwright
import requests

API_KEY = ''
SESSION_ID = 'my-session-id'
BROWSER_WS = f'wss://browser.scrapfly.io?api_key={API_KEY}&session={SESSION_ID}&auto_close=false'

# Step 1: authenticate inside Cloud Browser
with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp(BROWSER_WS)
    page = browser.contexts[0].new_page()
    page.goto('https://target.com/login')
    page.fill('#username', 'user@example.com')
    page.fill('#password', 'secret')
    page.click('#login-btn')
    page.wait_for_load_state('networkidle')
    browser.close()
```

--------------------------------

### Setup Virtual Environment and Install CrewAI

Source: https://scrapfly.io/docs/mcp/integrations/crewai?iframe=1

Use a virtual environment to manage dependencies and install the CrewAI package. This ensures a clean project setup.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install crewai
```

--------------------------------

### Dockerfile Example for Alpine

Source: https://scrapfly.io/docs/proxy-saver/certificates?view=markdown

Installs ca-certificates and curl, then downloads and installs the Scrapfly CA certificate for Alpine Docker images.

```dockerfile
FROM alpine:latest

# Install ca-certificates and curl
RUN apk add --no-cache ca-certificates curl

# Download and install the Proxy Saver certificate
RUN curl -fsSL -o /usr/local/share/ca-certificates/scrapfly-ca.crt https://scrapfly.io/ca.crt && \
update-ca-certificates

```

--------------------------------

### Install Browser Use

Source: https://scrapfly.io/docs/cloud-browser-api/browser-use

Install the required Python package via pip.

```bash
pip install browser-use
```

--------------------------------

### Get Content (cURL)

Source: https://scrapfly.io/docs/crawler-api/getting-started?view=markdown

Retrieve extracted content from crawled pages in the specified format. This example shows how to get content in markdown format.

```bash
# Get all content in markdown format
curl "https://api.scrapfly.io/crawl/{crawler_uuid}/contents?key={{ YOUR_API_KEY }}&format=markdown"
```

--------------------------------

### Install All Scrapfly SDK Modules

Source: https://scrapfly.io/docs/onboarding?view=markdown

Command to install the SDK with all optional extras included.

```bash
$ pip install 'scrapfly-sdk[all]'
```

--------------------------------

### Initialize Go WebSocket client

Source: https://scrapfly.io/docs/cloud-browser-api/errors?view=markdown

Boilerplate for setting up a Go WebSocket connection.

```go
package main

import (
    "context"
    "errors"
    "fmt"
    "time"

```

--------------------------------

### Get Extraction Monitoring Target Metrics (Go)

Source: https://scrapfly.io/docs/extraction-api/monitoring

Use the Go SDK to get monitoring statistics for a domain. This example demonstrates basic error handling and metric retrieval.

```go
package main

import (
    "fmt"
    "log"

    scrapfly "github.com/scrapfly/go-scrapfly"
)

func main() {
    client, err := scrapfly.New("")
    if err != nil {
        log.Fatal(err)
    }

    stats, err := client.GetExtractionMonitoringTargetMetrics(scrapfly.MonitoringTargetMetricsOptions{
        Domain: "web-scraping.dev",
        Period: scrapfly.MonitoringPeriodLast24h,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(stats)
}
```

--------------------------------

### Clone and Build from Source

Source: https://scrapfly.io/docs/tools/antibot-detector?__tab%5Burl-source-tab%5D=source-list

Commands to clone the repository and install dependencies for building the extension.

```bash
git clone https://github.com/scrapfly/Antibot-Detector.git
cd Antibot-Detector
```

```bash
npm install
```

```bash
npm run build
```

--------------------------------

### Get Screenshot Monitoring Metrics (Rust)

Source: https://scrapfly.io/docs/screenshot-api/monitoring?iframe=1

A Rust example using the scrapfly_sdk to get screenshot monitoring metrics. It utilizes Tokio for asynchronous operations and includes basic error handling.

```rust
use scrapfly_sdk::{Client, MonitoringPeriod, MonitoringTargetMetricsOptions};

#[tokio::main]
async fn main() -> Result<(), Box> {
    let client = Client::builder().api_key("").build()?;
    let stats = client
        .get_screenshot_monitoring_target_metrics(&MonitoringTargetMetricsOptions {
            domain: "web-scraping.dev".into(),
            group_subdomain: false,
            period: Some(MonitoringPeriod::Last24h),
            start: None,
            end: None,
            include_webhook: false,
        })
        .await?;
    println!("{}", stats);
    Ok(())
}
```

--------------------------------

### Install warcpp C++ Library

Source: https://scrapfly.io/docs/crawler-api/warc-format?__tab%5Burl-source-tab%5D=source-list

Commands to clone and build the warcpp library from source.

```bash
git clone https://github.com/pisa-engine/warcpp.git
cd warcpp
mkdir build && cd build
cmake ..
make
```

--------------------------------

### Trigger Scrape with Webhook via URL

Source: https://scrapfly.io/docs/scrape-api/webhook?__tab%5Burl-source-tab%5D=source-list

A direct GET request example for triggering a scrape with a webhook.

```http
https://api.scrapfly.io/scrape?webhook_name=example&key=&url=https%3A%2F%2Fhttpbin.dev%2Fhtml
```

--------------------------------

### Residential Proxy Request URL

Source: https://scrapfly.io/docs/scrape-api/proxy?__tab%5Burl-source-tab%5D=source-list

Example of a direct GET request URL configured for the public residential proxy pool.

```http
https://api.scrapfly.io/scrape?proxy_pool=public_residential_pool&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything
```

--------------------------------

### Enable Anti-Scraping Protection via API

Source: https://scrapfly.io/docs/scrape-api/anti-scraping-protection?view=markdown

Examples of how to enable ASP using cURL or a direct HTTP GET request.

```bash
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "asp=true" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.dev/anything"
```

```http
https://api.scrapfly.io/scrape?asp=true&key=&url=https%3A%2F%2Fhttpbin.dev%2Fanything
```

--------------------------------

### OpenAI Assistant Setup and Web Scraping

Source: https://scrapfly.io/docs/mcp/integrations/openai

This snippet demonstrates setting up an OpenAI Assistant, creating a thread, sending a user message to scrape and summarize web content, and handling function calls for web scraping. It includes a loop to poll the run status and submit tool outputs.

```python
from openai import OpenAI
from scrapfly import ScraperConfig, ScrapflyClient

# Initialize Scrapfly client with API key
client_scrapfly = ScrapflyClient(scrapfly_key="YOUR_SCRAPFLY_API_KEY")

def scrape_webpage(**kwargs):
    response = client_scrapfly.scrape(**kwargs)
    return response.json()['content']

# Initialize OpenAI client
client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

# Create or retrieve an assistant (replace with your assistant ID)
# assistant = client.beta.assistants.create(name="Web Scraper", ...)
assistant_id = "asst_YOUR_ASSISTANT_ID"
assistant = client.beta.assistants.retrieve(assistant_id)

tools=[{
    "type": "function",
    "function": {
        "name": "scrape_webpage",
        "description": "Scrape a given URL and return the content in JSON format.",
        "parameters": {
            "type": "object",
            "properties": {
                "url": {
                    "type": "string",
                    "description": "The URL to scrape."
                }
            },
            "required": ["url"]
        }
    }
}]

# Update assistant with tools if not already present
if assistant.tools is None or not any(t.type == 'function' and t.function.name == 'scrape_webpage' for t in assistant.tools):
    assistant = client.beta.assistants.update(assistant_id, tools=tools)

# Create a thread
thread = client.beta.threads.create()

# User message
client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Scrape the top posts from https://news.ycombinator.com and summarize them"
)

# Run the assistant
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id
)

# Handle function calling
import time
while run.status != "completed":
    run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)

    if run.status == "requires_action":
        tool_calls = run.required_action.submit_tool_outputs.tool_calls
        tool_outputs = []

        for tool_call in tool_calls:
            if tool_call.function.name == "scrape_webpage":
                import json
                args = json.loads(tool_call.function.arguments)
                result = scrape_webpage(**args)
                tool_outputs.append({
                    "tool_call_id": tool_call.id,
                    "output": result
                })

        # Submit function outputs
        client.beta.threads.runs.submit_tool_outputs(
            thread_id=thread.id,
            run_id=run.id,
            tool_outputs=tool_outputs
        )

    time.sleep(1)

# Get final response
messages = client.beta.threads.messages.list(thread_id=thread.id)
print(messages.data[0].content[0].text.value)

```

--------------------------------

### List Downloaded Files Response

Source: https://scrapfly.io/docs/cloud-browser-api/human-in-the-loop/embed

Example JSON response structure returned by the GET /run/{run_id}/downloads endpoint.

```json
{
  "files": [
    {
      "filename": "invoice-2026-Q1.pdf",
      "size": 184523,
      "mtime_unix": 1748284619
    },
    {
      "filename": "export.csv",
      "size": 9821,
      "mtime_unix": 1748284701
    }
  ]
}
```

--------------------------------

### Run Info Response Examples

Source: https://scrapfly.io/docs/cloud-browser-api/session-resume

Example JSON responses for running and finished browser sessions.

```json
{
  "run_id": "01KKBX...",
  "state": "running",
  "start_time": "2025-03-10T12:00:00Z",
  "runtime_ms": 45000,
  "bandwidth_up": 12345,
  "bandwidth_down": 67890,
  "proxy_pool": "public_datacenter_pool",
  "session_id": "my-session-id",
  "success": true,
  "api_credits": 2
}
```

```json
{
  "run_id": "01KKBX...",
  "state": "finished",
  "start_time": "2025-03-10T12:00:00Z",
  "end_time": "2025-03-10T12:01:30Z",
  "runtime_ms": 90000,
  "bandwidth_up": 24680,
  "bandwidth_down": 135780,
  "proxy_pool": "public_datacenter_pool",
  "session_id": "my-session-id",
  "success": true,
  "api_credits": 3
}
```

--------------------------------

### Build the Extension

Source: https://scrapfly.io/docs/tools/antibot-detector?__tab%5Burl-source-tab%5D=source-remote

Compile the project into a production-ready build located in the dist/ directory.

```bash
npm run build
```

--------------------------------

### Install Scrapfly SDK with Speedups

Source: https://scrapfly.io/docs/sdk/python?__tab%5Burl-source-tab%5D=source-remote

Installs the SDK with additional support for brotli compression and msgpack serialization.

```bash
pip install 'scrapfly-sdk[speedups]'
```

--------------------------------

### Connect to Cloud Browser with Playwright

Source: https://scrapfly.io/docs/cloud-browser-api/playwright?view=markdown

Quick start examples demonstrating how to connect to the Scrapfly Cloud Browser using the CDP protocol.

```javascript
const { chromium } = require('playwright');

const API_KEY = '';
const BROWSER_WS = `wss://browser.scrapfly.io?api_key=${API_KEY}&proxy_pool=public_datacenter_pool&os=linux`;

async function run() {
    let browser = null;
    try {
        // Connect to Cloud Browser
        browser = await chromium.connectOverCDP(BROWSER_WS);

        const context = await browser.newContext();
        const page = await context.newPage();

        // Navigate and interact
        await page.goto('https://web-scraping.dev');
        const title = await page.title();
        console.log('Page title:', title);

        // Take a screenshot
        await page.screenshot({ path: 'screenshot.png' });
    } finally {
        if (browser) await browser.close();
    }
}

run();
```

```python
from playwright.sync_api import sync_playwright

API_KEY = ''
BROWSER_WS = f'wss://browser.scrapfly.io?api_key={API_KEY}&proxy_pool=public_datacenter_pool&os=linux'

def run():
    with sync_playwright() as p:
        browser = None
        try:
            # Connect to Cloud Browser
            browser = p.chromium.connect_over_cdp(BROWSER_WS)

            context = browser.contexts[0]
            page = context.pages[0] if context.pages else context.new_page()

            # Navigate and interact
            page.goto('https://web-scraping.dev')
            print('Page title:', page.title())

            # Take a screenshot
            page.screenshot(path='screenshot.png')
        finally:
            if browser:
                browser.close()
```

--------------------------------

### Initialize Scrapfly Client

Source: https://scrapfly.io/docs/cloud-browser-api/captcha-solver?view=markdown

Basic setup for the Scrapfly Python SDK.

```python
from scrapfly import ScrapflyClient
```

--------------------------------

### Configure npx Path

Source: https://scrapfly.io/docs/mcp/integrations/crewai?__tab%5Burl-source-tab%5D=source-remote

Example of setting the full path for the npx command.

```bash
command="/usr/local/bin/npx"
```

--------------------------------

### Scrapfly Configuration Cost Examples

Source: https://scrapfly.io/docs/scrape-api/billing?__tab%5Burl-source-tab%5D=source-remote

JSON configurations demonstrating different cost scenarios based on proxy pool selection and JavaScript rendering.

```json
{
  "url": "https://example.com",
  "proxy_pool": "public_datacenter_pool"
}
```

```json
{
  "url": "https://example.com",
  "proxy_pool": "public_datacenter_pool",
  "render_js": true
}
```

```json
{
  "url": "https://example.com",
  "proxy_pool": "public_residential_pool"
}
```

```json
{
  "url": "https://example.com",
  "proxy_pool": "public_residential_pool",
  "render_js": true
}
```

--------------------------------

### Custom Extraction Prompt Example

Source: https://scrapfly.io/docs/mcp/faq

Use the 'extraction_prompt' parameter in web_scrape to guide LLM-based data extraction according to your specific needs.

```json
{
  "tool": "web_scrape",
  "parameters": {
    "url": "https://web-scraping.dev",
    "pow": "...",
    "extraction_prompt": "Extract all product names, prices, and availability status as a JSON array"
  }
}
```

--------------------------------

### Retrieve Browser Session Usage

Source: https://scrapfly.io/docs/cloud-browser-api/errors

Example JSON response from the GET /account endpoint showing concurrent session limits and monthly quota usage.

```json
{
  "account": {
    "plan": "professional",
    "browser_sessions": {
      "concurrent_limit": 10,
      "concurrent_active": 7,
      "monthly_quota_hours": 100,
      "monthly_quota_used_hours": 73.5,
      "monthly_quota_remaining_hours": 26.5
    }
  }
}
```

--------------------------------

### Manage Browser Sessions and Navigation

Source: https://scrapfly.io/docs/cli/commands

Examples for unblocking URLs, minting CDP sessions, and managing persistent browser sessions via the CLI.

```bash
# Unblock a URL and get an attach-ready browser session
scrapfly browser https://web-scraping.dev --unblock --country us

# Mint a fresh CDP URL (no target pre-navigated)
scrapfly browser --resolution 1920x1080 --pretty

# Persistent session: drive the page across calls
scrapfly browser --session demo start &
scrapfly browser navigate https://web-scraping.dev/login
scrapfly browser fill 'input[name=username]' user123
scrapfly browser click-ai "the submit button"   # LLM resolves the target
scrapfly browser content --raw > page.html
scrapfly browser close
```

--------------------------------

### Install and List WARC Records with warcio CLI

Source: https://scrapfly.io/docs/crawler-api/warc-format?view=markdown

Install the warcio Python package and use its command-line interface to index WARC files, listing all records.

```bash
# Install warcio
pip install warcio

# List all records
warcio index crawl.warc.gz

```

--------------------------------

### Basic ScrapflyReader Usage

Source: https://scrapfly.io/docs/integration/llamaindex

Initiate the ScrapflyReader with your API key and load documents from URLs as markdown. This snippet demonstrates the simplest way to get started.

```python
from llama_index.readers.web import ScrapflyReader

# Initiate ScrapflyReader with your ScrapFly API key
scrapfly_reader = ScrapflyReader(
    api_key="Your ScrapFly API key",  # Get your API key from https://www.scrapfly.io/
    ignore_scrape_failures=True,  # Ignore unprocessable web pages and log their exceptions
)

# Load documents from URLs as markdown
documents = scrapfly_reader.load_data(
    urls=["https://web-scraping.dev/products"]
)

```

--------------------------------

### Run Development Mode

Source: https://scrapfly.io/docs/tools/antibot-detector?__tab%5Burl-source-tab%5D=source-list

Command to start the development server with file watching and automatic rebuilding.

```bash
npm run dev
```

--------------------------------

### Scrape and Commit Data in GitHub Actions

Source: https://scrapfly.io/docs/cli/github-actions?__tab%5Burl-source-tab%5D=source-list

A complete workflow example that installs the CLI, scrapes a URL, processes the output with jq, and commits the results to the repository.

```yaml
name: Scrape data

on:
  schedule:
    - cron: '0 8 * * *'   # daily at 08:00 UTC
  workflow_dispatch:

jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Scrapfly CLI
        run: curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh

      - name: Scrape and save
        env:
          SCRAPFLY_API_KEY: ${{ secrets.SCRAPFLY_API_KEY }}
        run: |
          scrapfly scrape https://web-scraping.dev/products --format markdown \
            | jq '.data' > output/page.md

      - name: Commit results
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add output/
          git diff --staged --quiet || git commit -m "chore: update scraped data"
          git push
```
Tokens:
Raw