### Getting Started Prompt Example

Source: https://scrapfly.io/docs/mcp/examples?__tab%5Burl-source-tab%5D=source-remote

A simple starting prompt example to begin using Scrapfly MCP tools for AI-driven web scraping.

```markdown
Get me the top posts from Hacker News
```

--------------------------------

### Setup Webhook Server with Ngrok

Source: https://scrapfly.io/docs/sdk/python

This example demonstrates setting up a webhook server using the Scrapfly Python SDK, Flask, and ngrok for internet exposure. Ensure all dependencies are installed and ngrok is configured.

```python
import argparse
from typing import Dict
import flask
import ngrok
from scrapfly import webhook
from scrapfly.webhook import ResourceType

# Define the webhook callback function
def webhook_callback(data: Dict, resource_type: ResourceType, request: flask.Request):
    if resource_type == ResourceType.SCRAPE.value:
        # Process scrape result
        upstream_response = data['result']
        print(upstream_response)
    else:
        # Process other resource types
        print(data)

# Set up ngrok listener for tunneling
listener = ngrok.werkzeug_develop()

# Parse command-line arguments
parser = argparse.ArgumentParser(description="Webhook server with signing secret")
parser.add_argument("--signing-secret", required=True, help="Signing secret to verify webhook payload integrity")
args = parser.parse_args()

# Create Flask application and set up webhook server
app = flask.Flask("Scrapfly Webhook Server")
webhook.create_server(signing_secrets=(args.signing_secret,), callback=webhook_callback, app=app)

# Start the server and print the webhook endpoint URL
print("====== LISTENING ON ======")
print(listener.url() + "/webhook")
print("==========================")
app.run()
```

--------------------------------

### Install Scrapfly Go SDK

Source: https://scrapfly.io/docs/onboarding/golang?view=markdown

Use 'go get' to install the Scrapfly Go SDK. This command fetches and installs the SDK package.

```bash
go get github.com/scrapfly/go-scrapfly
```

--------------------------------

### Start MCP Server

Source: https://scrapfly.io/docs/mcp/faq?iframe=1

Run the MCP server after installing dependencies and configuring your API key. This command starts the local server instance.

```bash
npm start
```

--------------------------------

### Dockerfile Example for Debian-slim

Source: https://scrapfly.io/docs/proxy-saver/certificates?view=markdown

Installs ca-certificates and curl, then downloads and installs the Scrapfly CA certificate for Debian-slim Docker images.

```dockerfile
FROM debian:bullseye-slim

# Install ca-certificates and curl
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl && \
rm -rf /var/lib/apt/lists/*

# Download and install the Proxy Saver certificate
RUN curl -fsSL -o /usr/local/share/ca-certificates/scrapfly-ca.crt https://scrapfly.io/ca.crt && \
update-ca-certificates

```

--------------------------------

### Python Virtual Environment Setup

Source: https://scrapfly.io/docs/mcp/integrations/langchain?iframe=1

Demonstrates setting up a Python virtual environment and installing the Scrapfly MCP adapters package.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install langchain-mcp-adapters
```

--------------------------------

### Install Scrapfly CLI

Source: https://scrapfly.io/docs/cli

Installs the Scrapfly CLI using a curl command. This is the quickest way to get started.

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh
```

--------------------------------

### Pin Scrapfly CLI Version During Installation

Source: https://scrapfly.io/docs/cli/github-actions

This example shows how to install a specific version of the Scrapfly CLI using the installer script with the `--version` flag. This ensures reproducible workflows by avoiding the latest version.

```bash
curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --version v1.2.3
```

--------------------------------

### Set up Python Virtual Environment

Source: https://scrapfly.io/docs/mcp/integrations/llamaindex?__tab%5Burl-source-tab%5D=source-list

Use a virtual environment for Python projects to manage dependencies. This example shows how to create and activate a venv and install the MCP tools.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install llama-index-tools-mcp
```

--------------------------------

### Install CrewAI Package

Source: https://scrapfly.io/docs/mcp/integrations/crewai?__tab%5Burl-source-tab%5D=source-list

Install the CrewAI package to get started. It's recommended to use a virtual environment.

```bash
pip install crewai
```

--------------------------------

### Example with Options

Source: https://scrapfly.io/docs/screenshot-api/getting-started?iframe=1

Demonstrates using the 'options' parameter to enable dark mode and block cookie banners. The 'url' and 'key' parameters are required.

```bash
$ curl -G \
--request "GET" \
--url "https://api.scrapfly.io/screenshot" \
--data-urlencode "key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "options=dark_mode,block_banners" -o screenshot.jpg
```

--------------------------------

### Install Scrapfly SDK with All Features

Source: https://scrapfly.io/docs/sdk/python?iframe=1

Install the Scrapfly SDK with all optional features included.

```bash
pip install 'scrapfly-sdk[all]'
```

--------------------------------

### Setup Virtual Environment and Install CrewAI

Source: https://scrapfly.io/docs/mcp/integrations/crewai?iframe=1

Use a virtual environment to manage dependencies and install the CrewAI package. This ensures a clean project setup.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install crewai
```

--------------------------------

### Quick Use Example

Source: https://scrapfly.io/docs/sdk/golang

A basic example demonstrating how to create a Scrapfly client and perform a scrape operation. It shows how to access scraped HTML content and parse it using goquery.

```go
package main

import (
    "fmt"
    "log"

    "github.com/scrapfly/go-scrapfly"
)

func main() {
    key := "{{ YOUR_API_KEY }}"

    client, err := scrapfly.New(key)
    if err != nil {
        log.Fatalf("failed to create client: %v", err)
    }

    result, err := client.Scrape(&scrapfly.ScrapeConfig{
        URL:       "https://web-scraping.dev/product/1",
        ASP:       true,              // enable scraper blocking bypass
        Country:   "US",             // set proxy country
        RenderJS:  true,              // enable headless browser
        ProxyPool: scrapfly.PublicResidentialPool,
    })
    if err != nil {
        log.Fatalf("scrape failed: %v", err)
    }

    // 1) access scraped HTML content
    fmt.Println(result.Result.Content)
    // 2) or parse it with CSS selectors via goquery
    selector, _ := result.Selector()
    fmt.Println(selector.Find("h3").First().Text())
}
```

--------------------------------

### Dockerfile Example for Alpine

Source: https://scrapfly.io/docs/proxy-saver/certificates?view=markdown

Installs ca-certificates and curl, then downloads and installs the Scrapfly CA certificate for Alpine Docker images.

```dockerfile
FROM alpine:latest

# Install ca-certificates and curl
RUN apk add --no-cache ca-certificates curl

# Download and install the Proxy Saver certificate
RUN curl -fsSL -o /usr/local/share/ca-certificates/scrapfly-ca.crt https://scrapfly.io/ca.crt && \
update-ca-certificates

```

--------------------------------

### Install Browser Use

Source: https://scrapfly.io/docs/cloud-browser-api/browser-use

Install the Browser Use Python package using pip. Requires Python 3.11 or higher.

```bash
pip install browser-use
```

--------------------------------

### Get Content (cURL)

Source: https://scrapfly.io/docs/crawler-api/getting-started?view=markdown

Retrieve extracted content from crawled pages in the specified format. This example shows how to get content in markdown format.

```bash
# Get all content in markdown format
curl "https://api.scrapfly.io/crawl/{crawler_uuid}/contents?key={{ YOUR_API_KEY }}&format=markdown"
```

--------------------------------

### OpenAI Assistant Setup and Web Scraping

Source: https://scrapfly.io/docs/mcp/integrations/openai

This snippet demonstrates setting up an OpenAI Assistant, creating a thread, sending a user message to scrape and summarize web content, and handling function calls for web scraping. It includes a loop to poll the run status and submit tool outputs.

```python
from openai import OpenAI
from scrapfly import ScraperConfig, ScrapflyClient

# Initialize Scrapfly client with API key
client_scrapfly = ScrapflyClient(scrapfly_key="YOUR_SCRAPFLY_API_KEY")

def scrape_webpage(**kwargs):
    response = client_scrapfly.scrape(**kwargs)
    return response.json()['content']

# Initialize OpenAI client
client = OpenAI(api_key="YOUR_OPENAI_API_KEY")

# Create or retrieve an assistant (replace with your assistant ID)
# assistant = client.beta.assistants.create(name="Web Scraper", ...)
assistant_id = "asst_YOUR_ASSISTANT_ID"
assistant = client.beta.assistants.retrieve(assistant_id)

tools=[{
    "type": "function",
    "function": {
        "name": "scrape_webpage",
        "description": "Scrape a given URL and return the content in JSON format.",
        "parameters": {
            "type": "object",
            "properties": {
                "url": {
                    "type": "string",
                    "description": "The URL to scrape."
                }
            },
            "required": ["url"]
        }
    }
}]

# Update assistant with tools if not already present
if assistant.tools is None or not any(t.type == 'function' and t.function.name == 'scrape_webpage' for t in assistant.tools):
    assistant = client.beta.assistants.update(assistant_id, tools=tools)

# Create a thread
thread = client.beta.threads.create()

# User message
client.beta.threads.messages.create(
    thread_id=thread.id,
    role="user",
    content="Scrape the top posts from https://news.ycombinator.com and summarize them"
)

# Run the assistant
run = client.beta.threads.runs.create(
    thread_id=thread.id,
    assistant_id=assistant.id
)

# Handle function calling
import time
while run.status != "completed":
    run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)

    if run.status == "requires_action":
        tool_calls = run.required_action.submit_tool_outputs.tool_calls
        tool_outputs = []

        for tool_call in tool_calls:
            if tool_call.function.name == "scrape_webpage":
                import json
                args = json.loads(tool_call.function.arguments)
                result = scrape_webpage(**args)
                tool_outputs.append({
                    "tool_call_id": tool_call.id,
                    "output": result
                })

        # Submit function outputs
        client.beta.threads.runs.submit_tool_outputs(
            thread_id=thread.id,
            run_id=run.id,
            tool_outputs=tool_outputs
        )

    time.sleep(1)

# Get final response
messages = client.beta.threads.messages.list(thread_id=thread.id)
print(messages.data[0].content[0].text.value)

```

--------------------------------

### Install Scrapfly SDK with All Extras

Source: https://scrapfly.io/docs/onboarding

Install the Scrapfly SDK with all optional modules for enhanced functionality, including concurrency, Scrapy integration, and performance improvements.

```bash
$ pip install 'scrapfly-sdk[all]'
```

--------------------------------

### Install Specific Scrapfly Agent Skills

Source: https://scrapfly.io/docs/openapi?view=markdown

Install only the specific Scrapfly agent skills you need by listing them after the base command. This allows for a more tailored setup.

```bash
npx skills add scrapfly/skills -s Agent Rules -s Cloud Browser -s CLI
```

--------------------------------

### Example Options Parameter (Combined)

Source: https://scrapfly.io/docs/screenshot-api/getting-started?__tab%5Burl-source-tab%5D=source-list

Combine multiple options using a comma-separated string. This example enables both banner blocking and dark mode.

```text
block_banners,dark_mode
```

--------------------------------

### Get Extraction Monitoring Target Metrics (Go)

Source: https://scrapfly.io/docs/extraction-api/monitoring

Use the Go SDK to get monitoring statistics for a domain. This example demonstrates basic error handling and metric retrieval.

```go
package main

import (
    "fmt"
    "log"

    scrapfly "github.com/scrapfly/go-scrapfly"
)

func main() {
    client, err := scrapfly.New("")
    if err != nil {
        log.Fatal(err)
    }

    stats, err := client.GetExtractionMonitoringTargetMetrics(scrapfly.MonitoringTargetMetricsOptions{
        Domain: "web-scraping.dev",
        Period: scrapfly.MonitoringPeriodLast24h,
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(stats)
}
```

--------------------------------

### Install warcpp C++ Parser

Source: https://scrapfly.io/docs/crawler-api/warc-format?__tab%5Burl-source-tab%5D=source-list

Clone the warcpp repository, navigate to the directory, create a build directory, and then configure and build the project using CMake.

```bash
git clone https://github.com/pisa-engine/warcpp.git
cd warcpp
mkdir build && cd build
cmake ..
make
```

--------------------------------

### Get Crawler Status with Go

Source: https://scrapfly.io/docs/crawler-api/getting-started?iframe=1

Use the Scrapfly Go SDK to get the status of a crawler job. This example prints the status and URL counts, handling potential errors.

```go
client, _ := scrapfly.New("{{ YOUR_API_KEY }}")
status, err := client.CrawlStatus("{crawler_uuid}")
if err != nil {
    log.Fatal(err)
}
fmt.Println(status.Status, status.State.URLsVisited, "/", status.State.URLsExtracted)
```

--------------------------------

### Install Requests and Playwright (Python)

Source: https://scrapfly.io/docs/cloud-browser-api/selenium

Install the `requests` library for making HTTP requests and Playwright for browser automation. This setup is for connecting to Cloud Browser via its `/json/version` endpoint.

```bash
pip install requests playwright && playwright install chromium
```

--------------------------------

### Example with Custom Resolution

Source: https://scrapfly.io/docs/screenshot-api/getting-started?iframe=1

Sets a custom screen resolution for the screenshot. The 'url' and 'key' parameters are required.

```bash
$ curl -G \
--request "GET" \
--url "https://api.scrapfly.io/screenshot" \
--data-urlencode "key=YOUR_API_KEY" \
--data-urlencode "url=https://example.com" \
--data-urlencode "resolution=375x812" -o screenshot.jpg
```

--------------------------------

### Get Screenshot Monitoring Metrics (Rust)

Source: https://scrapfly.io/docs/screenshot-api/monitoring?iframe=1

A Rust example using the scrapfly_sdk to get screenshot monitoring metrics. It utilizes Tokio for asynchronous operations and includes basic error handling.

```rust
use scrapfly_sdk::{Client, MonitoringPeriod, MonitoringTargetMetricsOptions};

#[tokio::main]
async fn main() -> Result<(), Box> {
    let client = Client::builder().api_key("").build()?;
    let stats = client
        .get_screenshot_monitoring_target_metrics(&MonitoringTargetMetricsOptions {
            domain: "web-scraping.dev".into(),
            group_subdomain: false,
            period: Some(MonitoringPeriod::Last24h),
            start: None,
            end: None,
            include_webhook: false,
        })
        .await?;
    println!("{}", stats);
    Ok(())
}
```

--------------------------------

### Install go-warc Library

Source: https://scrapfly.io/docs/crawler-api/warc-format?__tab%5Burl-source-tab%5D=source-list

Command to install the go-warc library for reading and writing WARC files in Go.

```bash
go get github.com/nlnwa/gowarc
```

--------------------------------

### Remote URL List File Format Example

Source: https://scrapfly.io/docs/crawler-api/url-sources?__tab%5Burl-source-tab%5D=source-remote

Example of a plain-text file format for remote URL lists. Lines starting with '#' are comments, and blank lines are ignored. URLs with fragments are preserved.

```text
# Comments start with # at column zero.

https://web-scraping.dev/products/1
https://web-scraping.dev/products/2
https://web-scraping.dev/products/3

# Blank lines are ignored.
https://web-scraping.dev/api/products

```

--------------------------------

### Setting Headers

Source: https://scrapfly.io/docs/scrape-api/getting-started

Examples of how to set various headers like cookies, content type, and authorization. Values must be URL encoded.

```url
headers[Cookie]=test%3D1%3Bauth%3D1
```

```url
headers[content-type]=application%2Fjson
```

```url
headers[Authorization]=Bearer%20token123
```

--------------------------------

### Scrape API GET Request

Source: https://scrapfly.io/docs/scrape-api/getting-started?__tab%5Burl-source-tab%5D=source-list

This example demonstrates how to perform a GET request to the scrape API to retrieve web page content. It includes common parameters like URL, country, and JavaScript rendering.

```APIDOC
## GET /scrape

### Description
Retrieves web page content using a GET request. This is the primary method for scraping.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **url** (string) - Required - Target URL to scrape. Must be URL encoded.
- **key** (string) - Required - API Key for authentication.
- **proxy_pool** (string) - Optional - Select proxy pool. Popular default: `public_datacenter_pool`. Options: `public_datacenter_pool`, `public_residential_pool`.
- **country** (string) - Optional - Proxy country (ISO 3166-1 alpha-2). Supports exclusions and weighted distribution. Popular default: `random`. Examples: `us`, `us,ca,mx`, `-gb`, `us:10,gb:5`.
- **lang** (string) - Optional - Page language (sets `Accept-Language` header). Defaults to proxy location language. Examples: `en`, `fr-FR,en`.
- **os** (string) - Optional - Operating System. Cannot be set with custom `User-Agent` header. Options: `win11`, `mac`, `linux`.
- **browser_brand** (string) - Optional - Chromium-based browser brand for fingerprint generation (applies when `render_js=true`). Default: `chrome`. Options: `chrome`, `edge`, `brave`, `opera`.
- **headers** (string) - Optional - Custom HTTP headers. Must be URL encoded. Syntax: `headers[header-name]=encoded-value`. Multiple headers can be passed.
- **timeout** (integer) - Optional - Timeout in milliseconds. Default: 155000. See timeout documentation.
- **retry** (boolean) - Optional - Retry on failure (network errors, HTTP 5xx). Has impact on timeout. Default: `true`.
- **format** (string) - Optional - Response format. Default: `raw`.

### Request Example
```bash
curl -X GET "https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key="
```

### Response
#### Success Response (200)
- **result.content** (string) - The scraped content of the page.
- **config** (object) - Your scrape configuration.
- **context** (object) - Information about activated features.

#### Response Example
```json
{
  "result": {
    "content": "<html>...</html>"
  },
  "config": { ... },
  "context": { ... }
}
```
```

--------------------------------

### Install and List WARC Records with warcio CLI

Source: https://scrapfly.io/docs/crawler-api/warc-format?view=markdown

Install the warcio Python package and use its command-line interface to index WARC files, listing all records.

```bash
# Install warcio
pip install warcio

# List all records
warcio index crawl.warc.gz

```

--------------------------------

### Example Options Parameter (Print Media)

Source: https://scrapfly.io/docs/screenshot-api/getting-started?__tab%5Burl-source-tab%5D=source-list

Set 'print_media_format' in the 'options' parameter to render the page as it would appear in a print preview.

```text
print_media_format
```

--------------------------------

### Run Scrapfly CLI command with npx

Source: https://scrapfly.io/docs/cli/install?iframe=1

Example of running a Scrapfly CLI command using npx after installing via npm.

```bash
npx scrapfly scrape https://web-scraping.dev/products
```

--------------------------------

### Webhook Server Setup

Source: https://scrapfly.io/docs/sdk/python?__tab%5Burl-source-tab%5D=source-list

Sets up a local webhook server using Flask and ngrok for testing, with payload integrity verification.

```APIDOC
## Webhook Server Setup

### Description
Sets up a local webhook server that listens for incoming requests, verifies their integrity using a signing secret, and processes the data via a callback function. Ngrok is used to expose the local server to the internet for testing purposes.

### Dependencies
- `ngrok`
- `flask`
- `scrapfly`

### Setup Steps
1. Install dependencies: `pip install ngrok flask scrapfly`
2. Export ngrok auth token: `export NGROK_AUTHTOKEN=MY_NGROK_TOKEN`
3. Run the server script: `python your_script_name.py --signing-secret=MY_SIGNING_SECRET`
4. Once the server is running, copy the exposed ngrok URL and update your webhook endpoint on the Scrapfly dashboard.

### Code Example
```python
import argparse
from typing import Dict
import flask
import ngrok
from scrapfly import webhook
from scrapfly.webhook import ResourceType

# Define the webhook callback function
def webhook_callback(data: Dict, resource_type: ResourceType, request: flask.Request):
    if resource_type == ResourceType.SCRAPE.value:
        # Process scrape result
        upstream_response = data['result']
        print(upstream_response)
    else:
        # Process other resource types
        print(data)

# Set up ngrok listener for tunneling
listener = ngrok.werkzeug_develop()

# Parse command-line arguments
parser = argparse.ArgumentParser(description="Webhook server with signing secret")
parser.add_argument("--signing-secret", required=True, help="Signing secret to verify webhook payload integrity")
args = parser.parse_args()

# Create Flask application and set up webhook server
app = flask.Flask("Scrapfly Webhook Server")
webhook.create_server(signing_secrets=(args.signing_secret,), callback=webhook_callback, app=app)

# Start the server and print the webhook endpoint URL
print("====== LISTENING ON ======")
print(listener.url() + "/webhook")
print("==========================")
app.run()
```

### Callback Function Parameters
- **data** (dict) - The payload data received from the webhook.
- **resource_type** (string) - The type of resource the webhook is related to (e.g., 'scrape').
- **request** (flask.Request) - The Flask request object.
```

--------------------------------

### Read WARC Files with warcio in Python

Source: https://scrapfly.io/docs/crawler-api/warc-format?__tab%5Burl-source-tab%5D=source-list

Example of how to start reading a WARC file using the ArchiveIterator from the warcio library in Python.

```python
import gzip
from warcio.archiveiterator import ArchiveIterator


```

--------------------------------

### Basic Scraping Example

Source: https://scrapfly.io/docs/sdk/rust?view=markdown

A minimal end-to-end example demonstrating how to create a client, build a scrape configuration, and perform a scrape. It shows how to access the scraped content and status code.

```rust
use scrapfly_sdk::{Client, ScrapeConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::error("")>> {
    let client = Client::builder()
        .api_key("{{ YOUR_API_KEY }}")
        .build()?;

    let cfg = ScrapeConfig::builder("https://web-scraping.dev/product/1")
        .asp(true)             // enable scraper blocking bypass
        .country("us")         // set proxy country
        .render_js(true)       // enable headless browser
        .build()?;

    let result = client.scrape(&cfg).await?;

    // 1) access scraped HTML content
    println!("{}", result.result.content);
    // 2) inspect status code, metadata, response headers
    println!("status = {}", result.result.status_code);
    Ok(())
}
```

--------------------------------

### Custom Extraction Prompt Example

Source: https://scrapfly.io/docs/mcp/faq

Use the 'extraction_prompt' parameter in web_scrape to guide LLM-based data extraction according to your specific needs.

```json
{
  "tool": "web_scrape",
  "parameters": {
    "url": "https://web-scraping.dev",
    "pow": "...",
    "extraction_prompt": "Extract all product names, prices, and availability status as a JSON array"
  }
}
```

--------------------------------

### Install MCP SDK

Source: https://scrapfly.io/docs/mcp/integrations/custom-client?__tab%5Burl-source-tab%5D=source-remote

Install the official Model Context Protocol SDK for Python or JavaScript/TypeScript.

```bash
pip install mcp anthropic
```

```bash
npm install @modelcontextprotocol/sdk @anthropic-ai/sdk
```

--------------------------------

### Browser.downloadWillBegin Event

Source: https://scrapfly.io/docs/cloud-browser-api/cdp-reference/Browser

Fired when the page is about to start a download. It provides details about the download such as frame ID, GUID, URL, and suggested filename.

```APIDOC
## Browser.downloadWillBegin Event

### Description
Fired when page is about to start a download.

### Parameters
#### Path Parameters
* **frameId** (`Page.FrameId`) - Required - Id of the frame that caused the download to begin.
* **guid** (`string`) - Required - Global unique identifier of the download.
* **url** (`string`) - Required - URL of the resource being downloaded.
* **suggestedFilename** (`string`) - Required - Suggested file name of the resource (the actual name of the file saved on disk may differ).
```