Scrapfly Web Scraping API

https://scrapfly.io/docs/scrape-api/
Admin
Scrapfly's Web Scraping API provides developers with tools to scrape, capture, and extract web data...
Tokens:155,717
Snippets:1,082
Trust Score:5.6
Update:1 week ago
Show doc for...
Tokens:
Raw
### Unblocker Example with Python SDK

Source: https://scrapfly.io/docs/scrape-api/unblocker

An example demonstrating how to use the Unblocker with the Python SDK. This is the only integration example provided in the source.

```APIDOC
## Unblocker Example with Python SDK

### Description
This example shows how to use the Unblocker with the Python SDK. The source does not provide the actual code, but it indicates that an example is available.

### Method
Not specified in the source.

### Endpoint
Not specified in the source.

### Parameters
No parameters are documented in the source.

### Request Example
No request example is provided in the source.

### Response
No response example is provided in the source.

### Related Errors
The following error codes are associated with the Unblocker feature:

- **ERR::ASP::CAPTCHA_ERROR** - Something wrong happened with the captcha. We will figure out to fix the problem as soon as possible
- **ERR::ASP::CAPTCHA_TIMEOUT** - The budgeted time to solve the captcha is reached
- **ERR::ASP::SHIELD_ERROR** - The Unblocker encountered an unexpected problem. We will fix it as soon as possible. Our team has been alerted
- **ERR::ASP::SHIELD_EXPIRED** - The Unblocker shield previously set is expired, you must retry.
- **ERR::ASP::SHIELD_NOT_ELIGIBLE** - The feature requested is not eligible while using the Unblocker for the given protection/target
- **ERR::ASP::SHIELD_PROTECTION_FAILED** - The Unblocker shield failed to solve the challenge against the anti-bot protection
- **ERR::ASP::SHIELD_UNDER_MAINTENANCE** - This target is temporarily unavailable while we update the anti-bot bypass. No action is needed on your side and there is no need to contact support, we are aware and working on it. Please retry later.
- **ERR::ASP::TIMEOUT** - The Unblocker took too much time to solve or respond
- **ERR::ASP::UNABLE_TO_SOLVE_CAPTCHA** - Despite our effort, we were unable to solve the captcha. It can happened sporadically, please retry
- **ERR::ASP::UPSTREAM_UNEXPECTED_RESPONSE** - The response given by the upstream after challenge resolution is not expected. Our team has been alerted
```

--------------------------------

### Session name example

Source: https://scrapfly.io/docs/scrape-api/getting-started

Example session name value.

```text
my-session-123
```

--------------------------------

### Scrapfly API curl examples for GET, POST, PUT, PATCH, OPTIONS, and HEAD

Source: https://scrapfly.io/docs/scrape-api/getting-started

Example curl commands for the Scrapfly API covering GET, POST, PUT, PATCH, OPTIONS, and HEAD methods. The API has a default read timeout of 155 seconds; configure your HTTP client accordingly to avoid read timeout errors. Replace the key parameter with your API key.

```bash
curl -X GET https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key=
```

```bash
curl -X POST https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```bash
curl -X PUT https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```bash
curl -X PATCH https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```bash
curl -X OPTIONS https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything&country=us&render_js=true&key= -H content-type: text/json --data-raw "{\"test\": \"example\"}"
```

```bash
curl -I https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key=
```

--------------------------------

### Install CA certificate on Windows

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Installs the CA certificate on Windows using certutil to add it to the ROOT store. Must be run as Administrator.

```powershell
# PowerShell (run as Administrator)
certutil -addstore -f "ROOT" scrapfly-proxy-ca.crt
```

--------------------------------

### Client Configuration Examples

Source: https://scrapfly.io/docs/scrape-api/understand-timeout

Client configuration examples for Python and Node.js. The Python client uses a 95s timeout, while the TypeScript/Node.js client uses a 90s server timeout. These are placeholder snippets shown in the documentation.

```python
Python client with 95s timeout
```

```typescript
TypeScript / Node.js client with 90s server timeout
```

--------------------------------

### Example session response from Scrapfly API

Source: https://scrapfly.io/docs/scrape-api/session

Example response showing the session context with cookie jar, expiration, and identity fields.

```json
...
"context": {
    ...
    "session": {
        "name": "mysession1",
        "state": "FREE",
        "lease": null,
        "correlation_id": "default",
        "identity": "929f7fb918367df788717d825a3d75391e148c76",
        "created_at": "2020-09-15 12:44:15 UTC",
        "cookie_jar": [
            {
                "comment": null,
                "domain": "httpbin.dev",
                "expires": null,
                "http_only": false,
                "max_age": null,
                "name": "mycookie",
                "path": "/cookies",
                "secure": false,
                "size": null,
                "value": "123",
                "version": 0
            },
        ],
        "last_used_at": "2023-09-15 12:44:50 UTC",
        "expire_at": "2023-09-22 12:44:50 UTC",
        "referer": "https://httpbin.dev/"
    }
    ...
}
...
```

--------------------------------

### Geolocation format example

Source: https://scrapfly.io/docs/scrape-api/getting-started

Example geolocation values in latitude,longitude format.

```text
48.856614,2.3522219
```

```text
40.712784,-74.005941
```

--------------------------------

### Install CA certificate on Linux (Debian/Ubuntu)

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Installs the CA certificate on Debian/Ubuntu systems by copying it to the system trust store and updating certificates.

```bash
# Debian / Ubuntu
sudo cp scrapfly-proxy-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
```

--------------------------------

### Example scenario (before base64)

Source: https://scrapfly.io/docs/scrape-api/getting-started

Shows a JSON array with click and fill actions before base64 encoding.

```json
[{"click": {"selector": "#login-btn"}}, {"fill": {"selector": "#username", "value": "test"}}]
```

--------------------------------

### Install agent skills command

Source: https://scrapfly.io/docs/scrape-api/getting-started

Command to install Scrapfly agent skills using npx.

```bash
npx skills add scrapfly/skills
```

--------------------------------

### Verify proxy setup with cURL

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

cURL commands to verify proxy setup. The first uses basic options; the second enables JavaScript rendering and anti-bot protection via renderJs-true and unblocker-true options.

```bash
# Test with a simple request
curl -x http://country-us:YOUR_API_KEY@proxy.scrapfly.io:7777 \
    https://httpbin.dev/anything

# Test with JavaScript rendering and anti-bot protection
curl -x http://renderJs-true-unblocker-true-country-us:YOUR_API_KEY@proxy.scrapfly.io:7777 \
    https://web-scraping.dev/products
```

--------------------------------

### Create a schedule with Scrapfly SDKs, CLI, and curl

Source: https://scrapfly.io/docs/scrape-api/schedule

Creates a new schedule via the Scrapfly API. The body is the full scrape_config plus a recurrence and a webhook name. The Python, JavaScript, Go, and Rust SDK examples use the same configuration: URL, render_js, unblocker, country, webhook_name, cron recurrence, retry_on_failure, max_retries, and notes. The CLI example uses the `scrapfly scrape schedule create` command with `--config-inline`, `--webhook`, `--cron`, `--retry-on-failure`, `--max-retries`, and `--notes`. The curl example posts to `https://api.scrapfly.io/scrape/schedules?key=YOUR_API_KEY` with a JSON body that also includes `allow_concurrency: false`. All examples require an API key (empty in the SDK examples) and output the schedule ID and status.

```python
from scrapfly import ScrapflyClient, CreateScheduleRequest, ScheduleRecurrence

client = ScrapflyClient(key='')

sched = client.create_scrape_schedule(
    {
        'url': 'https://web-scraping.dev/products',
        'render_js': True,
        'unblocker': True,
        'country': 'us',
    },
    CreateScheduleRequest(
        webhook_name='my-webhook',
        recurrence=ScheduleRecurrence(cron='0 */6 * * *'),
        retry_on_failure=True,
        max_retries=3,
        notes='Refresh product catalog every 6 hours',
    ),
)
print(sched['id'], sched['status'])
```

```javascript
import { ScrapflyClient } from 'scrapfly-sdk';

const client = new ScrapflyClient({ key: '' });

const sched = await client.createScrapeSchedule(
  {
    url: 'https://web-scraping.dev/products',
    render_js: true,
    unblocker: true,
    country: 'us',
  },
  {
    webhook_name: 'my-webhook',
    recurrence: { cron: '0 */6 * * *' },
    retry_on_failure: true,
    max_retries: 3,
    notes: 'Refresh product catalog every 6 hours',
  },
);
console.log(sched.id, sched.status);
```

```go
package main

import (
    "fmt"
    "log"

    scrapfly "github.com/scrapfly/go-scrapfly"
)

func main() {
    client, err := scrapfly.New("")
    if err != nil {
        log.Fatal(err)
    }

    sched, err := client.CreateScrapeSchedule(
        map[string]interface{}{
            "url":       "https://web-scraping.dev/products",
            "render_js": true,
            "unblocker": true,
            "country":   "us",
        },
        &scrapfly.CreateScheduleRequest{
            WebhookName:    "my-webhook",
            Recurrence:     &scrapfly.ScheduleRecurrence{Cron: "0 */6 * * *"},
            RetryOnFailure: true,
            MaxRetries:     3,
            Notes:          "Refresh product catalog every 6 hours",
        },
    )
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(sched.ID, sched.Status)
}
```

```rust
use scrapfly_sdk::{Client, CreateScheduleRequest, ScheduleRecurrence};
use serde_json::{json, Value};
use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), Box> {
    let client = Client::builder().api_key("").build()?;

    let mut cfg: HashMap = HashMap::new();
    cfg.insert("url".into(), json!("https://web-scraping.dev/products"));
    cfg.insert("render_js".into(), json!(true));
    cfg.insert("unblocker".into(), json!(true));
    cfg.insert("country".into(), json!("us"));

    let sched = client.create_scrape_schedule(
        cfg,
        &CreateScheduleRequest {
            webhook_name: "my-webhook".into(),
            recurrence: Some(ScheduleRecurrence {
                cron: Some("0 */6 * * *".into()),
                ..Default::default()
            }),
            retry_on_failure: true,
            max_retries: Some(3),
            notes: Some("Refresh product catalog every 6 hours".into()),
            ..Default::default()
        },
    ).await?;
    println!("{} {}", sched.id, sched.status);
    Ok(())
}
```

```bash
scrapfly --api-key  scrape schedule create \
    --config-inline '{"url":"https://web-scraping.dev/products","render_js":true,"unblocker":true,"country":"us"}' \
    --webhook my-webhook \
    --cron '0 */6 * * *' \
    --retry-on-failure --max-retries 3 \
    --notes 'Refresh product catalog every 6 hours'
```

```bash
curl -X POST 'https://api.scrapfly.io/scrape/schedules?key=YOUR_API_KEY' \
    -H 'Content-Type: application/json' \
    -d '{
        "scrape_config": {
            "url": "https://web-scraping.dev/products",
            "render_js": true,
            "country": "us",
            "unblocker": true
        },
        "webhook_name": "my-webhook",
        "recurrence": {
            "cron": "0 */6 * * *"
        },
        "allow_concurrency": false,
        "retry_on_failure": true,
        "max_retries": 3,
        "notes": "Refresh product catalog every 6 hours"
    }'
```

--------------------------------

### Install CA certificate on macOS

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Installs the CA certificate on macOS using the security command to add it to the system keychain.

```bash
sudo security add-trusted-cert -d -r trustRoot \
    -k /Library/Keychains/System.keychain scrapfly-proxy-ca.crt
```

--------------------------------

### Example API response with debug object

Source: https://scrapfly.io/docs/scrape-api/debug

Example API response showing the debug object with screenshot_url, response_url, and content_replay_url. The screenshot_url is only available when JavaScript rendering is activated.

```json
...
context {
    debug: {
        // screenshot is available only if rendering javascript is activated
        screenshot_url: "https://api.scrapfly.io/11cd6abe-5061-4dce-8d37-5d50e667a071/scrape/screenshot/ee8484c6-ee5f-4775-a665-0a2b57631c1c/debug",
        response_url: "https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c",
        content_replay_url: "https://api.scrapfly.io/scrape/debug/ee8484c6-ee5f-4775-a665-0a2b57631c1c/replay",
    }
}
...
        
```

--------------------------------

### Example DNS response from Scrapfly API

Source: https://scrapfly.io/docs/scrape-api/dns

Example response showing the DNS details returned in the result.dns field when the DNS feature is enabled. Contains A, CAA, NS, SOA, and TXT records with host, ttl, and other fields.

```json
...
"result": {
    ...
    "dns": {
        "A": [
            {
                "host": "34.127.70.125",
                "ttl": 300
            }
        ],
        "CAA": [],
        "NS": [
            {
                "host": "ns-cloud-c4.googledomains.com",
                "ttl": -1
            },
            {
                "host": "ns-cloud-c2.googledomains.com",
                "ttl": -1
            },
            {
                "host": "ns-cloud-c1.googledomains.com",
                "ttl": -1
            },
            {
                "host": "ns-cloud-c3.googledomains.com",
                "ttl": -1
            }
        ],
        "SOA": [
            {
                "expires": 259200,
                "hostmaster": "cloud-dns-hostmaster.google.com",
                "minttl": 300,
                "nsname": "ns-cloud-c1.googledomains.com",
                "refresh": 21600,
                "serial": 1,
                "ttl": -1
            }
        ],
        "TXT": [
            {
                "text": "google-site-verification=WSRMzA3PyGeg1LYiNn979SCXSjVwfoqv9QuAXk-C_Dg",
                "ttl": -1
            }
        ]
    },
    ...
...
```

--------------------------------

### Example multipart/mixed batch response

Source: https://scrapfly.io/docs/scrape-api/batch

Shows the HTTP/2 200 response with Content-Type multipart/mixed and a boundary. Each part contains a JSON body with per-part headers like X-Scrapfly-Correlation-Id and X-Scrapfly-Scrape-Status. The example includes three parts: two successful scrapes and one failure with a 422 status and error envelope.

```http
HTTP/2 200
Content-Type: multipart/mixed; boundary=batch-abc123
Content-Encoding: gzip
X-Scrapfly-Batch-Uuid: 
X-Accel-Buffering: no
Transfer-Encoding: chunked

--batch-abc123
Content-Type: application/json
Content-Length: 6109
X-Scrapfly-Correlation-Id: job-1
X-Scrapfly-Scrape-Status: 200
X-Scrapfly-Log-Uuid: 01KPJXAJKEP62HD8YW48G3FF6Z

{ ...standard ScrapeResult JSON envelope (same shape as a single /scrape response)... }
--batch-abc123
Content-Type: application/json
Content-Length: 5873
X-Scrapfly-Correlation-Id: job-2
X-Scrapfly-Scrape-Status: 200
X-Scrapfly-Log-Uuid: 01KPJXAJKET0XGT1ESZBZRST28

{ ... }
--batch-abc123
Content-Type: application/json
Content-Length: 482
X-Scrapfly-Correlation-Id: job-3
X-Scrapfly-Scrape-Status: 422

{
  "code": "ERR::SCRAPE::ASP_SHIELD_PROTECTION_FAILED",
  "message": "...",
  "http_code": 422,
  "retryable": false,
  "links": { ... }
}
--batch-abc123--
```

--------------------------------

### Cache TTL values

Source: https://scrapfly.io/docs/scrape-api/getting-started

Example cache_ttl values in seconds.

```text
60
```

```text
3600
```

```text
86400
```

--------------------------------

### GET https://httpbin.dev/html

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Inspect response metadata headers from a Proxy Mode request using curl. This example sends a GET request through the Scrapfly proxy and greps for X-Scrapfly headers.

```APIDOC
## [GET] https://httpbin.dev/html

### Description
Retrieves the target page while returning X-Scrapfly-* metadata headers in the response. Use this to inspect request cost, remaining credits, timing, and error details.

### Method
GET

### Endpoint
https://httpbin.dev/html

### Parameters
#### Proxy Authentication
- **proxy** (string) - Required - Proxy URL in format `http://country-us:YOUR_API_KEY@proxy.scrapfly.io:7777`

### Request Example
```bash
curl -v -x http://country-us:YOUR_API_KEY@proxy.scrapfly.io:7777 \
    https://httpbin.dev/html 2>&1 | grep -i "x-scrapfly"
```

### Response
#### Success Response (200)
- **X-Scrapfly-Api-Cost** (string) - API credits consumed by this request.
- **X-Scrapfly-Response-Time** (string) - Scrape duration in seconds.
- **X-Scrapfly-Remaining-Api-Credit** (string) - Remaining API credits on your account.
- **X-Scrapfly-Account-Concurrent-Usage** (string) - Current concurrent request usage at the account level.
- **X-Scrapfly-Account-Remaining-Concurrent-Usage** (string) - Remaining concurrent request slots at the account level.
- **X-Scrapfly-Log** (string) - Scrape log UUID, use this to look up the request in the Dashboard Monitoring.
- **X-Scrapfly-Content-Format** (string) - Response content format (`text` or `binary`).
- **X-Scrapfly-Reject-Code** (string) - Error code when the scrape fails (see Error Reference).
- **X-Scrapfly-Reject-Description** (string) - Human-readable error description.
- **X-Scrapfly-Reject-Retryable** (string) - `true` if the request can be retried.
- **X-Scrapfly-Screenshot-{name}** (string) - Screenshot download URL (one header per screenshot). E.g., `X-Scrapfly-Screenshot-fullpage`. Requires `screenshot-true` option.

#### Response Example
```
< X-Scrapfly-Api-Cost: 5
< X-Scrapfly-Response-Time: 1.234000
< X-Scrapfly-Remaining-Api-Credit: 99995
< X-Scrapfly-Log: 550e8400-e29b-41d4-a716-446655440000
```
```

--------------------------------

### GET /scrape with country parameter

Source: https://scrapfly.io/docs/scrape-api/proxy

Example of a scrape request targeting Canada using the country parameter. The request is made to the Scrapfly scrape API endpoint.

```APIDOC
## GET /scrape

### Description
Performs a scrape request with geo-targeting by specifying the country parameter. The example targets Canada.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **country** (string) - Required - Country code to target. Example: `ca` for Canada.
- **key** (string) - Required - API key.
- **url** (string) - Required - URL to scrape.

### Request Example
```
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "country=ca" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.dev/anything"
```

### Response
#### Success Response (200)
No response schema provided in the source.

#### Response Example
No response example provided in the source.
```

--------------------------------

### Scrape .onion hidden service with JavaScript rendering and screenshot (Curl HTTP)

Source: https://scrapfly.io/docs/scrape-api/proxy

Use this example to scrape a `.onion` hidden service through the managed Tor pool with JavaScript rendering and a full-page screenshot. The request must target the Tor pool; other pools return `ERR::PROXY::POOL_NOT_AVAILABLE_FOR_TARGET`. The example uses the Curl HTTP API with a GET request and query parameters.

```bash
curl -G \
--request "GET" \
--url "https://api.scrape-it.cloud/v1/scrape" \
--data "url=https://example.onion" \
--data "proxy_pool=public_tor_pool" \
--data "country=us" \
--data 
```

--------------------------------

### Enable webhook with curl request

Source: https://scrapfly.io/docs/scrape-api/webhook

Use this curl command to send a scrape request with the `webhook_name` parameter set to 'example'. Replace `__API_KEY__` with your actual API key. The response is a promise; the webhook will be called when the scrape completes.

```bash
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "webhook_name=example" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.dev/html"
```

--------------------------------

### Javascript Scenario configuration for browser downloads

Source: https://scrapfly.io/docs/scrape-api/javascript-rendering

This scenario clicks the download button (#download-btn) three times with 2.5-second waits between clicks. Each click triggers the download of download-sample.pdf, and the browser automatically numbers duplicates as download-sample.pdf, download-sample (1).pdf, and download-sample (2).pdf.

```json
[
  {"click": {"selector":"#download-btn"}},
  {"wait": 2500},
  {"click": {"selector":"#download-btn"}},
  {"wait": 2500},
  {"click": {"selector":"#download-btn"}},
  {"wait": 2500}
]
```

--------------------------------

### Get a schedule with GET /scrape/schedules/{id}

Source: https://scrapfly.io/docs/scrape-api/schedule

Returns one schedule by its UUID.

```bash
curl 'https://api.scrapfly.io/scrape/schedules/SCHEDULE_UUID?key=YOUR_API_KEY'

```

--------------------------------

### Node.js: Use CA certificate with NODE_EXTRA_CA_CERTS

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Node.js example to use the CA certificate by setting NODE_EXTRA_CA_CERTS environment variable before running the script.

```bash
# Set NODE_EXTRA_CA_CERTS environment variable
export NODE_EXTRA_CA_CERTS=/path/to/scrapfly-proxy-ca.crt
node my_script.js
```

--------------------------------

### GET /scrape

Source: https://scrapfly.io/docs/scrape-api/custom

Sends a GET request to the Scrapfly API with custom headers. The headers parameter must be URL-encoded.

```APIDOC
## [GET] /scrape

### Description
Sends a GET request to the Scrapfly API with custom headers. The headers parameter must be URL-encoded.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **key** (string) - Required - Your API key.
- **url** (string) - Required - The target URL to scrape.
- **headers** (object) - Optional - Custom headers to send with the request. Must be URL-encoded. Can be repeated to send multiple values for the same header.

### Request Example
```
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://httpbin.dev/headers" \
--data-urlencode "headers[foo]=bar"
```

### Response
#### Success Response (200)
- **headers** (object) - The headers sent in the request, echoed back by the target.

#### Response Example
```
{
  "headers": {
    "foo": "bar"
  }
}
```

```

--------------------------------

### GET /scrape

Source: https://scrapfly.io/docs/scrape-api/getting-started

Scrapes the target URL using the GET method. The scraped content is returned in JSON format within the result.content field.

```APIDOC
## [GET] /scrape

### Description
Scrapes the target URL using the GET method.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **url** (string) - Required - Target URL to scrape. Must be URL encoded
- **key** (string) - Required - API Key for authentication. Find your key on dashboard
- **proxy_pool** (string) - Optional - Select proxy pool. See proxy dashboard for available pools and pricing. Default: public_datacenter_pool
- **country** (string) - Optional - Proxy country (ISO 3166-1 alpha-2). Supports exclusions (-gb) and weighted distribution (us:10,gb:5). Default: random
- **lang** (string) - Optional - Page language (sets Accept-Language header). Defaults to proxy location language
- **os** (string) - Optional - Default: null

### Request Example
```
curl -X GET https://api.scrapfly.io/scrape?url=https://httpbin.dev/anything?q=I%20want%20to%20Scrape%20this&country=us&render_js=true&key=
```

### Response
#### Success Response (200)
- **result.content** (string) - The scraped content
- **config** (object) - The scrape configuration
- **context** (object) - Other activated feature information

#### Response Example
```json
{
  "result": {
    "content": "<html>...</html>"
  },
  "config": {},
  "context": {}
}

```

--------------------------------

### Execute step usage with script and timeout

Source: https://scrapfly.io/docs/scrape-api/javascript-scenario

Executes a JavaScript script with a timeout of 1000 milliseconds. This example returns the user agent string.

```json
{"execute": {"script": "return navigator.userAgent", "timeout": "1000"}
```

--------------------------------

### Python: Use CA certificate with requests

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Python example to use the CA certificate with requests/httpx by setting REQUESTS_CA_BUNDLE or passing the certificate path to verify.

```python
# For requests / httpx: set REQUESTS_CA_BUNDLE
export REQUESTS_CA_BUNDLE=/path/to/scrapfly-proxy-ca.crt

# Or pass it directly
response = requests.get(url, proxies=proxies, verify="/path/to/scrapfly-proxy-ca.crt")
```

--------------------------------

### GET /scrape

Source: https://scrapfly.io/docs/scrape-api/screenshot

This snippet documents the GET /scrape endpoint for taking screenshots with the Scrapfly API, including parameters, flags, and error handling.

```APIDOC
## [GET] /scrape

### Description
This endpoint is used to scrape a web page with optional screenshots. The screenshot feature can be enabled by adding the `screenshots` parameter and additional options can be set with the `screenshot_flags` parameter.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **render_js** (boolean) - Required - Enables JavaScript rendering, which is required for screenshots.
- **screenshots[all]** (string) - Optional - Captures a full-page screenshot. Value: `fullpage`.
- **screenshots[reviews]** (string) - Optional - Captures a screenshot of the element matching the CSS selector `#reviews`.
- **screenshot_flags** (string) - Optional - Comma-separated list of flags to enable additional screenshot options. Supported flags: `load_images`, `dark_mode`, `block_banners`, `high_quality`, `print_media_format`.
- **key** (string) - Required - API key for authentication.
- **url** (string) - Required - The URL of the page to scrape.

### Request Example
```
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "screenshots[all]=fullpage" \
--data-urlencode "screenshots[reviews]=#reviews" \
--data-urlencode "screenshot_flags=load_images,block_banners,high_quality" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"
```

### Response
#### Success Response (200)
- **screenshots** (object) - Contains the screenshot data. The response includes the captured screenshots as specified in the request.

#### Response Example
(No response example provided in the source.)

### Error Handling
- **422** - ERR::SCRAPE::DOM_SELECTOR_INVISIBLE - The specified DOM selector is not visible on the page.
- **422** - ERR::SCRAPE::UNABLE_TO_TAKE_SCREENSHOT - The screenshot could not be taken.

### Limitations
- Screenshots require JavaScript rendering to be enabled.
- Only GET requests are eligible for screenshots.
- Screenshots may fail on unusually JavaScript-heavy pages.
- If the cache feature is used and the cache is hit, screenshots will be ignored.
```

--------------------------------

### Python: Disable SSL verification with requests

Source: https://scrapfly.io/docs/scrape-api/proxy-mode

Python requests example with proxy configuration and verify=False to skip SSL certificate verification.

```python
import requests

proxies = {
    "http": "http://scrape:YOUR_API_KEY@proxy.scrapfly.io:7777",
    "https": "http://scrape:YOUR_API_KEY@proxy.scrapfly.io:7777",
}
# Set verify=False to skip certificate verification
response = requests.get("https://httpbin.dev/anything", proxies=proxies, verify=False)
```

--------------------------------

### Execute

Source: https://scrapfly.io/docs/scrape-api/javascript-scenario

Execute scenario step for the Scrapfly Scrape API.

```APIDOC
## Execute

### Description
Executes a JavaScript script and stores the result if a value is returned. Supports async/await functions. The result is available in the API response under `result.browser_data.js_scenario.steps`.

### Parameters
- **script** (string) - Required - Script to execute. It can return a serializable value.
- **timeout** (int) - Optional - Timeout to wait after the script execution has started, in milliseconds (default: 3000).

### Usage
```json
{"execute": {"script": "document.querySelector(\"body\").style.backgroundColor = \"red\";"}}
```

```json
{"execute": {"script": "return navigator.userAgent", "timeout": "1000"}}
```
```

--------------------------------

### GET /scrape

Source: https://scrapfly.io/docs/scrape-api/screenshot

This snippet documents the GET /scrape endpoint for capturing screenshots, including the required query parameters and the response structure containing screenshot URLs.

```APIDOC
## [GET] /scrape

### Description
This endpoint captures screenshots of web pages. The `screenshots` parameter must be set with a screenshot name and the desired capture area: `fullpage` (reserved value for capturing the entire web page) or a CSS selector value to target specific HTML elements.

### Method
GET

### Endpoint
https://api.scrapfly.io/scrape

### Parameters
#### Query Parameters
- **render_js** (boolean) - Required - Enables JavaScript rendering.
- **screenshots[all]** (string) - Required - Set to `fullpage` to capture the whole page under the name `all`.
- **screenshots[reviews]** (string) - Required - CSS selector (e.g., `#reviews`) to capture the reviews section under the name `reviews`.
- **key** (string) - Required - Your Scrapfly API key.
- **url** (string) - Required - The URL of the page to capture.

### Request Example
```
curl -G \
--request "GET" \
--url "https://api.scrapfly.io/scrape" \
--data-urlencode "render_js=true" \
--data-urlencode "screenshots[all]=fullpage" \
--data-urlencode "screenshots[reviews]=#reviews" \
--data-urlencode "key=__API_KEY__" \
--data-urlencode "url=https://web-scraping.dev/product/1"
```

### Response
#### Success Response (200)
- **result.screenshots** (object) - Contains screenshot entries keyed by the provided names.
  - **{name}.extension** (string) - File extension (e.g., `png`).
  - **{name}.format** (string) - Capture format: `fullpage` or `element`.
  - **{name}.css_selector** (string or null) - The CSS selector used, or `null` for fullpage.
  - **{name}.size** (integer) - File size in bytes.
  - **{name}.url** (string) - Authenticated URL to download the screenshot.

#### Response Example
```
{
    ...
    "result": {
        ...
        "screenshots": {
            "all": {
                "extension": "png",
                "format": "fullpage",
                "css_selector": null,
                "size": 2867974,
                "url": "https://api.scrapfly.io/4d1b8e8f-3803-4aa6-88fa-39d5aa81b6b3/scrape/screenshot/db475202-a7c0-4d7b-9179-98089901fce3/main"
            },
            "reviews": {
                "extension": "png",
                "format": "element",
                "css_selector": ".similarities-widget",
                "size": 109817,
                "url": "https://api.scrapfly.io/4d1b8e8f-3803-4aa6-88fa-39d5aa81b6b3/scrape/screenshot/db475202-a7c0-4d7b-9179-98089901fce3/similarities"
            }
        }
        ...
    }
    ...
}
```
```
Tokens:
Raw