Scrapeless

https://docs.scrapeless.com/en
Admin
Scrapeless is a platform for powerful, compliant web data extraction, offering tools like Universal...
Tokens:143,975
Snippets:859
Trust Score:8.5
Update:8 months ago
Show doc for...
Tokens:
Raw
### Create Scraping Task

Source: https://docs.scrapeless.com/en/deep-serp-api/quickstart/getting-started

Initiates a web scraping task using a specified actor and input parameters. Returns a taskId for in-progress tasks or the scraped data directly upon success.

```APIDOC
## POST /api/v1/scraper/request

### Description
Initiates a web scraping task. If the task is successful, it returns the data directly. If the task is in progress, it returns a taskId to retrieve the results later.

### Method
POST

### Endpoint
/api/v1/scraper/request

### Parameters
#### Request Body
- **actor** (string) - Required - The identifier for the scraping service or actor to use.
- **input** (object) - Required - Parameters required by the actor to perform the scraping task.
- **proxy** (object) - Optional - Specifies the country for using a proxy to send the request.

### Request Example
```json
{
    "actor": "scraper.google.search",
    "input": {
        "q": "Top news headlines",
        "gl": "us",
        "hl": "en",
        "google_domain": "google.com"
    }
}
```

### Response
#### Success Response (200)
- **data** (object) - The scraped data from the task.

#### In Progress Response (201)
- **message** (string) - Indicates the task is in progress.
- **taskId** (string) - The ID to retrieve the task results.

#### Error Responses
- **400 Bad Request**: Invalid task parameters.
- **429 Too Many Requests**: Rate limit exceeded.
- **500 Internal Server Error**: Server encountered an error.
```

--------------------------------

### Install Scrapeless SDK using npm

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Install the Scrapeless SDK package using npm. This is the first step to using the Scrapeless API in your Node.js project.

```bash
npm install @scrapeless-ai/sdk

```

--------------------------------

### Quick Start

Source: https://docs.scrapeless.com/en/sdk/node-sdk

A basic example demonstrating how to initialize the Scrapeless client with an API key.

```APIDOC
## Quick Start

1. Log in to the Scrapeless dashboard and get the API Key 
2. Set the API key as an environment variable named `SCRAPELESS_API_KEY` or pass it as a parameter to the `Scrapeless` class.

Here’s a basic example:

```javascript
import { Scrapeless } from '@scrapeless-ai/sdk';
 
const client = new Scrapeless({
  apiKey: 'YOUR_API_KEY' // or use SCRAPELESS_API_KEY env variable
});
```
```

--------------------------------

### Install Scrapeless SDK using pnpm

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Install the Scrapeless SDK package using pnpm. This is an alternative package manager for installing the Scrapeless API in your Node.js project.

```bash
pnpm add @scrapeless-ai/sdk

```

--------------------------------

### Task In Progress Response Example

Source: https://docs.scrapeless.com/en/deep-serp-api/quickstart/getting-started

This JSON object represents a response indicating that a scraping task is currently in progress. It includes a message and a unique taskId for retrieving the results later.

```json
{"message": "task in progress", "taskId":"30681c8b-bfd3-48eb-a7c9-006e40b00591"}
```

--------------------------------

### Batch Scrape URLs with Scrapeless SDK

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Perform a batch scrape of multiple URLs simultaneously using the Scrapeless SDK. This example shows how to pass a list of URLs and apply scrape and browser configurations.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.batchScrapeUrls(
    ["https://example.com", "https://scrapeless.com"],
    {
      formats: ["markdown", "html", "links"],
      onlyMainContent: false,
      timeout: 15000,
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );

  console.log(result);
})();

```

--------------------------------

### Scrape a Single URL with Scrapeless SDK

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Scrape the content of a single URL using the Scrapeless SDK. This example demonstrates initializing the client and calling the scrapeUrl function.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.scrapeUrl(
    "https://example.com"
  );

  console.log(result);
})();

```

--------------------------------

### Verify Node.js and npm Installation

Source: https://docs.scrapeless.com/en/integrations/claude/getting-started

Commands to verify if Node.js and npm have been successfully installed on your system. This is a prerequisite for running the Scrapeless MCP Server.

```shell
node -v
npm -v
```

--------------------------------

### Retrieve Google Search Data with JavaScript SDK

Source: https://docs.scrapeless.com/en/deep-serp-api/quickstart/getting-started

This JavaScript code snippet shows how to use the Scrapeless SDK to fetch Google Search data. It initializes the Scrapeless client with an API key and calls the createTask method for the 'scraper.google.search' actor, providing search queries and language/region settings.

```javascript
import { Scrapeless } from '@scrapeless-ai/sdk';

const client = new Scrapeless({
  apiKey: 'YOUR_API_KEY'
});

client.deepserp.createTask({
    actor: "scraper.google.search",
    input: {
        q: "Top news headlines",
        gl: "us",
        hl: "en",
        google_domain: "google.com"
    }
}).then(async (result) => {
    console.log(result);
  })
  .catch((error) => {
    console.error('Error:', error);
  });

```

--------------------------------

### Retrieve Google Search Data with Python SDK

Source: https://docs.scrapeless.com/en/deep-serp-api/quickstart/getting-started

This Python code snippet demonstrates how to use the Scrapeless SDK to retrieve Google Search data. It initializes the client with an API key and sends a request to the 'scraper.google.search' actor with specified input parameters.

```python
import json

import requests

def send_request():
    host = "api.scrapeless.com"
    url = f"https://{host}/api/v1/scraper/request"
    token = "your_api_key"

    headers = {
        "x-api-token": token,
        "Content-Type": "application/json"
    }

    json_payload = json.dumps({
        "actor": "scraper.google.search",
        "input": {
            "q": "Top news headlines",
            "gl": "us",
            "hl": "en",
            "google_domain": "google.com"
        }
    })

    response = requests.post(url, headers=headers, data=json_payload)

    if response.status_code != 200:
        print("Error:", response.status_code, response.text)
        return

    print("body", response.text)

if __name__ == "__main__":
    send_request()

```

--------------------------------

### Initialize Node.js Project and Install Dependencies

Source: https://docs.scrapeless.com/en/scraping-browser/guides/cloudflare-bypass-in-web-scraping

This snippet demonstrates the initial setup for a Node.js project intended for web scraping. It includes initializing the project with npm and installing necessary dependencies like puppeteer-core for browser automation.

```bash
cd path/to/scrapeless-bypass
npm init -y
npm install puppeteer-core
# If Puppeteer is not already installed on your system, install the full version:
npm install puppeteer puppeteer-core
```

--------------------------------

### Install Svix Libraries for Webhook Verification

Source: https://docs.scrapeless.com/en/general/webhooks/verifying-webhooks

Install the Svix libraries required for webhook verification. This section provides installation commands for various package managers and platforms.

```bash
npm install svix
# Or
yarn add svix
```

```bash
pip install svix
```

```toml
http = "1.0.0"
svix = "1.20.0"
```

```bash
go get github.com/svix/svix-webhooks/go
```

```gradle
// Gradle: Add this dependency to your project's build file:
implementation "com.svix:svix:0.x.y"
```

```xml
// Maven: Add this dependency to your project's POM:
<dependency>
 <groupId>com.svix</groupId>
 <artifactId>svix</artifactId>
 <version>0.x.y</version>
</dependency>
```

```gradle
// Gradle: Add this dependency to your project's build file:
implementation "com.svix.kotlin:svix-kotlin:0.x.y"
```

```xml
// Maven: Add this dependency to your project's POM:
<dependency>
 <groupId>com.svix.kotlin</groupId>
 <artifactId>svix-kotlin</artifactId>
 <version>0.x.y</version>
</dependency>
```

```bash
gem install svix
```

```bash
composer require svix/svix
```

```bash
dotnet add package Svix
```

```bash
# On macOS install via [Homebrew](https://brew.sh/):
brew install svix/svix/svix

# On Windows install via [Scoop](https://scoop.sh/):
scoop bucket add svix https://github.com/svix/scoop-svix.git
scoop install svix

# For other platforms, such as linux, checkout the [CLI docs](https://github.com/svix/svix-cli#installation) on Github.
```

```bash
# Install cURL. E.g. on arch linux:
pacman -S curl
```

--------------------------------

### Submit and Poll for Scraper Results (Python)

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

Illustrates how to submit a scraping task and handle the response using Python's requests library. This example shows the initial POST request to submit a task and prepares for handling the response, including asynchronous processing.

```python
import requests
import json
import time

API_KEY = "YOUR_API_KEY"
HOST = "api.scrapeless.com"

# Submit task
task_url = f"https://{HOST}/api/v1/scraper/request"
payload = json.dumps({
    "actor": "scraper.shopee",
    "input": {"url": "https://shopee.tw/a-i.10228173.24803858474"}
})
headers = {'Content-Type': 'application/json', 'x-api-token': API_KEY}

response = requests.post(task_url, headers=headers, data=payload)

```

--------------------------------

### Scraping Browser WebSocket Connection Examples

Source: https://docs.scrapeless.com/en/scraping-browser/quickstart/getting-started

Demonstrates various ways to establish a WebSocket connection to the Scraping Browser API. These examples cover basic connection with an API key, advanced geographic proxy targeting, and using a custom proxy URL.

```websocket
wss://browser.scrapeless.com/api/v2/browser?token=APIKey&sessionTTL=180&proxyCountry=ANY
```

```websocket
wss://browser.scrapeless.com/api/v2/browser?token=APIKey&sessionTTL=180&proxyCountry=US&proxyState=CA&proxyCity=Los%20Angeles
```

```websocket
wss://browser.scrapeless.com/api/v2/browser?token=APIKey&sessionTTL=180&proxyURL=proxyURL
```

--------------------------------

### Installation

Source: https://docs.scrapeless.com/en/sdk/node-sdk

Instructions on how to install the Scrapeless Node.js SDK using npm.

```APIDOC
## Installation

To install the Scrapeless Node SDK, you can use npm:

```bash
npm install @scrapeless-ai/sdk
```
```

--------------------------------

### Full Browser Use SDK Example (Python)

Source: https://docs.scrapeless.com/en/scraping-browser/integrations/browser-use

This is a complete Python script demonstrating the Browser Use SDK. It includes setup for the browser and agent, execution of a web scraping task, and proper handling of browser connections and asynchronous operations.

```python
from dotenv import load_dotenv
import os
import asyncio
from urllib.parse import urlencode
from browser_use import Agent, Browser, ChatOpenAI
from pydantic import SecretStr

task = "Go to Google, search for 'Scrapeless', click on the first post and return to the title"

async def setup_browser() -> Browser:
    scrapeless_base_url = "wss://browser.scrapeless.com/api/v2/browser"
    query_params = {
        "token": os.environ.get("SCRAPELESS_API_KEY"),
        "sessionTTL": 900,
        "proxyCountry": "ANY"
    }
    browser_ws_endpoint = f"{scrapeless_base_url}?{urlencode(query_params)}"
    browser = Browser(cdp_url=browser_ws_endpoint)
    return browser

async def setup_agent(browser: Browser) -> Agent:
    llm = ChatOpenAI(
        model="gpt-4o", # Or choose the model you want to use
        api_key=SecretStr(os.environ.get("OPENAI_API_KEY")),
    )

    return Agent(
        task=task,
        llm=llm,
        browser=browser,
    )

async def main():
    load_dotenv()
    browser = await setup_browser()
    await browser.start()
    agent = await setup_agent(browser)
    result = await agent.run()
    print(result)
    await browser.kill()

asyncio.run(main())
```

--------------------------------

### Launch Browser with Puppeteer and Proxy Authentication

Source: https://docs.scrapeless.com/en/proxies/features/integrations

This Node.js example uses Puppeteer to launch a browser instance configured to use a proxy server. It also demonstrates how to handle proxy authentication within the page context. Ensure 'puppeteer' is installed.

```javascript
const puppeteer =require('puppeteer');

(async() => {
    const proxyUrl = 'http://gw-us.scrapeless.com:8789';
    const username = 'username';
    const password = 'password';

    const browser = await puppeteer.launch({
        args: [`--proxy-server=${proxyUrl}`],
        headless: false
    });

    const page = await browser.newPage();

    await page.authenticate({ username, password });
    await page.goto('API_URL');

    await browser.close();
})();
```

--------------------------------

### Create Browser Session with Scrapeless SDK and Puppeteer

Source: https://docs.scrapeless.com/en/proxies/features/integrations

This example uses the Scrapeless SDK and Puppeteer to create a browser instance connected via a proxy. It requires API keys and specifies proxy details like country and type. Ensure '@scrapeless-ai/sdk' and 'puppeteer-core' are installed.

```javascript
import { Scrapeless } from '@scrapeless-ai/sdk';
import puppeteer from 'puppeteer-core';
const client = new Scrapeless({
apiKey: 'YOUR_API_KEY',
});
(async () => {
const proxyUrl = client.proxies.proxy({
  country: 'US',
  sessionDuration: 30,
  sessionId: 'test-session-123',
  gateway: 'gw-us.scrapeless.com:8789',
  type: 'residential' | 'ipv6' | 'datacenter'
});
const { browserWSEndpoint } = await client.browser.create({
  sessionName: 'sessionName',
  sessionTTL: 180,
  proxyCountry: 'US',
  proxyURL: proxyUrl,
  sessionRecording: true,
  defaultViewport: null,
});
const browser = await puppeteer.connect({
  browserWSEndpoint: browserWSEndpoint,
});
const page = await browser.newPage();
await page.goto('API_URL');
await browser.close();
})();
```

--------------------------------

### Set up Svix Webhook Route in Ruby on Rails

Source: https://docs.scrapeless.com/en/general/webhooks/verifying-webhooks

This Ruby on Rails example shows how to set up a route for incoming Svix webhooks and implement the verification logic. It first defines a `POST /webhook` route in `config/routes.rb` and then adds the verification code to the `WebhookController`'s `index` action in `app/controllers/webhook_controller.rb`. It disables CSRF protection for the endpoint and handles verification errors.

```ruby
Rails.application.routes.draw do
  post "/webhook", to: "webhook#index"

  # For details on the DSL available within this file, see https://guides.rubyonrails.org/routing.html
end
```

```ruby
require  'svix'  
  
class  WebhookController  < ApplicationController  
 protect_from_forgery with:  :null_session  # disables CSRF middleware; required for API endpoints  
 def  index begin payload = request.body.read headers = request.headers wh = Svix::Webhook.new("whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw")  

 json = wh.verify(payload, headers)  
 # Do something with the message...  
 head :no_content rescue head :bad_request end endend  
```

--------------------------------

### Capture Data with Universal Scraping API (JavaScript SDK)

Source: https://docs.scrapeless.com/en/universal-scraping-api/quickstart/getting-started

Demonstrates how to capture data using the Universal Scraping API with the JavaScript SDK. It requires an API key and specifies the actor, input parameters (URL, method, redirect), and proxy country. The output is logged to the console or an error is caught.

```javascript
import { Scrapeless } from '@scrapeless-ai/sdk';

const client = new Scrapeless({
  apiKey: 'YOUR_API_KEY'
});

client.universal.scrape({
   actor: "unlocker.webunlocker",
   input: {
      url: "https://httpbin.io/get",
      redirect: false,
      method: "GET",
   },
   proxy: {
      country: "ANY",
   }
}).then(async (result) => {
    console.log(result);
  })
  .catch((error) => {
    console.error('Error:', error);
  });
```

--------------------------------

### Install Scrapeless Node.js SDK

Source: https://docs.scrapeless.com/en/sdk/node-sdk

Installs the Scrapeless Node.js SDK using npm. This is the first step to integrate Scrapeless services into your Node.js application.

```bash
npm install @scrapeless-ai/sdk
```

--------------------------------

### Universal Scraping API - POST /api/v2/unlocker/request

Source: https://docs.scrapeless.com/en/universal-scraping-api/quickstart/getting-started

This endpoint allows you to perform universal scraping tasks by specifying an actor, input parameters for the scraping task, and optional proxy settings.

```APIDOC
## POST /api/v2/unlocker/request

### Description
This endpoint allows you to perform universal scraping tasks by specifying an actor, input parameters for the scraping task, and optional proxy settings.

### Method
POST

### Endpoint
/api/v2/unlocker/request

### Parameters
#### Query Parameters
- **apiKey** (string) - Required - Your API key for authentication.

#### Request Body
- **actor** (string) - Required - The specific scraping actor to use (e.g., "unlocker.webunlocker").
- **input** (object) - Required - Parameters required by the actor to complete the scraping task.
  - **url** (string) - Required - The target website URL.
  - **method** (string) - Optional - The HTTP request method (e.g., "GET", "POST"). Defaults to "GET".
  - **redirect** (boolean) - Optional - Whether to allow redirection. Defaults to `false`.
  - **headers** (object) - Optional - Custom request header fields.
- **proxy** (object) - Optional - Proxy settings for the request.
  - **country** (string) - Optional - The country code for the proxy region (e.g., "ANY").

### Request Example
```json
{
  "actor": "unlocker.webunlocker",
  "input": {
    "url": "https://httpbin.io/get",
    "redirect": false,
    "method": "GET"
  },
  "proxy": {
    "country": "ANY"
  }
}
```

### Response
#### Success Response (200)
- **result** (object) - The scraped data or result of the scraping task.
- **status** (string) - The status of the scraping request.

#### Response Example
```json
{
  "result": {
    "url": "https://httpbin.io/get",
    "headers": {
      "Accept": "*/*",
      "Accept-Encoding": "gzip, deflate",
      "Host": "httpbin.io",
      "User-Agent": "python-requests/2.28.1",
      "X-Amzn-Trace-Id": "Root=1-658a7d1b-7f1a0d0e1a2b3c4d5e6f7a8b"
    },
    "args": {},
    "origin": "1.2.3.4",
    "data": "",
    "files": {},
    "form": {},
    "headers": {
      "Accept": "*/*",
      "Accept-Encoding": "gzip, deflate",
      "Host": "httpbin.io",
      "User-Agent": "python-requests/2.28.1",
      "X-Amzn-Trace-Id": "Root=1-658a7d1b-7f1a0d0e1a2b3c4d5e6f7a8b"
    },
    "json": null,
    "method": "GET",
    "origin": "1.2.3.4",
    "url": "https://httpbin.io/get"
  },
  "status": "completed"
}
```
```

--------------------------------

### Create LLM Chat Scraper Task with cURL

Source: https://docs.scrapeless.com/en/llm-chat-scraper/quickstart/getting-started

This example demonstrates how to create a scraping task for the LLM Chat Scraper API using cURL. It sends a POST request to the API endpoint with actor-specific input and an optional webhook URL for asynchronous result delivery. The input includes a prompt, country, and a flag for web search.

```bash
curl '{api_host}/api/v2/scraper/request' \
  --header 'Content-Type: application/json' \
  --header 'x-api-token: {you_api_key}' \
  --data '{
    "actor": "scraper.chatgpt",
    "input": {
      "prompt": "Most reliable proxy service for data extraction",
      "country": "US",
      "web_search": true
    },
    "webhook": {
      "url": "http://www.youwebhook.com"
    }
  }'
```

--------------------------------

### Install Browser Use SDK (Python)

Source: https://docs.scrapeless.com/en/scraping-browser/integrations/browser-use

Instructions for installing the Browser Use Python SDK using pip. It also provides a command for installing with memory functionality, noting a Python version compatibility requirement.

```bash
pip install browser-use

pip install "browser-use[memory]"
```

--------------------------------

### Poll Scrapeless API for Results (cURL)

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

This cURL command demonstrates how to poll the Scrapeless API for the results of a scraping task. It sends a GET request to the result endpoint, including the task ID and the necessary API token in the headers. This is useful for manual checks or integration with systems that do not use the provided SDKs.

```bash
curl --location --request GET 'https://api.scrapeless.com/api/v1/scraper/result/30681c8b-bfd3-48eb-a7c9-006e40b00591' \
--header 'x-api-token: YOUR_API_KEY' \
--header 'Content-Type: application/json'

```

--------------------------------

### Configure Scrapeless MCP Server in Cline

Source: https://docs.scrapeless.com/en/integrations/cline/getting-started

This JSON configuration is used to set up the Scrapeless MCP Server within Cline. It specifies the command to run the server, necessary arguments, and environment variables, including the crucial SCRAPELESS_KEY which must be replaced with your actual API token.

```json
{
  "mcpServers": {
    "scrapelessMcpServer": {
      "command": "npx",
      "args": ["-y", "scrapeless-mcp-server"],
      "env": {
        "SCRAPELESS_KEY": "YOUR_SCRAPELESS_KEY" 
      }
    }
  }
}
```

--------------------------------

### Capture Data with Universal Scraping API (Python Requests)

Source: https://docs.scrapeless.com/en/universal-scraping-api/quickstart/getting-started

Shows how to capture data using the Universal Scraping API via a POST request with Python's requests library. It requires an API key, specifies the actor, input parameters (URL, method, redirect), and proxy country. The response text is printed to the console.

```python
import requests
import json

API_KEY = ""
host = "api.scrapeless.com"
url = f"https://{host}/api/v2/unlocker/request"

payload = json.dumps({
   "actor": "unlocker.webunlocker",
   "input": {
      "url": "https://httpbin.io/get",
      "redirect": False,
      "method": "GET",
   },
   "proxy":{
       "country": "ANY",
   }
})
headers = {
   'Content-Type': 'application/json',
   'x-api-token': f'{API_KEY}'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)
```

--------------------------------

### Install Playwright Core

Source: https://docs.scrapeless.com/en/scraping-browser/libraries/playwright

Installs the playwright-core library, a lightweight version of Playwright used for connecting to existing browser instances. This is a prerequisite for using Playwright with Scraping Browser.

```bash
npm install playwright-core

```

--------------------------------

### Install Scrapling and Fetcher Dependencies

Source: https://docs.scrapeless.com/en/scraping-browser/integrations/scrapling

Installs the Scrapling library and optional fetcher dependencies using pip. The `scrapling install` command is used to set up browser-related dependencies required for Scrapling's functionality.

```bash
pip install scrapling
# If you need dynamic or stealth fetchers:
pip install "scrapling[fetchers]"
# Install browser dependencies
scrapling install
```

--------------------------------

### Install Puppeteer Core

Source: https://docs.scrapeless.com/en/scraping-browser/libraries/puppeteer

Installs the `puppeteer-core` package, a lightweight version of Puppeteer for connecting to existing browser instances. This is a prerequisite for using Puppeteer with Scraping Browser.

```bash
npm install puppeteer-core
```

--------------------------------

### Challenge Bypass Example (Axios)

Source: https://docs.scrapeless.com/en/scraping-browser/guides/cloudflare-bypass-in-web-scraping

An example demonstrating how to use the Scrapeless Web Unlocker API with Axios to bypass Cloudflare challenges by enabling JavaScript rendering and using the `waitFor` instruction.

```APIDOC
## Challenge Bypass Example using Axios

### Description
This example shows how to programmatically bypass Cloudflare challenges using the Scrapeless Web Unlocker API. It utilizes the `axios` library to send a POST request, enabling JavaScript rendering and specifying an instruction to wait for a specific element on the page, ensuring the challenge is resolved before content is returned.

### Method
POST

### Endpoint
/api/v2/unlocker/request

### Parameters
#### Request Body (Illustrative Payload)
- **actor**: "unlocker.webunlocker"
- **proxy**: { "country": "US" }
- **input**: {
    "url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "jsRender": {
      "enabled": true,
      "instructions": [
        {
          "waitFor": [
            "main.page-content .challenge-info",
            30000
          ]
        }
      ]
    }
  }

### Request Example (JavaScript with Axios)
```javascript
import axios from 'axios'
 
async function sendRequest() {
  const host = "api.scrapeless.com";
  const url = `https://${host}/api/v2/unlocker/request`;
  const API_KEY = 'your_api_key' // Replace with your actual API key

  const payload = {
    actor: "unlocker.webunlocker",
    proxy: {
      country: "US"
    },
    input: {
      url: "https://www.scrapingcourse.com/cloudflare-challenge",
      jsRender: {
        enabled: true,
        instructions: [
          {
            waitFor: [
              "main.page-content .challenge-info",
              30000
            ]
          }
        ]
      }
    }
  }

  try {
    const response = await axios.post(url, payload, {
      headers: {
        'Content-Type': 'application/json',
        'x-api-token': API_KEY
      }
    });
    console.log("[page_html_body] =>", response.data);
  } catch (error) {
    console.error('Error:', error);
  }
}
sendRequest();
```

### Response
#### Success Response (200)
- **page_html_body** (string) - The HTML content of the page after successfully bypassing the Cloudflare challenge.

#### Response Example
```json
{
  "page_html_body": "<html>...</html>"
}
```
```

--------------------------------

### URL Crawling with Scrapeless AI

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

This section demonstrates how to initialize the Scrapeless AI client and crawl a given URL with various scraping and browser options.

```APIDOC
## POST /websites/scrapeless_en/crawl

### Description
Initiates a web crawl for a given URL, allowing for detailed configuration of scraping and browser options.

### Method
POST

### Endpoint
/websites/scrapeless_en/crawl

### Parameters
#### Query Parameters
- **apiKey** (string) - Required - Your API key for authentication.

#### Request Body
- **url** (string) - Required - The URL to crawl.
- **limit** (integer) - Optional - The maximum number of pages to crawl.
- **scrapeOptions** (object) - Optional - Configuration for the scraping process.
  - **formats** (array of strings) - Optional - Desired output formats (e.g., "markdown", "html", "links").
  - **onlyMainContent** (boolean) - Optional - Whether to extract only the main content of the page.
  - **timeout** (integer) - Optional - Timeout in milliseconds for page navigation.
- **browserOptions** (object) - Optional - Configuration for the browser session.
  - **proxyCountry** (string) - Optional - The country to use for the proxy (e.g., "ANY").
  - **sessionName** (string) - Optional - A name for the browser session.
  - **sessionRecording** (boolean) - Optional - Whether to record the session.
  - **sessionTTL** (integer) - Optional - Time-to-live for the session in seconds.

### Request Example
```json
{
  "url": "https://example.com",
  "limit": 2,
  "scrapeOptions": {
    "formats": ["markdown", "html", "links"],
    "onlyMainContent": false,
    "timeout": 15000
  },
  "browserOptions": {
    "proxyCountry": "ANY",
    "sessionName": "Crawl",
    "sessionRecording": true,
    "sessionTTL": 900
  }
}
```

### Response
#### Success Response (200)
- **success** (boolean) - Indicates if the operation was successful.
- **status** (string) - The current status of the crawl (e.g., "completed", "failed").
- **completed** (integer) - The number of pages successfully crawled.
- **total** (integer) - The total number of pages targeted for crawling.
- **data** (array) - An array of objects, each representing a crawled page.
  - **url** (string) - The URL of the crawled page.
  - **metadata** (object) - Metadata about the crawled page.
    - **title** (string) - The title of the page.
    - **description** (string) - The description of the page.
  - **markdown** (string) - The content of the page in markdown format (if requested).
  - **error** (object) - An error object if the page crawl failed.

#### Response Example
```json
{
  "success": true,
  "status": "completed",
  "completed": 2,
  "total": 2,
  "data": [
    {
      "url": "https://example.com",
      "metadata": {
        "title": "Example Page",
        "description": "A sample webpage"
      },
      "markdown": "# Example Page\nThis is content...",
      "html": "<html>...</html>",
      "links": ["http://link1.com", "http://link2.com"]
    }
  ]
}
```
```

--------------------------------

### Verify Svix Webhook in Node.js (NestJS Controller)

Source: https://docs.scrapeless.com/en/general/webhooks/verifying-webhooks

This NestJS controller example demonstrates Svix webhook verification. It utilizes the `@Req()` decorator to access the raw request body, which must be enabled in the NestJS application setup. The webhook secret is hardcoded in this example but should ideally be managed securely.

```typescript
// webhook.controller.ts
import { Controller, Post, RawBodyRequest, Req } from '@nestjs/common';
import { Request } from 'express';
import { Webhook } from 'svix';

@Controller('webhook')
class WebhookController {
  @Post()
  webhook(@Req() request: RawBodyRequest<Request>) {
    const secret = 'whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw';
    const wh = new Webhook(secret);

    const payload = request.rawBody.toString('utf8');
    const headers = request.headers;

    let msg;
    try {
      msg = wh.verify(payload, headers);
    } catch (err) {
      // handle error
    }

    // Do something with the message...
  }
}

```

--------------------------------

### Crawl URL with Browser Configurations (JavaScript)

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Demonstrates initializing the Scrapeless AI client and crawling a URL with specific browser configurations. This includes setting proxy country, session name, recording, and time-to-live for the browser session.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );

  console.log(result);
})();

```

--------------------------------

### Crawl URL with Scrape Configurations (JavaScript)

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Shows how to initialize the Scrapeless AI client and crawl a URL while specifying scrape job configurations. This includes setting the desired output formats, enabling or disabling main content extraction, and defining a page navigation timeout.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      scrapeOptions: {
        formats: ["markdown", "html", "links"],
        onlyMainContent: false,
        timeout: 15000,
      }
    }
  );

  console.log(result);
})();

```

--------------------------------

### Crawl URL with Scrapeless AI SDK (JavaScript)

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Initializes the Scrapeless AI client and crawls a given URL. It demonstrates setting scrape options like formats and timeout, and browser options such as proxy country and session settings. The result of the crawl is then logged to the console.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.crawlUrl(
    "https://example.com",
    {
      limit: 2,
      scrapeOptions: {
        formats: ["markdown", "html", "links"],
        onlyMainContent: false,
        timeout: 15000,
      },
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );

  console.log(result);
})();

```

--------------------------------

### Retrieve LLM Chat Scraper Task Result with cURL

Source: https://docs.scrapeless.com/en/llm-chat-scraper/quickstart/getting-started

This example shows how to retrieve the result of a previously created LLM Chat Scraper task using cURL. It sends a GET request to the API endpoint, requiring the task ID obtained from the task creation step. This method is used for polling the result if a webhook was not configured.

```bash
curl --request GET '{api_host}/api/v2/scraper/result/{task_id}' \
  --header 'Content-Type: application/json' \
  --header 'x-api-token: {you_api_key}'
```

--------------------------------

### Start Email Listener with Task ID (JavaScript)

Source: https://docs.scrapeless.com/en/scraping-browser/guides/github-mfa-automation

This code snippet shows how to instantiate and start an EmailListener. It assumes a `taskId` variable is available and passed to the `start` method. The `.then()` is used to handle the promise returned by the `start` method, though no specific callback is provided in this example.

```javascript
const listener = new EmailListener();
listener.start({taskId}).then(); // Replace with taskId from the first step
```

--------------------------------

### Scrape URL with Scrape Configurations using Scrapeless SDK

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Scrape a URL using the Scrapeless SDK with specific scrape job configurations, including response formats, main content extraction, and timeout settings.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.scrapeUrl(
    "https://example.com",
    {
      formats: ["markdown", "html", "links"],
      onlyMainContent: false,
      timeout: 15000,
    }
  );

  console.log(result);
})();

```

--------------------------------

### GET /api/v1/scraper/result/{taskId}

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

Poll for the results of an asynchronous scraping task using the provided taskId.

```APIDOC
## GET /api/v1/scraper/result/{taskId}

### Description
Poll for the results of an asynchronous scraping task. Use this endpoint with the `taskId` obtained from a `POST /api/v1/scraper/request` call when the initial response was HTTP 201.

### Method
GET

### Endpoint
/api/v1/scraper/result/{taskId}

### Parameters
#### Path Parameters
- **taskId** (string) - Required - The ID of the task to retrieve results for.

#### Query Parameters
None

#### Request Body
None

### Request Example
(No request body for GET requests)

### Response
#### Success Response (200)
- **data** (object) - The scraped data. This will be returned once the task is complete.

#### Processing Response (201)
- **message** (string) - Indicates the task is still in progress.

#### Response Example (200)
```json
{
  "data": {
    "product_name": "Example Product",
    "price": "100.00"
  }
}
```

#### Response Example (201)
```json
{
  "message": "Task is still processing. Please poll again later."
}
```
```

--------------------------------

### Scrape URL with Browser Configurations using Scrapeless SDK

Source: https://docs.scrapeless.com/en/crawl/quickstart/getting-started

Scrape a URL using the Scrapeless SDK while providing custom browser configurations, such as proxy country and session settings.

```javascript
import { ScrapingCrawl } from "@scrapeless-ai/sdk";

// Initialize the client
const client = new ScrapingCrawl({
  apiKey: "your-api-key", // Get your API key from https://scrapeless.com
});

(async () => {
  const result = await client.scrapeUrl(
    "https://example.com",
    {
      browserOptions: {
        proxyCountry: "ANY",
        sessionName: "Crawl",
        sessionRecording: true,
        sessionTTL: 900,
      },
    }
  );

  console.log(result);
})();

```

--------------------------------

### Submit and Poll for Scraper Results (Node.js)

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

Demonstrates the full workflow of submitting a scraping task and handling the response using the Scrapeless SDK in Node.js. It shows how to initiate a scrape request and process the returned data or errors.

```javascript
import { Scrapeless } from '@scrapeless-ai/sdk';
const client = new Scrapeless({
  apiKey: 'YOUR_API_KEY',
});

async function runTask() {
  const result = await client.scraping.scrape({
    actor: 'scraper.shopee',
    input: {
      url: 'https://shopee.tw/a-i.10228173.24803858474'
    }
  });

  return result;
}

runTask()
  .then(data => {
    console.log('Data:', data);
  })
  .catch(error => {
    console.error('Error:', error);
  });
```

--------------------------------

### POST /api/v1/scraper/request

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

Submit a scraping task to the API. This endpoint initiates the scraping process.

```APIDOC
## POST /api/v1/scraper/request

### Description
Submit a scraping task to the API. This endpoint initiates the scraping process. Depending on the task, you might receive data directly or a taskId to poll for results.

### Method
POST

### Endpoint
/api/v1/scraper/request

### Parameters
#### Query Parameters
None

#### Request Body
- **actor** (string) - Required - The identifier for the scraper to use.
- **input** (object) - Required - The input parameters for the scraper.
  - **url** (string) - Required - The URL to scrape.

### Request Example
```json
{
  "actor": "scraper.shopee",
  "input": {
    "url": "https://shopee.tw/a-i.10228173.24803858474"
  }
}
```

### Response
#### Success Response (200)
- **data** (object) - The scraped data.

#### Processing Response (201)
- **taskId** (string) - The ID of the submitted task, used for polling.

#### Response Example (200)
```json
{
  "data": {
    "product_name": "Example Product",
    "price": "100.00"
  }
}
```

#### Response Example (201)
```json
{
  "taskId": "a1b2c3d4-e5f6-7890-1234-567890abcdef"
}
```
```

--------------------------------

### Get Live URL with Agent.liveURL (Node.js/Puppeteer, Python/Playwright)

Source: https://docs.scrapeless.com/en/scraping-browser/features/advanced-privacy-anti-detection/cdp-api

Retrieves the live URL of the current session's page. This functionality is available for both Node.js (Puppeteer) and Python (Playwright). The Node.js example uses `liveURL` via a CDP session and includes error handling. The Python examples directly use `Agent.liveURL`.

```javascript
const { Puppeteer, log as Log, createPuppeteerCDPSession } = require('@scrapeless-ai/sdk');
const logger = Log.withPrefix('puppeteer-example');

(async () => {
    const browser = await Puppeteer.connect({
      sessionName: 'sdk_test',
      sessionTTL: 180,
      proxyCountry: 'US',
      sessionRecording: true,
      defaultViewport: null
    });
    const page = await browser.newPage();
    await page.goto('https://www.scrapeless.com');
    const cdpSession = await createPuppeteerCDPSession(page);

    const { error, liveURL } = await cdpSession.liveURL();
    if (error) {
      logger.error('Failed to get current page URL:', error);
    } else {
      logger.info('Current page URL:', liveURL);
    }
    await browser.close();
})();
```

```javascript
await page.goto("https://www.scrapeless.com", {
    waitUntil: "networkidle0",
    timeout: 30000,
});
const client = await page.createCDPSession();
const result = await client.send("Agent.liveURL");
console.log(result);
```

```python
await page.goto("https://www.scrapeless.com", timeout=60000, wait_until="load")
client = await page.target.createCDPSession()
result = await client.send('Agent.liveURL')
print(result)
```

--------------------------------

### GET /api/v1/scraper/result/{taskId}

Source: https://docs.scrapeless.com/en/scraping-api/quickstart/getting-started

Polls the Scrapeless API for the results of a previously submitted scraping task using its taskId. Returns the result if available (200) or indicates processing is ongoing (201).

```APIDOC
## GET /api/v1/scraper/result/{taskId}

### Description
Polls the Scrapeless API for the results of a previously submitted scraping task using its taskId. Returns the result if available (200) or indicates processing is ongoing (201).

### Method
GET

### Endpoint
https://api.scrapeless.com/api/v1/scraper/result/{taskId}

### Parameters
#### Path Parameters
- **taskId** (string) - Required - The ID of the scraping task to retrieve results for.

### Response
#### Success Response (200)
- **data** (object) - The final scraping result.

#### Success Response (201)
- Indicates that the task is still being processed.

#### Response Example (200)
```json
{
  "data": { ... }
}
```
```

--------------------------------

### Command-Line Arguments

Source: https://docs.scrapeless.com/en/scraping-browser/features/advanced-privacy-anti-detection/custom-fingerprint

Configure browser behavior using command-line switches.

```APIDOC
## Command-Line Arguments

### Description
Run browser with command-line switches. Please note that the following args are not supported: `--load-extension`, `--user-data-dir`, `--fonts-dir`, `--remote-debugging-port`.

### Method
N/A (Configuration Object)

### Endpoint
N/A (Configuration Object)

### Parameters
#### Object: `args`
**Type**: `object`
**Description**: Run browser with command-line switches.

##### Fields
- **proxy bypass** (`string`) - Optional - Example: `'--proxy-bypass-list': 'example.com'`.
- **window size** (`string`) - Optional - Example: `'--window-size': '1280,1024'`. It is recommended to set the same as the `screen` fingerprint.

### Request Example
```json
{
  "args": {
    "--proxy-bypass-list": "example.com",
    "--window-size": "1920,1080"
  }
}
```

### Response
#### Success Response (200)
N/A (Configuration Object)

#### Response Example
N/A (Configuration Object)
```

--------------------------------

### Make API Request with Rust Reqwest Proxy

Source: https://docs.scrapeless.com/en/proxies/features/integrations

This Rust example uses the 'reqwest' crate to perform an asynchronous HTTP GET request through a proxy with basic authentication. Ensure 'reqwest' and 'tokio' are added to your Cargo.toml.

```rust
#[tokio::main]
async fn main() {

let http_proxy = reqwest::Proxy::http("http://gw-us.scrapeless.io:8789")
  .unwrap().basic_auth("user", "password");

let client = reqwest::ClientBuilder::new().proxy(http_proxy).build().unwrap();

let resp = client.get("API_URL").send().await.unwrap();

println!("{:?}", resp.text().await.unwrap());

}
```

--------------------------------

### Make API Request with Node.js Axios Proxy

Source: https://docs.scrapeless.com/en/proxies/features/integrations

This Node.js example uses the 'axios' library to fetch data from an API via a proxy. It includes basic authentication for the proxy. Make sure to install 'axios'.

```javascript
const axios = require("axios");

const url = "API_URL";
axios.get(url, {
proxy: {
    protocol: 'http',
    host: 'gw-us.scrapeless.io',
    port: 8789,
    auth: {
        username: 'username',
        password: 'password',
    },
},
})
.then((res) => {
    console.log(res.data);
})
.catch((err) => {
    console.log('[err]:', err);
});
console.log(resp.data);
```
Tokens:
Raw