Scraping Robot

https://scrapingrobot.readme.io/reference
Admin
Scraping Robot is an API service that enables users to scrape web content by sending HTTP requests...
Tokens:15,057
Snippets:88
Trust Score:9.9
Update:6 months ago
Show doc for...
Tokens:
Raw
### Get Credit Balance using Ruby

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

Illustrates how to get your ScrapingRobot credit balance using Ruby. This example uses the built-in 'net/http' library to perform the GET request.

```ruby
require 'net/http'
require 'uri'

uri = URI.parse('https://api.scrapingrobot.com/balance?token=<YOUR_SR_TOKEN>')

response = Net::HTTP.get_response(uri)

if response.is_a?(Net::HTTPSuccess)
  puts response.body
else
  puts "Error: #{response.code} #{response.message}"
end
```

--------------------------------

### Scrape Web Pages using Ruby

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example shows how to scrape a web page using Ruby. It uses the 'net/http' library to send a GET request to the Scraping Robot API with the required URL parameter.

```ruby
require 'net/http'
require 'uri'

def scrape_page(url)
  uri = URI.parse("https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>&url=#{URI.encode_www_form_component(url)}")
  response = Net::HTTP.get_response(uri)
  response.body if response.is_a?(Net::HTTPSuccess)
end

# Example usage:
# puts scrape_page('https://scrapingrobot.com/about-us')
```

--------------------------------

### Scrape Web Pages using cURL

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example shows how to scrape a web page using the cURL command-line tool. It includes the necessary GET request, URL, and headers for interacting with the Scraping Robot API.

```shell
curl --request GET \
     --url 'https://api.scrapingrobot.com/?url=https://scrapingrobot.com/about-us/' \
     --header 'accept: application/json'
```

--------------------------------

### Scrape Web Pages using PHP

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example demonstrates how to scrape a web page using PHP. It uses the 'file_get_contents' function to make a GET request to the Scraping Robot API, including the necessary URL parameter.

```php
<?php

function scrape_page($url) {
    $token = '<YOUR_SR_TOKEN>';
    $encoded_url = urlencode($url);
    $api_url = "https://api.scrapingrobot.com/?token={$token}&url={$encoded_url}";
    
    $content = file_get_contents($api_url);
    return $content;
}

// Example usage:
// echo scrape_page('https://scrapingrobot.com/about-us');

?>
```

--------------------------------

### Scrape Web Pages using Node.js

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example demonstrates how to scrape a web page using Node.js. It utilizes the 'axios' library to make a GET request to the Scraping Robot API, including the necessary URL parameter.

```javascript
const axios = require('axios');

async function scrapePage(url) {
  try {
    const response = await axios.get('https://api.scrapingrobot.com/', {
      params: {
        token: '<YOUR_SR_TOKEN>',
        url: url
      }
    });
    return response.data;
  } catch (error) {
    console.error('Error scraping page:', error);
    return null;
  }
}

// Example usage:
// scrapePage('https://scrapingrobot.com/about-us').then(data => console.log(data));
```

--------------------------------

### Get Credit Balance using Python

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

A Python example for fetching your ScrapingRobot credit balance. This code uses the 'requests' library to send the GET request.

```python
import requests

def get_credit_balance(token):
    url = "https://api.scrapingrobot.com/balance"
    headers = {
        "accept": "application/json"
    }
    params = {
        "token": token
    }
    
    try:
        response = requests.get(url, headers=headers, params=params)
        response.raise_for_status()  # Raise an exception for bad status codes
        return response.json()
    except requests.exceptions.RequestException as e:
        print(f"Error fetching credit balance: {e}")
        return None

# Example usage:
# YOUR_SR_TOKEN = 'YOUR_SR_TOKEN'
# balance = get_credit_balance(YOUR_SR_TOKEN)
# if balance:
#     print(balance)
```

--------------------------------

### GET /

Source: https://scrapingrobot.readme.io/reference/get_

The primary endpoint for initiating a web scraping task using a simple GET request.

```APIDOC
## GET /

### Description
This endpoint allows you to retrieve web page content by sending a GET request with your authentication token and the target URL. It supports optional parameters for JS rendering and targeted element extraction.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/

### Parameters
#### Query Parameters
- **token** (string) - Required - Your Scraping Robot API token.
- **url** (string) - Required - The URL of the page to scrape. Must be the last parameter.
- **render** (boolean) - Optional - Set to true to enable JS rendering.
- **scrapeXpath** (string) - Optional - URL-encoded XPath of the specific element to extract.
- **scrapeSelector** (string) - Optional - URL-encoded CSS selector of the specific element to extract.

### Request Example
GET https://api.scrapingrobot.com/?token=YOUR_TOKEN&url=https://example.com&render=true

### Response
#### Success Response (200)
- **status** (string) - Returns "SUCCESS".
- **date** (string) - Timestamp of the request.
- **httpCode** (number) - HTTP status code.
- **result** (object/string) - The scraped HTML content or parsed element data.

#### Response Example
{
  "status": "SUCCESS",
  "date": "2023-10-27T10:00:00Z",
  "httpCode": 200,
  "result": "<html>...</html>"
}
```

--------------------------------

### Get Credit Balance using Node.js

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

Example of retrieving your ScrapingRobot credit balance with Node.js. This code snippet utilizes the 'axios' library to make the GET request.

```javascript
const axios = require('axios');

const getCreditBalance = async (token) => {
  try {
    const response = await axios.get('https://api.scrapingrobot.com/balance', {
      params: {
        token: token
      },
      headers: {
        'accept': 'application/json'
      }
    });
    return response.data;
  } catch (error) {
    console.error('Error fetching credit balance:', error);
    throw error;
  }
};

// Example usage:
// const YOUR_SR_TOKEN = 'YOUR_SR_TOKEN';
// getCreditBalance(YOUR_SR_TOKEN).then(balance => console.log(balance));
```

--------------------------------

### GET /balance

Source: https://scrapingrobot.readme.io/reference/get_balance

Retrieves the current credits balance for your account. This endpoint is useful for monitoring your usage and ensuring you have sufficient credits for scraping tasks.

```APIDOC
## GET /balance

### Description
Get your current credits balance.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/balance

### Parameters
#### Query Parameters
- **token** (string) - Required - Your API token for authentication.

### Request Example
```
GET /balance?token=<YOUR_TOKEN>
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the request, should be "SUCCESS".
- **date** (string) - The date and time the balance was retrieved.
- **token** (string) - The API token used for the request.
- **result** (string) - The current credit balance.

#### Response Example
```json
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "token": "<YOUR_TOKEN>",
  "result": "998.35"
}
```

#### Error Response (500)
- **status** (string) - Indicates the status of the request, should be "FAIL".
- **date** (string) - The date and time the error occurred.
- **token** (string) - The API token used for the request.
- **error** (string) - A message describing the error.

#### Response Example
```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "token": "<INVALID_TOKEN>",
  "error": "Account does not exist or no data available"
}
```
```

--------------------------------

### Scrape Web Pages with Simple GET API

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example demonstrates the basic usage of the Scraping Robot API to scrape a web page. It requires a user token and the target URL as query parameters. The URL must be the last parameter for correct processing.

```text
GET https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>&url=https://scrapingrobot.com/about-us?user=false
```

--------------------------------

### Scrape Web Pages using Python

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This example shows how to scrape a web page using Python. It utilizes the 'requests' library to send a GET request to the Scraping Robot API, including the necessary URL parameter.

```python
import requests

def scrape_page(url):
    token = '<YOUR_SR_TOKEN>'
    api_url = f'https://api.scrapingrobot.com/?token={token}&url={requests.utils.quote(url)}'
    response = requests.get(api_url)
    return response.text if response.status_code == 200 else None

# Example usage:
# print(scrape_page('https://scrapingrobot.com/about-us'))
```

--------------------------------

### Get Credit Balance using PHP

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

Provides a PHP code snippet to retrieve your ScrapingRobot credit balance. This example uses cURL to make the HTTP GET request.

```php
<?php

$token = '<YOUR_SR_TOKEN>';
$url = "https://api.scrapingrobot.com/balance?token={$token}";

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('accept: application/json'));

$response = curl_exec($ch);

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
} else {
    echo $response;
}

curl_close($ch);
?>
```

--------------------------------

### Invalid Task Example (JSON)

Source: https://scrapingrobot.readme.io/reference/post_

An example of an invalid scraping task configuration. This highlights potential errors in request formatting, such as missing required fields, which would result in a 400 Bad Request response.

```json
{
  "params": {
    "search": "pizza"
  }
}
```

--------------------------------

### Scrape Web Page with Additional Parameters (Node.js)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This example demonstrates how to scrape a web page using Node.js, passing additional parameters within the 'params' object. This allows for more granular control over the scraping process, such as specifying the request type.

```JavaScript
const axios = require('axios');

const scrapePage = async (token, url, module, params = {}) => {
  try {
    const response = await axios.post(`https://api.scrapingrobot.com/?token=${token}`, {
      url: url,
      module: module,
      params: params
    });
    return response.data;
  } catch (error) {
    console.error('Error scraping page:', error);
    return null;
  }
};

// Example usage:
const token = '<YOUR_SR_TOKEN>';
const url = 'https://scrapingrobot.com/about-us';
const module = 'HtmlRequestScraper';
const additionalParams = {
  requestType: 'post'
};

scrapePage(token, url, module, additionalParams).then(data => {
  console.log(data);
});
```

--------------------------------

### GET /balance

Source: https://scrapingrobot.readme.io/reference/get-credit-balance

Retrieves the current credit balance associated with your Scraping Robot API token.

```APIDOC
## GET /balance

### Description
Retrieves the current number of available credits remaining on your account using your unique API token.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/balance

### Parameters
#### Query Parameters
- **token** (string) - Required - The API key from your Scraping Robot dashboard.

### Request Example
GET https://api.scrapingrobot.com/balance?token=<YOUR_SR_TOKEN>

### Response
#### Success Response (200)
- **status** (string) - The status of the request (e.g., SUCCESS).
- **date** (string) - The timestamp of the request.
- **token** (string) - The API token used.
- **credits** (integer) - The current number of available credits.

#### Response Example
{
  "status": "SUCCESS",
  "date": "Fri, 01 May 2020 00:00:00 GMT",
  "token": "abcdef-0123-4567-89ab-cdef01234567",
  "credits": 120
}
```

--------------------------------

### Scrape Web Page with Additional Parameters (Python)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This Python example illustrates how to scrape a web page and include extra configuration options via the 'params' object. This is useful for advanced scraping scenarios, including setting the request type for POST requests.

```Python
import requests
import json

def scrape_page(token, url, module, params={}):
    headers = {
        'Content-Type': 'application/json'
    }
    payload = {
        'url': url,
        'module': module,
        'params': params
    }
    try:
        response = requests.post(f'https://api.scrapingrobot.com/?token={token}', headers=headers, data=json.dumps(payload))
        response.raise_for_status()  # Raise an exception for bad status codes
        return response.json()
    except requests.exceptions.RequestException as e:
        print(f'Error scraping page: {e}')
        return None

# Example usage:
token = '<YOUR_SR_TOKEN>'
url = 'https://scrapingrobot.com/about-us'
module = 'HtmlRequestScraper'
additional_params = {
    'requestType': 'post'
}

result = scrape_page(token, url, module, additional_params)
if result:
    print(result)
```

--------------------------------

### Handle Successful Scraping Responses

Source: https://scrapingrobot.readme.io/reference/get_

Examples of JSON responses returned when a scraping task completes successfully. Includes both raw HTML output and structured selector-based data extraction.

```json
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 200,
  "result": "<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>\n\n<meta charset=\"utf-8\"/> ... </body></html>"
}
```

```json
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 200,
  "result": {
    "selectorElements": [
      {
        "selector": "h1",
        "textNodes": ["Example Domain"],
        "htmlElements": ["<h1>Example Domain</h1>"]
      }
    ],
    "html": "<!DOCTYPE html><html><head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\">...</a></p>\n</div>\n\n\n</body></html>"
  }
}
```

--------------------------------

### Get Credit Balance using cURL

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

Demonstrates how to fetch your ScrapingRobot credit balance using a cURL command. This requires your API token and specifies the JSON content type for the response.

```shell
curl --request GET \
     --url 'https://api.scrapingrobot.com/balance?token=<YOUR_SR_TOKEN>' \
     --header 'accept: application/json'
```

--------------------------------

### GET /balance

Source: https://scrapingrobot.readme.io/reference/credits-endpoints

Retrieves the current credit balance for the authenticated Scraping Robot account.

```APIDOC
## GET /balance

### Description
Retrieves the current credit balance associated with the provided API token.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/balance

### Parameters
#### Query Parameters
- **token** (string) - Required - The API key from your Scraping Robot dashboard.

### Request Example
```bash
curl --request GET \
     --url 'https://api.scrapingrobot.com/balance?token=<YOUR_SR_TOKEN>' \
     --header 'accept: application/json'
```

### Response
#### Success Response (200)
- **balance** (number) - The remaining credit balance for the account.

#### Response Example
{
  "balance": 1000
}
```

--------------------------------

### POST API Scraping Task Example (JSON)

Source: https://scrapingrobot.readme.io/reference/post_

Illustrates how to perform a POST request for scraping using the HtmlRequestScraper module. This is useful for interacting with APIs that require POST data and specific content types. JS-rendering must be disabled for this to work.

```json
{
  "url": "https://www.example.com",
  "module": "HtmlRequestScraper",
  "params": {
    "contentType": "application/x-www-form-urlencoded"
  },
  "postPayload": "param1=value1&param2=value2"
}
```

--------------------------------

### Get Credit Usage Statistics

Source: https://scrapingrobot.readme.io/reference/get-credit-usage-statistics

This endpoint retrieves your credit usage statistics over a specified time period. You can filter the results by daily, weekly, or monthly usage.

```APIDOC
## GET /stats

### Description
This endpoint will show you your credit usage over the specified time period.

### Method
GET

### Endpoint
`https://api.scrapingrobot.com/stats`

### Parameters
#### Query Parameters
- **token** (string) - Required - The API key from your Scraping Robot dashboard.
- **type** (string) - Required - This sets the time period for credit usage statistics that will be displayed. Accepted values are "daily", "weekly", and "monthly". Defaults to "daily".

### Request Example
```bash
curl -X GET "https://api.scrapingrobot.com/stats?token=<YOUR_SR_TOKEN>&type=daily"
```

### Response
#### Success Response (200)
- **status** (string) - Indicates the status of the request (e.g., "SUCCESS").
- **date** (string) - The date associated with the statistics.
- **token** (string) - The API token used for the request.
- **result** (object) - An object containing the credit usage statistics, where keys are dates and values are the credit usage for that date.

#### Response Example
```json
{
  "status": "SUCCESS",
  "date": "Fri, 01 May 2020 00:00:00 GMT",
  "token": "abcdef-0123-4567-89ab-cdef01234567",
  "result": {
    "Sat May 09 2020": 0,
    "Fri May 08 2020": 3,
    "Thu May 07 2020": 0,
    "Wed May 06 2020": 144,
    "Tue May 05 2020": 12,
    "Mon May 04 2020": 43,
    "Sun May 03 2020": 1
  }
}
```

#### Error Response (400)
- An empty JSON object `{}` is returned for errors.
```

--------------------------------

### GET /balance

Source: https://scrapingrobot.readme.io/reference/get-current-credits-balance

Retrieves the current remaining credit balance associated with the provided API token.

```APIDOC
## GET /balance

### Description
Fetches the current credit balance for the authenticated user account.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/balance

### Parameters
#### Path Parameters
- None

#### Query Parameters
- None

#### Request Body
- None

### Request Example
```bash
curl --request GET \
     --url https://api.scrapingrobot.com/balance \
     --header 'accept: application/json'
```

### Response
#### Success Response (200)
- **balance** (number) - The current number of credits remaining in the account.

#### Response Example
{
  "balance": 1000
}

#### Error Handling
- **500** - The system was unable to process the request.
```

--------------------------------

### Simple HTML Scraping Task Example (JSON)

Source: https://scrapingrobot.readme.io/reference/post_

Demonstrates a basic scraping task using the HtmlChromeScraper module to extract the content of an h1 tag from a given URL. This is a common use case for extracting specific elements from a webpage.

```json
{
  "url": "https://www.example.com",
  "module": "HtmlChromeScraper",
  "params": {
    "scrapeSelector": "h1"
  }
}
```

--------------------------------

### Retrieve current credits balance via API

Source: https://scrapingrobot.readme.io/reference/get-current-credits-balance

Fetches the current credit balance for the authenticated user. The request requires no parameters and returns a JSON object containing the balance information.

```Shell
curl --request GET \
     --url https://api.scrapingrobot.com/balance \
     --header 'accept: application/json'
```

--------------------------------

### Scrape Web Page with Additional Parameters (PHP)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This PHP example demonstrates how to scrape web pages using the Scraping Robot API with extended functionalities provided by the 'params' object. It allows for custom settings, like defining the request type for POST requests.

```PHP
<?php

function scrape_page($token, $url, $module, $params = []) {
    $ch = curl_init();

    $post_data = json_encode([
        'url' => $url,
        'module' => $module,
        'params' => $params
    ]);

    curl_setopt($ch, CURLOPT_URL, "https://api.scrapingrobot.com/?token={$token}");
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));

    $response = curl_exec($ch);
    $http_status = curl_getinfo($ch, CURLINFO_HTTP_CODE);

    if (curl_errno($ch)) {
        echo 'Curl error: ' . curl_error($ch);
        return null;
    }

    curl_close($ch);

    if ($http_status >= 200 && $http_status < 300) {
        return json_decode($response, true);
    } else {
        echo "HTTP Error: {$http_status}";
        return null;
    }
}

// Example usage:
$token = '<YOUR_SR_TOKEN>';
$url = 'https://scrapingrobot.com/about-us';
$module = 'HtmlRequestScraper';
$additional_params = [
    'requestType' => 'post'
];

$result = scrape_page($token, $url, $module, $additional_params);

if ($result) {
    print_r($result);
}

?>

```

--------------------------------

### Scrape Web Pages

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-get-request

This endpoint allows you to scrape web page content using a simple GET HTTP request. You need to provide your token and the URL of the desired page as query parameters. The URL of the desired page must be sent as the last parameter.

```APIDOC
## GET / 

### Description
Scrape web-pages using simple GET API. This API will allow you to use the Scraping Robot service to retrieve web page content using a simple GET HTTP request. To do this, you just need to send a request to the Scraping Robot server and transfer your token, as well as the URL of the desired page as query parameters.

> **Note!** Due to the specifics of the HTTP protocol, for the correct perception of all address parameters, the URL of the desired page must be sent as the last parameter.

### Method
GET

### Endpoint
https://api.scrapingrobot.com/

### Parameters
#### Query Parameters
- **token** (string) - Required - Your Scraping Robot API token.
- **url** (string) - Required - URL of the target web-page to be scraped. This parameter should always be sent as the last one.
- **render** (boolean) - Optional - Enables JS rendering. Defaults to false.
- **scrapeXpath** (string) - Optional - URL-encoded xpath of the element, content of which you want to scrape.
- **scrapeSelector** (string) - Optional - URL-encoded CSS-selector of the element, content of which you want to scrape.

### Request Example
```
GET https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>&url=https://scrapingrobot.com/about-us?user=false
```

### Response
#### Success Response (200)
- **content** (string) - The scraped content of the web page.

#### Response Example
```json
{
  "content": "<html>...</html>"
}
```

#### Error Responses
- **400** - The user made a mistake/typo while forming the request body.
- **401** - The system denied access to the service. The reason for this may be either an incorrect access token or an insufficient amount of funds in the user's account.
- **429** - The system is overloaded. Please perform a retry. You will not be charged for failed attempts.
- **500** - The system was unable to process the task within the allotted time. Our system was unable to scrape the site due to anti-scraping technology. This can occur when we have not optimized a particular use case. Please contact support and we will resolve it on a site-by-site basis. You will not be charged for failed attempts.
```

--------------------------------

### Invalid Task Format Error Response (JSON)

Source: https://scrapingrobot.readme.io/reference/post_

An example of an error response when a task is invalid due to missing required fields. This response indicates a 'FAIL' status, the HTTP code, and a specific error message detailing the issue, such as a missing 'url' property.

```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 0,
  "error": "Invalid task format: should have required property url"
}
```

--------------------------------

### Example Invalid JSON Response

Source: https://scrapingrobot.readme.io/reference/request-body-is-not-valid-json

This code snippet demonstrates the expected JSON error response when a request body is not valid JSON. It includes status, date, and a specific error message indicating the issue. This format helps developers identify and debug JSON formatting problems.

```text
{
 "status": "FAIL",
 "date":"Thu, 01 Mar 2020 10:00:00 GMT",
 "error": "Request-body is not a valid JSON"
}
```

--------------------------------

### Initiate web scraping via POST request

Source: https://scrapingrobot.readme.io/reference/post_

Demonstrates the simplest way to trigger a scraping task by sending a POST request to the Scraping Robot API. The request requires a token as a query parameter and a JSON body containing the target URL and the scraper module.

```http
POST https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>

{
  "url": "https://scrapingrobot.com/about-us",
  "module": "HtmlRequestScraper"
}
```

--------------------------------

### POST /

Source: https://scrapingrobot.readme.io/reference

The primary endpoint for initiating a scraping task. It requires an API token passed as a query parameter and a JSON body specifying the target URL and scraper module.

```APIDOC
## POST /

### Description
Initiates a web scraping task using the specified module and target URL.

### Method
POST

### Endpoint
https://api.scrapingrobot.com

### Parameters
#### Query Parameters
- **token** (string) - Required - Your unique Scraping Robot API key.

#### Request Body
- **url** (string) - Required - The target website URL to scrape.
- **module** (string) - Required - The scraping module to use (e.g., "HtmlRequestScraper").

### Request Example
{
	"url": "https://www.scrapingrobot.com/",
	"module": "HtmlRequestScraper"
}

### Response
#### Success Response (200)
- **data** (object) - The scraped content returned by the specified module.

#### Response Example
{
	"status": "success",
	"data": "<html>...</html>"
}
```

--------------------------------

### Scraping Configuration Parameters

Source: https://scrapingrobot.readme.io/reference/post_

This section details the configuration parameters available for web scraping requests, including response types and rendering options.

```APIDOC
## GET /websites/scrapingrobot_readme_io_reference

### Description
This endpoint allows for configuring web scraping tasks with various options.

### Method
GET

### Endpoint
/websites/scrapingrobot_readme_io_reference

### Parameters
#### Query Parameters
- **country** (string) - Optional - Specifies the country code for the scraping request. Accepted values are ISO 3166-1 alpha-2 codes (e.g., "AF", "AL", "DZ", etc.).
- **mobile** (boolean) - Optional - If true, the system will attempt to scrape the mobile version of the page.
- **screenshot** (boolean) - Optional - If true, the system will take a screenshot of the entire web page. Requires JS rendering.
- **responseType** (string) - Optional - Specifies the format of the response data. Accepted values are "json" or "zip". "zip" response type requires JS rendering. Default: "json".

### Request Example
```json
{
  "example": "/websites/scrapingrobot_readme_io_reference?country=US&mobile=true&screenshot=true&responseType=zip"
}
```

### Response
#### Success Response (200)
- **data** (object) - Contains the scraped data or a link to the zipped data.
- **screenshot** (string) - A URL to the generated screenshot if requested.

#### Response Example
```json
{
  "example": {
    "data": {
      "title": "Example Title",
      "content": "Scraped content..."
    },
    "screenshot": "https://example.com/screenshots/screenshot.png"
  }
}
```
```

--------------------------------

### Handling Insufficient Credits

Source: https://scrapingrobot.readme.io/reference/not-enough-credits

This documentation describes the error response received when a scraping request fails due to a lack of available credits in the user account.

```APIDOC
## Error: Not Enough Credits

### Description
This error occurs when the requested scraping task cannot be completed because the account balance is insufficient. Users must add credits via the dashboard or contact support.

### Response
#### Error Response (402/400)
- **status** (string) - The status of the request, returns "FAIL".
- **date** (string) - The timestamp of the error occurrence.
- **error** (string) - A descriptive message: "You do not have enough credits".

### Response Example
{
  "status": "FAIL",
  "date": "Thu, 01 Mar 2020 10:00:00 GMT",
  "error": "You do not have enough credits"
}
```

--------------------------------

### Web Scraping Parameters

Source: https://scrapingrobot.readme.io/reference/post_

This section details the various parameters that can be used to configure web scraping requests, including element selection, waiting conditions, form submission, and resource loading control.

```APIDOC
## Web Scraping Configuration Parameters

### Description
This endpoint allows for detailed configuration of web scraping tasks using a variety of parameters to control how pages are loaded, elements are interacted with, and data is extracted.

### Method
POST (Implied, as these are parameters for a request)

### Endpoint
`/api/v1/scrape` (Example endpoint, actual may vary)

### Parameters
#### Request Body Parameters
- **customHeaders** (boolean) - Optional - If specified, the system will add the headers received in the client request to the target request.
- **scrapeXpath** (string) - Optional - XPath of the element, content of which you want to scrape.
- **scrapeSelector** (string) - Optional - CSS-selector of the element, content of which you want to scrape.
- **waitUntil** (string) - Optional - Browser event that determines that the page has loaded successfully. Enum: `load`, `domcontentloaded`, `networkiddle0`, `networkiddle2`.
- **waitForSeconds** (number) - Optional - The system waits for the specified number of seconds after loading the page before starting to collect data. Minimum: 1, Maximum: 60.
- **waitForSelector** (string) - Optional - CSS-selector of a HTML element. The system waits for this element to appear on the page after the page has loaded, before starting to collect data.
- **scrollDown** (number) - Optional - Scroll to the very bottom of the page. The accepted value is the number of scrolls that the module should do. Minimum: 1, Maximum: 10.
- **clickOnElement** (string) - Optional - CSS-selector for the element to be clicked on.
- **formSelector** (string) - Optional - CSS-selector for the text field to be filled in on the page. Dependencies: `query`, `submitSelector`.
- **submitSelector** (string) - Optional - CSS-selector of the form submit button. Dependencies: `query`, `formSelector`.
- **query** (string) - Optional - The text to be entered into the form on the page. Dependencies: `formSelector`, `submitSelector`.
- **noScripts** (boolean) - Optional - Setting this parameter to true disables loading scripts. Warning! In this case, JS rendering will not work! Default value is false.
- **noImages** (boolean) - Optional - Setting this parameter to true disables loading images. Default value is true.
- **noFonts** (boolean) - Optional - Setting this parameter to true disables loading fonts. Default value is true.
- **noCss** (boolean) - Optional - Setting this parameter to true disables loading CSS. Default value is true.
- **requestType** (string) - Optional - To send POST request using Scraping Robot, please set this parameter to "post" and additionally set the "postBody" parameter. Enum: `post`, `get`.
- **contentType** (string) - Optional - Content type that will be used when sending POST-requests using SR (see `postBody`). Default value: `text/plain`. When using `application/x-www-form-urlencoded`, please pass POST-payload without URL-encoding (the application handles this itself). Enum: `application/json`, `application/x-www-form-urlencoded`, `text/plain`.

### Request Example
```json
{
  "url": "https://example.com",
  "scrapeSelector": ".product-title",
  "waitUntil": "domcontentloaded",
  "waitForSeconds": 5,
  "clickOnElement": ".accept-cookies-button",
  "formSelector": "#search-input",
  "submitSelector": "#search-button",
  "query": "example search term",
  "noImages": false,
  "requestType": "post",
  "contentType": "application/json",
  "postBody": {
    "key": "value"
  }
}
```

### Response
#### Success Response (200)
- **data** (object) - The scraped data from the website.
- **url** (string) - The URL that was scraped.

#### Response Example
```json
{
  "data": [
    {
      "product-title": "Example Product"
    }
  ],
  "url": "https://example.com"
}
```
```

--------------------------------

### OpenAPI Specification Definition

Source: https://scrapingrobot.readme.io/reference/get_

The complete OpenAPI 3.0.0 schema defining the API structure, security schemes, and response models for the Scraping Robot service.

```JSON
{
  "openapi": "3.0.0",
  "info": {
    "title": "HTML API",
    "version": "1.0.0"
  },
  "paths": {
    "/": {
      "get": {
        "summary": "Main API endpoint"
      }
    }
  }
}
```

--------------------------------

### API Error Handling Reference

Source: https://scrapingrobot.readme.io/reference/get_

Details regarding the standard error responses, including 500 internal server errors and service overload scenarios.

```APIDOC
## Error Handling Reference

### Description
This section describes common error responses returned by the API when a scraping task fails. Users are not charged for failed attempts.

### Error Scenarios
- **Service Overloaded (0)**: The service is currently at capacity. Please implement a retry mechanism.
- **Anti-scraping Block (500)**: The system encountered anti-scraping technology. Contact support for site-specific optimization.
- **Internal Error (500)**: An unexpected server-side error occurred. Please retry the request.

### Response Structure
- **status** (string) - The result status (e.g., "FAIL")
- **date** (string) - Timestamp of the error
- **httpCode** (integer) - The HTTP status code
- **error** (string) - A descriptive message explaining the failure

### Response Example (Anti-scraping)
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 500,
  "error": "Our system was unable to scrape the site due to anti-scraping technology."
}
```

--------------------------------

### Scrape Web Page with Additional Parameters (Ruby)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This Ruby code snippet shows how to perform web scraping with additional parameters. It sends a POST request to the Scraping Robot API, including a 'params' object to customize the scraping task, such as setting the request type.

```Ruby
require 'net/http'
require 'uri'
require 'json'

def scrape_page(token, url, module_type, params = {})
  uri = URI.parse("https://api.scrapingrobot.com/?token=#{token}")
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true

  request = Net::HTTP::Post.new(uri.request_uri)
  request['Content-Type'] = 'application/json'
  request.body = {
    'url' => url,
    'module' => module_type,
    'params' => params
  }.to_json

  begin
    response = http.request(request)
    return JSON.parse(response.body)
  rescue StandardError => e
    puts "Error scraping page: #{e.message}"
    return nil
  end
end

# Example usage:
token = '<YOUR_SR_TOKEN>'
url = 'https://scrapingrobot.com/about-us'
module_type = 'HtmlRequestScraper'
additional_params = {
  'requestType' => 'post'
}

result = scrape_page(token, url, module_type, additional_params)
puts result
```

--------------------------------

### Scrape Web Page (HTML Request)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This snippet demonstrates the simplest way to scrape a web page using the HtmlRequestScraper module. It requires a URL and the module type, sent as a JSON object in the POST request body. This method is suitable for pages that do not require JavaScript rendering.

```Shell
POST https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>

# Request body:
{
  "url":"https://scrapingrobot.com/about-us",
  "module":"HtmlRequestScraper"
}
```

```Shell
curl --request POST \
     --url https://api.scrapingrobot.com/ \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '{
  "module": "HtmlRequestScraper"
}'
```

--------------------------------

### POST /scrape

Source: https://scrapingrobot.readme.io/reference/post_

Submits a new web scraping task to the ScrapingRobot service. Requires a target URL and a specific scraper module.

```APIDOC
## POST /scrape

### Description
Submits a scraping task to the service. Users must specify the target URL and the module to be used for scraping. Supports JS-rendering and custom POST payloads.

### Method
POST

### Endpoint
/scrape

### Parameters
#### Request Body
- **url** (string) - Required - The target URL to scrape.
- **module** (string) - Required - The scraper module to use (e.g., "HtmlChromeScraper").
- **params** (object) - Optional - Configuration parameters for the scraper.
- **postPayload** (string) - Optional - Data to send in a POST request. Works only when JS-rendering is disabled.

### Request Example
{
  "url": "https://www.example.com",
  "module": "HtmlChromeScraper",
  "params": {
    "scrapeSelector": "h1"
  }
}

### Response
#### Success Response (200)
- **status** (string) - The status of the task.
- **date** (string) - Timestamp of the response.
- **httpCode** (integer) - The HTTP status code.
- **result** (object) - The scraped data, including selector elements and raw HTML.

#### Response Example
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 200,
  "result": {
    "selectorElements": [
      {
        "selector": "h1",
        "textNodes": ["Example Domain"]
      }
    ]
  }
}
```

--------------------------------

### Scrape Web Page (JS Rendering)

Source: https://scrapingrobot.readme.io/reference/create-a-task-using-post-request

This snippet shows how to scrape a web page that requires JavaScript rendering using the HtmlChromeScraper module. Similar to the basic scraping, it involves sending a POST request with the URL and specifying 'HtmlChromeScraper' as the module.

```Shell
POST https://api.scrapingrobot.com/?token=<YOUR_SR_TOKEN>

# Request body:
{
  "url":"https://scrapingrobot.com/about-us",
  "module":"HtmlChromeScraper"
}
```

--------------------------------

### OpenAPI Specification Definition

Source: https://scrapingrobot.readme.io/reference/post_

The complete OpenAPI 3.0.0 schema defining the Scraping Robot API structure, including security schemes, error models, and endpoint configurations.

```json
{
  "openapi": "3.0.0",
  "info": {
    "title": "HTML API",
    "version": "1.0.0"
  },
  "paths": {
    "/": {
      "post": {
        "summary": "Create a task using POST-request"
      }
    }
  }
}
```

--------------------------------

### Web Scraping Task

Source: https://scrapingrobot.readme.io/reference/get_

This endpoint allows you to scrape a target web page. The URL of the target page should be provided as a query parameter. The API supports various scraping options, including extracting specific elements using CSS selectors.

```APIDOC
## POST /api/scrape

### Description
Initiates a web scraping task for a specified URL. The URL of the target web page is a required parameter and should be sent as the last one.

### Method
POST

### Endpoint
/api/scrape

### Parameters
#### Query Parameters
- **url** (string) - Required - URL of the target web-page to be scraped.

### Request Example
```json
{
  "url": "https://www.example.com/path?param=value"
}
```

### Response
#### Success Response (200)
- **status** (string) - The status of the scraping task (e.g., "SUCCESS").
- **date** (string) - The timestamp when the task was processed.
- **httpCode** (integer) - The HTTP status code of the scraped page.
- **result** (object|string) - The scraped content. This can be the full HTML or a structured object containing extracted elements if selectors were used.

#### Response Example (Simplest example)
```json
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 200,
  "result": "<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>\n\n<meta charset=\"utf-8"/> ... </body></html>"
}
```

#### Response Example (Selector scraping)
```json
{
  "status": "SUCCESS",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 200,
  "result": {
    "selectorElements": [
      {
        "selector": "h1",
        "textNodes": [
          "Example Domain"
        ],
        "htmlElements": [
          "<h1>Example Domain</h1>"
        ]
      }
    ],
    "html": "<!DOCTYPE html><html><head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8">...</a></p>\n</div>\n\n\n</body></html>"
  }
}
```

#### Error Response (400)
- **status** (string) - The status of the scraping task (e.g., "FAIL").
- **date** (string) - The timestamp when the error occurred.
- **httpCode** (integer) - The HTTP status code indicating the error.
- **error** (string) - A message describing the error.

#### Response Example (Invalid task)
```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 0,
  "error": "Invalid task format: should have required property url"
}
```

#### Error Response (401)
- **status** (string) - The status of the scraping task (e.g., "FAIL").
- **date** (string) - The timestamp when the error occurred.
- **httpCode** (integer) - The HTTP status code indicating the error.
- **error** (string) - A message describing the error.

#### Response Example (No access token)
```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 0,
  "error": "Token query parameter not found"
}
```

#### Response Example (Invalid token or balance too low)
```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 0,
  "error": "Incorrect token or you do not have enough credits"
}
```

#### Error Response (429)
- **status** (string) - The status of the scraping task (e.g., "FAIL").
- **date** (string) - The timestamp when the error occurred.
- **httpCode** (integer) - The HTTP status code indicating the error.
- **error** (string) - A message describing the error.

#### Response Example (System is overloaded)
```json
{
  "status": "FAIL",
  "date": "Mon, 1 Jan 2021 12:00:00 GMT",
  "httpCode": 0,
  "error": "System is overloaded. Please try again later."
}
```
```

--------------------------------

### API Error: System Overload

Source: https://scrapingrobot.readme.io/reference/post_

This response indicates that the system is currently overloaded and unable to process the request. It suggests retrying the request at a later time. The error message will reflect the system's overloaded state.

```json
{
  "description": "The system is overloaded",
  "content": {
    "application/json": {
      "schema": {
        "$ref": "#/components/schemas/GeneralError"
      },
      "examples": {
        "System is overloaded": {
          "description": "Invalid task",
          "value": {
            "status": "FAIL",
            "date": "Mon, 1 Jan 2021 12:00:00 GMT",
            "httpCode": 0,
            "error": "The service is overloaded. Try again later"
          }
        }
      }
    }
  }
}
```

--------------------------------

### Not Enough Credits Error Response (Text)

Source: https://scrapingrobot.readme.io/reference/not-enough-credits

This code snippet shows the JSON response received when a scraping task fails due to insufficient credits. It includes the status, timestamp, and a specific error message indicating the lack of credits. No external dependencies are required to interpret this response.

```text
{
 "status": "FAIL",
 "date": "Thu, 01 Mar 2020 10:00:00 GMT",
 "error": "You do not have enough credits"
}
```
Tokens:
Raw