ScrapingAnt Documentation

https://docs.scrapingant.com
Admin
ScrapingAnt provides a reliable Web Scraping API powered by a cluster of Chrome browsers, enabling...
Tokens:28,683
Snippets:219
Trust Score:10
Update:8 months ago
Show doc for...
Tokens:
Raw
### Usage Examples

Source: https://docs.scrapingant.com/mcp-server

Examples demonstrating how to use the ScrapingAnt MCP server with AI assistants for various web scraping tasks.

```APIDOC
## Usage Examples

Once configured, you can instruct your AI assistant to scrape web pages naturally using the available tools.

### Basic Scraping

*   "Fetch the content from https://example.com and summarize it."
*   "Get the HTML from https://news.ycombinator.com."

### Using Markdown Output

*   "Scrape https://docs.python.org/3/tutorial/index.html as markdown and explain the main topics."

### With Residential Proxies

*   "Fetch https://example.com using residential proxies."

### Geo-targeted Requests

*   "Get the content from https://example.com using a proxy from Germany."
```

--------------------------------

### ScrapingAnt API Endpoint and Request Example

Source: https://docs.scrapingant.com/api-basics

This snippet shows the base URL for the ScrapingAnt general API and an example of how to make a GET request using cURL. It requires a URL to scrape and an API key as query parameters. The output is the HTML content of the requested page.

```bash
curl --request GET \
--url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR-API-KEY>'
```

--------------------------------

### Python Example: Using the Markdown Transformation Endpoint

Source: https://docs.scrapingant.com/llm-markdown

A Python script demonstrating how to use the requests library to call the Markdown Transformation Endpoint. It sends a GET request with the URL and API key, then prints the Markdown content or any errors.

```python
import requests

api_key = "YOUR_SCRAPINGANT_API_KEY"
url = "https://example.com"

response = requests.get("https://api.scrapingant.com/v2/markdown", params={"url": url, "x-api-key": api_key})

if response.status_code == 200:
    markdown_content = response.json()["markdown"]
    print(markdown_content)
else:
    print("Error:", response.text)
```

--------------------------------

### Install ScrapingAnt Python Client

Source: https://docs.scrapingant.com/python-client

Installs the official ScrapingAnt Python client library using pip. This is the first step to integrating the ScrapingAnt API into your Python applications.

```shell
pip install scrapingant-client
```

--------------------------------

### Install ScrapingAnt JS Client via Yarn

Source: https://docs.scrapingant.com/js-client

Installs the official ScrapingAnt JavaScript client library using Yarn. This is an alternative package manager for installing the client.

```bash
yarn add @scrapingant/scrapingant-client
```

--------------------------------

### MCP Server Configuration

Source: https://docs.scrapingant.com/mcp-server

Configuration examples for integrating ScrapingAnt's MCP server with different AI assistants.

```APIDOC
## MCP Server Configuration

This section provides configuration details for integrating ScrapingAnt's MCP server with various AI assistants.

### Claude Desktop

Add the following to your `claude_desktop_config.json` file:

*   **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
*   **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

```json
{
  "mcpServers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp",
      "transport": "streamableHttp",
      "headers": {
        "x-api-key": "<YOUR-API-KEY>"
      }
    }
  }
}
```

### VS Code / GitHub Copilot

Add the following to your VS Code settings or create `.vscode/mcp.json` in your workspace:

```json
{
  "servers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp/",
      "requestInit": {
        "headers": {
          "x-api-key": "<YOUR-API-KEY>"
        }
      }
    }
  }
}
```

### Cursor

1.  Open **Settings** → **MCP** → **Add new MCP Server**.
2.  Enter the following configuration:
    *   **Name**: `scrapingant`
    *   **URL**: `https://api.scrapingant.com/mcp`
    *   **Transport**: `streamableHttp`
    *   **Headers**: `x-api-key: <YOUR-API-KEY>`

### Claude Code (CLI)

Run the following command in your terminal:

```bash
claude mcp add scrapingant --transport http https://api.scrapingant.com/mcp -H "x-api-key: <YOUR-API-KEY>"
```

### Cline

Add the following to your `cline_mcp_settings.json`:

```json
{
  "mcpServers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp",
      "transport": "streamableHttp",
      "headers": {
        "x-api-key": "<YOUR-API-KEY>"
      }
    }
  }
}
```

### Windsurf

Follow the Windsurf MCP documentation using the standard configuration:

```json
{
  "mcpServers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp",
      "transport": "streamableHttp",
      "headers": {
        "x-api-key": "<YOUR-API-KEY>"
      }
    }
  }
}
```
```

--------------------------------

### Install ScrapingAnt JS Client via NPM

Source: https://docs.scrapingant.com/js-client

Installs the official ScrapingAnt JavaScript client library using the Node Package Manager (NPM). This is the first step to integrating the API into your project.

```bash
npm i @scrapingant/scrapingant-client
```

--------------------------------

### ScrapingAnt Proxy Mode Username with Parameters

Source: https://docs.scrapingant.com/proxy-mode

Example of constructing the username for ScrapingAnt Proxy Mode when passing additional parameters. This example disables browser rendering and enables residential proxies.

```bash
scrapingant&browser=false&proxy_type=residential
```

--------------------------------

### GET /v2/general

Source: https://docs.scrapingant.com/migration-guides/scrapingbee-migration

This endpoint is used to fetch web page content. It supports various parameters for controlling browser rendering, proxies, and other scraping options. The provided code examples demonstrate how to use this endpoint and implement retry logic for handling rate limits and detections.

```APIDOC
## GET /v2/general

### Description
Fetches web page content using the ScrapingAnt API. This endpoint allows for flexible scraping by supporting various parameters to control rendering, proxies, and more. It's designed to help bypass anti-bot systems and extract data efficiently.

### Method
GET

### Endpoint
`https://api.scrapingant.com/v2/general`

### Parameters
#### Query Parameters
- **x-api-key** (string) - Required - Your ScrapingAnt API key.
- **url** (string) - Required - The URL of the web page to scrape.
- **browser** (boolean) - Optional - Whether to use browser rendering (defaults to true).
- **proxy_type** (string) - Optional - The type of proxy to use (e.g., "residential", "datacenter").

### Request Example
```python
import requests

api_key = "YOUR_API_KEY"
url = "http://example.com"

response = requests.get(
    f"https://api.scrapingant.com/v2/general?x-api-key={api_key}&url={url}"
)

print(response.text)
```

### Response
#### Success Response (200)
- **content** (string) - The HTML content of the requested web page.

#### Response Example
```json
{
  "content": "<html>...</html>"
}
```

### Error Handling
- **409 Conflict**: Rate limit exceeded (especially for free plans). Implement a retry mechanism with a delay.
- **403 Forbidden**: Detection by anti-bot systems. Consider adjusting parameters like `browser` or `proxy_type` and retrying.
- Other status codes indicate different errors. Refer to ScrapingAnt documentation for specific error codes and their meanings.
```

--------------------------------

### GET /v2/general

Source: https://docs.scrapingant.com/migration-guides/scraperapi-migration

This endpoint is used to fetch data from a given URL. It supports various parameters for browser rendering and proxy settings. The provided code examples demonstrate how to implement retry mechanisms for handling rate limits and detections.

```APIDOC
## GET /v2/general

### Description
Fetches data from a specified URL using the ScrapingAnt API. Includes parameters for browser rendering and proxy types, with examples of retry logic for common error codes.

### Method
GET

### Endpoint
https://api.scrapingant.com/v2/general

### Parameters
#### Query Parameters
- **x-api-key** (string) - Required - Your ScrapingAnt API key.
- **url** (string) - Required - The URL to scrape.
- **browser** (boolean) - Optional - Whether to enable browser rendering (defaults to true).
- **proxy_type** (string) - Optional - The type of proxy to use (e.g., 'residential', 'datacenter').

### Request Example
```python
import requests
import time

def get_response(url, api_key, browser=True, proxy_type='residential'):
    params = {
        'x-api-key': api_key,
        'url': url,
        'browser': browser,
        'proxy_type': proxy_type
    }
    response = requests.get("https://api.scrapingant.com/v2/general", params=params)
    
    if response.status_code == 200:
        return response
    elif response.status_code == 409: # Rate limit for free plan
        time.sleep(1)
        return get_response(url, api_key, browser, proxy_type)
    elif response.status_code == 403: # Handle detection
        if browser == True:
            # Try without browser rendering
            return get_response(url, api_key, False, proxy_type)
        elif proxy_type == "residential":
            # Try with datacenter proxy
            return get_response(url, api_key, browser, "datacenter")
        else:
            # Try with residential proxy
            return get_response(url, api_key, browser, "residential")
    else:
        # Handle other errors
        return None
```

### Response
#### Success Response (200)
- **content** (string) - The HTML content of the scraped page.

#### Response Example
```json
{
  "content": "<html>...</html>"
}
```

#### Error Responses
- **403** - Forbidden (Detection)
- **409** - Conflict (Rate Limit for free plan)
- Other status codes indicate different errors.
```

--------------------------------

### GET /v2/usage

Source: https://docs.scrapingant.com/api-credits-usage

Retrieves the current status of API credits usage, including subscription plan details, start and end dates, total credits, and remaining credits.

```APIDOC
## GET /v2/usage

### Description
Retrieves the current status of API credits usage. This endpoint allows you to check your active subscription plan, the start and end dates of your subscription period, the total credits allocated for the plan, and the number of credits remaining.

### Method
GET

### Endpoint
`https://api.scrapingant.com/v2/usage`

### Parameters
#### Query Parameters
- **x-api-key** (string) - Required - Your Scrapingant API key.

### Request Example
```
GET https://api.scrapingant.com/v2/usage?x-api-key=<YOUR_SCRAPINGANT_API_KEY>
```

### Response
#### Success Response (200)
- **plan_name** (string) - The name of the active subscription plan.
- **start_date** (string) - The start date of the active subscription period (ISO 8601 format).
- **end_date** (string) - The end date of the active subscription period (ISO 8601 format).
- **plan_total_credits** (integer) - The total number of credits allocated for the active subscription plan.
- **remained_credits** (integer) - The number of credits remaining for the active subscription.

#### Response Example
```json
{
  "plan_name": "Enthusiast",
  "start_date": "2022-01-11T00:00:00",
  "end_date": "2022-02-11T00:00:00",
  "plan_total_credits": 100000,
  "remained_credits": 96740
}
```
```

--------------------------------

### Example HTML Response from ScrapingAnt API

Source: https://docs.scrapingant.com/api-basics

This is an example of the HTML content that the ScrapingAnt API returns when successfully scraping a web page. It represents the structure and content of the target website.

```html
<!DOCTYPE html><html><head>
    <title>Example Domain</title>

    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style type="text/css">
        body {
            background-color: #f0f0f2;
            margin: 0;
            padding: 0;
            font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;

        }
        div {
            width: 600px;
            margin: 5em auto;
            padding: 2em;
            background-color: #fdfdff;
            border-radius: 0.5em;
            box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
        }
        a:link, a:visited {
            color: #38488f;
            text-decoration: none;
        }
        @media (max-width: 700px) {
            div {
                margin: 0 auto;
                width: auto;
            }
        }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
        domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>


</body></html>

```

--------------------------------

### AI Extractor Request Example (cURL)

Source: https://docs.scrapingant.com/ai-data-extraction/ai-extractor

Demonstrates a simple cURL request to the AI extractor endpoint to fetch the title and content of a web page. It requires the URL, extract properties, and an API key.

```bash
curl --request GET \
--url 'https://api.scrapingant.com/v2/extract?url=https%3A%2F%2Fexample.com&extract_properties=title%2C%20content&x-api-key=<YOUR-API-KEY>'
```

--------------------------------

### Basic Request Comparison: ScraperAPI vs. ScrapingAnt (With Rendering)

Source: https://docs.scrapingant.com/migration-guides/scraperapi-migration

Illustrates the cURL command for enabling browser rendering in ScrapingAnt API, contrasting it with ScraperAPI's default behavior. ScrapingAnt enables rendering by default or with the 'browser=true' parameter, while ScraperAPI requires specific configuration.

```curl
curl "https://api.scrapingant.com/v2/general?x-api-key=SCRAPINGANT_API_KEY&url=https%3A%2F%2Fexample.com"
```

```curl
curl "https://api.scrapingant.com/v2/general?x-api-key=SCRAPINGANT_API_KEY&url=https%3A%2F%2Fexample.com&browser=true"
```

--------------------------------

### ScrapingAnt Proxy Mode with Header Forwarding

Source: https://docs.scrapingant.com/proxy-mode

Example of constructing the username for ScrapingAnt Proxy Mode to enable header forwarding. This ensures that all headers sent to the proxy are forwarded to the API endpoint.

```bash
scrapingant&forward_headers=true
```

--------------------------------

### cURL Request with ScrapingAnt Proxy Mode

Source: https://docs.scrapingant.com/proxy-mode

Example cURL request demonstrating how to use ScrapingAnt Proxy Mode with a username and API key. It forwards the request to httpbin.org/ip. Ensure SSL certificate verification is disabled in your client.

```bash
curl -x "http://scrapingant:YOUR-API-KEY@proxy.scrapingant.com:8080" -k "http://httpbin.org/ip"
```

--------------------------------

### AI Extractor JSON Output Example

Source: https://docs.scrapingant.com/ai-data-extraction/ai-extractor

Illustrates the expected JSON output from the AI extractor when requesting the 'title' and 'content' of a web page. The structure mirrors the 'extract_properties' provided in the request.

```json
{
  "title": "Example Domain",
  "content": "This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission."
}
```

--------------------------------

### URL Encoding for Query Parameters

Source: https://docs.scrapingant.com/request-response-format

Provides examples of how to URL-encode query parameters in various programming languages for constructing API requests. Proper encoding ensures parameters are correctly interpreted.

```Go
url.QueryEscape
```

```Java
URLEncoder.encode
```

```NodeJS
encodeURIComponent
```

```PHP
urlencode
```

```Python
urllib.parse.quote
```

```Ruby
URI::escape
```

--------------------------------

### General Scraping Endpoint

Source: https://docs.scrapingant.com/api-basics

The primary endpoint for general web scraping. It requires a URL to scrape and your API key for authentication. Supports GET, POST, PUT, and DELETE methods.

```APIDOC
## GET /v2/general

### Description
This endpoint scrapes the content of a given URL. It requires an API key for authentication and the target URL as query parameters.

### Method
GET

### Endpoint
https://api.scrapingant.com/v2/general

### Parameters
#### Query Parameters
- **url** (string) - Required - The URL of the website to scrape.
- **x-api-key** (string) - Required - Your ScrapingAnt API key.
- **timeout** (integer) - Optional - The execution time for the request in seconds (5-60).

### Request Example
```curl
curl --request GET \
  --url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR-API-KEY>'
```

### Response
#### Success Response (200)
- **html_content** (string) - The HTML content of the scraped web page.

#### Response Example
```html
<!DOCTYPE html><html><head>  
    <title>Example Domain</title>  
  
    <meta charset="utf-8">
    <meta http-equiv="Content-type" content="text/html; charset=utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style type="text/css">
        body {
            background-color: #f0f0f2;
            margin: 0;
            padding: 0;
            font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;

        }
        div {
            width: 600px;
            margin: 5em auto;
            padding: 2em;
            background-color: #fdfdff;
            border-radius: 0.5em;
            box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
        }
        a:link, a:visited {
            color: #38488f;
            text-decoration: none;
        }
        @media (max-width: 700px) {
            div {
                margin: 0 auto;
                width: auto;
            }
        }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
        domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>


</body></html>
```
```

--------------------------------

### Markdown Transformation Endpoint Response Format

Source: https://docs.scrapingant.com/llm-markdown

Example JSON response structure from the Markdown Transformation Endpoint. It includes the original URL and the extracted content in Markdown format.

```json
{
    "url": "https://example.com",
    "markdown": "# Heading 1\n\nThis is a paragraph of text.\n\n## Heading 2\n\nAnother paragraph of text."
}
```

--------------------------------

### Performing POST/PUT/DELETE Requests

Source: https://docs.scrapingant.com/post-put-delete

This section details how to send POST, PUT, or DELETE requests to the main Scrapingant API endpoint. It covers the necessary parameters, custom headers for content type, and provides an example using cURL.

```APIDOC
## POST / PUT / DELETE /websites/scrapingant

### Description
Send POST, PUT, or DELETE requests to the main endpoint to forward data transparently to the target web page. Ensure you specify the `Content-Type` using custom headers.

### Method
POST, PUT, DELETE

### Endpoint
`/websites/scrapingant`

### Parameters
#### Query Parameters
- **url** (string) - Required - The URL to send the request to.
- **x-api-key** (string) - Required - Your Scrapingant API key.

#### Request Body
Data will be forwarded transparently to the target web page.

#### Custom Headers
- **Ant-Content-Type** (string) - Required - Specifies the `Content-Type` of the data being sent (e.g., `application/x-www-form-urlencoded`, `application/json`).

### Request Example
```bash
curl -X "POST" "https://api.scrapingant.com/v2/general?url=https:%2F%2Fhttpbin.org%2Fanything&x-api-key=<YOUR_SCRAPINGANT_API_KEY>" \
     -H "Ant-Content-Type: application/x-www-form-urlencoded" \
     -d "key1=value1&key2=value2"
```

### Response
#### Success Response (200)
- **body** (string) - The response body from the target URL.
- **headers** (object) - The headers returned by the target URL.
- **status** (integer) - The HTTP status code returned by the target URL.

#### Response Example
```json
{
  "body": "{\n  \"args\": {\n    \"url\": \"https://httpbin.org/anything\",\n    \"x-api-key\": \"<YOUR_SCRAPINGANT_API_KEY>\"\n  },\n  \"data\": \"key1=value1&key2=value2\",\n  \"files\": {},\n  \"form\": {\n    \"key1\": \"value1\",\n    \"key2\": \"value2\"\n  },\n  \"headers\": {\n    \"Accept\": "*/*",\n    \"Ant-Content-Type\": \"application/x-www-form-urlencoded\",\n    \"Content-Length\": \"24\",\n    \"Content-Type\": \"application/x-www-form-urlencoded\",\n    \"Host\": \"httpbin.org\",\n    \"User-Agent\": \"Scrapingant/1.0\"\n  },\n  \"json\": null,\n  \"origin\": \"1.2.3.4\",\n  \"url\": \"https://httpbin.org/anything\"\n}",
  "headers": {
    "Content-Type": "application/json",
    "Date": "Mon, 23 Oct 2023 10:00:00 GMT",
    "Server": "nginx"
  },
  "status": 200
}
```
```

--------------------------------

### API Credits Usage Status Response (JSON)

Source: https://docs.scrapingant.com/api-credits-usage

This snippet illustrates the JSON output format for the API credits usage status endpoint. It includes fields such as the active subscription plan name, start and end dates of the subscription, the total number of credits allocated for the plan, and the number of credits remaining.

```JSON
{
  "plan_name": "Enthusiast",
  "start_date": "2022-01-11T00:00:00",
  "end_date": "2022-02-11T00:00:00",
  "plan_total_credits": 100000,
  "remained_credits": 96740
}
```

--------------------------------

### Basic Request Comparison: ScraperAPI vs. ScrapingAnt (No Rendering)

Source: https://docs.scrapingant.com/migration-guides/scraperapi-migration

Compares the cURL command for a basic web scraping request without browser rendering. ScraperAPI uses a standard endpoint and api_key, while ScrapingAnt uses a specific endpoint, an 'x-api-key' header, and a 'browser=false' parameter to disable rendering.

```curl
curl "http://api.scraperapi.com?api_key=SCRAPERAPI_API_KEY&url=https%3A%2F%2Fexample.com"
```

```curl
curl "https://api.scrapingant.com/v2/general?x-api-key=SCRAPINGANT_API_KEY&url=https%3A%2F%2Fexample.com&browser=false"
```

--------------------------------

### Basic ScrapingDog Request with JS Rendering (cURL)

Source: https://docs.scrapingant.com/migration-guides/scrapingdog-migration

This cURL command demonstrates a basic web scraping request to ScrapingDog's API, specifically enabling JavaScript rendering using the 'dynamic=true' parameter. It includes the API key and the target URL.

```bash
curl "https://api.scrapingdog.com/scrape?api_key=SCRAPINGDOG_API_KEY&url=https%3A%2F%2Fexample.com&dynamic=true"
```

--------------------------------

### Get API Credits Usage Status (HTTP)

Source: https://docs.scrapingant.com/api-credits-usage

This snippet shows how to make an HTTP GET request to the Scrapingant API endpoint to retrieve the current status of API credits usage. It requires your unique API key for authentication. The response provides details about your subscription plan, including start and end dates, total credits, and remaining credits.

```HTTP
GET https://api.scrapingant.com/v2/usage?x-api-key=<YOUR_SCRAPINGANT_API_KEY>
```

--------------------------------

### Send Custom Headers with cURL to ScrapingAnt

Source: https://docs.scrapingant.com/custom-headers

Demonstrates sending a custom header ('Ant-Custom-Header') in a GET request to the ScrapingAnt API using cURL. The header is prefixed with 'Ant-' to be forwarded to the target URL. The example shows the request and the subsequent response containing the custom header.

```bash
curl --request GET \
--url 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fhttpbin.org%2Fheaders&x-api-key=<YOUR_SCRAPINGANT_API_KEY>' \
--header 'Ant-Custom-Header: I <3 ScrapingAnt'
```

--------------------------------

### Scrape Website with ScrapingAnt Python Client

Source: https://docs.scrapingant.com/python-client

Demonstrates how to use the ScrapingAnt Python client to scrape a website. It initializes the client with an API token and makes a general request to a given URL, then prints the content of the result.

```python
from scrapingant_client import ScrapingAntClient

client = ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')
# Scrape the example.com site.
result = client.general_request('https://example.com')
print(result.content)
```

--------------------------------

### Configure VS Code / GitHub Copilot MCP Server

Source: https://docs.scrapingant.com/mcp-server

Configure ScrapingAnt's MCP server for VS Code or GitHub Copilot by adding it to your VS Code settings or a dedicated MCP JSON file. This enables AI-powered web scraping within your development environment. Remember to substitute '<YOUR-API-KEY>' with your key.

```json
{
  "servers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp/",
      "requestInit": {
        "headers": {
          "x-api-key": "<YOUR-API-KEY>"
        }
      }
    }
  }
}
```

--------------------------------

### Specify Product Properties for AI Extraction

Source: https://docs.scrapingant.com/ai-data-extraction/ai-extractor-best-practices

Defines how to specify product properties for extraction, including data types and nested structures. This helps the AI model understand the desired output format, such as extracting a product title, a numerical price, and a list of reviews with their content.

```text
product title, price(number), full description
```

```text
product title, price(number), full description, reviews(list: review title, review content)
```

--------------------------------

### API Authentication using cURL

Source: https://docs.scrapingant.com/request-response-format

Demonstrates how to authenticate API requests using the `x-api-key` query parameter with cURL. This is a common method for direct API interaction.

```bash
curl 'https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&x-api-key=<YOUR_SCRAPINGANT_API_KEY>'
```

--------------------------------

### Web Scraping Parameters

Source: https://docs.scrapingant.com/mcp-server

Details on the parameters available for all web scraping tools, including URL, browser rendering, and proxy options.

```APIDOC
## Web Scraping Parameters

All three scraping tools (`get_web_page_html`, `get_web_page_markdown`, `get_web_page_text`) accept the following parameters:

### Parameters Table

| Parameter       | Type    | Required | Default   | Description                                       |
|-----------------|---------|----------|-----------|---------------------------------------------------|
| `url`           | string  | ✅       | -         | The URL of the page to scrape.                    |
| `browser`       | boolean |          | `true`    | Whether to use browser rendering.                 |
| `proxy_type`    | string  |          | `datacenter`| Proxy type: `datacenter` or `residential`.        |
| `proxy_country` | string  |          | Random    | ISO-3166 country code for proxy location.         |

### Parameter Details

#### `browser`

*   **Description**: By default, ScrapingAnt uses a headless browser to render pages, which is essential for JavaScript-heavy websites and SPAs. Set to `false` for lightweight HTML parsing of static pages.

#### `proxy_type`

*   **`datacenter`** (default): High-speed datacenter proxies. Suitable for most use cases.
*   **`residential`**: Premium residential proxies. Recommended when encountering anti-bot detection for improved success rates.

#### `proxy_country`

*   **Description**: Specify an ISO-3166 country code to route requests through proxies in that specific region. This is useful for accessing geo-restricted content or testing location-specific website behavior.
```

--------------------------------

### General Scraping Endpoint

Source: https://docs.scrapingant.com/migration-guides/scrapingdog-migration

This endpoint serves as a direct replacement for the ScrapingDog API, proxying responses from the target website to the client.

```APIDOC
## GET /v2/general

### Description
Proxies responses from the target website to the client. This is the primary endpoint for migrating from ScrapingDog's general scraping functionality.

### Method
GET

### Endpoint
`/v2/general`

### Parameters
#### Query Parameters
- **x-api-key** (string) - Required - Your ScrapingAnt API key for authentication.
- **url** (string) - Required - The URL of the target website to scrape.
- **browser** (boolean) - Optional - Set to `true` to enable headless browser rendering for JavaScript execution (default is `true`). Set to `false` to disable browser rendering.
- **proxy_country** (string) - Optional - ISO country code of the proxy to use for the request (e.g., `US`, `DE`).
- **proxy_type** (string) - Optional - Configures the proxy type. Accepted values include `datacenter` and `residential`.
- **wait_for_selector** (string) - Optional - A CSS selector to wait for before returning the response. This is a more reliable alternative to time-based waiting.

#### Custom Headers
To forward custom headers, prepend them with `ant-`. For example, a `User-Agent` header would be sent as `ant-User-Agent`.

### Request Example
```bash
curl "https://api.scrapingant.com/v2/general?x-api-key=YOUR_SCRAPINGANT_API_KEY&url=https%3A%2F%2Fexample.com&browser=false"
```

### Response
#### Success Response (200)
- **content** (string) - The HTML content of the scraped page.
- **cookies** (object) - Cookies received from the target website.
- **headers** (object) - Headers received from the target website.
- **status** (integer) - HTTP status code of the response.
- **url** (string) - The final URL after any redirects.

#### Response Example
```json
{
  "content": "<html>...</html>",
  "cookies": {},
  "headers": {},
  "status": 200,
  "url": "https://example.com"
}
```
```

--------------------------------

### Example Error Response JSON

Source: https://docs.scrapingant.com/errors

Demonstrates the structure of a typical error response from the ScrapingAnt API. It includes a 'detail' field providing a human-readable explanation of the error.

```json
{
  "detail": "Wrong host part of url. Please check out the documentation https://scrapingant.com/documentation/"
}
```

--------------------------------

### Headless Browser (JS Rendering Enabled)

Source: https://docs.scrapingant.com/headless-browser

Enables headless browser rendering for dynamic websites, SPAs, and AJAX content. This mode opens a real browser, loads the page, and then extracts the HTML content, cookies, and other data. It can also help bypass anti-scraping protections like Cloudflare.

```APIDOC
## GET /v2/general

### Description
Performs web scraping using a headless browser with JavaScript rendering enabled. This is ideal for dynamic websites, Single-Page Applications (SPAs), and sites using AJAX.

### Method
GET

### Endpoint
/v2/general

### Query Parameters
- **url** (string) - Required - The URL of the website to scrape.
- **browser** (boolean) - Optional - Set to `true` to enable headless browser rendering (default behavior).
- **js_snippet** (string) - Optional - A JavaScript snippet to execute in the browser context.
- **wait_for_selector** (string) - Optional - A CSS selector to wait for before rendering the page.
- **return_page_source** (boolean) - Optional - If set to `true` along with `browser=true`, returns the raw HTTP response data without JS rendering.

### Request Example
```
https://api.scrapingant.com/v2/general?url=https%3A%2F%2Fexample.com&browser=true
```

### Response
#### Success Response (200)
- **html** (string) - The HTML content of the rendered page.
- **cookies** (array) - An array of cookies associated with the page.

#### Response Example
```json
{
  "html": "<html>...</html>",
  "cookies": [
    {
      "name": "session_id",
      "value": "abcdef12345"
    }
  ]
}
```
```

--------------------------------

### Scrape Website using ScrapingAnt JS Client

Source: https://docs.scrapingant.com/js-client

Demonstrates how to use the ScrapingAnt JavaScript client to scrape a website. It requires an API key and handles the asynchronous scraping process, logging the result or any errors.

```javascript
const ScrapingAntClient = require('@scrapingant/scrapingant-client');

const client = new ScrapingAntClient({ apiKey: '<YOUR-SCRAPINGANT-API-KEY>' });

// Scrape the example.com site.
client.scrape('https://example.com')
    .then(res => console.log(res))
    .catch(err => console.error(err.message));
```

--------------------------------

### Extended Scraping Endpoint

Source: https://docs.scrapingant.com/migration-guides/scrapingdog-migration

This endpoint provides a more comprehensive response, including JSON data, text content, cookies, headers, and details about XHR requests and iframes.

```APIDOC
## GET /v2/extended

### Description
Provides an extended response format, including JSON data, text content, cookies, headers, and details about XHR requests and iframes. Useful for complex scraping scenarios.

### Method
GET

### Endpoint
`/v2/extended`

### Parameters
#### Query Parameters
- **x-api-key** (string) - Required - Your ScrapingAnt API key for authentication.
- **url** (string) - Required - The URL of the target website to scrape.
- **browser** (boolean) - Optional - Set to `true` to enable headless browser rendering for JavaScript execution (default is `true`). Set to `false` to disable browser rendering.
- **proxy_country** (string) - Optional - ISO country code of the proxy to use for the request (e.g., `US`, `DE`).
- **proxy_type** (string) - Optional - Configures the proxy type. Accepted values include `datacenter` and `residential`.
- **wait_for_selector** (string) - Optional - A CSS selector to wait for before returning the response. This is a more reliable alternative to time-based waiting.

#### Custom Headers
To forward custom headers, prepend them with `ant-`. For example, a `User-Agent` header would be sent as `ant-User-Agent`.

### Request Example
```bash
curl "https://api.scrapingant.com/v2/extended?x-api-key=YOUR_SCRAPINGANT_API_KEY&url=https%3A%2F%2Fexample.com"
```

### Response
#### Success Response (200)
- **json_data** (object) - Extracted JSON data from the page, if available.
- **text** (string) - Text representation of the page content.
- **cookies** (object) - Cookies received from the target website.
- **headers** (object) - Headers received from the target website.
- **status** (integer) - HTTP status code of the response.
- **url** (string) - The final URL after any redirects.
- **xhrs** (array) - Details of any XHR requests made by the page.
- **iframes** (array) - Details of any iframes found on the page.

#### Response Example
```json
{
  "json_data": {},
  "text": "Page text content...",
  "cookies": {},
  "headers": {},
  "status": 200,
  "url": "https://example.com",
  "xhrs": [],
  "iframes": []
}
```
```

--------------------------------

### Web Scraping Tools

Source: https://docs.scrapingant.com/mcp-server

The ScrapingAnt MCP server provides three tools for web scraping: fetching HTML, Markdown, or plain text content.

```APIDOC
## Web Scraping Tools

The ScrapingAnt MCP server offers the following tools for web scraping:

### `get_web_page_html`

*   **Description**: Fetch a URL and return raw HTML content.

### `get_web_page_markdown`

*   **Description**: Fetch a URL and return content as Markdown.

### `get_web_page_text`

*   **Description**: Fetch a URL and return plain text content.
```

--------------------------------

### ScrapingAnt Proxy Mode Integration

Source: https://docs.scrapingant.com/proxy-mode

This section explains how to integrate with ScrapingAnt Proxy Mode using HTTP clients. It details the proxy addresses, username format, and password (API key).

```APIDOC
## ScrapingAnt Proxy Mode Integration

### Description
ScrapingAnt Proxy Mode allows you to use the ScrapingAnt API through a simple HTTP/HTTPS proxy. This mode is compatible with any HTTP client that supports proxy configuration, making it easier to integrate with third-party libraries and tools. The proxy mode functions as a lightweight front-end for the scraping API, offering the same functionality and performance.

### Method
N/A (Proxy Mode)

### Endpoint
- **HTTP Address:** `proxy.scrapingant.com:8080`
- **HTTPS Address:** `proxy.scrapingant.com:443`

### Parameters
#### Username Format
`scrapingant` + API parameters separated by `&` delimiter

#### Password
`YOUR-API-KEY`

### Request Example (cURL)
```bash
curl -x "http://scrapingant:YOUR-API-KEY@proxy.scrapingant.com:8080" -k "http://httpbin.org/ip"
```

### Important Note
Ensure your code is configured to **not verify** SSL certificates when using this proxy mode.
```

--------------------------------

### GET /v2/extended

Source: https://docs.scrapingant.com/json-response

Retrieves an extended JSON response from a web page, including HTML content, plain text, cookies, status code, headers, XHR requests, and iframe content.

```APIDOC
## GET /v2/extended

### Description
This endpoint provides an extended JSON response containing detailed information about the scraped web page. It includes the full HTML content, extracted text, cookies, HTTP status code, response headers, details of any XHR or FETCH requests made, and the content of any iframes found on the page.

### Method
GET

### Endpoint
https://api.scrapingant.com/v2/extended

### Parameters
#### Query Parameters
This endpoint uses the same request structure as the general endpoint. Please refer to the documentation for the general endpoint for details on available query parameters (e.g., `url`, `api_key`).

### Request Example
```json
{
  "example": "https://api.scrapingant.com/v2/extended?url=https://example.com&api_key=YOUR_API_KEY"
}
```

### Response
#### Success Response (200)
- **html** (String) - The full HTML content of the scraped web page.
- **text** (String) - The plain text content extracted from the scraped web page.
- **cookies** (String) - The cookies received from the scraped web page, formatted as a string (e.g., `cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2`).
- **status_code** (Number) - The HTTP status code received from the scraped page.
- **headers** (Array) - An array of objects, where each object represents a response header with `name` and `value` properties.
- **xhrs** (Array) - An array of objects detailing XHR and FETCH requests made during page rendering. Each object includes `url`, `status`, `method`, `headers`, `request_body`, and `body`.
- **iframes** (Array) - An array of objects detailing iframe content. Each object includes `src` and `html` properties.

#### Response Example
```json
{
  "html": "<html><body><h1>Hello, World!</h1></body></html>",
  "text": "Hello, World!",
  "cookies": "cookie_name_1=cookie_value_1;cookie_name_2=cookie_value_2",
  "status_code": 200,
  "headers": [
    {
      "name": "Content-Type",
      "value": "text/html; charset=utf-8"
    }
  ],
  "xhrs": [
    {
      "url": "https://example.com/xhr1",
      "status": 200,
      "method": "GET",
      "headers": [
        {
          "name": "Content-Type",
          "value": "application/json"
        }
      ],
      "request_body": "",
      "body": "Hello, World!"
    }
  ],
  "iframes": [
    {
      "src": "https://example.com/iframe1",
      "html": "<html><body><h1>Hello, World!</h1></body></html>"
    }
  ]
}
```
```

--------------------------------

### Proxy Settings Configuration

Source: https://docs.scrapingant.com/proxy-settings

This section details how to configure proxy settings for your web scraping requests. You can specify the proxy country and type to tailor your scraping strategy.

```APIDOC
## GET /websites/scrapingant

### Description
Configure proxy settings for web scraping requests. By default, ScrapingAnt uses a random proxy from the standard datacenter pool. You can specify `proxy_country` and `proxy_type` for more control.

### Method
GET

### Endpoint
/websites/scrapingant

### Parameters
#### Query Parameters
- **proxy_country** (string) - Optional - The country code for the proxy (e.g., 'US', 'GB', 'DE'). If not provided, a random country is selected.
- **proxy_type** (string) - Optional - The type of proxy to use. Available options are 'datacenter' (default) and 'residential'.

### Request Example
```json
{
  "proxy_country": "US",
  "proxy_type": "residential"
}
```

### Response
#### Success Response (200)
- **message** (string) - A confirmation message indicating the proxy settings have been applied.

#### Response Example
```json
{
  "message": "Proxy settings updated successfully."
}
```

## Available Proxy Types

### Description
Lists the available proxy types that can be used with ScrapingAnt.

### Method
GET

### Endpoint
/websites/scrapingant/proxy-types

### Response
#### Success Response (200)
- **proxy_types** (array) - A list of available proxy types, each with a `type` and `description`.

#### Response Example
```json
{
  "proxy_types": [
    {
      "type": "datacenter",
      "description": "Default option. High-speed proxy server located in one of the datacenters across the world. Our datacenter proxy pool contains over 50.000 proxies (different IP addresses)."
    },
    {
      "type": "residential",
      "description": "High-quality proxy server located in one of the real people houses across the world. This option is perfect for scraping social media and search engines. Our residential proxy pool contains over 2.000.000 proxies (different IP addresses)."
    }
  ]
}
```

## Available Proxy Countries

### Description
Lists the available proxy countries that can be used with ScrapingAnt.

### Method
GET

### Endpoint
/websites/scrapingant/proxy-countries

### Response
#### Success Response (200)
- **proxy_countries** (array) - A list of available proxy countries, each with a `country` name and `value` (country code).

#### Response Example
```json
{
  "proxy_countries": [
    {
      "country": "World",
      "value": null
    },
    {
      "country": "Brazil",
      "value": "BR"
    },
    {
      "country": "Canada",
      "value": "CA"
    }
    // ... other countries
  ]
}
```
```

--------------------------------

### Configure Claude Desktop MCP Server

Source: https://docs.scrapingant.com/mcp-server

Add ScrapingAnt's MCP server configuration to your Claude Desktop settings. This allows Claude to use ScrapingAnt for web scraping. Ensure you replace '<YOUR-API-KEY>' with your actual ScrapingAnt API key.

```json
{
  "mcpServers": {
    "scrapingant": {
      "url": "https://api.scrapingant.com/mcp",
      "transport": "streamableHttp",
      "headers": {
        "x-api-key": "<YOUR-API-KEY>"
      }
    }
  }
}
```
Tokens:
Raw