Scraping Fish API

https://scrapingfish.com/docs
Admin
Scraping Fish API powers data gathering pipelines, providing tools and features for efficient web...
Tokens:17,208
Snippets:146
Trust Score:8
Update:3 months ago
Show doc for...
Tokens:
Raw
### Execute JS Scenario with Python

Source: https://scrapingfish.com/docs/js-scenario

This example demonstrates how to use the `requests` library in Python to send a GET request to the Scraping Fish API with a JS Scenario defined. The scenario waits for 1 second and then clicks on the first link within a paragraph.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "js_scenario": json.dumps({
    "steps": [
      {"wait": 1000},
      {"click_and_wait_for_navigation": "p > a"}
    ]
  })
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Example: Preload localStorage for an Origin

Source: https://scrapingfish.com/docs/preload-local-storage

Illustrates how to set `localStorage` with specific key-value pairs for a given origin using the `preload_local_storage` parameter.

```json
[
  {
    "origin": "https://example.com"
    "localStorage": [
      {
        "name": "key1",
        "value": "value1"
      },
      {
        "name": "key2",
        "value": "value2"
      }
    ]
  }
]

```

--------------------------------

### API Request Usage Response Example

Source: https://scrapingfish.com/docs/usage

Example of the response format when you have active API request packs. It details the total requests, remaining requests, and the expiration date for each pack.

```json
[
  {
    "total": 100000,
    "left": 69420,
    "expires": "24 Feb 2024 20:00:00 GMT"
  }
]
```

--------------------------------

### Making a POST Request

Source: https://scrapingfish.com/docs/request-method

This example demonstrates how to make a POST request to the Scraping Fish API. PUT requests can be made similarly by replacing POST with PUT.

```APIDOC
## POST /api/v1/

### Description
To make a POST or PUT request with Scraping Fish API just use POST or PUT method when calling `/api/v1/` endpoint along with optional body.

### Method
POST

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **headers** (string) - Optional - A JSON string representing custom headers to include in the request.

#### Request Body
- **key1** (string) - Required - Example key for the request body.
- **key2** (string) - Required - Example value for the request body.

### Request Example
```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/post",
}

data = {"key1": "value1", "key2": "value2"}

response = requests.post(f"https://api.scrapingfish.com/api/v1/", params=payload, json=data)
print(response.content)
```

### Response
#### Success Response (200)
Returns the content of the scraped URL, often including data and headers sent in the request.
```

--------------------------------

### Retrieve Cookies in Python

Source: https://scrapingfish.com/docs/add-cookies

This Python example shows how to inspect the `Sf-Cookies` header in the API response to retrieve cookies set by the target website.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "cookies": json.dumps(
    [
      {"name": "cookie1-name", "value": "cookie1-value"},
      {"name": "cookie2-name", "value": "cookie2-value"},
    ]
  ),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.headers)
```

--------------------------------

### Get Screenshot

Source: https://scrapingfish.com/docs/screenshot

Retrieve a screenshot of a webpage as a PNG image file by setting the `screenshot` query parameter to true.

```APIDOC
## GET /api/v1/

### Description
Retrieves a screenshot of the specified URL as a PNG image.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL of the webpage to capture.
- **screenshot** (boolean) - Optional - Set to `true` to enable screenshot retrieval.

### Request Example
```
GET https://api.scrapingfish.com/api/v1/?api_key=[your API key]&url=https://www.example.com&screenshot=true
```

### Response
#### Success Response (200)
- **content** (bytes) - The raw PNG image data of the screenshot.
```

--------------------------------

### Example Header Parameter Value

Source: https://scrapingfish.com/docs/forward-headers

This JSON object shows how to format the 'headers' parameter to include custom headers or remove existing ones by setting their value to an empty string.

```json
{"x-custom-header": "value", "host": ""}
```

--------------------------------

### Making a POST Request with Custom Headers

Source: https://scrapingfish.com/docs/request-method

This example shows how to include custom headers in a POST request to the Scraping Fish API.

```APIDOC
## POST /api/v1/ with Custom Headers

### Description
This example demonstrates how to make a POST request to `httpbin.org` and include custom header `X-Custom-Header: value`.

### Method
POST

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **headers** (string) - Optional - A JSON string representing custom headers to include in the request. For example: `json.dumps({"x-custom-header": "value"})`.

#### Request Body
- **key1** (string) - Required - Example key for the request body.
- **key2** (string) - Required - Example value for the request body.

### Request Example
```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/post",
  "headers": json.dumps({"x-custom-header": "value"}),
}

data = {"key1": "value1", "key2": "value2"}

response = requests.post(f"https://api.scrapingfish.com/api/v1/", params=payload, json=data)
print(response.content)
```

### Response
#### Success Response (200)
Returns the content of the scraped URL, often including data and headers sent in the request.
```

--------------------------------

### Example Cookie Structure

Source: https://scrapingfish.com/docs/add-cookies

Cookies are provided as an array of objects. 'name' and 'value' are mandatory. Optional fields like 'domain', 'path', 'expires', 'secure', 'httpOnly', and 'sameSite' can also be specified.

```json
[
  {
    "name": "cookie1",
    "value": "cookie1value"
  },
  {
    "name": "cookie2",
    "value": "cookie2value",
    "domain": "example.com",
    "path": "/",
    "expires": 100000,
    "secure": true,
    "httpOnly": false,
    "sameSite": "Lax"
  }
]
```

--------------------------------

### Wait for Request and Get HTML Content (Python)

Source: https://scrapingfish.com/docs/intercept-request

Utilize the `wait_for_response` step in a JS Scenario to wait for a specific XHR/Fetch request to complete and then retrieve the HTML content of the requested URL.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "http://httpbin.org",
  "js_scenario": json.dumps(
    {"steps": [{"wait_for_response": "**/spec**"}]}
  ),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Intercept Request and Get Response (Python)

Source: https://scrapingfish.com/docs/intercept-request

Use the `intercept_request` parameter to match a specific XHR/Fetch request and get its response. Scraping Fish automatically waits for the request to complete.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "http://httpbin.org",
  "intercept_request": "**/spec*",
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.json())
```

--------------------------------

### Get Webpage Screenshot as PNG

Source: https://scrapingfish.com/docs/screenshot

Use this snippet to retrieve a webpage's screenshot as a PNG image file. The screenshot is saved directly to a local file.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "screenshot": True,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
with open("./screenshot.png", "wb") as f:
  f.write(response.content)
```

--------------------------------

### Sf-Cookies Header Example

Source: https://scrapingfish.com/docs/add-cookies

The `Sf-Cookies` header contains cookies set by the website, separated by newlines. Cookie attributes are separated by semicolons.

```text
Sf-Cookies: "cookie1-name=cookie1-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax\ncookie2-name=cookie2-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax"
```

--------------------------------

### Wait for Response and Get HTML Content

Source: https://scrapingfish.com/docs/intercept-request

Wait for a specific XHR/Fetch request to complete and then retrieve the HTML content of the requested URL. This is achieved using the `js_scenario` parameter with the `wait_for_response` step.

```APIDOC
## GET /api/v1/

### Description
Waits for a specified XHR/Fetch request to complete and then returns the HTML content of the requested URL. This is useful when you need to ensure a background request has finished before processing the page.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **js_scenario** (string) - Required - A JSON string defining the JavaScript scenario. Use `{"steps": [{"wait_for_response": "**/pattern**"}]}` to specify the request to wait for.

### Request Example
```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "http://httpbin.org",
  "js_scenario": json.dumps(
    {"steps": [{"wait_for_response": "**/spec**"}]}
  ),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

### Response
#### Success Response (200)
- The response content is the HTML of the requested URL, extracted after the specified background request has completed.
```

--------------------------------

### Set Total and Trial Timeouts for Long JS Scenarios

Source: https://scrapingfish.com/docs/timeouts

Use `total_timeout_ms` and `trial_timeout_ms` to allow sufficient time for complex JavaScript scenarios and website loading. Both parameters are set to 110,000 ms in this example.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "js_scenario": json.dumps(
    {"steps": [{"wait": 100_000}]}
  ),
  "trial_timeout_ms": 110_000,
  "total_timeout_ms": 110_000,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Get Screenshot and HTML Content

Source: https://scrapingfish.com/docs/screenshot

Retrieve both the HTML content of a webpage and a base64-encoded screenshot. The response is a JSON object containing 'content' and 'screenshot' keys.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "screenshot_base64": True,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.json())
```

```json
{
    "content": "<!DOCTYPE html><html><head>\n    <title>Example Domain</title>...",
    "screenshot": "iVBORw0KGgoAAAANSUhEUgAAAmwAAAF1CAIAAADaxiG8AAAAAXNSR0..."
}
```

--------------------------------

### Single Trial Timeout Configuration

Source: https://scrapingfish.com/docs/timeouts

The `trial_timeout_ms` parameter defines the timeout for a single attempt to execute a request. It defaults to 30 seconds and can be adjusted between 10 and 600 seconds, but must not exceed `total_timeout_ms`. If a trial fails within this limit, Scraping Fish will retry. This example sets `trial_timeout_ms` to 10,000 ms to retry quickly on failure.

```APIDOC
## GET /api/v1/

### Description
Sets the timeout for a single trial of request execution.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **trial_timeout_ms** (integer) - Optional - Timeout for a single trial in milliseconds. Defaults to 30000.

### Request Example
```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "trial_timeout_ms": 10_000,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

### Response
#### Success Response (200)
- **content** (bytes) - The scraped content of the page.
```

--------------------------------

### Make a Basic API Request in Python

Source: https://scrapingfish.com/docs/quickstart

Use this snippet to send a GET request to the Scraping Fish API with your API key and the target URL. Parameters are automatically URL-encoded by the requests library.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Scrape Website with cURL

Source: https://scrapingfish.com/docs/api-reference

This cURL command shows how to make a GET request to the Scraping Fish API to retrieve website HTML. Remember to substitute '[your API key]' with your actual API key.

```cURL
curl -G "https://api.scrapingfish.com/api/v1/" -d "api_key=[your API key]" -d "url=https://example.com"
```

--------------------------------

### Get Webpage Screenshot with Custom Dimensions

Source: https://scrapingfish.com/docs/screenshot

Customize the dimensions of the captured screenshot by providing a JSON object with 'width' and 'height' properties. Both values must be positive integers.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "screenshot": json.dumps({"width": 1920, "height": 1080}),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
with open("./screenshot.png", "wb") as f:
  f.write(response.content)
```

--------------------------------

### Get API Usage

Source: https://scrapingfish.com/docs/usage

Retrieve the current status of API requests available for your account. This endpoint returns a list of your valid API request packs, sorted by expiration date.

```APIDOC
## GET /api/v1/usage

### Description
This endpoint allows you to programmatically check the status of active request packs associated with your account. Each successful request to the Scraping Fish API consumes one API request credit.

### Method
GET

### Endpoint
/api/v1/usage/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.

### Request Example
```http
GET /api/v1/usage/?api_key=[your API key]
```

### Response
#### Success Response (200)
- **total** (integer) - The total number of API requests purchased in this pack.
- **left** (integer) - The number of API requests remaining in this pack.
- **expires** (string) - The expiration date and time of the API requests pack in GMT format.

#### Response Example
```json
[
  {
    "total": 100000,
    "left": 69420,
    "expires": "24 Feb 2024 20:00:00 GMT"
  }
]
```
```

--------------------------------

### Get Cookies

Source: https://scrapingfish.com/docs/add-cookies

Retrieve all cookies set by the website, including dynamically set cookies, via the `Sf-Cookies` header. Cookies are separated by newline characters (`\n`), and cookie elements are separated by semicolons (`;`).

```APIDOC
## Get Cookies

### Description
Retrieves all cookies set by the website, including dynamically set cookies, in the `Sf-Cookies` response header.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **cookies** (string) - Optional - A JSON-encoded array of cookie objects to send with the request. (This parameter is shown in the example for context but is not directly related to *retrieving* cookies set by the site).

### Request Example
```json
{
  "api_key": "[your API key]",
  "url": "https://example.com"
}
```

### Response
#### Success Response (200)
- **response.headers** - Contains various headers, including `Sf-Cookies`.

#### Response Example
```
Sf-Cookies: "cookie1-name=cookie1-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax\ncookie2-name=cookie2-value; expires=Wed, 31 Dec 1969 23:59:59 GMT; domain=example.com; path=/; SameSite=Lax"
```
```

--------------------------------

### Total Timeout Configuration

Source: https://scrapingfish.com/docs/timeouts

The `total_timeout_ms` parameter sets the maximum time for the entire request, including JS scenario execution. It defaults to 90 seconds and can be set between 10 and 600 seconds. Ensure it's not smaller than `trial_timeout_ms`. This example demonstrates setting both `trial_timeout_ms` and `total_timeout_ms` to 110,000 ms for a long JS scenario.

```APIDOC
## GET /api/v1/

### Description
Sets the total timeout for a request, including JS scenario execution.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **js_scenario** (string) - Optional - A JSON string defining JavaScript execution steps.
- **trial_timeout_ms** (integer) - Optional - Timeout for a single trial in milliseconds. Defaults to 30000.
- **total_timeout_ms** (integer) - Optional - Timeout for the entire request in milliseconds. Defaults to 90000.

### Request Example
```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "js_scenario": json.dumps(
    {"steps": [{"wait": 100_000}]}
  ),
  "trial_timeout_ms": 110_000,
  "total_timeout_ms": 110_000,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

### Response
#### Success Response (200)
- **content** (bytes) - The scraped content of the page.
```

--------------------------------

### Encoding Other Parameters with cURL

Source: https://scrapingfish.com/docs/url-encoding

When using `cURL` or similar clients that don't auto-encode, ensure all API parameters containing non-ASCII or reserved characters are encoded. This example shows encoding a `js_scenario` parameter.

```bash
curl -G --data-urlencode 'url=https://example.com' \
--data-urlencode 'js_scenario={"steps": [{"wait": 1000}, {"click_and_wait_for_navigation": "p > a"}]}' \
'https://api.scrapingfish.com/api/v1/?api_key=[your API key]'
```

--------------------------------

### Python POST Request with Custom Headers

Source: https://scrapingfish.com/docs/request-method

This example demonstrates how to include custom headers in a POST request to the Scraping Fish API. The 'headers' parameter in the payload should be a JSON string representing the custom headers.

```Python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/post",
  "headers": json.dumps({"x-custom-header": "value"}),
}

data = {"key1": "value1", "key2": "value2"}

response = requests.post(f"https://api.scrapingfish.com/api/v1/", params=payload, json=data)
print(response.content)

```

--------------------------------

### Enable Sticky Session with Python

Source: https://scrapingfish.com/docs/sticky-session

Demonstrates enabling sticky sessions and preserving cookies between requests using the Python client. The first request sets a cookie, and the second request retrieves it using the same session ID.

```python
import requests
import json

session_id = "my_session"

payload1 = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/cookies",
  "cookies": json.dumps([{"name": "cookie-name", "value": "cookie-value"}]),
  "session": session_id,
}

response1 = requests.get("https://api.scrapingfish.com/api/v1/", params=payload1)

payload2 = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/cookies",
  "session": session_id,
}

response2 = requests.get("https://api.scrapingfish.com/api/v1/", params=payload2)
print(response2.content)
```

--------------------------------

### Python: Preload localStorage via API

Source: https://scrapingfish.com/docs/preload-local-storage

Demonstrates how to use the Python `requests` library to call the Scrapingfish API and preload localStorage for a specified URL.

```python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "preload_local_storage": json.dumps(
    [{"origin": "https://example.com", "localStorage": [{"name": "key1", "value": "value1"}, {"name": "key2", "value": "value2"}]}]
  ),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)

```

--------------------------------

### Extract First Paragraph Text

Source: https://scrapingfish.com/docs/extract-rules

Uses the simple form of extraction rules to get the text content of the first paragraph element.

```Python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({"paragraph": "p"}),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

```cURL
GET
/api/v1/

import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({"paragraph": "p"}),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

```JavaScript
GET
/api/v1/

import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({"paragraph": "p"}),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Set Single Trial Timeout for Quick Loading Websites

Source: https://scrapingfish.com/docs/timeouts

Configure `trial_timeout_ms` to a lower value (e.g., 10,000 ms) when expecting a website to load quickly. This parameter controls the timeout for a single request execution trial.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "trial_timeout_ms": 10_000,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Enable JavaScript Rendering

Source: https://scrapingfish.com/docs/js-rendering

Add `render_js=true` to your request parameters to enable JavaScript rendering. This ensures dynamic content is loaded before the HTML is returned. It does not incur extra costs.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "render_js": "true",
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

--------------------------------

### Generate PDF from Webpage

Source: https://scrapingfish.com/docs/pdf

Use this Python snippet to generate a PDF from a given URL and save the response content to a file. Ensure you replace '[your API key]' with your actual API key.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "pdf": True,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
with open("./page.pdf", "wb") as f:
  f.write(response.content)
```

--------------------------------

### Select options from a dropdown

Source: https://scrapingfish.com/docs/js-scenario

Selects one or more options from a `<select>` element. Provide the selector for the dropdown and the option(s) to select. Multiple options can be selected using an array.

```json
{
  "steps": [
    {
      "select": {
        "selector": "#select1",
        "options": "1"
      }
    }
  ]
}
```

--------------------------------

### Scrape Website with JavaScript

Source: https://scrapingfish.com/docs/api-reference

This JavaScript snippet demonstrates how to fetch website HTML using the Scraping Fish API. Replace '[your API key]' with your actual API key and adjust the URL as needed.

```JavaScript
const url = require('url');
const https = require('https');

const options = {
  hostname: 'api.scrapingfish.com',
  port: 443,
  path: '/api/v1/?api_key=[your API key]&url=https://example.com',
  method: 'GET'
};

const req = https.request(options, (res) => {
  console.log(`statusCode: ${res.statusCode}`);

  let data = '';
  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('end', () => {
    console.log(data);
  });
});

req.on('error', (error) => {
  console.error(error);
});

req.end();
```

--------------------------------

### POST to website

Source: https://scrapingfish.com/docs/api-reference

This endpoint allows you to retrieve HTML content of the desired website by responding to a POST request. It supports various parameters for customization, including API key, URL, JavaScript rendering, headers, and more.

```APIDOC
## POST /api/v1/

### Description
This endpoint allows you to retrieve HTML content of the desired website responding to POST request.

### Method
POST

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key. If you don't have it, you can get one by buying a Request Pack.
- **url** (string) - Required - URL to scrape. If it contains non-ASCII or reserved characters, it needs to be URL encoded.
- **render_js** (bool) - Optional - Enable JavaScript rendering to wait for background network requests. See: JS Rendering.
- **js_scenario** (object) - Optional - Actions to execute after the website is loaded. See: JS Scenario.
- **intercept_request** (string) - Optional - Intercept background XHR/Fetch request matching specified pattern and obtain its response. See: Intercept XHR request.
- **headers** (object) - Optional - Headers to be set or overridden. See: Forward HTTP headers.
- **forward_original_status** (bool) - Optional - Whether to forward original status in `Sf-Original-Status-Code` response header. See: Original status code
- **extract_rules** (object) - Optional - Rules to be applied to resulting HTML to extract desired content. See: Extraction rules
- **cookies** (array[object]) - Optional - Cookies to be sent with the request. See: Cookies
- **total_timeout_ms** (integer) - Optional - Maximum total number of milliseconds before timing out. It must be between `10000` (10 seconds) and `600000` (600 seconds) and greater than or equal to `trial_timeout_ms`. Defaults to `90000` (90 seconds). See: Timeouts
- **trial_timeout_ms** (integer) - Optional - Maximum number of milliseconds before timing out for a single request execution trial. It must be between `10000` (10 seconds) and `600000` (600 seconds) and lower than or equal to `total_timeout_ms`. Defaults to `30000` (30 seconds). See: Timeouts
- **render_js_timeout_ms** (integer) - Optional - Maximum number of milliseconds to wait for background network requests to finish when JS rendering is enabled. See: JS Rendering timeout
- **screenshot** (bool | object) - Optional - If set to true, screenshot of the website will be taken and returned as bytes. You can also pass a JSON object with `width` and `height` properties to customize the screenshot dimensions. See: Screenshot
- **screenshot_base64** (bool) - Optional - If set to true, screenshot of the website will be taken and returned as base64-encoded image data together with the HTML content in JSON response. See: Screenshot
- **preload_local_storage** (object) - Optional - Object of key/values that will be preloaded as `localStorage`. See: Preload localStorage
- **session** (string) - Optional - A session identifier. For requests with the same value of this parameter, cookies and localStorage will be preserved. Must be between 1 and 512 characters long. See: Sticky Session.

#### Request Body
This endpoint accepts a JSON object for the request body.

### Request Example
```python
import requests
import json

payload = {
"api_key": "[your API key]",
"url": "https://httpbin.org/post",
"headers": json.dumps({"x-custom-header": "value"}),
}

data = {"key1": "value1", "key2": "value2"}

response = requests.post(f"https://api.scrapingfish.com/api/v1/", params=payload, json=data)
print(response.content)
```

### Response
#### Success Response (200)
This endpoint returns a JSON object containing the scraped content and other relevant information.

#### Response Example
```json
{
"args": {},
"data": "{\"key1\":\"value1\",\"key2\":\"value2\"}",
"files": {},
"form": {},
"headers": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US",
"Content-Length": "33",
"Content-Type": "application/octet-stream",
"Host": "httpbin.org",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Upgrade-Insecure-Requests": "1",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; rv:124.0) Gecko/20100101 Firefox/124.0",
"X-Amzn-Trace-Id": "Root=1-661c1f6c-4c8f5f7f312dd7d91700cd06",
"X-Custom-Header": "value"
},
"json": {
"key1": "value1",
"key2": "value2"
},
"origin": "1.2.3.4",
"url": "https://httpbin.org/post"
}
```
```

--------------------------------

### Generate PDF from Webpage

Source: https://scrapingfish.com/docs/pdf

Generates a PDF file from a given webpage URL. The PDF content is returned as bytes.

```APIDOC
## GET /api/v1/

### Description
Generates a PDF file from a given webpage URL. The PDF content is returned as bytes.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL of the webpage to convert to PDF.
- **pdf** (boolean) - Required - Set to `true` to enable PDF generation.

### Request Example
```json
{
  "api_key": "[your API key]",
  "url": "https://www.example.com",
  "pdf": true
}
```

### Response
#### Success Response (200)
- **bytes** - The PDF content of the webpage.
```

--------------------------------

### Accessing Response Headers with Python

Source: https://scrapingfish.com/docs/headers

Use this Python snippet to retrieve all response headers, including those from the original website prefixed with 'Sf-'.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "http://google.com",
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.headers)
```

--------------------------------

### Get Response from Intercepted Request

Source: https://scrapingfish.com/docs/intercept-request

Intercept an XHR/Fetch request and retrieve its response. Scraping Fish automatically waits for the request to complete. Use the `intercept_request` query parameter with a pattern to match the desired request.

```APIDOC
## GET /api/v1/

### Description
Intercepts an XHR/Fetch request and returns its response. The API waits for the request matching the provided pattern to complete and then forwards its response.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL to scrape.
- **intercept_request** (string) - Required - A pattern to match the XHR/Fetch request you want to intercept. Example: `**/spec**`.

### Request Example
```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "http://httpbin.org",
  "intercept_request": "**/spec*",
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.json())
```

### Response
#### Success Response (200)
- The response content is the direct response from the intercepted background request.
```

--------------------------------

### Screenshot with HTML Content

Source: https://scrapingfish.com/docs/screenshot

Obtain both the screenshot and HTML content by setting `screenshot_base64` to true. The response is a JSON object containing `content` and `screenshot` keys.

```APIDOC
## GET /api/v1/

### Description
Retrieves both the HTML content and a base64-encoded screenshot of the specified URL.

### Method
GET

### Endpoint
/api/v1/

### Parameters
#### Query Parameters
- **api_key** (string) - Required - Your Scraping Fish API key.
- **url** (string) - Required - The URL of the webpage to capture.
- **screenshot_base64** (boolean) - Optional - Set to `true` to receive both HTML content and a base64-encoded screenshot.

### Request Example
```
GET https://api.scrapingfish.com/api/v1/?api_key=[your API key]&url=https://www.example.com&screenshot_base64=true
```

### Response
#### Success Response (200)
- **content** (string) - The HTML content of the webpage.
- **screenshot** (string) - The base64-encoded screenshot image.
```

--------------------------------

### Forwarding Original Status Code with Python

Source: https://scrapingfish.com/docs/headers

This Python snippet demonstrates how to enable and access the original status code from the website using the `forward_original_status` parameter. The original status code is available in the `Sf-Original-Status-Code` header.

```python
import requests

payload = {
  "api_key": "[your API key]",
  "url": "https://httpbin.org/status/202",
  "forward_original_status": True,
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.headers)
```

--------------------------------

### URL Encoding with cURL for Query Parameters

Source: https://scrapingfish.com/docs/url-encoding

When using `cURL`, it is recommended to always encode your URL, especially when it contains query parameters. Use `--data-urlencode` to pass parameters.

```bash
curl -G --data-urlencode 'url=https://example.com?example=param&second=parameter' \
'https://api.scrapingfish.com/api/v1/?api_key=[your API key]'
```

--------------------------------

### Get API Request Usage Status

Source: https://scrapingfish.com/docs/usage

Call the /usage endpoint with your API key to retrieve the current status of your API request packs. This returns a list of valid API request packs sorted by expiration date.

```python
import requests

response = requests.get("https://api.scrapingfish.com/api/v1/usage/?api_key=[your API key]")
print(response.json())
```

--------------------------------

### Click an element

Source: https://scrapingfish.com/docs/js-scenario

Use this action to click on an element identified by a CSS selector. Ensure the selector accurately targets the desired element.

```json
{
  "steps": [
    {"click": "#a-button"}
  ]
}
```

--------------------------------

### Click and wait for navigation

Source: https://scrapingfish.com/docs/js-scenario

Performs a click on a specified element and then waits for the subsequent page navigation to complete before proceeding. This is useful when a click triggers a new page load.

```json
{
  "steps": [
    {"click_and_wait_for_navigation": "#a-button"}
  ]
}
```

--------------------------------

### Execute custom JavaScript

Source: https://scrapingfish.com/docs/js-scenario

Allows execution of arbitrary JavaScript code within the browser context. Use this for custom logic not covered by predefined actions.

```javascript
{
  "steps": [
    {
      "evaluate": "console.log('Hello from Scraping Fish!')"
    }
  ]
}
```

--------------------------------

### Create Nested Output Structure

Source: https://scrapingfish.com/docs/extract-rules

Shows how to define nested extraction rules to create a hierarchical JSON output from nested HTML elements.

```JSON
{
  "extract_rules": {
    "items": {
      "type": "all",
      "selector": ".item",
      "output": {
        "price": ".price",
        "date": ".date",
        "details": {
          "selector": ".details",
          "output": {
            "title": ".title",
            "description": ".description"
          }
        }
      }
    }
  }
}
```

--------------------------------

### JS Scenario Steps Definition

Source: https://scrapingfish.com/docs/js-scenario

This JSON structure defines a sequence of steps for the JS Scenario. It includes waiting for an element, selecting an option from a dropdown, and clicking a button.

```json
{
  "steps": [
    {
      "wait_for": "#button-id"
    },
    {
      "select": {
        "selector": "#select-id",
        "options": "value1"
      }
    },
    {
      "click": "#button-id"
    }
  ]
}
```

--------------------------------

### Preload localStorage Schema

Source: https://scrapingfish.com/docs/preload-local-storage

Defines the structure for the `preload_local_storage` parameter, specifying origins and their corresponding localStorage key-value pairs.

```json
[
  {
    "origin": "<origin for which the following localStorage key/values should be preloaded>"
    "localStorage": [
      {
        "name": "<a key>",
        "value": "<a value>"
      },
      {
        "name": "<another key>",
        "value": "<another value>"
      }
    ]
  },
  {
    "origin": "<another origin>"
    "localStorage": [
      {
        "name": "<a key>",
        "value": "<a value>"
      },
      {
        "name": "<another key>",
        "value": "<another value>"
      }
    ]
  },
]

```

--------------------------------

### Extract All Links

Source: https://scrapingfish.com/docs/extract-rules

Demonstrates extracting all hyperlink URLs from a webpage using the extended rule form with `type: "all"` and `output: "@href"`.

```Python
import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({
    "links": {"type": "all","selector": "a", "output": "@href"}
  }),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

```cURL
GET
/api/v1/

import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({
    "links": {"type": "all","selector": "a", "output": "@href"}
  }),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```

```JavaScript
GET
/api/v1/

import requests
import json

payload = {
  "api_key": "[your API key]",
  "url": "https://example.com",
  "extract_rules": json.dumps({
    "links": {"type": "all","selector": "a", "output": "@href"}
  }),
}

response = requests.get("https://api.scrapingfish.com/api/v1/", params=payload)
print(response.content)
```
Tokens:
Raw