Scrapy ZenRows Middleware

https://github.com/swistaczek/scrapy-zenrows-middleware
Admin
A Scrapy middleware that integrates Scrapy with the ZenRows Scraper API, enabling easy web scraping...
Tokens:15,069
Snippets:111
Trust Score:8.9
License:MIT
Update:9 months ago
Show doc for...
Tokens:
Raw
### Install scrapy-zenrows

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Installs the necessary scrapy-zenrows library. Ensure Python and pip are set up correctly before running.

```bash
pip install scrapy-zenrows
```

--------------------------------

### ZenRows Scrapy Middleware - Example Configurations

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

Provides example configurations for common scraping scenarios, including high-performance, session-based, and AI/LLM pipelines.

```APIDOC
## Example Configurations

### High-Performance Scraping

Optimize scraping speed by blocking resources and enabling retries.

```python
# settings.py
ZENROWS_BLOCK_RESOURCES = "image,media,font,stylesheet"
USE_ZENROWS_PREMIUM_PROXY = True
ZENROWS_RETRY_ENABLED = True
ZENROWS_MAX_RETRIES = 5
```

### Session-Based Scraping

Maintain a consistent IP address for a user session using `session_id`.

```python
# In your spider
from scrapy_zenrows import ZenRowsRequest

# ...

    yield ZenRowsRequest(
        url="https://example.com/login",
        params={
            "session_id": 12345,
            "premium_proxy": "true",
            "js_render": "true",
        },
        callback=self.after_login,
    )
```

### AI/LLM Pipeline

(This section is incomplete in the provided text and requires further details for a full example.)

```

--------------------------------

### Run Scrapy Spider

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Executes a specific Scrapy spider from the command line. This is used to initiate the scraping process.

```bash
scrapy crawl session_spider
```

--------------------------------

### Scrapy ZenRows Middleware Usage Examples

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

This section provides links to example Scrapy spiders demonstrating various use cases and features of the `scrapy-zenrows-middleware`, such as basic usage, concurrent scraping, custom headers, pagination, screenshots, table parsing, session management, dynamic content handling, and optimized scraping.

```APIDOC
## Scrapy ZenRows Middleware Usage Examples

### Description
This section provides links to example spider implementations showcasing different ways to utilize the `scrapy-zenrows-middleware` with the ZenRows Scraper API.

### Method
N/A (Examples of Scrapy spider configurations)

### Endpoint
N/A

### Parameters
N/A

### Request Example
N/A

### Response
N/A

### Usage Examples
- **Antibot Bypass:** Demonstrates basic usage for bypassing anti-bot measures.
  - Link: [antibot_bypass_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/antibot_bypass_spider.py)
- **Concurrent E-commerce Scraping:** Shows concurrent scraping using `ZenRowsRequest`.
  - Link: [concurrent_ecommerce_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/concurrent_ecommerce_spider.py)
- **Custom Headers:** Illustrates specifying custom headers, including cookies.
  - Link: [custom_headers_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/custom_headers_spider.py)
- **Pagination:** Demonstrates implementing pagination within Scrapy.
  - Link: [pagination_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/pagination_spider.py)
- **Screenshot Capability:** Shows how to add screenshot functionality.
  - Link: [screenshot_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/screenshot_spider.py)
- **Table Parsing:** Examples for parsing tables using the `outputs` feature.
  - Link: [table_parsing_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/table_parsing_spider.py)
- **Session-Based Scraping (v1.1.0+):** Multi-step scraping with IP persistence using `session_id`.
  - Link: [session_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/session_spider.py)
- **Dynamic Content Scraping (v1.1.0+):** Scraping SPAs and dynamic content using `wait_for`.
  - Link: [dynamic_content_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/dynamic_content_spider.py)
- **Optimized Scraping (v1.1.0+):** High-performance scraping with `block_resources`.
  - Link: [optimized_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/optimized_spider.py)
- **AI/LLM Pipeline Extraction (v1.1.0+):** Extracting content for AI/LLM pipelines using `response_type`.
  - Link: [ai_pipeline_spider.py](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/ai_pipeline_spider.py)

### Further Information
- **Full Examples Directory:** [examples](https://github.com/ZenRows/scrapy-zenrows-middleware/blob/main/examples/examples/examples/spiders/)
- **Official Scrapy Integration Documentation:** [docs.zenrows.com](https://docs.zenrows.com/scraping-api/integrations/scrapy)
- **Full Feature Roadmap:** [ROADMAP.md](../docs/ROADMAP.md)

```

--------------------------------

### ZenRows Scrapy Middleware - Migration and Settings

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

Guides users on migrating from v1.0.0 and details new optional settings for enhanced features.

```APIDOC
## Migration from v1.0.0 & New Settings

### Breaking Changes

None. v1.1.0 is fully backward compatible.

### New Settings

Add these optional settings to `settings.py` to enable new features:

- **`ZENROWS_SESSION_ID`**: `None` (int: 1-99999) - For session persistence and maintaining the same IP.
- **`USE_ZENROWS_ORIGINAL_STATUS`**: `False` - To get the original HTTP status from the target.
- **`ZENROWS_BLOCK_RESOURCES`**: `None` (str: "image,media,font,stylesheet") - To block resources globally and speed up scraping.
- **`ZENROWS_ALLOWED_STATUS_CODES`**: `None` (str: "404,500,503") - To allow content retrieval from error pages.
- **Retry Middleware Settings**:
    - `ZENROWS_RETRY_ENABLED`: `True`
    - `ZENROWS_MAX_RETRIES`: `3`
    - `ZENROWS_RETRY_BACKOFF`: `1.0`
    - `ZENROWS_RETRY_STATUS_CODES`: `[429, 500, 502, 503, 504]`

### New Middleware Registration

To enable the retry middleware with exponential backoff, update `DOWNLOADER_MIDDLEWARES`:

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,  # NEW
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}
```
```

--------------------------------

### Run Scrapy Spider with JSON Output

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Runs a Scrapy spider and saves the scraped data to a JSON file. Useful for data analysis and further processing.

```bash
scrapy crawl ai_pipeline_spider -o output.json
```

--------------------------------

### Set ZenRows API Key

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Sets the ZenRows API key as an environment variable. This is crucial for authentication when using the middleware.

```bash
export ZENROWS_API_KEY="your_api_key_here"
```

--------------------------------

### ZenRows Request Parameters

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Applies ZenRows configurations on a per-request basis using the `params` dictionary within `ZenRowsRequest`. This provides granular control over scraping parameters.

```python
yield ZenRowsRequest(
    url="...",
    params={"premium_proxy": "true", "js_render": "true"},
)
```

--------------------------------

### Optimize Scraping with Block Resources

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This example illustrates how to block unnecessary resources like images, media, and fonts to improve scraping speed and reduce costs. It also shows the use of `wait_for` to ensure the page content is loaded before proceeding.

```python
params = {
    "js_render": "true",
    "block_resources": "image,media,font",
    "wait_for": ".main-content",
}
```

--------------------------------

### Session Spider Request

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Demonstrates making a request with session persistence using `session_id` and a premium proxy. This is useful for handling multi-step processes like login flows.

```python
yield ZenRowsRequest(
    url="https://example.com/login",
    params={"session_id": 12345, "premium_proxy": "true"},
)
```

--------------------------------

### Optimized Spider Request

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Enables faster scraping by blocking specific resource types like images and stylesheets. Reduces bandwidth usage and speeds up page load.

```python
yield ZenRowsRequest(
    url="https://example.com",
    params={"block_resources": "image,font,media,stylesheet"},
)
```

--------------------------------

### Maintain IP Consistency with Sessions

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This example demonstrates using sessions for multi-step flows, such as login processes. By using the same `SESSION_ID` across multiple `ZenRowsRequest` calls, you maintain IP consistency, which is crucial for stateful operations.

```python
SESSION_ID = 12345  # Use same ID across requests

yield ZenRowsRequest(url="/login", params={"session_id": SESSION_ID})
yield ZenRowsRequest(url="/dashboard", params={"session_id": SESSION_ID})
yield ZenRowsRequest(url="/export", params={"session_id": SESSION_ID})
```

--------------------------------

### Dynamic Content Spider Request

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Scrapes JavaScript-rendered content by enabling `js_render` and specifying a `wait_for` selector. Useful for single-page applications (SPAs).

```python
yield ZenRowsRequest(
    url="https://example.com/spa",
    params={"js_render": "true", "wait_for": ".content-loaded"},
)
```

--------------------------------

### Scrapy Spider Custom Settings

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Configures ZenRows middleware settings globally for a Scrapy spider using `custom_settings`. This allows overriding defaults like API key and retry counts.

```python
class MySpider(scrapy.Spider):
    custom_settings = {
        "ZENROWS_API_KEY": "...",
        "USE_ZENROWS_PREMIUM_PROXY": True,
        "ZENROWS_MAX_RETRIES": 5,
    }
```

--------------------------------

### AI Pipeline Spider Request

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/examples/README.md

Fetches clean content suitable for LLM processing, available in markdown or plaintext format using `response_type`. Ideal for natural language processing tasks.

```python
yield ZenRowsRequest(
    url="https://example.com/article",
    params={"response_type": "markdown"},
)
```

--------------------------------

### Get Clean Markdown for AI Processing

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This snippet demonstrates how to fetch content in markdown format for AI processing. It utilizes ZenRowsRequest with specific parameters to block resources and specify the response type. This is useful for extracting structured text data from web pages.

```python
yield ZenRowsRequest(
    url="https://blog.example.com/article",
    params={
        "response_type": "markdown",
        "block_resources": "image,media,font",
    },
    callback=self.process_for_llm,
)
```

--------------------------------

### Implement Session-Based Scraping with ZenRowsRequest

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This example demonstrates how to use `ZenRowsRequest` within a Scrapy spider to leverage session-based scraping. By passing `session_id`, `premium_proxy`, and `js_render` in the `params` dictionary, you can maintain the same IP address for a period and enable JavaScript rendering for the target URL. The response is handled by the `after_login` callback.

```python
# In spider
from scrapy_zenrows import ZenRowsRequest

yield ZenRowsRequest(
    url="https://example.com/login",
    params={
        "session_id": 12345,
        "premium_proxy": "true",
        "js_render": "true",
    },
    callback=self.after_login,
)
```

--------------------------------

### Configure High-Performance Scraping with ZenRows

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This configuration in `settings.py` optimizes scraping for performance. It blocks non-essential resources like images and stylesheets, enables premium proxies, and configures the retry middleware with an increased number of max retries. This setup is ideal for scenarios where speed and reliability are critical.

```python
# settings.py
ZENROWS_BLOCK_RESOURCES = "image,media,font,stylesheet"
USE_ZENROWS_PREMIUM_PROXY = True
ZENROWS_RETRY_ENABLED = True
ZENROWS_MAX_RETRIES = 5
```

--------------------------------

### Maintain IP Persistence Across Requests with ZenRows

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

This Python example showcases session-based scraping using `ZenRowsRequest` to maintain a consistent IP address across multiple requests. This is crucial for scenarios like login flows or multi-step processes where session integrity is required. The `session_id` parameter ensures that ZenRows uses the same proxy for the duration of the session.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class SessionSpider(scrapy.Spider):
    name = "session_spider"
    SESSION_ID = 12345  # Integer between 1-99999, persists for 10 minutes

    def start_requests(self):
        # Step 1: Access login page
        yield ZenRowsRequest(
            url="https://example.com/login",
            params={
                "session_id": self.SESSION_ID,
                "premium_proxy": "true",
                "js_render": "true",
            },
            callback=self.parse_login,
        )

    def parse_login(self, response):
        # Step 2: Submit login form (same IP maintained)
        yield ZenRowsRequest(
            url="https://example.com/dashboard",
            params={
                "session_id": self.SESSION_ID,  # Same session = same IP
                "premium_proxy": "true",
                "custom_headers": "true",
            },
            cookies={
                "session_token": "abc123",
            },
            callback=self.parse_dashboard,
        )

    def parse_dashboard(self, response):
        # Step 3: Access protected content (still same IP)
        user_data = response.css('.user-info::text').getall()

        yield {
            'user_data': user_data,
            'session_id': self.SESSION_ID,
        }
```

--------------------------------

### Handle Errors and Get Original Status Codes with ZenRowsRequest

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

This Python snippet demonstrates how to use `ZenRowsRequest` to handle error pages and retrieve original HTTP status codes from target websites. It configures ZenRows to allow specific error codes and retrieve the target's actual status, with a dedicated error callback for handling failures.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class ErrorHandlingSpider(scrapy.Spider):
    name = "error_handling_spider"

    def start_requests(self):
        yield ZenRowsRequest(
            url="https://example.com/may-not-exist",
            params={
                "original_status": "true",        # Get target's actual status
                "allowed_status_codes": "404,500,503",  # Allow error codes
                "premium_proxy": "true",
            },
            callback=self.parse,
            errback=self.handle_error,
        )

    def parse(self, response):
        # Response is returned even if target returned 404/500
        yield {
            "url": response.url,
            "status": response.status,
            "content": response.text[:200],
        }

    def handle_error(self, failure):
        self.logger.error(f"Request failed: {failure.value}")
```

--------------------------------

### ZenRows Scrapy Middleware - Complete Parameter Reference

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

A comprehensive reference of all available parameters for configuring ZenRows Scrapy Middleware, categorized by functionality.

```APIDOC
## Complete Parameter Reference

### Page Loading Parameters

- **`js_render`** (bool, default: `false`): Enable JavaScript rendering.
- **`wait`** (int, default: `0`): Wait milliseconds after page load.
- **`wait_for`** (str, default: `-`): Wait for a CSS selector to appear in the DOM.
- **`block_resources`** (str, default: `-`): Block resources like image, media, font, stylesheet, script.
- **`js_instructions`** (str, default: `-`): JSON array of JavaScript actions to execute.

### Proxy Settings

- **`premium_proxy`** (bool, default: `false`): Use residential IPs for proxies.
- **`proxy_country`** (str, default: `-`): Specify proxy country code (e.g., 'us', 'de', 'jp').
- **`proxy_city`** (str, default: `-`): Specify proxy city (requires `premium_proxy`).
- **`session_id`** (int, default: `-`): Maintain the same IP for a session (1-99999, max 10 minutes).

### Data Extraction Parameters

- **`autoparse`** (bool, default: `false`): Automatically extract structured data.
- **`css_extractor`** (str, default: `-`): CSS selectors in JSON format for data extraction.
- **`outputs`** (str, default: `-`): Specify data types to extract (e.g., 'tables', 'emails', 'hashtags').
- **`response_type`** (str, default: `-`): Set response format to 'markdown', 'plaintext', or 'pdf'.
- **`json_response`** (bool, default: `false`): Capture XHR/Fetch requests as JSON.

### Screenshots Parameters

- **`screenshot`** (bool, default: `false`): Capture an above-the-fold screenshot.
- **`screenshot_fullpage`** (bool, default: `false`): Capture a full-page screenshot.
- **`screenshot_selector`** (str, default: `-`): Capture a screenshot of a specific element identified by a CSS selector.
- **`screenshot_format`** (str, default: 'png'): Image format for screenshots ('png' or 'jpeg').
- **`screenshot_quality`** (int, default: `-`): JPEG quality for screenshots (1-100).

### Response Handling Parameters

- **`original_status`** (bool, default: `false`): Return the target URL's original HTTP status code.
- **`allowed_status_codes`** (str, default: `-`): Retrieve content even for specified error status codes.
- **`custom_headers`** (bool, default: `false`): Enable the use of custom headers.
```

--------------------------------

### Advanced JavaScript Interactions with Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Execute custom JavaScript instructions on a webpage before scraping content. This allows for dynamic interactions like clicking buttons, waiting for elements, and scrolling. The 'js_instructions' parameter takes a JSON array of commands to be executed.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class JSInteractionSpider(scrapy.Spider):
    name = "js_interaction_spider"

    def start_requests(self):
        yield ZenRowsRequest(
            url="https://example.com/products",
            params={
                "js_render": "true",
                "premium_proxy": "true",
                "js_instructions": '''[
                    {"click": ".load-more-btn"},
                    {"wait": 1000},
                    {"click": ".load-more-btn"},
                    {"wait": 1000},
                    {"scroll": 500}
                ]''',
            },
            callback=self.parse,
        )

    def parse(self, response):
        # Parse content loaded via JavaScript interactions
        products = response.css(".product")
        for product in products:
            yield {
                "name": product.css(".product-name::text").get(),
                "price": product.css(".product-price::text").get(),
            }
```

--------------------------------

### ZenRows Scraper API Parameters

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

This section details the various parameters available for configuring the ZenRows Scraper API when used with the scrapy-zenrows-middleware. These parameters allow for fine-grained control over proxy selection, JavaScript rendering, resource blocking, session management, and output formatting.

```APIDOC
## ZenRows Scraper API Parameters

### Description
Reference for all available parameters to configure the ZenRows Scraper API within the scrapy-zenrows-middleware.

### Method
N/A (Configuration parameters for middleware)

### Endpoint
N/A

### Parameters
#### Query Parameters
- **js_render** (bool) - Optional - Enable JavaScript rendering.
- **premium_proxy** (bool) - Optional - Use residential IPs.
- **proxy_country** (str) - Optional - Country code (e.g., 'us', 'de', 'jp').
- **proxy_city** (str) - Optional - City name (requires `premium_proxy`).
- **session_id** (int) - Optional - Maintain IP for a session (1-99999, lasts 10 minutes).
- **wait** (int) - Optional - Wait in milliseconds after the page has loaded.
- **wait_for** (str) - Optional - Wait for a specific CSS selector to be present.
- **block_resources** (str) - Optional - Block specific resource types (e.g., 'image,media,font').
- **js_instructions** (str) - Optional - JSON array of JavaScript execution instructions.
- **autoparse** (bool) - Optional - Automatically extract structured data from the page.
- **css_extractor** (str) - Optional - CSS selectors to extract data, provided as a JSON string.
- **outputs** (str) - Optional - Specify data types to extract (e.g., 'tables', 'emails'). See [supported outputs](https://docs.zenrows.com/scraper-api/features/output).
- **response_type** (str) - Optional - Desired response format (e.g., 'markdown', 'plaintext', 'pdf').
- **json_response** (bool) - Optional - Capture XHR/Fetch requests as JSON.
- **screenshot** (bool) - Optional - Capture a screenshot of the page.
- **screenshot_fullpage** (bool) - Optional - Capture a full-page screenshot.
- **screenshot_selector** (str) - Optional - Capture a screenshot of a specific element identified by a CSS selector.
- **screenshot_format** (str) - Optional - Image format for the screenshot ('png' or 'jpeg').
- **screenshot_quality** (int) - Optional - JPEG quality setting (1-100).
- **original_status** (bool) - Optional - Return the original HTTP status code of the target URL.
- **allowed_status_codes** (str) - Optional - Retrieve content even if the page returns an error status code.
- **custom_headers** (bool) - Optional - Enable the use of custom headers.

### Request Example
N/A (These are configuration parameters, not a direct request body)

### Response
N/A (These are configuration parameters, not a direct response)

```

--------------------------------

### Configure Global Settings for Scrapy ZenRows

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Configure global settings for the ZenRows middleware in your Scrapy project's settings.py file. This includes enabling the middleware and setting your ZenRows API key.

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}

# ZenRows API Key
ZENROWS_API_KEY = "<YOUR_ZENROWS_API_KEY>"
```

--------------------------------

### Enable Retry Middleware for Production

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This configuration snippet shows how to enable the `ZenRowsRetryMiddleware` for production environments. It includes settings for maximum retries and retry backoff, ensuring resilience against transient network issues.

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}
ZENROWS_MAX_RETRIES = 3
ZENROWS_RETRY_BACKOFF = 1.0
```

--------------------------------

### Configure Global ZenRows Settings in Scrapy

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

This Python snippet shows how to configure ZenRows middleware globally within a Scrapy project's `settings.py` file. It includes settings for enabling the middleware, API key configuration, global ZenRows parameters like proxy usage and JS rendering, and retry settings.

```python
# settings.py
import os

BOT_NAME = "my_scraping_project"
SPIDER_MODULES = ["my_project.spiders"]

# Enable ZenRows middleware
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}

# ZenRows API configuration
ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY", "your_api_key")

# Global ZenRows settings (can be overridden per-request)
USE_ZENROWS_PREMIUM_PROXY = True          # Enable proxies globally
USE_ZENROWS_JS_RENDER = True              # Enable JS rendering globally
USE_ZENROWS_ORIGINAL_STATUS = False       # Return original status codes
ZENROWS_SESSION_ID = None                 # Default session ID (1-99999)
ZENROWS_BLOCK_RESOURCES = "image,font"    # Resources to block globally
ZENROWS_ALLOWED_STATUS_CODES = None       # Status codes to allow globally

# Retry settings
ZENROWS_RETRY_ENABLED = True
ZENROWS_MAX_RETRIES = 3
ZENROWS_RETRY_BACKOFF = 1.0
ZENROWS_RETRY_STATUS_CODES = [429, 500, 502, 503, 504]

# Standard Scrapy settings
ROBOTSTXT_OBEY = False
LOG_LEVEL = "INFO"
CONCURRENT_REQUESTS = 16
REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
```

--------------------------------

### Reliable Dynamic Content Waiting with wait_for

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This snippet compares the reliability of `wait_for` with a fixed `wait` for handling dynamic content. `wait_for` is recommended as it waits for a specific element to appear, making scraping more robust.

```python
# Better - waits for specific element
params = {"wait_for": ".product-list"}

# Less reliable - fixed delay
params = {"wait": "5000"}
```

--------------------------------

### Enable Global Premium Proxy and JS Rendering in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Globally enable premium proxy and JavaScript rendering for all requests processed by the ZenRows middleware. These settings are false by default.

```python
# ...

USE_ZENROWS_PREMIUM_PROXY = True # to turn on premium proxy (False by default)
USE_ZENROWS_JS_RENDER = True # to turn on JS rendering (False by default)
```

--------------------------------

### Bypass Anti-Bot Protection with ZenRowsRequest

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

This Python code demonstrates how to make a single request using `ZenRowsRequest` to bypass anti-bot challenges. It utilizes ZenRows' premium proxies and JavaScript rendering capabilities, with specific instructions to wait after the page loads. The middleware handles the complexities of anti-bot detection.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class AntiBotSpider(scrapy.Spider):
    name = "antibot_spider"

    def start_requests(self):
        yield ZenRowsRequest(
            url="https://www.scrapingcourse.com/antibot-challenge",
            params={
                "js_render": "true",       # Enable JavaScript rendering
                "premium_proxy": "true",   # Use residential proxy
                "custom_headers": "true",  # Enable custom headers
                "js_instructions": '[{"wait": 500}]',  # Wait 500ms after load
            },
            headers={
                "Referer": "https://www.google.com/",
            },
            callback=self.parse,
        )

    def parse(self, response):
        # Extract data from the protected page
        title = response.css('h1::text').get()
        content = response.css('.content::text').getall()

        yield {
            'title': title,
            'content': content,
            'status': response.status,
        }
```

--------------------------------

### Configure ZenRows Retry Middleware in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Shows how to enable and configure the `ZenRowsRetryMiddleware` in Scrapy's `settings.py`. This middleware provides robust scraping by implementing exponential backoff for retrying requests. Key settings include enabling/disabling retries, setting maximum retry attempts, configuring the backoff factor, and specifying which HTTP status codes should trigger a retry.

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,  # Add retry middleware
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}

# Retry settings (all optional)
ZENROWS_RETRY_ENABLED = True  # Enable/disable retry (default: True)
ZENROWS_MAX_RETRIES = 3  # Max retry attempts (default: 3)
ZENROWS_RETRY_BACKOFF = 1.0  # Backoff factor in seconds (default: 1.0)
ZENROWS_RETRY_STATUS_CODES = [429, 500, 502, 503, 504]  # Status codes to retry
```

--------------------------------

### Retry Middleware Configuration with Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Configure automatic request retries with exponential backoff for failed HTTP requests. This is achieved by adding the ZenRowsRetryMiddleware to Scrapy's settings and defining retry-related parameters such as maximum retries and backoff factor.

```python
# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,  # Add retry middleware
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}

# Retry configuration
ZENROWS_RETRY_ENABLED = True               # Enable/disable retry
ZENROWS_MAX_RETRIES = 3                    # Max retry attempts
ZENROWS_RETRY_BACKOFF = 1.0               # Backoff factor (seconds)
ZENROWS_RETRY_STATUS_CODES = [429, 500, 502, 503, 504]  # Status codes to retry

# The middleware automatically retries with exponential backoff:
# 1st retry: 1.0 * (2^0) = 1 second
# 2nd retry: 1.0 * (2^1) = 2 seconds
# 3rd retry: 1.0 * (2^2) = 4 seconds
```

--------------------------------

### Content Transformation for AI/LLM Pipelines with Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Convert HTML content to markdown or plaintext formats suitable for AI and NLP processing. This involves using ZenRowsRequest with specific response_type parameters. The output can be directly used for tasks like summarization or general text analysis.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class AIPipelineSpider(scrapy.Spider):
    name = "ai_pipeline_spider"

    def start_requests(self):
        url = "https://quotes.toscrape.com/"

        # Get markdown for LLM processing
        yield ZenRowsRequest(
            url=url,
            params={
                "response_type": "markdown",
                "block_resources": "image,media,font",
            },
            callback=self.parse_markdown,
        )

        # Get plaintext for NLP
        yield ZenRowsRequest(
            url=url,
            params={
                "response_type": "plaintext",
            },
            callback=self.parse_plaintext,
        )

    def parse_markdown(self, response):
        markdown_content = response.text

        yield {
            "format": "markdown",
            "content_length": len(markdown_content),
            "content": markdown_content,
            # Ready for LLM processing
            "llm_prompt": f"Summarize this content:\n\n{markdown_content[:2000]}",
        }

    def parse_plaintext(self, response):
        text_content = response.text
        words = text_content.split()

        yield {
            "format": "plaintext",
            "word_count": len(words),
            "content": text_content,
        }
```

--------------------------------

### Dynamic Content Scraping with wait_for in Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Handles SPAs and AJAX-loaded content by waiting for specific CSS selectors to appear in the DOM before extracting data. This is useful for sites that render content dynamically after the initial page load. It requires the `scrapy-zenrows` library.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class DynamicContentSpider(scrapy.Spider):
    name = "dynamic_content_spider"

    def start_requests(self):
        # Wait for specific CSS selector to load
        yield ZenRowsRequest(
            url="https://quotes.toscrape.com/js/",
            params={
                "js_render": "true",
                "wait_for": ".quote",  # Wait for quotes to appear in DOM
                "block_resources": "image,font,stylesheet",  # Speed optimization
                "premium_proxy": "true",
            },
            callback=self.parse_quotes,
        )

    def parse_quotes(self, response):
        # Extract dynamically loaded content
        quotes = response.css(".quote")

        for quote in quotes:
            yield {
                "text": quote.css(".text::text").get(),
                "author": quote.css(".author::text").get(),
                "tags": quote.css(".tag::text").getall(),
            }

        # Handle pagination with same parameters
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield ZenRowsRequest(
                url=response.urljoin(next_page),
                params={
                    "js_render": "true",
                    "wait_for": ".quote",
                    "block_resources": "image,font,stylesheet",
                },
                callback=self.parse_quotes,
            )

```

--------------------------------

### Geolocation-Specific Scraping with Targeted Proxies in Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Performs web scraping from specific geographic locations (countries or cities) by utilizing targeted proxy settings. This allows for testing geo-specific content or bypassing geo-restrictions. It requires the `scrapy-zenrows` library and premium proxy access.

```python
import scrapy
from scrapy_zenrows import ZenRowsRequest

class GeoLocationSpider(scrapy.Spider):
    name = "geolocation_spider"

    def start_requests(self):
        # Scrape from US location
        yield ZenRowsRequest(
            url="https://www.google.com/search?q=weather",
            params={
                "premium_proxy": "true",
                "proxy_country": "us",     # Country code: us, de, jp, etc.
                "js_render": "true",
            },
            callback=self.parse,
            meta={"country": "US"},
        )

        # Scrape from specific city
        yield ZenRowsRequest(
            url="https://www.google.com/search?q=weather",
            params={
                "premium_proxy": "true",
                "proxy_country": "us",
                "proxy_city": "new york",  # Requires premium_proxy
                "js_render": "true",
            },
            callback=self.parse,
            meta={"country": "US", "city": "New York"},
        )

    def parse(self, response):
        yield {
            "location": response.meta.get("city") or response.meta.get("country"),
            "content": response.css("body::text").get()[:500],
        }

```

--------------------------------

### Set Custom Headers with ZenRowsRequest in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Demonstrates how to set custom headers, like 'Referer', when making requests using ZenRowsRequest in a Scrapy spider. The `custom_headers` parameter must be set to 'true' to enable this functionality. This allows for fine-grained control over request headers beyond what ZenRows manages automatically.

```python
# ...
class YourSpider(scrapy.Spider):
    # ...

    def start_requests(self):
        # use ZenRowsRequest for customization
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                # overrides the settings config for this specific spider
                params={
                    "custom_headers": "true",  # to use custom headers
                },
                # add a referer header
                headers={
                    "Referer": "https://www.google.com/",
                },
            )
```

--------------------------------

### ZenRows Scrapy Middleware - Version 1.1.0 Features

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

Details new API parameters and the introduction of the Retry Middleware in version 1.1.0.

```APIDOC
## ZenRows Scrapy Middleware - Version 1.1.0 Features

### New API Parameters

- **session_id** (int 1-99999): Maintain the same IP for up to 10 minutes. Global Setting: `ZENROWS_SESSION_ID`.
- **wait_for** (str CSS selector): Wait for an element to appear in the DOM.
- **original_status** (bool): Return the target's original HTTP status. Global Setting: `USE_ZENROWS_ORIGINAL_STATUS`.
- **block_resources** (str): Block specific resources (e.g., image, media, font). Global Setting: `ZENROWS_BLOCK_RESOURCES`.
- **response_type** (str): Convert the response to markdown, plaintext, or PDF.
- **allowed_status_codes** (str): Retrieve content from error pages. Global Setting: `ZENROWS_ALLOWED_STATUS_CODES`.
- **screenshot_selector** (str): Take a screenshot of a specific element.
- **proxy_city** (str): Use city-level geolocation for proxies.

### Retry Middleware

Introduces `ZenRowsRetryMiddleware` with exponential backoff.

#### Configuration Example

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}

ZENROWS_RETRY_ENABLED = True
ZENROWS_MAX_RETRIES = 3
ZENROWS_RETRY_BACKOFF = 1.0
ZENROWS_RETRY_STATUS_CODES = [429, 500, 502, 503, 504]
```

### Parameter Validation

Automatic validation warnings for:
- `session_id` range (1-99999).
- `screenshot_selector` and `screenshot_fullpage` mutual exclusion.
- `proxy_city` requires `premium_proxy`.
- `screenshot` requires `js_render`.
```

--------------------------------

### Block Unnecessary Resources with block_resources in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Shows how to use the `block_resources` parameter in `ZenRowsRequest` to speed up scraping by preventing the loading of specific resource types like images, media, fonts, or stylesheets. This can significantly reduce bandwidth usage and improve parsing efficiency.

```python
yield ZenRowsRequest(
    url="https://example.com",
    params={
        "js_render": "true",
        "block_resources": "image,media,font,stylesheet",
    },
)
```

--------------------------------

### Enable ZenRows Features with Scrapy Settings

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

These settings configure various features of the ZenRows Scrapy middleware. They include enabling session persistence with `ZENROWS_SESSION_ID`, retrieving original HTTP status codes, blocking resources to speed up scraping, allowing specific error status codes, and configuring retry middleware settings. These settings are typically placed in Scrapy's `settings.py` file.

```python
# settings.py

# Session persistence (maintains same IP)
ZENROWS_SESSION_ID = None  # int: 1-99999

# Get original HTTP status from target
USE_ZENROWS_ORIGINAL_STATUS = False

# Block resources globally (speeds up scraping)
ZENROWS_BLOCK_RESOURCES = None  # "image,media,font,stylesheet"

# Allow specific error status codes
ZENROWS_ALLOWED_STATUS_CODES = None  # "404,500,503"

# Retry middleware settings
ZENROWS_RETRY_ENABLED = True
ZENROWS_MAX_RETRIES = 3
ZENROWS_RETRY_BACKOFF = 1.0
ZENROWS_RETRY_STATUS_CODES = [429, 500, 502, 503, 504]
```

--------------------------------

### Retrieve Content from Error Pages with allowed_status_codes in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Demonstrates the use of the `allowed_status_codes` parameter in `ZenRowsRequest`, which permits scraping content even from pages that return error status codes like 404, 500, or 503. This feature is valuable when you need to inspect error pages or extract information from them.

```python
yield ZenRowsRequest(
    url="https://example.com/may-404",
    params={
        "allowed_status_codes": "404,500,503",
    },
)
```

--------------------------------

### Register ZenRowsRetryMiddleware in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This snippet shows how to register the `ZenRowsRetryMiddleware` in Scrapy's `DOWNLOADER_MIDDLEWARES` setting. By adding it with a specific order (e.g., 550), you enable the retry functionality with exponential backoff for failed requests. Ensure `ZenRowsMiddleware` is also present.

```python
DOWNLOADER_MIDDLEWARES = {
    "scrapy_zenrows.ZenRowsRetryMiddleware": 550,  # NEW
    "scrapy_zenrows.ZenRowsMiddleware": 543,
}
```

--------------------------------

### Wait for Element Visibility with wait_for Parameter in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Explains the use of the `wait_for` parameter in `ZenRowsRequest` to instruct ZenRows to wait until a specific CSS selector appears in the DOM before returning the page content. This is a more reliable method than using a fixed `wait` time for JavaScript-rendered pages.

```python
yield ZenRowsRequest(
    url="https://spa-example.com",
    params={
        "js_render": "true",
        "wait_for": ".product-list",  # Wait for this element
    },
)
```

--------------------------------

### Convert HTML to Other Formats with response_type in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Details the `response_type` parameter for `ZenRowsRequest`, enabling the conversion of HTML responses into formats like Markdown, plain text, or PDF. This is particularly useful for integrating scraped data into AI/LLM pipelines or other content processing workflows.

```python
yield ZenRowsRequest(
    url="https://blog.example.com/article",
    params={
        "response_type": "markdown",  # or "plaintext", "pdf"
    },
)
```

--------------------------------

### Configure Other ZenRows API Parameters with ZenRowsRequest

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Utilize ZenRowsRequest to pass additional parameters to the ZenRows Scraper API for specific requests, such as proxy country, JS instructions, and output formats.

```python
# ...
class YourSpider(scrapy.Spider):
    # ...

    def start_requests(self):
        # use ZenRowsRequest for customization
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                params={
                    # ...,
                    "proxy_country": "ca", # use proxy from a specific country
                    "js_instructions": '[{"wait": 500}]', # pass JS instructions
                    "autparse": "true", # for supported websites
                    "outputs": "tables" # extract specific data,
                    'css_extractor': '{"links":"a @href","images":"img @src"}'
                    ""
                },
            )
```

--------------------------------

### Screenshot Capture during Scraping in Python

Source: https://context7.com/swistaczek/scrapy-zenrows-middleware/llms.txt

Captures screenshots of web pages during scraping. It supports full-page or element-specific screenshots, configurable quality, and format. The screenshot data is returned as base64 encoded JSON. This requires the `scrapy-zenrows` library and standard Python libraries like `base64` and `json`.

```python
import base64
import json
import scrapy
from scrapy_zenrows import ZenRowsRequest

class ScreenshotSpider(scrapy.Spider):
    name = "screenshot_spider"

    def start_requests(self):
        yield ZenRowsRequest(
            url="https://www.scrapingcourse.com/ecommerce/",
            params={
                "js_render": "true",
                "json_response": "true",     # Get JSON response with screenshot
                "screenshot": "true",
                "screenshot_fullpage": "true",
                "screenshot_format": "jpeg",
                "screenshot_quality": "80",  # 1-100 quality
                "wait": "5000",              # Wait 5 seconds before screenshot
            },
            callback=self.parse_screenshot,
        )

    def parse_screenshot(self, response):
        data = json.loads(response.text)
        screenshot_data = data.get("screenshot", {})

        if screenshot_data:
            # Decode base64 screenshot data
            image_data = base64.b64decode(screenshot_data["data"])
            image_type = screenshot_data["type"].split("/")[-1]
            filename = f"screenshot.{image_type}"

            # Save screenshot to file
            with open(filename, "wb") as f:
                f.write(image_data)

            yield {
                "screenshot_filename": filename,
                "dimensions": f"{screenshot_data['width']}x{screenshot_data['height']}",
                "image_type": screenshot_data["type"],
            }

```

--------------------------------

### Add Cookies Header with ZenRowsRequest in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Illustrates how to pass cookies as a meta parameter when using ZenRowsRequest in Scrapy. Similar to custom headers, `custom_headers` must be set to 'true'. This feature allows for managing session cookies or other necessary cookie data for the scraping requests.

```python
# ...
class YourSpider(scrapy.Spider):
    # ...

    def start_requests(self):
        # use ZenRowsRequest for customization
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                # overrides the settings config for this specific spider
                params={
                    "custom_headers": "true",  # to use custom headers
                },
                cookies={
                    "currency": "USD",
                    "country": "UY",
                },
            )
```

--------------------------------

### Maintain IP Address with Session Persistence (session_id) in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Demonstrates how to use the `session_id` parameter with `ZenRowsRequest` to maintain the same IP address across multiple requests for up to 10 minutes. This is useful for scenarios requiring persistent sessions, such as logging into a website. The `session_id` should be an integer between 1 and 99999.

```python
yield ZenRowsRequest(
    url="https://example.com/login",
    params={
        "session_id": 12345,  # Integer 1-99999
        "premium_proxy": "true",
    },
)
```

--------------------------------

### Retrieve Original Status Code with original_status in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Explains how to use the `original_status` parameter with `ZenRowsRequest` to obtain the original HTTP status code from the target page, rather than ZenRows' status code. This is helpful for debugging and understanding the actual response from the server.

```python
yield ZenRowsRequest(
    url="https://example.com",
    params={
        "original_status": "true",
    },
)
```

--------------------------------

### Debug Failed Requests with ZenRows

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/docs/ROADMAP.md

This code snippet shows how to retrieve content even from error pages by configuring ZenRowsRequest to accept specific non-200 status codes. It uses the `original_status` and `allowed_status_codes` parameters to handle potential request failures gracefully.

```python
# Get content even from error pages
yield ZenRowsRequest(
    url="https://example.com/may-fail",
    params={
        "original_status": "true",
        "allowed_status_codes": "404,500,503",
    },
    callback=self.handle_response,
)
```

--------------------------------

### Override Global Settings with ZenRowsRequest in Scrapy

Source: https://github.com/swistaczek/scrapy-zenrows-middleware/blob/main/scrapy_zenrows/README.md

Override global ZenRows settings for specific requests by using ZenRowsRequest. This allows for per-request customization of parameters like premium proxy and JS rendering.

```python
# pip install scrapy-zenrows
from scrapy_zenrows import ZenRowsRequest

class YourSpider(scrapy.Spider):
    # ...

    def start_requests(self):
        # use ZenRowsRequest for customization
        for url in self.start_urls:
            yield ZenRowsRequest(
                url=url,
                # overrides the settings config for this specific spider
                params={
                    "js_render": "true",  # enable JavaScript rendering (if needed)
                    "premium_proxy": "true",  # use the proxy (if needed)
                },
            )
```
Tokens:
Raw