### Install Scrapy-Splash via pip

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This command installs the `scrapy-splash` library using Python's package installer, pip. It's the first step to set up the integration with Scrapy.

```shell
$ pip install scrapy-splash
```

--------------------------------

### Scrapy Spider to Get HTML and Screenshot with SplashRequest

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This example extends `SplashRequest` to retrieve both HTML content and a PNG screenshot using the `render.json` endpoint. It illustrates how to configure `splash_args` for various rendering options and access different data types (HTML, PNG bytes) from `response.body` and `response.data`.

```python
import json
import base64
import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):

    # ...
            splash_args = {
                'wait': 1,
                'html': 1,
                'png': 1,
                'width': 600,
                'render_all': 1,
            }
            yield SplashRequest(url, self.parse_result, endpoint='render.json',
                                args=splash_args)

        # ...
        def parse_result(self, response):
            # magic responses are turned ON by default,
            # so the result under 'html' key is available as response.body
            html = response.body

            # you can also query the html result as usual
            title = response.css('title').extract_first()

            # full decoded JSON data is available as response.data:
            png_bytes = base64.b64decode(response.data['png'])

            # ...
```

--------------------------------

### Run Splash Docker Container

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This command starts a Splash instance using Docker, exposing it on port 8050. A running Splash server is a prerequisite for `scrapy-splash` to function, as it handles JavaScript rendering.

```shell
$ docker run -p 8050:8050 scrapinghub/splash
```

--------------------------------

### Scrapy Spider to Get HTML Content with SplashRequest

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Scrapy spider example shows basic integration of `SplashRequest` to fetch HTML content from multiple URLs. It demonstrates how to yield `SplashRequest` instances with arguments like `wait` and parse the `response.body` which contains the browser-processed HTML.

```python
import scrapy
from scrapy_splash import SplashRequest

class MySpider(scrapy.Spider):
    name = "MySpider"
    start_urls = ["http://example.com", "http://example.com/foo"]

    def start_requests(self):
        for url in self.start_urls:
                yield SplashRequest(url, self.parse, args={'wait': 0.5})

        def parse(self, response):
            # response.body is a result of render.html call; it
            # contains HTML processed by a browser.
            # ...
```

--------------------------------

### Python Scrapy-SplashFormRequest Basic Usage

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Python example shows how to instantiate `scrapy_splash.SplashFormRequest` to send a POST request with form data. It functions similarly to Scrapy's built-in `FormRequest`, accepting a `formdata` dictionary to populate the request body.

```Python
from scrapy_splash import SplashFormRequest
SplashFormRequest('http://example.com', formdata={'foo': 'bar'})
```

--------------------------------

### Create a Scrapy SplashRequest

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This example demonstrates how to use `scrapy_splash.SplashRequest` to send a request to Splash for rendering. It allows passing arguments directly to the Splash HTTP API, specifying endpoints, and overriding the default Splash URL or slot policy.

```python
yield SplashRequest(url, self.parse_result,
    args={
        # optional; parameters passed to Splash HTTP API
        'wait': 0.5,

        # 'url' is prefilled from request url
        # 'http_method' is set to 'POST' for POST requests
        # 'body' is set to request body for POST requests
    },
    endpoint='render.json', # optional; default is render.html
    splash_url='<url>',     # optional; overrides SPLASH_URL
    slot_policy=scrapy_splash.SlotPolicy.PER_DOMAIN  # optional
)
```

--------------------------------

### Run Scrapy-Splash Integration Tests with Docker

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This snippet provides commands to set up and execute integration tests for the `scrapy-splash` project. It involves launching a Splash Docker container and then configuring the `SPLASH_URL` environment variable before running the `tox` test suite for a specific Python version.

```bash
docker run -d --rm -p8050:8050 scrapinghub/splash:3.0
SPLASH_URL=http://127.0.0.1:8050 tox -e py36
```

--------------------------------

### Enable SplashDeduplicateArgsMiddleware in Scrapy Settings

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/CHANGES.rst

Configuration snippet to enable the `SplashDeduplicateArgsMiddleware` in Scrapy's `settings.py`. This middleware helps optimize network traffic and disk storage by deduplicating Splash arguments, requiring Splash 2.1+.

```Python
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
```

--------------------------------

### Enable Scrapy-Splash Downloader Middlewares

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This configuration in `settings.py` activates the necessary Splash middlewares for handling requests and responses. It also adjusts the priority of `HttpCompressionMiddleware` to ensure proper processing of Splash-rendered content.

```python
DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
```

--------------------------------

### Apply Per-Request HTTP Basic Authentication to Splash

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

Shows how to dynamically apply HTTP Basic Authentication to individual Splash requests using the `splash_headers` argument within `SplashRequest`. This allows for different credentials per request and avoids exposing credentials globally, which is safer than using `HttpAuthMiddleware`.

```python
import scrapy
from w3lib.http import basic_auth_header

class MySpider(scrapy.Spider):
        # ...
        def start_requests(self):
            auth = basic_auth_header('user', 'userpass')
            yield SplashRequest(url, self.parse,
                                splash_headers={'Authorization': auth})
```

--------------------------------

### Configure Global HTTP Basic Authentication for Splash

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

Illustrates how to set up HTTP Basic Authentication credentials for Splash globally using `SPLASH_USER` and `SPLASH_PASS` in Scrapy settings. This method applies authentication to all Splash requests from the spider.

```python
SPLASH_USER = 'user'
SPLASH_PASS = 'userpass'
```

--------------------------------

### request.meta['splash'] API Reference

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

Comprehensive reference for the `request.meta['splash']` dictionary, which configures how Scrapy requests are handled by the Splash rendering service. This dictionary is used by `SplashMiddleware` to modify request behavior, including arguments passed to Splash, endpoint selection, concurrency policies, and response processing.

```APIDOC
request.meta['splash'] (dict): Configuration dictionary for Splash requests.

  Parameters:
    args (dict): Arguments sent to Splash.
      'url' (str): Request URL. Prefilled from request.url.
      'http_method' (str): HTTP method. Set to 'POST' for POST requests.
      'body' (bytes): Request body. Set to request.body for POST requests.
      Note: Default values can be overridden. URL fragments require manual handling if not using SplashRequest. Splash 1.8+ required for POST requests.

    cache_args (list of str): List of argument names to cache on Splash side.
      These arguments are sent only once, then cached values are used to save network traffic and decrease request queue disk memory usage.
      Recommended for large, static arguments like 'lua_source'. Requires Splash 2.1+.

    endpoint (str): The Splash endpoint to use.
      Default: 'render.json' for raw scrapy.Request, 'render.html' for SplashRequest.
      It is recommended to always pass endpoint explicitly.

    splash_url (str): Overrides the global SPLASH_URL setting for this request.

    splash_headers (dict): Headers sent to the Splash server itself (not to the remote website).

    slot_policy (scrapy_splash.SlotPolicy): Customizes concurrency and politeness for Splash requests.
      Options:
        PER_DOMAIN (default): Send Splash requests to downloader slots based on the URL being rendered, maintaining per-domain politeness and concurrency.
        SINGLE_SLOT: Send all Splash requests to a single downloader slot, useful for throttling requests to Splash.
        SCRAPY_DEFAULT: No specific slot handling; similar to SINGLE_SLOT but can differ if other services are on the same Splash address.

    dont_process_response (bool): If True, SplashMiddleware will not change the response to a custom scrapy.Response subclass (e.g., SplashResponse, SplashTextResponse, SplashJsonResponse). Default: False.

    dont_send_headers (bool): If True, scrapy-splash will not pass request headers to Splash in the 'headers' JSON POST field. Default: False.

    magic_response (bool): Optional parameter. Default: True.
```

--------------------------------

### Scrapy Spider to Execute Simple Splash Lua Script

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Scrapy spider demonstrates how to execute a simple inline Lua script using `SplashRequest` with the `execute` endpoint. The script navigates to a URL and evaluates JavaScript to retrieve the document title, showcasing how to pass `lua_source` as an argument.

```python
import json
import base64
from scrapy_splash import SplashRequest


class MySpider(scrapy.Spider):

    # ...
            script = """
            function main(splash)
                assert(splash:go(splash.args.url))
                return splash:evaljs("document.title")
            end
            """
            yield SplashRequest(url, self.parse_result, endpoint='execute',
                                args={'lua_source': script})

        # ...
        def parse_result(self, response):
            doc_title = response.text
            # ...
```

--------------------------------

### Scrapy Spider to Execute Complex Lua Script for Element Screenshot

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Scrapy spider integrates the complex Lua script for taking an element screenshot. It shows how to pass multiple arguments, including the `lua_source`, CSS selector, and padding, to the `SplashRequest` using the `execute` endpoint, and how to handle the binary image data in the response.

```python
class MySpider(scrapy.Spider):


        # ...
            yield SplashRequest(url, self.parse_element_screenshot,
                endpoint='execute',
                args={
                    'lua_source': script,
                    'pad': 32,
                    'css': 'a.title'
                }
             )

        # ...
        def parse_element_screenshot(self, response):
            image_data = response.body  # binary image data in PNG format
            # ...
```

--------------------------------

### Lua Script for Scrapy-Splash Session Management

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Lua script demonstrates how to initialize and retrieve cookies within a Splash `execute` endpoint. It's crucial for maintaining session state across requests when using `scrapy-splash` and ensures cookie handling is enabled by default with `SplashRequest`.

```lua
function main(splash)
    splash:init_cookies(splash.args.cookies)

    -- ... your script

    return {
        cookies = splash:get_cookies(),
        -- ... other results, e.g. html
    }
end
```

--------------------------------

### Interact Directly with Splash HTTP API from Scrapy

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

Presents an alternative approach to using `scrapy-splash` by sending direct HTTP POST requests to the Splash API endpoints (e.g., `/render.html`) using standard `scrapy.Request`. It highlights the boilerplate involved and potential issues with Scrapy's concurrency, politeness settings, and URL handling when not using the `scrapy-splash` library.

```python
import json

import scrapy
from scrapy.http.headers import Headers

RENDER_HTML_URL = "http://127.0.0.1:8050/render.html"

class MySpider(scrapy.Spider):
        start_urls = ["http://example.com", "http://example.com/foo"]

        def start_requests(self):
            for url in self.start_urls:
                body = json.dumps({"url": url, "wait": 0.5}, sort_keys=True)
                headers = Headers({'Content-Type': 'application/json'})
                yield scrapy.Request(RENDER_HTML_URL, self.parse, method="POST",
                                     body=body, headers=headers)

        def parse(self, response):
            # response.body is a result of render.html call; it
            # contains HTML processed by a browser.
            # ...
```

--------------------------------

### Scrapy-Splash Request Meta Options Reference

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This section documents the various keys available within the `request.meta['splash']` dictionary, which control specific behaviors of Scrapy-Splash requests. These options include managing header transmission, mapping HTTP error codes, enabling 'magic response' processing for JSON results, and configuring session handling.

```APIDOC
meta['splash'] Options:
  dont_send_headers: bool
    Description: If True, prevents passing 'headers' from the Scrapy Request to Splash.
  http_status_from_error_code: bool
    Description: When True, sets response.status to the HTTP error code if assert(splash:go(..)) fails. Requires 'magic_response' to be True. Default: False (raw meta API), True (SplashRequest).
  magic_response: bool
    Description: When True and a JSON response is received from Splash, automatically populates Scrapy Response attributes (headers, body, url, status code) from the JSON data. Default: True (SplashRequest).
    Populated Attributes:
      response.headers: From 'headers' key in JSON.
      response.url: From 'url' key in JSON.
      response.body: From 'html' key or base64-decoded 'body' key in JSON.
      response.status: From 'http_status' key in JSON.
    Note: Original URL, status, and headers are available as response.real_url, response.splash_response_status, and response.splash_response_headers.
    Note: 'render.json' and 'execute' endpoints might not provide all necessary keys. For non-JSON endpoints, only 'url' is filled regardless of this setting.
  dont_process_response: bool
    Description: If True, forces Scrapy-Splash to return standard Scrapy Response classes instead of its specialized SplashResponse subclasses.
  session_id: string
    Description: A unique identifier for session management. When set, Scrapy sends current cookies to Splash and merges updated cookies back from the Splash response's 'cookies' field.
```

--------------------------------

### Send Scrapy Request with Splash Meta Key

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This snippet shows an alternative way to use Splash with a standard `scrapy.Request` by including a `'splash'` key in the request's `meta` dictionary. This allows specifying Splash rendering arguments directly within a regular Scrapy request.

```python
yield scrapy.Request(url, self.parse_result, meta={
    'splash': {
        'args': {
            # set rendering arguments here
            'html': 1,
            'png': 1,

```

--------------------------------

### Scrapy-Splash Specialized Response Classes

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This documentation outlines the specialized `Response` subclasses returned by Scrapy-Splash, which are tailored to handle different types of Splash outputs (binary, text, JSON). It details their unique attributes and behaviors, including how JSON responses are processed and how session cookies are managed.

```APIDOC
Scrapy-Splash Response Subclasses:
  SplashResponse:
    Description: Returned for binary Splash responses, such as images from /render.png.
  SplashTextResponse:
    Description: Returned for text-based Splash responses, like HTML content from /render.html.
  SplashJsonResponse:
    Description: Returned when Splash provides a JSON object, typically from /render.json or /execute endpoints (if a Lua table is returned).
    Attributes & Features:
      response.data: dict
        Description: Contains the entire response data decoded from the JSON object. Example: response.data['html'].
      response.cookiejar: CookieJar
        Description: A CookieJar instance containing current cookies, available if Splash session handling is configured.
      Magic Response Processing (if enabled by 'meta['splash']['magic_response']=True'):
        response.headers: Populated from 'headers' keys in the JSON.
        response.url: Set from the 'url' key in the JSON.
        response.body: Set from the 'html' key or base64-decoded 'body' key in the JSON.
        response.status: Set from the 'http_status' key in the JSON.
      Methods:
        response.css(): Available for parsing if response.body is updated from 'html' or 'body' keys.
        response.xpath(): Available for parsing if response.body is updated from 'html' or 'body' keys.
    Note: To disable special handling of JSON result keys, set 'meta['splash']['magic_response']=False' or pass 'magic_response=False' to SplashRequest.
  Common Attributes for all Splash Response Subclasses:
    response.url: The URL of the original Scrapy request (the target website URL).
    response.real_url: The actual URL of the requested Splash endpoint.
```

--------------------------------

### Splash Lua Script Session Management Functions

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This section describes the essential Lua functions, `splash:get_cookies()` and `splash:init_cookies()`, provided by Splash for managing HTTP cookies within Lua scripts. These functions are crucial for maintaining session state across multiple requests within a Splash script.

```APIDOC
Splash Lua Session Handling Methods:
  splash:get_cookies():
    Description: Retrieves the current set of cookies from the Splash environment.
  splash:init_cookies():
    Description: Initializes or updates the cookie store within the Splash environment, typically used with cookies received from the client.
```

--------------------------------

### Lua Splash:go Function with Custom Headers

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This Lua code snippet demonstrates how to pass custom headers to the `splash:go` function. It utilizes `splash.args.headers` to dynamically apply headers provided to the Splash request, allowing for fine-grained control over HTTP requests made within Splash scripts.

```Lua
splash:go{url, headers=splash.args.headers}
```

--------------------------------

### Enable Splash Deduplication Spider Middleware

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This middleware, configured in `settings.py`, helps optimize disk space and network traffic by preventing duplicate Splash arguments from being stored or sent repeatedly, especially useful with the `cache_args` feature.

```python
SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
```

--------------------------------

### Complex Splash Lua Script for Element Screenshot

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This advanced Lua script captures a screenshot of a specific HTML element identified by a CSS selector. It includes helper functions for padding and retrieving bounding box coordinates, demonstrating how to pass arguments to the script and utilize Splash's full viewport and PNG rendering capabilities.

```lua
script = """
-- Arguments:
-- * url - URL to render;
-- * css - CSS selector to render;
-- * pad - screenshot padding size.

-- this function adds padding around region
function pad(r, pad)
  return {r[1]-pad, r[2]-pad, r[3]+pad, r[4]+pad}
end

-- main script
function main(splash)

  -- this function returns element bounding box
  local get_bbox = splash:jsfunc([[
        function(css) {
          var el = document.querySelector(css);
          var r = el.getBoundingClientRect();
          return [r.left, r.top, r.right, r.bottom];
        }
      ]])

  assert(splash:go(splash.args.url))
  assert(splash:wait(0.5))

  -- don't crop image by a viewport
  splash:set_viewport_full()

  local region = pad(get_bbox(splash.args.css), splash.args.pad)
  return splash:png{region=region}
end
"""
```

--------------------------------

### Configure Splash Server URL in Scrapy Settings

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This setting in `settings.py` specifies the URL where the Splash server is accessible. Scrapy-Splash uses this URL to send requests for rendering.

```python
SPLASH_URL = 'http://192.168.59.103:8050'
```

--------------------------------

### Execute Custom Lua Script for Enhanced Scrapy Requests

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

Demonstrates how to use a custom Lua script within `scrapy-splash` to control browser behavior, retrieve full response details (HTML, headers, cookies, status), and pass arguments like URL, headers, and body. The `lua_source` argument value is cached on the Splash server for efficiency, requiring Splash 2.1+.

```python
import scrapy
from scrapy_splash import SplashRequest

script = """
function main(splash)
  splash:init_cookies(splash.args.cookies)
  assert(splash:go{
    splash.args.url,
    headers=splash.args.headers,
    http_method=splash.args.http_method,
    body=splash.args.body,
    })
  assert(splash:wait(0.5))

  local entries = splash:history()
  local last_response = entries[#entries].response
  return {
    url = splash:url(),
    headers = last_response.headers,
    http_status = last_response.status,
    cookies = splash:get_cookies(),
    html = splash:html(),
  }
end
"""

class MySpider(scrapy.Spider):


        # ...
            yield SplashRequest(url, self.parse_result,
                endpoint='execute',
                cache_args=['lua_source'],
                args={'lua_source': script},
                headers={'X-My-Header': 'value'},
            )

        def parse_result(self, response):
            # here response.body contains result HTML;
            # response.headers are filled with headers from last
            # web page loaded to Splash;
            # cookies from all responses and from JavaScript are collected
            # and put into Set-Cookie response header, so that Scrapy
            # can remember them.
```

--------------------------------

### Accessing Splash Response Status and Headers

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/CHANGES.rst

Describes how to access HTTP status code and headers from Splash responses via new attributes on the Scrapy response object. These attributes provide direct access to the original Splash response details, introduced in version 0.8.0.

```APIDOC
Scrapy Response Object:
  Attributes:
    splash_response_status: int
      The HTTP status code returned by Splash.
    splash_response_headers: dict
      The HTTP headers returned by Splash.
```

--------------------------------

### Set Custom Request Fingerprinter Class

Source: https://github.com/scrapy-plugins/scrapy-splash/blob/master/README.rst

This setting in `settings.py` specifies a custom request fingerprinter provided by `scrapy-splash`. It is crucial for correctly identifying and deduplicating requests when using Splash.

```python
REQUEST_FINGERPRINTER_CLASS = 'scrapy_splash.SplashRequestFingerprinter'
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.