### Install scrapy-impersonate

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/00-START-HERE.md

Install the library using pip. This is the first step before configuring or using the library.

```bash
pip install scrapy-impersonate
```

--------------------------------

### Install Asyncio Reactor for Twisted

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/README.md

Install the asyncio-based Twisted reactor for proper asynchronous execution. This is a required setup step for scrapy-impersonate.

```python
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

--------------------------------

### Initialize Crawler with Reactor Setting

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

This example shows how to initialize a Scrapy CrawlerProcess with the TWISTED_REACTOR setting applied. This ensures the correct reactor is used before the crawler starts.

```python
from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    "DOWNLOAD_HANDLERS": {
        "http": "scrapy_impersonate.ImpersonateDownloadHandler",
        "https": "scrapy_impersonate.ImpersonateDownloadHandler",
    }
})
```

--------------------------------

### Complete Scrapy Settings Example for Scrapy-Impersonate

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/configuration.md

A comprehensive example of how to configure scrapy-impersonate within your Scrapy project's settings.py file. Includes required, recommended, and optional settings for browser rotation and other Scrapy configurations.

```python
# settings.py

# Required
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

# Recommended
USER_AGENT = ""

# Optional - browser rotation
DOWNLOADER_MIDDLEWARES = {
    "scrapy_impersonate.RandomBrowserMiddleware": 1000,
}

IMPERSONATE_BROWSERS = ["chrome", "firefox", "safari"]

# Other Scrapy settings
ROBOTSTXT_OBEY = False
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 0.5
COOKIES_ENABLED = True
```

--------------------------------

### Corrected Browser Configuration Example

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

Shows the correct way to configure the IMPERSONATE_BROWSERS setting with valid browser family names. This prevents errors caused by misspellings or unsupported browser types.

```python
# Correct
IMPERSONATE_BROWSERS = ["chrome", "firefox", "safari"]

# Incorrect
IMPERSONATE_BROWSERS = ["crome", "ffox"]
```

--------------------------------

### RandomBrowserMiddleware Integration Example

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RandomBrowserMiddleware.md

This example demonstrates how to integrate RandomBrowserMiddleware into a Scrapy project. Ensure the middleware and download handlers are correctly configured in `custom_settings`. The middleware automatically assigns a random browser impersonation to each request's meta, which is then used by the ImpersonateDownloadHandler.

```python
import scrapy

class MySpider(scrapy.Spider):
    name = "example"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
        "IMPERSONATE_BROWSERS": ["chrome", "firefox"]
    }
    
    def start_requests(self):
        # Middleware will automatically add random impersonate to meta
        for i in range(10):
            yield scrapy.Request(f"https://example.com/page{i}")
    
    def parse(self, response):
        # Each request was sent with a random browser
        browser = response.request.meta.get("impersonate")
        print(f"Downloaded with: {browser}")
        return {"browser": browser, "status": response.status}
```

--------------------------------

### Verify Installed Reactor

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

This code snippet verifies if the required AsyncioSelectorReactor is installed. It is used during the initialization of ImpersonateDownloadHandler.

```python
verify_installed_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
```

--------------------------------

### Minimal Scrapy-Impersonate Setup

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/USAGE_PATTERNS.md

This is the most basic configuration required to use scrapy-impersonate. It involves setting the TWISTED_REACTOR and DOWNLOAD_HANDLERS in settings.py and making a request with the 'impersonate' meta key in your spider.

```python
# settings.py
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

# spider.py
import scrapy

class SimpleSpider(scrapy.Spider):
    name = "simple"
    
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={"impersonate": "chrome124"}
        )
    
    def parse(self, response):
        yield {"status": response.status}

```

--------------------------------

### Advanced Download Request with curl_cffi Arguments

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/ImpersonateDownloadHandler.md

This example demonstrates using `_download_request` for more advanced scenarios, including POST requests and custom `curl_cffi` arguments like disabling SSL verification and setting a proxy. It also shows how to access the download latency.

```python
import scrapy
from scrapy.http import Request

class AdvancedSpider(scrapy.Spider):
    def start_requests(self):
        yield Request(
            "https://api.example.com/data",
            method="POST",
            body=b'{"key": "value"}',
            headers={"Content-Type": "application/json"},
            meta={
                "impersonate": "chrome124",
                "impersonate_args": {
                    "verify": False,  # Disable SSL verification
                    "timeout": 30,
                    "proxy": "http://proxy.example.com:8080"
                }
            }
        )
    
    def parse(self, response):
        latency = response.meta.get("download_latency")
        print(f"Request took {latency:.2f} seconds")
        return {"status": response.status}
```

--------------------------------

### Basic Impersonation Setup

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Configures Scrapy to use ImpersonateDownloadHandler for HTTP/HTTPS requests and sets a specific browser impersonation in the request meta. This is useful for mimicking a particular browser.

```python
import scrapy

class MySpider(scrapy.Spider):
    name = "example"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
    }
    
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={"impersonate": "chrome124"}
        )
    
    def parse(self, response):
        return {"url": response.url, "status": response.status}
```

--------------------------------

### Example of Incorrect Browser Configuration

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

Demonstrates a common configuration error where IMPERSONATE_BROWSERS contains typos, leading to an empty browser list and subsequent IndexError. Ensure browser family names are spelled correctly.

```python
# This matches no browsers (typo: "crome" instead of "chrome")
IMPERSONATE_BROWSERS = ["crome", "firefox"]

# Result: RandomBrowserMiddleware.browsers will be empty list
# random.choice([]) will raise IndexError when processing requests
```

--------------------------------

### Basic Scrapy Spider Usage

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/00-START-HERE.md

Example of a Scrapy spider using scrapy-impersonate. A Request is yielded with meta specifying the impersonated browser.

```python
import scrapy

class MySpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={"impersonate": "chrome124"}
        )
```

--------------------------------

### Configuring RandomBrowserMiddleware

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RandomBrowserMiddleware.md

Example of how to configure RandomBrowserMiddleware in Scrapy's settings.py. It shows both default usage and custom browser subset configuration.

```python
# Using default browser rotation (all browser families)
DOWNLOADER_MIDDLEWARES = {
    "scrapy_impersonate.RandomBrowserMiddleware": 1000,
}

# Using custom browser subset
DOWNLOADER_MIDDLEWARES = {
    "scrapy_impersonate.RandomBrowserMiddleware": 1000,
}
IMPERSONATE_BROWSERS = ["chrome", "firefox"]  # Only these families
```

--------------------------------

### Example Usage of @curl_option_method

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Demonstrates how to apply the @curl_option_method decorator to a class method. Only methods decorated with @curl_option_method will be invoked by the as_dict() method.

```python
class CurlOptionsParser:
    @curl_option_method
    def _set_proxy_auth(self):
        # This method will be called by as_dict()
        pass
    
    def some_other_method(self):
        # This method will NOT be called by as_dict()
        pass
```

--------------------------------

### Instantiate and Use CurlOptionsParser

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Instantiate CurlOptionsParser with a Scrapy Request object and call as_dict() to get accumulated curl options. This is useful for translating Scrapy request metadata into a format compatible with curl_cffi.

```python
from scrapy.http import Request
from scrapy_impersonate.parser import CurlOptionsParser
from curl_cffi import CurlOpt

request = Request(
    "https://api.example.com",
    meta={"proxy": "http://proxy:8080"},
    headers={"Proxy-Authorization": "Basic dXNlcjpwYXNz"}
)

parser = CurlOptionsParser(request)
curl_opts = parser.as_dict()

# Returns something like:
# {
#     CurlOpt.PROXYHEADER: [b"Proxy-Authorization: Basic dXNlcjpwYXNz"]
# }
```

--------------------------------

### Invalid Impersonate Browser Values

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

Shows examples of invalid browser names passed to the 'impersonate' meta key, which can cause configuration errors. Use only supported browser names.

```python
import scrapy

# Invalid browser names
yield scrapy.Request(
    "https://example.com",
    meta={"impersonate": "chrome999"}  # Doesn't exist
)

yield scrapy.Request(
    "https://example.com",
    meta={"impersonate": ""}  # Empty string
)

yield scrapy.Request(
    "https://example.com",
    meta={"impersonate": "internetexplorer"}  # Not supported
)
```

--------------------------------

### Basic Spider Usage with Scrapy-Impersonate

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/README.md

Example of a Scrapy spider that uses scrapy-impersonate. Ensure all necessary settings like TWISTED_REACTOR, USER_AGENT, and DOWNLOAD_HANDLERS are configured. The RandomBrowserMiddleware is also included for rotating user agents.

```python
import scrapy


class ImpersonateSpider(scrapy.Spider):
    name = "impersonate_spider"
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "USER_AGENT": "",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
    }

    def start_requests(self):
        for _ in range(5):
            yield scrapy.Request(
                "https://tls.browserleaks.com/json",
                dont_filter=True,
            )

    def parse(self, response):
        # ja3_hash: 98cc085d47985d3cca9ec1415bbbf0d1 (chrome133a)
        # ja3_hash: 2d692a4485ca2f5f2b10ecb2d2909ad3 (firefox133)
        # ja3_hash: c11ab92a9db8107e2a0b0486f35b80b9 (chrome124)
        # ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37 (safari15_5)
        # ja3_hash: cd08e31494f9531f560d64c695473da9 (chrome99_android)

        yield {"ja3_hash": response.json()["ja3_hash"]}
```

--------------------------------

### Download Request with Browser Impersonation

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/ImpersonateDownloadHandler.md

The `download_request` method processes a Scrapy Request. It routes to a curl_cffi-based download if `impersonate` is set in `request.meta`, otherwise it delegates to the default Scrapy HTTP handler. This example shows how to configure a request for impersonation.

```python
import scrapy

class MySpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={
                "impersonate": "chrome124",
                "impersonate_args": {"timeout": 10}
            }
        )
    
    def parse(self, response):
        # response is a standard Scrapy Response
        # response.flags contains "impersonate" if from curl_cffi
        if "impersonate" in response.flags:
            print(f"Downloaded via impersonation: {response.url}")
```

--------------------------------

### Initialize RequestParser with URL

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Demonstrates initializing RequestParser and accessing the full URL of the request.

```python
parser = RequestParser(Request("https://example.com/api/users"))
assert parser.url == "https://example.com/api/users"
```

--------------------------------

### Advanced Configuration with Proxy and Timeout

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Demonstrates advanced configuration by setting a proxy, a specific impersonation, and passing custom arguments like timeout and SSL verification settings to the impersonation handler.

```python
import scrapy

class MySpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request(
            "https://api.example.com/data",
            method="POST",
            body=b'{"key": "value"}',
            meta={
                "impersonate": "chrome124",
                "proxy": "http://proxy.example.com:8080",
                "impersonate_args": {
                    "timeout": 30,
                    "verify": False,
                }
            }
        )
    
    def parse(self, response):
        return {}
```

--------------------------------

### Initialize RequestParser with Method

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Demonstrates initializing RequestParser and accessing the HTTP method of the request.

```python
parser = RequestParser(Request("https://example.com", method="POST"))
assert parser.method == "POST"
```

--------------------------------

### Instantiate and Use RequestParser

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Demonstrates how to create a Scrapy Request object, initialize RequestParser with it, and convert the request into curl_cffi compatible arguments using the as_dict() method.

```python
from scrapy.http import Request
from scrapy_impersonate.parser import RequestParser

request = Request(
    "https://example.com/api",
    method="POST",
    body=b"key=value",
    headers={\"Authorization\": \"Bearer token\"},
    meta={"impersonate": "chrome124"}
)

parser = RequestParser(request)
curl_args = parser.as_dict()
# curl_args can now be passed to curl_cffi AsyncSession.request()
```

--------------------------------

### Configure SOCKS5 Proxy with Proxy-Authorization Header

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/configuration.md

Shows how to use a SOCKS5 proxy with authentication provided via the 'Proxy-Authorization' header. The credentials must be base64 encoded.

```python
import scrapy
import base64
creds = base64.b64encode(b"user:pass").decode()
yield scrapy.Request(
    "https://example.com",
    meta={
        "impersonate": "chrome124",
        "proxy": "socks5://proxy.example.com:1080"
    },
    headers={"Proxy-Authorization": f"Basic {creds}"}
)
```

--------------------------------

### Accessing Query Parameters

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Shows how to initialize RequestParser with query parameters passed via impersonate_args and access them.

```python
request = Request(
    "https://example.com/search",
    meta={
        "impersonate": "chrome124",
        "impersonate_args": {
            "params": {"q": "python", "limit": 10}
        }
    }
)

parser = RequestParser(request)
assert parser.params == {"q": "python", "limit": 10}
```

--------------------------------

### Scrapy Request with Impersonation Meta

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Example of a Scrapy Request object configured for impersonation. The 'impersonate' key in the meta dictionary specifies the browser fingerprint to use, and 'impersonate_args' can pass additional arguments like timeout.

```python
Scrapy Request
├─ url: "https://example.com"
├─ method: "GET"
├─ headers: {"User-Agent": "..."}
├─ meta:
│  ├─ impersonate: "chrome124"
│  └─ impersonate_args: {"timeout": 10}
└─ meta:
   └─ proxy: "http://proxy:8080"
```

--------------------------------

### Configure HTTP, HTTPS, and SOCKS5 Proxies

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/configuration.md

Demonstrates how to set different proxy types (HTTP, HTTPS with auth, SOCKS5) using the 'proxy' meta key in scrapy.Request. Ensure the 'impersonate' key is also set.

```python
import scrapy

class MySpider(scrapy.Spider):
    def start_requests(self):
        # HTTP proxy
        yield scrapy.Request(
            "https://example.com",
            meta={
                "impersonate": "chrome124",
                "proxy": "http://proxy.example.com:8080"
            }
        )
        
        # HTTPS proxy with embedded auth
        yield scrapy.Request(
            "https://example.com",
            meta={
                "impersonate": "chrome124",
                "proxy": "https://user:password@proxy.example.com:8443"
            }
        )
        
        # SOCKS5 proxy
        yield scrapy.Request(
            "https://example.com",
            meta={
                "impersonate": "chrome124",
                "proxy": "socks5://socks-proxy.example.com:1080"
            }
        )
```

--------------------------------

### Import Core Scrapy-Impersonate Components

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/README.md

Import the main download handler, browser rotation middleware, and request parsing utility from the scrapy_impersonate library.

```python
from scrapy_impersonate import (
    ImpersonateDownloadHandler,    # Main download handler
    RandomBrowserMiddleware,        # Browser rotation middleware
    RequestParser,                  # Request parsing utility
)
```

--------------------------------

### Configure Scrapy Settings

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/00-START-HERE.md

Configure your Scrapy project's settings.py file to enable scrapy-impersonate. This involves setting the TWISTED_REACTOR and defining DOWNLOAD_HANDLERS.

```python
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
```

--------------------------------

### Initialize CurlOptionsParser

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Instantiate CurlOptionsParser with a Scrapy Request object. The options are parsed lazily when as_dict() is called.

```python
from scrapy.http import Request
from scrapy_impersonate.parser import CurlOptionsParser

request = Request(
    "https://example.com",
    headers={"Proxy-Authorization": "Basic dXNlcjpwYXNz"}
)

parser = CurlOptionsParser(request)
curl_opts = parser.as_dict()
```

--------------------------------

### CurlOptionsParser.as_dict()

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Discovers and executes all curl option setter methods, returning the accumulated curl_options as a dictionary.

```APIDOC
## as_dict()

### Description
Discovers and executes all curl option setter methods, returning the accumulated curl_options.

### Method
Instance Method

### Returns
`dict` — Dictionary mapping curl_cffi CurlOpt enum values to configuration values

### Behavior
1. Iterates through all attributes of the class instance
2. Identifies methods decorated with @curl_option_method
3. Executes each method in order
4. Collects resulting curl_options into a dict
5. Returns the accumulated options

### Example
```python
from scrapy.http import Request
from scrapy_impersonate.parser import CurlOptionsParser
from curl_cffi import CurlOpt

request = Request(
    "https://api.example.com",
    meta={"proxy": "http://proxy:8080"},
    headers={"Proxy-Authorization": "Basic dXNlcjpwYXNz"}
)

parser = CurlOptionsParser(request)
curl_opts = parser.as_dict()

# Returns something like:
# {
#     CurlOpt.PROXYHEADER: [b"Proxy-Authorization: Basic dXNlcjpwYXNz"]
# }
```
```

--------------------------------

### Checking Supported Browser Families

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

Provides a Python snippet to list all supported browser families from curl_cffi. Use this to verify correct spelling and available options for the IMPERSONATE_BROWSERS setting.

```python
from curl_cffi import BrowserType

supported_families = set()
for b in BrowserType:
    family = b.value.split('_')[0]  # e.g., "chrome" from "chrome124"
    supported_families.add(family)

print("Supported families:", supported_families)
```

--------------------------------

### Handling Non-Impersonated Requests with RandomBrowserMiddleware

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RandomBrowserMiddleware.md

Demonstrates how the middleware handles requests. If 'request.meta["impersonate"]' is already set, the middleware does not override it, preserving explicitly chosen browsers.

```python
import scrapy

class MySpider(scrapy.Spider):
    def start_requests(self):
        # This request gets a random browser from middleware
        yield scrapy.Request("https://example.com/1")
        
        # This request uses the explicitly set browser (not randomized)
        yield scrapy.Request(
            "https://example.com/2",
            meta={"impersonate": "chrome124"}  # Explicit, won't be overridden
        )
```

--------------------------------

### Minimal Scrapy-Impersonate Settings

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

These are the minimum required settings to enable scrapy-impersonate. Ensure you set the TWISTED_REACTOR to use the asyncio reactor and configure the DOWNLOAD_HANDLERS for both http and https protocols.

```python
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
```

--------------------------------

### CurlOptionsParser Constructor

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Initializes the CurlOptionsParser with a Scrapy Request object. This prepares the parser to extract curl options lazily.

```APIDOC
## __init__(request: scrapy.http.Request)

### Description
Initializes the parser with a Scrapy Request object and prepares to extract curl options.

### Parameters
- **request** (scrapy.http.Request) - Required - The Scrapy request to parse

### Behavior
- Stores the request for processing
- Initializes empty curl_options dict
- Does not immediately extract options (lazy evaluation)

### Example
```python
from scrapy.http import Request
from scrapy_impersonate.parser import CurlOptionsParser

request = Request(
    "https://example.com",
    headers={"Proxy-Authorization": "Basic dXNlcjpwYXNz"}
)

parser = CurlOptionsParser(request)
curl_opts = parser.as_dict()
```
```

--------------------------------

### Integrating CurlOptionsParser with curl_cffi

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Shows how the parsed curl options are passed to curl_cffi's AsyncSession for request execution. This is typically done within a download handler.

```python
# In ImpersonateDownloadHandler._download_request()
curl_options = CurlOptionsParser(request.copy()).as_dict()
async with AsyncSession(max_clients=1, curl_options=curl_options) as client:
    response = await client.request(**request_args)
```

--------------------------------

### Configure ImpersonateDownloadHandler in Scrapy Settings

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/ImpersonateDownloadHandler.md

Set TWISTED_REACTOR and DOWNLOAD_HANDLERS to enable browser impersonation. USER_AGENT is recommended to be empty for auto-selection.

```python
# Required settings
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

# Recommended settings
USER_AGENT = ""  # Let curl_cffi auto-select based on impersonated browser
```

--------------------------------

### Scrapy-Impersonate Request Flow

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Illustrates the request processing flow within scrapy-impersonate, showing how requests are handled by the ImpersonateDownloadHandler and potentially routed through curl_cffi or the default Scrapy handler.

```text
Request with impersonate meta
    ↓
ImpersonateDownloadHandler.download_request()
    ↓ (if impersonate is set)
    ├─→ CurlOptionsParser (extract curl-specific options)
    │   └─→ CurlOpt configuration
    │
    ├─→ RequestParser (extract curl_cffi arguments)
    │   └─→ method, url, headers, cookies, etc.
    │
    └─→ AsyncSession.request() (curl_cffi)
        └─→ HTTP response
        
    ↓ (if impersonate is not set)
    └─→ Default Scrapy HTTP11DownloadHandler
        └─→ HTTP response

Response returned to Scrapy
    ↓
RandomBrowserMiddleware (optional; sets random impersonate)
    ↓
Next request
```

--------------------------------

### ImpersonateDownloadHandler Request Processing

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Illustrates the internal processing within the ImpersonateDownloadHandler. It shows how CurlOptionsParser and RequestParser extract and format request details, including impersonation settings and proxy information, before passing them to AsyncSession for the actual HTTP request.

```python
ImpersonateDownloadHandler._download_request()

├─ CurlOptionsParser(request).as_dict()
│  └─ {CurlOpt.PROXYHEADER: [...]}
│
├─ RequestParser(request).as_dict()
│  └─ {
│     method: "GET",
│     url: "https://example.com",
│     headers: {"User-Agent": "..."},
│     impersonate: "chrome124",
│     timeout: 10,
│     proxy: "http://proxy:8080",
│     ...
│  }
│
└─ AsyncSession.request(**args)
   └─ curl_cffi performs HTTP request with chrome124 TLS fingerprint
```

--------------------------------

### Extending CurlOptionsParser with Custom Methods

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Demonstrates how to extend CurlOptionsParser by subclassing and adding custom curl option methods using decorators. This allows for custom TLS configurations.

```python
class CustomCurlOptionsParser(CurlOptionsParser):
    @curl_option_method
    def _set_custom_tls(self):
        # Custom TLS configuration
        if "custom_tls" in self.request.meta:
            self.curl_options[CurlOpt.SSLCERT] = self.request.meta["custom_tls"]["cert"]

# Usage in spider
class MySpider(scrapy.Spider):
    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={
                "impersonate": "chrome124",
                "custom_tls": {"cert": "/path/to/cert.pem"}
            }
        )
```

--------------------------------

### Recommended Scrapy-Impersonate Settings

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

These recommended settings enhance scrapy-impersonate's functionality. Setting USER_AGENT to an empty string allows curl_cffi to auto-select a browser, and the RandomBrowserMiddleware is added for random browser selection.

```python
USER_AGENT = ""  # Let curl_cffi auto-select based on browser

DOWNLOADER_MIDDLEWARES = {
    "scrapy_impersonate.RandomBrowserMiddleware": 1000,
}

IMPERSONATE_BROWSERS = ["chrome", "firefox"]
```

--------------------------------

### Correct SOCKS Authentication Credentials

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

Illustrates the correct format for SOCKS proxy authentication credentials, ensuring both username and password are included for proper base64 encoding and parsing.

```python
import base64

# Correct
creds = base64.b64encode(b"socks_user:socks_password").decode()

# Incorrect
creds = base64.b64encode(b"socks_user").decode()  # Missing password
```

--------------------------------

### Combine Random Middleware with Per-Request Overrides

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/USAGE_PATTERNS.md

Utilize RandomBrowserMiddleware for default request behavior, while allowing specific requests to override the middleware by setting the 'impersonate' key in their meta dictionary. This enables a mix of random and explicitly defined browser choices.

```python
import scrapy

class MixedBrowserSpider(scrapy.Spider):
    name = "mixed_browser"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "DOWNLOADER_MIDDLEWARES": {
            "scrapy_impersonate.RandomBrowserMiddleware": 1000,
        },
    }
    
    def start_requests(self):
        # These use random browser from middleware
        yield scrapy.Request("https://example.com/page1")
        yield scrapy.Request("https://example.com/page2")
        
        # This overrides middleware and uses specific browser
        yield scrapy.Request(
            "https://api.example.com/v1",
            meta={"impersonate": "chrome124"}
        )
        
        # This overrides middleware
        yield scrapy.Request(
            "https://api.example.com/v2",
            meta={"impersonate": "firefox147"}
        )
    
    def parse(self, response):
        browser = response.request.meta["impersonate"]
        yield {"url": response.url, "browser": browser}
```

--------------------------------

### Minimal Scrapy Configuration

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/README.md

Configure Scrapy settings to use the ImpersonateDownloadHandler for HTTP and HTTPS protocols. This requires setting the TWISTED_REACTOR to AsyncioSelectorReactor.

```python
# settings.py
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
```

--------------------------------

### Handle Query Parameters in Scrapy Requests

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/USAGE_PATTERNS.md

Demonstrates three methods for including query parameters in Scrapy requests when using the impersonate middleware. Method 1 includes params directly in the URL. Method 2 uses the 'impersonate_args' with a 'params' dictionary. Method 3 shows how to handle complex parameters like lists.

```python
import scrapy

class QueryParamSpider(scrapy.Spider):
    name = "query_params"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
    }
    
    def start_requests(self):
        # Method 1: Include params in URL
        yield scrapy.Request(
            "https://example.com/search?q=python&limit=10",
            meta={"impersonate": "chrome124"}
        )
        
        # Method 2: Use params in impersonate_args (curl_cffi encodes them)
        yield scrapy.Request(
            "https://example.com/search",
            meta={
                "impersonate": "chrome124",
                "impersonate_args": {
                    "params": {"q": "scrapy", "limit": 20}
                }
            }
        )
        
        # Method 3: Complex params (lists, nested structures)
        yield scrapy.Request(
            "https://api.example.com/filter",
            meta={
                "impersonate": "firefox147",
                "impersonate_args": {
                    "params": [
                        ("category", "books"),
                        ("category", "videos"),
                        ("sort", "date")
                    ]
                }
            }
        )
    
    def parse(self, response):
        yield {"url": response.url, "status": response.status}
```

--------------------------------

### Documentation Structure

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/00-START-HERE.md

Overview of the documentation directory structure. This helps in understanding where to find specific information.

```text
/workspace/home/output/
├── 00-START-HERE.md              ← You are here
├── README.md                      ← Navigation guide
├── OVERVIEW.md                    ← Architecture overview
├── USAGE_PATTERNS.md              ← 15 code examples
├── INDEX.md                       ← Documentation index
├── configuration.md               ← Settings reference
├── types.md                       ← Type definitions
├── errors.md                      ← Error handling
└── api-reference/
    ├── ImpersonateDownloadHandler.md
    ├── RandomBrowserMiddleware.md
    ├── RequestParser.md
    └── CurlOptionsParser.md
```

--------------------------------

### Configure Browser Set

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RandomBrowserMiddleware.md

Control which browser families are rotated by setting IMPERSONATE_BROWSERS. This list accepts prefixes that match browser family names, expanding to all available versions in curl_cffi. By default, all families are included.

```python
# Use only specific browser families
IMPERSONATE_BROWSERS = ["chrome", "firefox"]
```

--------------------------------

### Browser Selection Algorithm

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RandomBrowserMiddleware.md

The middleware uses Python's random.choice() for uniform random selection from the configured browsers. Each request is independently randomized and respects the global random seed if set.

```python
browser = random.choice(self.browsers)  # Uniform random selection
request.meta["impersonate"] = browser
```

--------------------------------

### Custom Headers and SSL Verification with Scrapy-Impersonate

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/USAGE_PATTERNS.md

Shows how to set custom request headers, including authorization tokens, and how to disable SSL verification for requests to servers with self-signed certificates.

```python
import scrapy

class CustomHeadersSpider(scrapy.Spider):
    name = "custom_headers"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
    }
    
    def start_requests(self):
        # Custom headers
        yield scrapy.Request(
            "https://api.example.com/data",
            headers={
                "Accept-Language": "en-US,en;q=0.9",
                "Accept": "application/json",
                "X-Custom-Header": "value"
            },
            meta={"impersonate": "chrome124"}
        )
        
        # With API token
        yield scrapy.Request(
            "https://api.example.com/protected",
            headers={"Authorization": "Bearer YOUR_TOKEN"},
            meta={"impersonate": "firefox147"}
        )
        
        # Disable SSL verification (self-signed certificates)
        yield scrapy.Request(
            "https://internal-server.local/api",
            meta={
                "impersonate": "chrome124",
                "impersonate_args": {"verify": False}
            }
        )
    
    def parse(self, response):
        yield {"status": response.status}
```

--------------------------------

### Verifying Supported Browsers with curl_cffi

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/errors.md

This snippet demonstrates how to list and check for supported browser names using the curl_cffi library's BrowserType enum. This helps in identifying valid values for the 'impersonate' meta key.

```python
from curl_cffi import BrowserType

# List all supported browsers
supported = [b.value for b in BrowserType]
print(supported)

# Check if browser is supported
browser = "chrome124"
if browser in supported:
    print(f"{browser} is supported")
else:
    print(f"{browser} is NOT supported")
```

--------------------------------

### Registering ImpersonateDownloadHandler in Scrapy Settings

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/ImpersonateDownloadHandler.md

Shows how to configure Scrapy to use the ImpersonateDownloadHandler for both HTTP and HTTPS schemes. Ensure the AsyncioSelectorReactor is also specified for proper async operation.

```python
# Registration via settings
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
```

--------------------------------

### Import Scrapy Impersonate Classes

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/OVERVIEW.md

Imports the main classes from the scrapy-impersonate library. Ensure these are available in your project.

```python
from scrapy_impersonate import (
    ImpersonateDownloadHandler,
    RandomBrowserMiddleware,
    RequestParser,
)
```

--------------------------------

### Managing Cookies with Scrapy-Impersonate

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/USAGE_PATTERNS.md

Demonstrates how to enable cookie handling and explicitly pass cookies in requests. Scrapy middleware manages cookies automatically, but they can also be provided directly.

```python
import scrapy

class CookieSpider(scrapy.Spider):
    name = "cookie_management"
    
    custom_settings = {
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "COOKIES_ENABLED": True,
    }
    
    def start_requests(self):
        # Request 1: Get session cookie
        yield scrapy.Request(
            "https://example.com/login",
            method="POST",
            body=b"username=user&password=pass",
            meta={"impersonate": "chrome124"}
        )
    
    def parse(self, response):
        # Scrapy middleware handles cookies automatically
        # But we can also pass cookies explicitly
        yield scrapy.Request(
            "https://example.com/dashboard",
            cookies={
                "session_id": "abc123",
                "user_id": "42"
            },
            meta={"impersonate": "chrome124"}
        )
    
    def parse_dashboard(self, response):
        yield {"user": response.css(".username::text").get()}
```

--------------------------------

### Subclassing ImpersonateDownloadHandler

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/types.md

Demonstrates how subclassing ImpersonateDownloadHandler with a custom handler preserves type information. The from_crawler method correctly infers the subclass type, enhancing type safety in projects that extend the base handler.

```python
class CustomHandler(ImpersonateDownloadHandler):
    pass

# Type checker infers: CustomHandler (not ImpersonateDownloadHandler)
handler = CustomHandler.from_crawler(crawler)
```

--------------------------------

### Accessing Proxy URL

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Shows how to retrieve the proxy URL from request metadata using RequestParser.

```python
request = Request(
    "https://example.com",
    meta={"proxy": "http://proxy.example.com:8080"}
)

parser = RequestParser(request)
assert parser.proxy == "http://proxy.example.com:8080"
```

--------------------------------

### Importable Components

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/README.md

These are the core components you can import directly from the `scrapy_impersonate` package to integrate its functionality into your Scrapy spiders and pipelines.

```APIDOC
## Exported API

Everything importable from scrapy_impersonate:

```python
from scrapy_impersonate import (
    ImpersonateDownloadHandler,    # Main download handler
    RandomBrowserMiddleware,        # Browser rotation middleware
    RequestParser,                  # Request parsing utility
)
```

Note: `CurlOptionsParser` is not re-exported but is documented separately.
```

--------------------------------

### HTTP Proxy Authentication with CurlOptionsParser

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/CurlOptionsParser.md

Parses proxy authentication for HTTP/HTTPS proxies using the Proxy-Authorization header. Sets CurlOpt.PROXYHEADER for curl_cffi.

```python
import base64
from scrapy.http import Request
from scrapy_impersonate.parser import CurlOptionsParser

# Create request with proxy auth
creds = base64.b64encode(b"user:password").decode()
request = Request(
    "https://api.example.com/data",
    meta={"proxy": "http://proxy.example.com:8080"},
    headers={"Proxy-Authorization": f"Basic {creds}"}
)

parser = CurlOptionsParser(request)
curl_opts = parser.as_dict()
# curl_opts[CurlOpt.PROXYHEADER] == [b"Proxy-Authorization: Basic dXNlcjpwYXNzd29yZA=="]
```

--------------------------------

### Extracting Impersonation from Request Meta

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/_autodocs/api-reference/RequestParser.md

Demonstrates how to set the 'impersonate' key in request.meta to specify a browser fingerprint for curl_cffi. This is required for impersonation to be active.

```python
from scrapy.http import Request
from scrapy_impersonate.parser import RequestParser

request = Request(
    "https://example.com",
    meta={"impersonate": "chrome124"}
)

parser = RequestParser(request)
assert parser.impersonate == "chrome124"
```

--------------------------------

### Configure Download Handlers in Scrapy

Source: https://github.com/jxlil/scrapy-impersonate/blob/master/README.md

Replace the default http and https download handlers with scrapy-impersonate's handler by updating the DOWNLOAD_HANDLERS setting in your Scrapy project.

```python
DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}
```