### Request Queue Configuration Examples

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

Configure how Redis stores start URLs for spiders. Examples show default FIFO, unordered SET, and priority-based ZSET configurations.

```python
# FIFO queue (default)
REDIS_START_URLS_KEY = 'myspider:start_urls'

# Set-based (unordered, unique)
REDIS_START_URLS_AS_SET = True

# Sorted set (priority-based)
REDIS_START_URLS_AS_ZSET = True

# With priority key checking
REDIS_KEY_CHECK_INTERVAL = 60  # Check every 60 seconds

# Auto-close after idle
MAX_IDLE_TIME_BEFORE_CLOSE = 300  # 5 minutes
```

--------------------------------

### Start Docker Compose for Example Project

Source: https://github.com/rmax/scrapy-redis/blob/master/example-project/README.rst

Launches the Scrapy Redis example project using Docker Compose. The '-d' flag runs the containers in detached mode (background).

```bash
docker-compose up
```

--------------------------------

### Basic RedisStatsCollector Setup

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/stats.md

Example settings.py configuration to use RedisStatsCollector, including specifying the Redis host and port. Stats are automatically tracked and stored in Redis.

```python
# settings.py
STATS_CLASS = 'scrapy_redis.stats.RedisStatsCollector'
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
```

--------------------------------

### Basic Scrapy-Redis Pipeline Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

A basic setup for using RedisPipeline with Scrapy. This includes the necessary settings in settings.py and a simple spider to yield items.

```python
# settings.py
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 400,
}
REDIS_HOST = 'localhost'
REDIS_PORT = 6379

# spider.py
import scrapy

class MySpider(scrapy.Spider):
    name = 'myspider'
    start_urls = ['https://example.com']
    
    def parse(self, response):
        yield {
            'url': response.url,
            'title': response.css('title::text').get(),
        }
```

--------------------------------

### RFPDupeFilter Usage Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/dupefilter.md

Demonstrates how to instantiate and use RFPDupeFilter to check for duplicate requests. Includes setup, checking, and cleanup.

```python
from scrapy_redis.dupefilter import RFPDupeFilter
from scrapy_redis.connection import get_redis
from scrapy import Request
from scrapy.settings import Settings

# Setup
settings = Settings({'DUPEFILTER_DEBUG': True})
server = get_redis(host='localhost')
dupefilter = RFPDupeFilter(server, key='myspider:dupefilter', debug=True)

# Check for duplicates
req1 = Request('https://example.com')
req2 = Request('https://example.com')
req3 = Request('https://other.com')

print(dupefilter.request_seen(req1))  # False (first time)
print(dupefilter.request_seen(req2))  # True (duplicate)
print(dupefilter.request_seen(req3))  # False (different URL)

# Cleanup
dupefilter.clear()
```

--------------------------------

### Install Scrapy-Redis in Virtual Environment

Source: https://github.com/rmax/scrapy-redis/blob/master/CONTRIBUTING.rst

Set up a virtual environment and install Scrapy-Redis locally for development. This includes installing virtualenv, creating an environment, activating it, and installing project dependencies.

```shell
pip install virtualenv==20.0.23
virtualenv --python=/usr/bin/python3 ~/scrapy_redis
source ~/scrapy_redis/bin/activate
cd scrapy-redis/
pip install -r requirements-install.txt
pip install .
```

--------------------------------

### Install Scrapy-Redis from Source

Source: https://github.com/rmax/scrapy-redis/wiki/Installation

Install Scrapy-Redis in editable mode after downloading the source code. This is useful for development.

```bash
pip install -e .
```

--------------------------------

### Install Redis Server on Debian/Ubuntu

Source: https://github.com/rmax/scrapy-redis/wiki/Installation

Set up a Redis server on Debian-based systems by adding the official Redis repository and installing the Redis package.

```bash
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee 
/etc/apt/sources.list.d/redis.list
sudo apt update && sudo apt install -y redis
```

--------------------------------

### Pipeline Configuration Examples

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

Configure item storage in Redis. Examples show setting the pipeline class, custom items key, and custom serializer.

```python
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 400,
}

# Custom items key
REDIS_ITEMS_KEY = 'output:%(spider)s'

# Custom serializer
REDIS_ITEMS_SERIALIZER = 'myproject.serializers.custom_json'
```

--------------------------------

### Stats Settings Configuration Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

Configure statistics collection in Redis. Example shows enabling the Redis stats collector and setting the stats key.

```python
# Enable Redis stats collector
STATS_CLASS = 'scrapy_redis.stats.RedisStatsCollector'
STATS_KEY = 'stats:%(spider)s'
```

--------------------------------

### Install Testing Dependencies

Source: https://github.com/rmax/scrapy-redis/blob/master/CONTRIBUTING.rst

Install the necessary dependencies for running tests and linters, including flake8 and pytest.

```shell
pip install -r requirements-tests.txt
```

--------------------------------

### Verify Pyenv Installation

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Check if pyenv is installed correctly and list the available Python versions.

```bash
pyenv versions
```

--------------------------------

### PriorityQueue Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Demonstrates adding requests with different priorities to a PriorityQueue and retrieving them in order.

```python
from scrapy_redis.queue import PriorityQueue
from scrapy_redis.connection import get_redis
from scrapy import Request, Spider

server = get_redis(host='localhost')
spider = Spider(name='myspider')

queue = PriorityQueue(server, spider, key='%(spider)s:requests')

# Add requests with different priorities
queue.push(Request('https://example.com/low', priority=1))
queue.push(Request('https://example.com/high', priority=100))
queue.push(Request('https://example.com/medium', priority=50))

# Pop in priority order (highest first)
req1 = queue.pop()  # priority=100
req2 = queue.pop()  # priority=50
req3 = queue.pop()  # priority=1
```

--------------------------------

### Complete Scrapy-Redis settings.py Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

A comprehensive example of a settings.py file for a Scrapy project using scrapy-redis, including connection, scheduler, dupefilter, spider, pipeline, and stats configurations.

```python
# Project settings for Scrapy with Redis

BOT_NAME = 'myproject'
SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

# ===== Redis Connection =====
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 0
REDIS_ENCODING = 'utf-8'

REDIS_PARAMS = {
    'socket_timeout': 30,
    'socket_connect_timeout': 30,
    'retry_on_timeout': True,
}

# ===== Scheduler =====
SCHEDULER = 'scrapy_redis.scheduler.Scheduler'
SCHEDULER_PERSIST = True
SCHEDULER_FLUSH_ON_START = False
SCHEDULER_IDLE_BEFORE_CLOSE = 10
SCHEDULER_QUEUE_CLASS = 'scrapy_redis.queue.PriorityQueue'
SCHEDULER_DUPEFILTER_CLASS = 'scrapy_redis.dupefilter.RFPDupeFilter'

# ===== Dupefilter =====
DUPEFILTER_CLASS = 'scrapy_redis.dupefilter.RFPDupeFilter'
DUPEFILTER_DEBUG = False

# ===== Spider Configuration =====
REDIS_START_URLS_KEY = '%(name)s:start_urls'
REDIS_START_URLS_AS_SET = False
REDIS_START_URLS_AS_ZSET = False
REDIS_KEY_CHECK_INTERVAL = None
MAX_IDLE_TIME_BEFORE_CLOSE = 0

# ===== Pipelines =====
ITEM_PIPELINES = {
    'myproject.pipelines.ProcessPipeline': 300,
    'scrapy_redis.pipelines.RedisPipeline': 400,
}
REDIS_ITEMS_KEY = '%(spider)s:items'

# ===== Stats =====
STATS_CLASS = 'scrapy_redis.stats.RedisStatsCollector'
STATS_KEY = '%(spider)s:stats'

```

--------------------------------

### FifoQueue Example Usage

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Demonstrates how to initialize and use the FifoQueue to add and retrieve requests, showing the FIFO behavior.

```python
from scrapy_redis.queue import FifoQueue
from scrapy_redis.connection import get_redis
from scrapy import Request, Spider

server = get_redis(host='localhost')
spider = Spider(name='myspider')

queue = FifoQueue(server, spider, key='%(spider)s:requests')

# Add requests
queue.push(Request('https://example.com/1'))
queue.push(Request('https://example.com/2'))

print(len(queue))  # 2

# Pop requests in FIFO order
req1 = queue.pop()  # First pushed
req2 = queue.pop()  # Second pushed
```

--------------------------------

### Install Scrapy-Redis from GitHub

Source: https://github.com/rmax/scrapy-redis/blob/master/README.rst

Clone the scrapy-redis repository from GitHub and install it locally. This is recommended for using the JSON data feature.

```bash
git clone https://github.com/darkrho/scrapy-redis.git
cd scrapy-redis
python setup.py install
```

--------------------------------

### Install Scrapy-Redis in Virtual Environment

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Set up a virtual environment and install scrapy-redis along with its test dependencies. Ensure virtualenv version is at least 20.23.

```bash
pip install virtualenv>=20.23 # tox require version >=20.23
virtualenv --python=/usr/bin/python3 ~/
scrapy_redis
source ~/scrapy_redis/bin/activate
cd scrapy-redis/
pip install -r requirements-tests.txt
pip install .
```

--------------------------------

### Basic RedisMixin Spider Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

Example of a Scrapy spider inheriting from RedisMixin and Spider. The setup_redis method is automatically called by from_crawler.

```python
from scrapy_redis.spiders import RedisMixin
from scrapy import Spider

class MySpider(RedisMixin, Spider):
    name = 'myspider'
    
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # setup_redis called by from_crawler
```

--------------------------------

### Scale Crawler Instances with Docker Compose

Source: https://github.com/rmax/scrapy-redis/blob/master/example-project/README.rst

Adjusts the number of 'crawler' service instances running in the Docker Compose setup. This example scales to 4 instances.

```bash
docker-compose scale crawler=4
```

--------------------------------

### Initial Crawl and Stop

Source: https://github.com/rmax/scrapy-redis/wiki/Running-the-example-project

Start the crawler for the first time and then stop it. This initializes the request queue.

```bash
cd example-project
scrapy crawl dmoz
...
^C
```

--------------------------------

### Install Build Dependencies for Pyenv

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Install necessary system dependencies required for building Python versions with pyenv.

```bash
sudo apt update && sudo install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm
```

--------------------------------

### LifoQueue Example

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Illustrates using LifoQueue to add and remove requests in a Last-In-First-Out manner.

```python
from scrapy_redis.queue import LifoQueue
from scrapy_redis.connection import get_redis
from scrapy import Request, Spider

server = get_redis(host='localhost')
spider = Spider(name='myspider')

queue = LifoQueue(server, spider, key='%(spider)s:requests')

# Add requests
queue.push(Request('https://example.com/1'))
queue.push(Request('https://example.com/2'))

# Pop requests in LIFO order
req1 = queue.pop()  # Second pushed
req2 = queue.pop()  # First pushed
```

--------------------------------

### Example Python Code Block for picklecompat.dumps

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet shows how to use picklecompat.dumps for serialization.

```python
from scrapy_redis import picklecompat

my_object = {'data': 123}
serialized_data = picklecompat.dumps(my_object)
print(serialized_data)

```

--------------------------------

### Example Python Code Block

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This is a generic example of a Python code block. It demonstrates basic syntax and structure.

```python
import scrapy

class MySpider(scrapy.Spider):
    name = 'my_spider'
    # ... spider logic ...

```

--------------------------------

### Example Python Code Block for picklecompat.loads

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet shows how to use picklecompat.loads for deserialization.

```python
from scrapy_redis import picklecompat

serialized_data = b'...' # Assume this is valid serialized data
original_object = picklecompat.loads(serialized_data)
print(original_object)

```

--------------------------------

### Install Scrapy and redis-py

Source: https://github.com/rmax/scrapy-redis/wiki/Installation

Install the core dependencies Scrapy and redis-py using pip. Ensure you have versions compatible with Scrapy-Redis.

```bash
pip install scrapy>=2.6.0 redis>=4.2
```

--------------------------------

### Scrapy-Redis Start URLs Key

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Defines the Redis key format for start URLs using the spider name placeholder.

```text
%(spider)s:start_urls
```

--------------------------------

### RedisMixin make_request_from_data Example

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/scrapy_redis.md

Example of JSON data structure for creating a Request from Redis. Supports url, meta, and other optional parameters like method and cookies.

```json
{
    "url": "https://example.com",
    "meta": {
        "job-id":"123xsd",
        "start-date":"dd/mm/yy",
    },
    "url_cookie_key":"fertxsas",
    "method":"POST",
}
```

--------------------------------

### Multi-Key Priority Queue Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Example of configuring multiple priority keys for a Redis queue using a list.

```python
redis_key = [
    'high_priority:urls',
    'medium_priority:urls',
    'low_priority:urls',
]
```

--------------------------------

### Basic RedisSpider Implementation

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

Example of a custom RedisSpider. Configure name, redis_key, and redis_batch_size. Implement the parse method to process responses.

```python
from scrapy_redis.spiders import RedisSpider

class MySpider(RedisSpider):
    name = 'myspider'
    redis_key = 'myspider:start_urls'
    redis_batch_size = 16
    
    def parse(self, response):
        return {
            'url': response.url,
            'title': response.css('title::text').get(),
        }
```

--------------------------------

### Install Scrapy-Redis Stable Release

Source: https://github.com/rmax/scrapy-redis/wiki/Installation

Install the latest stable version of Scrapy-Redis directly from PyPI using pip.

```bash
pip install scrapy-redis
```

--------------------------------

### Run Tox Tests

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Install and run tox to test your changes against flake8, bandit, pylint, and pytest. Ensure scrapy-redis is installed.

```bash
pip install -U tox
tox
```

--------------------------------

### Custom Queue Key by Environment

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Example of a custom `SCHEDULER_QUEUE_KEY` that includes an environment prefix for segregation.

```python
# By environment
SCHEDULER_QUEUE_KEY = 'prod:%(spider)s:requests'
```

--------------------------------

### Custom Queue Key by Date

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Example of a custom `SCHEDULER_QUEUE_KEY` that includes the date for organization.

```python
# By date
SCHEDULER_QUEUE_KEY = '%(spider)s:requests:2024-01-15'
```

--------------------------------

### Start URLs Key Template Substitution

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Demonstrates template substitution for `REDIS_START_URLS_KEY`, using `%(name)s` which typically resolves to the spider's name.

```python
REDIS_START_URLS_KEY = '%(name)s:start_urls'
# Becomes: myspider:start_urls
```

--------------------------------

### Install Testing Dependencies and Run Linters/Tests

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/contributing.md

Install testing requirements, run flake8 for code linting, and execute pytest for testing. Tox is used to test across multiple Python versions.

```shell
pip install -r requirements-tests.txt
flake8 src/ tests/
python -m pytest --ignore=setup.py
tox
```

--------------------------------

### Push URLs to Redis

Source: https://github.com/rmax/scrapy-redis/wiki/Feeding-a-Spider-from-Redis

Add starting URLs to the Redis queue for your spider using the `redis-cli` tool. The URL is pushed to a list named after your spider and ':start_urls'.

```bash
redis-cli lpush myspider:start_urls http://google.com
```

--------------------------------

### Example Python Code Block with Imports

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This example shows a Python code block that includes necessary imports for a Scrapy spider.

```python
from scrapy_redis.spiders import RedisSpider

class MyRedisSpider(RedisSpider):
    name = 'my_redis_spider'
    # ... spider logic ...

```

--------------------------------

### Configure RedisPipeline in settings.py

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Example configuration for enabling RedisPipeline in Scrapy's settings.py. Includes optional settings for customizing the Redis key and serializer.

```python
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 400,
}

# Optional: customize items key
REDIS_ITEMS_KEY = '%(spider)s:items'

# Optional: custom serializer
REDIS_ITEMS_SERIALIZER = 'myproject.serializers.custom_json_encoder'
```

--------------------------------

### Basic RedisCrawlSpider Implementation

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

Example of a custom RedisCrawlSpider. Configure name, redis_key, and rules for crawling. Implement parse_item to process extracted data.

```python
from scrapy_redis.spiders import RedisCrawlSpider
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import Rule

class MyCrawler(RedisCrawlSpider):
    name = 'mycrawler'
    redis_key = 'urls:tocrawl'
    
    rules = (
        Rule(LinkExtractor(allow=r'/item/'), callback='parse_item'),
    )
    
    def parse_item(self, response):
        return {
            'url': response.url,
            'text': response.css('::text').getall(),
        }
```

--------------------------------

### Run Scrapy Crawler

Source: https://github.com/rmax/scrapy-redis/blob/master/example-project/README.rst

Starts the 'dmoz' Scrapy spider. Use this command to initiate a crawl. The crawler will schedule requests through Redis.

```bash
cd example-project
scrapy crawl dmoz
```

--------------------------------

### Distributed Crawling Setup

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/README.md

Illustrates how multiple spider instances concurrently crawl from a shared Redis queue and utilize a shared deduplication filter. Shows the Redis keys used for the queue and filter.

```python
# Multiple instances crawling from shared queue
# Instance 1: scrapy crawl myspider
# Instance 2: scrapy crawl myspider
# Instance 3: scrapy crawl myspider

# All pull from redis key: myspider:start_urls
# All check dupefilter: myspider:dupefilter
```

--------------------------------

### Example: TypeError from Scheduler Initialization

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/errors.md

Demonstrates how a TypeError is raised when an invalid negative value is provided for 'idle_before_close' during Scheduler initialization.

```python
from scrapy_redis.scheduler import Scheduler

# Raises TypeError: idle_before_close cannot be negative
scheduler = Scheduler(
    server=redis_client,
    idle_before_close=-10  # Invalid
)
```

--------------------------------

### Example Python Code Block for bytes_to_str

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet demonstrates the usage of the bytes_to_str utility for data conversion.

```python
from scrapy_redis.utils import bytes_to_str

byte_data = b'hello'
string_data = bytes_to_str(byte_data)
print(string_data)

```

--------------------------------

### Example Python Code Block for convert_bytes_to_str

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet demonstrates the recursive conversion of bytes to strings.

```python
from scrapy_redis.utils import convert_bytes_to_str

complex_data = {'key': b'value'}
converted_data = convert_bytes_to_str(complex_data)
print(converted_data)

```

--------------------------------

### Import Redis Connection Utilities

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/README.md

Import functions to get a Redis connection instance, either directly or from Scrapy settings.

```python
from scrapy_redis.connection import get_redis, get_redis_from_settings
```

--------------------------------

### Clone Scrapy-Redis Repository

Source: https://github.com/rmax/scrapy-redis/wiki/Home

Clone the official scrapy-redis GitHub repository to get started.

```sh
$ git clone https://github.com/rmax/scrapy-redis.git
```

--------------------------------

### Multiple Pipelines with Validation

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Example of using multiple item pipelines, including a custom validation pipeline before the RedisPipeline. This ensures data integrity before storage.

```python
# settings.py
ITEM_PIPELINES = {
    'myproject.pipelines.ValidationPipeline': 300,
    'scrapy_redis.pipelines.RedisPipeline': 400,
}
```

```python
# pipelines.py
from scrapy_redis.pipelines import RedisPipeline

class ValidationPipeline:
    def process_item(self, item, spider):
        # Validate before storing
        if not item.get('url'):
            raise DropItem(f"Missing URL in {item}")
        return item
```

--------------------------------

### Handle Missing Settings Configuration Error

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/errors.md

Catch KeyError when initializing the Scrapy-Redis scheduler if required settings are missing. The example shows how to add missing settings.

```python
# settings.py - incomplete

from scrapy_redis.scheduler import Scheduler

try:
    scheduler = Scheduler.from_settings(settings)
except KeyError as e:
    print(f"Missing required setting: {e}")
    # Add to settings:
    # REDIS_HOST = 'localhost'
    # REDIS_PORT = 6379
```

--------------------------------

### Example: ValueError from Invalid Queue Class

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/errors.md

Illustrates a ValueError raised when attempting to instantiate a Scheduler with a non-existent queue class path.

```python
from scrapy_redis.scheduler import Scheduler
from scrapy_redis.connection import get_redis

redis_client = get_redis(host='localhost')

# Raises ValueError if queue class doesn't exist
try:
    scheduler = Scheduler(
        server=redis_client,
        queue_cls='nonexistent.QueueClass',  # Invalid class path
    )
except ValueError as e:
    print(f"Failed to instantiate queue: {e}")
```

--------------------------------

### Example Python Code Block for TextColor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet shows how to use the TextColor utility for ANSI color codes.

```python
from scrapy_redis.utils import TextColor

print(TextColor.RED('This is red text'))

```

--------------------------------

### Retrieving and Deserializing Items from Redis

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Provides a Python example for retrieving JSON-encoded items from a Redis list using LPOP and then deserializing them.

```python
import redis
import json

client = redis.StrictRedis(host='localhost', port=6379)

while True:
    item_data = client.lpop('myspider:items')
    if not item_data:
        break
    item = json.loads(item_data.decode('utf-8'))
    print(item)
```

--------------------------------

### Pushing URLs to Redis for Scrapy-Redis

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/MANIFEST.txt

Example of how to push URLs to a Redis list that your scrapy-redis spider will consume. Requires the redis-py library.

```python
import redis
import json

client = redis.StrictRedis()
client.lpush('myspider:start_urls', json.dumps({'url': 'https://example.com'}))
```

--------------------------------

### JSON Data Example for Scrapy-Redis

Source: https://github.com/rmax/scrapy-redis/wiki/Introduction

This JSON structure represents data that can be stored in Redis for distributed crawling with Scrapy-Redis. It includes the URL to be requested, meta-information such as job ID and start date, and an optional cookie key.

```json
{"url": "https://exaple.com", "meta": {"job-id":"123xsd", "start-date":"dd/mm/yy"}, "url_cookie_key":"fertxsas" }
```

--------------------------------

### Configure Pyenv Environment Variables

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Add PYENV_ROOT to your PATH and initialize pyenv by adding the necessary configuration to your .bashrc file.

```bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bashrc
```

--------------------------------

### Start Post-Processing Workers

Source: https://github.com/rmax/scrapy-redis/wiki/Running-the-example-project

Start one or more workers to process items collected by the crawlers. This script consumes items from the queue.

```bash
python process_items.py dmoz:items -v
...
Processing: Kilani Giftware (http://www.dmoz.org/Computers/Shopping/Gifts/)
Processing: NinjaGizmos.com (http://www.dmoz.org/Computers/Shopping/Gifts/)
...
```

--------------------------------

### Start Additional Scrapy Crawlers

Source: https://github.com/rmax/scrapy-redis/wiki/Running-the-example-project

Start one or more additional scrapy crawlers. These will also resume the crawl from the shared queue.

```bash
scrapy crawl dmoz
...
... [dmoz] DEBUG: Resuming crawl (8712 requests scheduled)
```

--------------------------------

### Uninstall Scrapy-Redis

Source: https://github.com/rmax/scrapy-redis/blob/master/README.rst

Uninstall the scrapy-redis package if it was previously installed via pip, especially before installing from source to use the JSON data feature.

```bash
pip uninstall scrapy-redis
```

--------------------------------

### Create Scheduler from Settings

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/scheduler.md

Instantiate the Scheduler using Scrapy settings. Ensure Redis is reachable.

```python
from scrapy_redis.scheduler import Scheduler

scheduler = Scheduler.from_settings(crawler.settings)
```

--------------------------------

### Simple GET Request JSON Structure

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Defines the basic JSON structure for a simple GET request, including the required URL field.

```json
{
  "url": "https://example.com"
}
```

--------------------------------

### RedisPipeline Constructor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Initializes the RedisPipeline with a Redis server instance, an optional key template, and an optional serialization function.

```APIDOC
## RedisPipeline Constructor

### Description
Initializes the RedisPipeline with a Redis server instance, an optional key template, and an optional serialization function.

### Parameters
- **server** (`redis.StrictRedis`) - Required - Redis client instance
- **key** (`str`) - Optional - Redis key template for items (default: `%(spider)s:items`)
- **serialize_func** (`callable`) - Optional - Function to serialize items (default: JSON encoder)
```

--------------------------------

### Import Queue Implementations

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/README.md

Import various queue implementations for different scheduling strategies, including FIFO, LIFO, PriorityQueue, and their aliases.

```python
from scrapy_redis.queue import (
    Base,
    FifoQueue,
    LifoQueue,
    PriorityQueue,
    SpiderQueue,      # Alias for FifoQueue
    SpiderStack,      # Alias for LifoQueue
    SpiderPriorityQueue,  # Alias for PriorityQueue
)
```

--------------------------------

### Redis Connection with Authentication

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

Set up Redis connection parameters, including a password for authentication.

```python
REDIS_PARAMS = {
    'password': 'secret_password',
}
```

--------------------------------

### Get Redis Client with Custom Parameters

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/connection.md

This function allows you to get a Redis client instance by providing custom connection parameters directly or via a URL. It also supports specifying a custom Redis client class.

```python
from scrapy_redis.connection import get_redis

server = get_redis(host='localhost', port=6379)
```

```python
from scrapy_redis.connection import get_redis

# Using explicit parameters
server = get_redis(
    host='redis.example.com',
    port=6379,
    db=0,
    encoding='utf-8',
    socket_timeout=30,
    socket_connect_timeout=30,
    retry_on_timeout=True
)

# Using connection URL
server = get_redis(url='redis://localhost:6379/0')

# With custom Redis class
import redis
server = get_redis(
    redis_cls=redis.Redis,
    host='localhost',
    port=6379
)
```

--------------------------------

### RedisPipeline from_settings Method

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Creates a RedisPipeline instance using Scrapy settings. Allows customization of the items key and serializer via settings.

```python
@classmethod
def from_settings(cls, settings: scrapy.settings.Settings) -> RedisPipeline
```

```python
from scrapy_redis.pipelines import RedisPipeline

pipeline = RedisPipeline.from_settings(settings)
```

--------------------------------

### scrapy_redis.spiders.RedisCrawlSpider

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/modules.md

A Scrapy spider that supports reading starting URLs from Redis.

```APIDOC
## Class `RedisCrawlSpider` in `scrapy_redis.spiders`

### Attributes

#### `redis_key` (string)

The Redis key used for storing starting URLs.

#### `redis_batch_size` (int)

The number of URLs to fetch from Redis in a single batch.

#### `redis_encoding` (string)

The encoding to use when reading URLs from Redis.

#### `REDIS_START_URLS_KEY` (string)

Default Redis key for starting URLs if `redis_key` is not specified.
```

--------------------------------

### RedisCrawlSpider Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/modules.md

Configuration options for RedisCrawlSpider to manage start URLs and encoding.

```APIDOC
## RedisCrawlSpider Configuration

### Description
Configuration attributes for `RedisCrawlSpider` that control how initial URLs are fetched and encoded.

### Attributes

- **REDIS_START_URLS_BATCH_SIZE** (int) - The number of start URLs to fetch from Redis at once.
- **REDIS_START_URLS_AS_SET** (bool) - If True, start URLs are treated as a set; otherwise, as a list.
- **REDIS_ENCODING** (str) - The encoding to use for Redis keys and values.
```

--------------------------------

### Get Queue Length

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/scheduler.md

Returns the current number of pending requests in the queue.

```python
pending = len(scheduler)
print(f"{pending} requests in queue")
```

--------------------------------

### RFPDupeFilter Constructor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/dupefilter.md

Initializes the RFPDupeFilter with a Redis server instance, a key for storing fingerprints, and an optional debug flag.

```python
def __init__(
    server: redis.StrictRedis,
    key: str,
    debug: bool = False,
) -> None:
    pass
```

--------------------------------

### Import FifoQueue Class

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Import the FifoQueue class, which implements a First-In-First-Out queue using Redis lists.

```python
from scrapy_redis.queue import FifoQueue
```

--------------------------------

### RFPDupeFilter Constructor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/dupefilter.md

Initializes the RFPDupeFilter with a Redis server instance, a key for storing fingerprints, and an optional debug flag.

```APIDOC
## RFPDupeFilter Constructor

### Description
Initializes the RFPDupeFilter with a Redis server instance, a key for storing fingerprints, and an optional debug flag.

### Parameters
#### Path Parameters
- **server** (redis.StrictRedis) - Required - Redis server instance
- **key** (str) - Required - Redis key where fingerprints are stored
- **debug** (bool) - Optional - Log all filtered duplicates (not just first)

### Returns
`RFPDupeFilter` — Dupefilter instance.
```

--------------------------------

### RedisPipeline Constructor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Defines the parameters for initializing the RedisPipeline, including the Redis server, key template, and serialization function.

```python
def __init__(
    server: redis.StrictRedis,
    key: str = "% (spider)s:items",
    serialize_func: callable = ScrapyJSONEncoder().encode,
) -> None
```

--------------------------------

### Example Python Code Block for RedisStatsCollector

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet demonstrates the basic structure for a RedisStatsCollector in Scrapy.

```python
from scrapy_redis.stats import RedisStatsCollector

class MyStatsCollector(RedisStatsCollector):
    # ... stats collection logic ...
    pass

```

--------------------------------

### Example Python Code Block for RedisPipeline

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet illustrates the basic structure for a RedisPipeline in Scrapy.

```python
from scrapy_redis.pipelines import RedisPipeline

class MyPipeline(RedisPipeline):
    # ... pipeline logic ...
    pass

```

--------------------------------

### Base Queue Constructor Parameters

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Defines the parameters for initializing the abstract Base queue class, including Redis server, spider instance, key template, and an optional serializer.

```python
def __init__(
    server: redis.StrictRedis,
    spider: scrapy.Spider,
    key: str,
    serializer=None,
) -> None
```

--------------------------------

### RedisPipeline.from_settings()

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Class method to create a RedisPipeline instance from Scrapy settings. It allows customization of the Redis items key and the serializer function via settings.

```APIDOC
## RedisPipeline.from_settings()

### Description
Class method to create a RedisPipeline instance from Scrapy settings. It allows customization of the Redis items key and the serializer function via settings.

### Method
`@classmethod from_settings(cls, settings: scrapy.settings.Settings) -> RedisPipeline`

### Parameters
- **settings** (`scrapy.settings.Settings`) - Required - Scrapy settings object

### Returns
`RedisPipeline` instance

### Settings
- `REDIS_ITEMS_KEY` (str, optional) — Override default items key
- `REDIS_ITEMS_SERIALIZER` (str, optional) — Path to custom serializer function

### Example
```python
from scrapy_redis.pipelines import RedisPipeline

pipeline = RedisPipeline.from_settings(settings)
```
```

--------------------------------

### Scheduler.from_settings()

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/scheduler.md

Creates a Scheduler instance using Scrapy settings. This method is useful for configuring the scheduler through a Scrapy project's settings file.

```APIDOC
## Scheduler.from_settings()

### Description
Creates a Scheduler instance using Scrapy settings. This method is useful for configuring the scheduler through a Scrapy project's settings file.

### Method
`classmethod`

### Parameters
- **settings** (`scrapy.settings.Settings`) - Required - Scrapy settings object

### Returns
`Scheduler` instance

### Throws
Connection errors if Redis is unreachable

### Example
```python
from scrapy_redis.scheduler import Scheduler

scheduler = Scheduler.from_settings(crawler.settings)
```
```

--------------------------------

### Example Python Code Block for is_dict

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/INDEX.md

This snippet shows how to use the is_dict utility for JSON validation.

```python
from scrapy_redis.utils import is_dict

my_dict = {'key': 'value'}
print(is_dict(my_dict))

```

--------------------------------

### Basic Local Redis Connection

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/configuration.md

Configure connection to a local Redis instance using host, port, and database number.

```python
REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 0
```

--------------------------------

### RedisStatsCollector.inc_value()

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/stats.md

Increments a statistics counter in Redis. It can initialize the counter to a specified start value if it doesn't exist.

```APIDOC
## RedisStatsCollector.inc_value()

### Description
Increment a stats counter.

### Parameters
- **key** (str) - Required - Stats key
- **count** (int, optional) - Amount to increment (default: 1)
- **start** (int, optional) - Initial value if key doesn't exist (default: 0)
- **spider** (`scrapy.Spider`, optional) - Spider

### Behavior
- Initializes to `start` if key doesn't exist
- Uses Redis HINCRBY for atomic increment

### Request Example
```python
collector.inc_value('item_count')  # +1
collector.inc_value('request_count', 5)  # +5

# In scheduler
if stats:
    stats.inc_value('scheduler/enqueued/redis')
```
```

--------------------------------

### Import Base Queue Class

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/queue.md

Import the abstract base class for per-spider Redis queues.

```python
from scrapy_redis.queue import Base
```

--------------------------------

### Clone Pyenv Repository

Source: https://github.com/rmax/scrapy-redis/wiki/Getting-Started

Download the pyenv tool from its GitHub repository to your local machine.

```bash
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
```

--------------------------------

### RedisSpider

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

A Scrapy Spider that reads URLs from a Redis queue when idle. It integrates with Redis for managing the start URLs.

```APIDOC
## RedisSpider

Spider reading URLs from Redis queue when idle.

```python
from scrapy_redis.spiders import RedisSpider
```

### Class Attributes

| Attribute | Type | Default | Description |
|-----------|------|---------|-------------|
| redis_key | str | — | Redis key (default: `<spider_name>:start_urls`) |
| redis_batch_size | int | — | Batch size (default: CONCURRENT_REQUESTS) |
| redis_encoding | str | — | Encoding (default: utf-8) |

### Methods

#### from_crawler()

Create spider from crawler with Redis integration.

```python
@classmethod
def from_crawler(cls, crawler: scrapy.Crawler, *args, **kwargs) -> RedisSpider
```

**Parameters:**
- `crawler` (`scrapy.Crawler`) — Crawler object
- `*args`, `**kwargs` — Spider arguments

**Returns:** RedisSpider instance

**Behavior:** Calls `setup_redis()` automatically
```

--------------------------------

### RedisSpider Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/modules.md

Configuration options for RedisSpider, including Redis key names and batch sizes for start URLs.

```APIDOC
## RedisSpider Configuration

### Description
Configuration attributes specific to `RedisSpider` for managing start URLs and encoding, extending `RedisMixin`'s capabilities.

### Attributes

- **redis_key** (str) - The Redis key used for storing requests.
- **redis_batch_size** (int) - The number of requests to fetch from Redis in a single batch.
- **redis_encoding** (str) - The encoding to use for Redis keys and values.
- **REDIS_START_URLS_KEY** (str) - The Redis key used for storing the initial list of URLs.
- **REDIS_START_URLS_BATCH_SIZE** (int) - The number of start URLs to fetch from Redis at once.
- **REDIS_START_URLS_AS_SET** (bool) - If True, start URLs are treated as a set; otherwise, as a list.
- **REDIS_ENCODING** (str) - The encoding to use for Redis keys and values.
```

--------------------------------

### Basic Redis Pipeline Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/pipelines.md

Configure the default RedisPipeline in your settings.py. You can also specify a custom serializer for items.

```python
# settings.py
ITEM_PIPELINES = {
    'scrapy_redis.pipelines.RedisPipeline': 400,
}
REDIS_ITEMS_SERIALIZER = 'myproject.serializers.custom_serializer'
```

--------------------------------

### Multi-Key Prioritization Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/README.md

Set up multiple Redis keys for different priority queues and configure the interval for checking these keys. This allows for managing different levels of crawl urgency.

```python
# Multiple queues with priority checking
redis_key = [
    'urgent:urls',
    'normal:urls',
    'low:urls',
]

REDIS_KEY_CHECK_INTERVAL = 60  # Check every 60s
```

--------------------------------

### Get All Stats for a Spider

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/stats.md

Retrieve all statistics stored for a given spider as a dictionary. If no spider is specified, it uses the current spider.

```python
all_stats = collector.get_stats()
for key, value in all_stats.items():
    print(f"{key}: {value}")
```

--------------------------------

### Serialize and Deserialize Object

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/utils.md

Demonstrates the basic usage of `dumps` and `loads` from the `picklecompat` module to serialize and then deserialize a Python object.

```python
from scrapy_redis import picklecompat

encoded = picklecompat.dumps(obj)
decoded = picklecompat.loads(encoded)
```

--------------------------------

### Custom Items Key with Nested Structure

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/types.md

Example of a custom `REDIS_ITEMS_KEY` using a nested directory-like structure for organizing scraped items.

```python
# Nested structure
REDIS_ITEMS_KEY = 'output/%(spider)s/items'
```

--------------------------------

### Project Requirements

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/README.md

List of Python packages and their minimum version requirements for running scrapy-redis.

```text
scrapy>=2.6.0
redis>=4.2
six>=1.15
```

--------------------------------

### Handling Redis Connection Error

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/errors.md

Provides an example of catching and handling a ConnectionError when attempting to connect to a Redis server that is not running or unreachable.

```python
from scrapy_redis.connection import get_redis

try:
    server = get_redis(host='localhost', port=6379)
    server.ping()
except ConnectionError as e:
    print(f"Cannot connect to Redis: {e}")
    # Start Redis: redis-server
```

--------------------------------

### Example: ValueError from Invalid Spider Configuration

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/errors.md

Shows a ValueError occurring when a RedisSpider is configured with an invalid value for 'redis_batch_size' that cannot be converted to an integer.

```python
from scrapy_redis.spiders import RedisSpider

class MySpider(RedisSpider):
    name = 'myspider'
    redis_batch_size = "not_an_int"  # Invalid

try:
    spider = MySpider()
except ValueError as e:
    print(f"Invalid spider configuration: {e}")
```

--------------------------------

### RedisSpider from_crawler Method

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

Class method to create a RedisSpider instance from a crawler object. It automatically calls setup_redis().

```python
@classmethod
def from_crawler(cls, crawler: scrapy.Crawler, *args, **kwargs) -> RedisSpider
```

--------------------------------

### Initialize Scrapy-Redis Scheduler

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/scheduler.md

Configures and initializes the Scrapy-Redis scheduler using provided settings. Supports persistent queues and priority queues.

```python
from scrapy_redis.scheduler import Scheduler
from scrapy.settings import Settings
from scrapy import Spider, Request

# Configure in settings.py
class MySpider(Spider):
    name = 'myspider'

# Use scheduler
settings = Settings({
    'SCHEDULER': 'scrapy_redis.scheduler.Scheduler',
    'SCHEDULER_PERSIST': True,
    'SCHEDULER_QUEUE_CLASS': 'scrapy_redis.queue.PriorityQueue',
    'SCHEDULER_IDLE_BEFORE_CLOSE': 10,
})

scheduler = Scheduler.from_settings(settings)
```

--------------------------------

### Create a New Branch for Development

Source: https://github.com/rmax/scrapy-redis/blob/master/CONTRIBUTING.rst

Create a new branch for your bugfix or feature development work after setting up the local environment.

```shell
git checkout -b name-of-your-bugfix-or-feature
```

--------------------------------

### Download Source Tarball

Source: https://github.com/rmax/scrapy-redis/blob/master/docs/installation.md

Download the Scrapy-Redis source code as a tarball using curl. This is an alternative method for obtaining the source code before installation.

```console
curl  -OL https://github.com/rolando/scrapy-redis/tarball/master
```

--------------------------------

### Scheduler Constructor

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/scheduler.md

Initializes a new Scheduler instance. It requires a Redis server connection and allows configuration of persistence, queueing, deduplication, and serialization.

```APIDOC
## Scheduler Constructor

### Description
Initializes a new Scheduler instance. It requires a Redis server connection and allows configuration of persistence, queueing, deduplication, and serialization.

### Parameters

#### Parameters
- **server** (`redis.StrictRedis`) - Required - Redis server instance
- **persist** (bool) - Optional - Keep requests in Redis queue after spider closes. Default: `False`
- **flush_on_start** (bool) - Optional - Clear queue and dupefilter when spider opens. Default: `False`
- **queue_key** (str) - Optional - Redis key template for request queue (supports `%(spider)s` placeholder). Default: `"%%(spider)s:requests"`
- **queue_cls** (str) - Optional - Importable path to queue class. Default: `"scrapy_redis.queue.PriorityQueue"`
- **dupefilter** (object) - Optional - Pre-instantiated dupefilter object. Default: `None`
- **dupefilter_key** (str) - Optional - Redis key template for dupefilter (supports `%(spider)s` placeholder). Default: `"%%(spider)s:dupefilter"`
- **dupefilter_cls** (str) - Optional - Importable path to dupefilter class. Default: `"scrapy_redis.dupefilter.RFPDupeFilter"`
- **idle_before_close** (int) - Optional - Seconds to wait before giving up when queue is empty. Default: `0`
- **serializer** (module) - Optional - Serializer module with `loads()` and `dumps()` functions. Default: `None`

### Returns
`Scheduler` — Scheduler instance.

### Throws
- `TypeError` — if `idle_before_close < 0`
- `ValueError` — if queue class cannot be instantiated
```

--------------------------------

### Push URLs to Redis for Distributed Crawling

Source: https://github.com/rmax/scrapy-redis/blob/master/_autodocs/api-reference/spiders.md

Example of pushing URLs to a Redis list using the redis-py client. URLs are JSON-encoded before being pushed.

```python
import redis
import json

client = redis.StrictRedis(host='localhost', port=6379)

urls = [
    {"url": "https://example.com/1"},
    {"url": "https://example.com/2"},
    {"url": "https://example.com/3"},
]

for url_data in urls:
    client.lpush('myspider:start_urls', json.dumps(url_data))
```