Scrapy

https://docs.scrapy.org/en/latest/
Admin
Scrapy is a fast high-level web crawling and web scraping framework used to crawl websites and...
Tokens:373,440
Snippets:3,582
Trust Score:9.8
Update:4 months ago
Show doc for...
Tokens:
Raw
### Start a New Scrapy Project (Example)

Source: https://docs.scrapy.org/en/latest/topics/commands.html

An example of how to use the `startproject` command to create a project named 'myproject'.

```bash
$ scrapy startproject myproject

```

--------------------------------

### Async Spider Middleware `process_start` Example

Source: https://docs.scrapy.org/en/latest/topics/spider-middleware.html

Implement an asynchronous `process_start` method to iterate over and potentially modify the output of `start()` or an earlier middleware.

```python
async def process_start(self, start):
    async for item_or_request in start:
        yield item_or_request
```

--------------------------------

### Inspect Scrapy Shell Output

Source: https://docs.scrapy.org/en/latest/intro/tutorial.html

Example of the typical output when starting the Scrapy shell, showing available objects and shortcuts.

```text
[ ... Scrapy log here ... ]
2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/page/1/> (referer: None)
[s] Available Scrapy objects:
[s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s]   crawler    <scrapy.crawler.Crawler object at 0x7fa91d888c90>
[s]   item       {}
[s]   request    <GET https://quotes.toscrape.com/page/1/>
[s]   response   <200 https://quotes.toscrape.com/page/1/>
[s]   settings   <scrapy.settings.Settings object at 0x7fa91d888c10>
[s]   spider     <DefaultSpider 'default' at 0x7fa91c8af990>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

```

--------------------------------

### Install form2request

Source: https://docs.scrapy.org/en/latest/topics/request-response.html

Install the form2request library using pip.

```bash
pip install form2request

```

--------------------------------

### Default Start Method Behavior

Source: https://docs.scrapy.org/en/latest/topics/spiders.html

Illustrates the default behavior of the `start` method when using `start_urls`, which is equivalent to yielding requests with `dont_filter=True`.

```python
async def start(self):
    for url in self.start_urls:
        yield Request(url, dont_filter=True)

```

--------------------------------

### Starting CrawlerProcess with Reactor

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html

Starts the Twisted reactor, adjusts its thread pool size, and installs a DNS resolver. Optionally stops the reactor after all crawlers finish.

```python
from twisted.internet import reactor

if stop_after_crawl:
    d = self.join()
    # Don't start the reactor if the deferreds are already fired
    if d.called:
        return
    d.addBoth(self._stop_reactor)

self._setup_reactor(install_signal_handlers)
reactor.run(installSignalHandlers=install_signal_handlers)  # blocking call
```

--------------------------------

### Install Asyncio Reactor

Source: https://docs.scrapy.org/en/latest/topics/asyncio.html

Installs the AsyncioSelectorReactor for Scrapy. Use this when manually setting up the reactor or if Scrapy complains about a pre-installed reactor.

```python
from twisted.internet.asyncioreactor import install_reactor

install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")

```

--------------------------------

### Install and Configure RerpRobotParser

Source: https://docs.scrapy.org/en/latest/topics/downloader-middleware.html

To use the Robotexclusionrulesparser, install it and set ROBOTSTXT_PARSER.

```bash
pip install robotexclusionrulesparser
```

```python
ROBOTSTXT_PARSER = 'scrapy.robotstxt.RerpRobotParser'
```

--------------------------------

### Yielding Items from Start Method

Source: https://docs.scrapy.org/en/latest/topics/spiders.html

Shows how to yield items directly from the `start` method, bypassing request processing for simple data extraction.

```python
async def start(self):
    yield {"foo": "bar"}

```

--------------------------------

### Install Scrapy with Pip

Source: https://docs.scrapy.org/en/latest/intro/install.html

Install Scrapy and its dependencies from PyPI using pip. It's strongly recommended to do this within a virtual environment.

```bash
pip install Scrapy

```

--------------------------------

### Install Xcode Command Line Tools on macOS

Source: https://docs.scrapy.org/en/latest/intro/install.html

Run this command in the terminal to install the necessary C compiler and development headers on macOS.

```bash
xcode-select --install

```

--------------------------------

### Start Scrapy Engine (Deprecated)

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/engine.html

Starts the Scrapy engine. This method is deprecated and `start_async()` should be used instead.

```python
def start(
        self,
        _start_request_processing: bool = True
    ) -> Deferred[None]:  # pragma: no cover
        warnings.warn(
            "ExecutionEngine.start() is deprecated, use start_async() instead",
            ScrapyDeprecationWarning,
            stacklevel=2,
        )
        return deferred_from_coro(
            self.start_async(_start_request_processing=_start_request_processing)
        )
```

--------------------------------

### Instantiate and Start Crawler Synchronously

Source: https://docs.scrapy.org/en/latest/topics/api.html

Use `crawl` to start the crawler with a spider class and arguments. This method should be called only once and returns a deferred that fires when the crawl is finished.

```python
crawl(_* args: Any_, _** kwargs: Any_) → Generator[Deferred[Any], Any, None][source]
    
    Start the crawler by instantiating its spider class with the given _args_ and _kwargs_ arguments, while setting the execution engine in motion. Should be called only once.
    Return a deferred that is fired when the crawl is finished. 
```

--------------------------------

### Install Twisted Reactor

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html

Installs the Twisted reactor, optionally with a specific asyncio event loop. It handles different reactor types and suppresses errors if a reactor is already installed.

```python
from twisted.internet import asyncioreactor, error
from scrapy.utils.misc import load_object

def install_reactor(reactor_path: str, event_loop_path: str | None = None) -> None:
    """Installs the :mod:`~twisted.internet.reactor` with the specified
    import path. Also installs the asyncio event loop with the specified import
    path if the asyncio reactor is enabled"""
    reactor_class = load_object(reactor_path)
    if reactor_class is asyncioreactor.AsyncioSelectorReactor:
        set_asyncio_event_loop_policy()
        with suppress(error.ReactorAlreadyInstalledError):
            event_loop = set_asyncio_event_loop(event_loop_path)
            asyncioreactor.install(eventloop=event_loop)
    else:
        *module, _ = reactor_path.split(".")
        installer_path = [*module, "install"]
        installer = load_object(".".join(installer_path))
        with suppress(error.ReactorAlreadyInstalledError):
            installer()
```

--------------------------------

### Start Crawling with CrawlerProcess

Source: https://docs.scrapy.org/en/latest/topics/api.html

Start a crawler using the CrawlerProcess. This method accepts a Crawler instance, Spider subclass, or spider name. It returns a Deferred that completes when crawling finishes.

```python
process.crawl("my_spider", start_url="http://example.com")
```

--------------------------------

### Install Pre-commit Hooks

Source: https://docs.scrapy.org/en/latest/contributing.html

Run this command in the root of your local Scrapy repository clone to install the pre-commit hooks. This ensures code is formatted and checked before each commit.

```bash
pre-commit install

```

--------------------------------

### Instantiate and Start Crawler Asynchronously

Source: https://docs.scrapy.org/en/latest/topics/api.html

Use `crawl_async` to start the crawler with a spider class and arguments. This method should be called only once and completes when the crawl is finished.

```python
async crawl_async(_* args: Any_, _** kwargs: Any_) → None[source]
    
    Start the crawler by instantiating its spider class with the given _args_ and _kwargs_ arguments, while setting the execution engine in motion. Should be called only once.
    Added in version 2.14.
    Complete when the crawl is finished. 
```

--------------------------------

### Example Component Initialization with Settings

Source: https://docs.scrapy.org/en/latest/topics/components.html

Demonstrates how a component can access crawler settings during initialization. Use this to enable or disable component features based on project settings.

```python
class MyExtension:
    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(settings.getbool("LOG_ENABLED"))

    def __init__(self, log_is_enabled=False):
        if log_is_enabled:
            print("log is enabled!")

```

--------------------------------

### Get Start Queue Class Helper

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/scheduler.html

A helper method to determine the class for the starting queue (disk or memory) based on crawler settings.

```python
def _get_start_queue_cls(
        self, crawler: Crawler | None, queue: str
    ) -> type[BaseQueue] | None:
        if crawler is None:
            return None
        cls = crawler.settings[f"SCHEDULER_START_{queue}_QUEUE"]
        if not cls:
            return None
        return cast("type[BaseQueue]", load_object(cls))

```

--------------------------------

### Spider Yielding Items with Async Start

Source: https://docs.scrapy.org/en/latest/topics/spiders.html

An example of a spider using an asynchronous `start` method to yield requests and processing them into structured items using `MyItem`.

```python
import scrapy
from myproject.items import MyItem


class MySpider(scrapy.Spider):
    name = "example.com"
    allowed_domains = ["example.com"]

    async def start(self):
        yield scrapy.Request("http://www.example.com/1.html", self.parse)
        yield scrapy.Request("http://www.example.com/2.html", self.parse)
        yield scrapy.Request("http://www.example.com/3.html", self.parse)

    def parse(self, response):
        for h3 in response.xpath("//h3").getall():
            yield MyItem(title=h3)

        for href in response.xpath("//a/@href").getall():
            yield scrapy.Request(response.urljoin(href), self.parse)

```

--------------------------------

### Activating and Configuring Add-ons

Source: https://docs.scrapy.org/en/latest/topics/addons.html

Example of enabling two add-ons in a project's settings.py file. The keys are add-on import paths or class names, and values are their priorities.

```python
ADDONS = {
    'path.to.someaddon': 0,
    SomeAddonClass: 1,
}

```

--------------------------------

### Importing BaseSettings and Settings

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html

Demonstrates the basic import statements for the core settings classes.

```python
from scrapy.settings import BaseSettings, Settings
```

--------------------------------

### Start Reactor and Signal Handlers

Source: https://docs.scrapy.org/en/latest/topics/api.html

Start the reactor or asyncio event loop and optionally install signal handlers for graceful shutdown. If stop_after_crawl is True, the reactor stops once all crawlers are done.

```python
process.start(stop_after_crawl=True, install_signal_handlers=True)
```

--------------------------------

### Create a New Scrapy Project

Source: https://docs.scrapy.org/en/latest/intro/tutorial.html

Run this command in your terminal to create a new Scrapy project named 'tutorial'. This sets up the basic directory structure for your project.

```bash
scrapy startproject tutorial

```

--------------------------------

### Get All Href Attribute Values

Source: https://docs.scrapy.org/en/latest/topics/selectors.html

Example of using `::attr(href)` to extract all `href` attribute values from `<a>` elements.

```python
>>> response.css("a::attr(href)").getall()
['image1.html',
'image2.html',
'image3.html',
'image4.html',
'image5.html']
```

--------------------------------

### Get Text Content of a Title Element

Source: https://docs.scrapy.org/en/latest/topics/selectors.html

Example of using `::text` to extract the text content from a `<title>` element.

```python
>>> response.css("title::text").get()
'Example website'
```

--------------------------------

### RefererMiddleware Initialization

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/spidermiddlewares/referer.html

Initializes the RefererMiddleware, setting up default policies and loading custom policies from settings.

```python
def __init__(self, settings: BaseSettings | None = None):
    self.default_policy: type[ReferrerPolicy] = DefaultReferrerPolicy
    self.policies: dict[str, type[ReferrerPolicy]] = {
        p.name: p
        for p in (
            NoReferrerPolicy,
            NoReferrerWhenDowngradePolicy,
            SameOriginPolicy,
            OriginPolicy,
            StrictOriginPolicy,
            OriginWhenCrossOriginPolicy,
            StrictOriginWhenCrossOriginPolicy,
            UnsafeUrlPolicy,
            DefaultReferrerPolicy,
        )
    }
    # Reference: https://www.w3.org/TR/referrer-policy/#referrer-policy-empty-string
    self.policies[""] = NoReferrerWhenDowngradePolicy
    if settings is None:
        return
    setting_policies = settings.getdict("REFERRER_POLICIES")
    for policy_name, policy_class_import_path in setting_policies.items():
        if policy_class_import_path is None:
            del self.policies[policy_name]
        else:
            self.policies[policy_name] = load_object(policy_class_import_path)
    settings_policy = self._load_policy_class(
        settings.get("REFERRER_POLICY"), allow_import_path=True
    )
    assert settings_policy
    self.default_policy = settings_policy
```

--------------------------------

### Prepare URL Partitions for Distributed Crawls

Source: https://docs.scrapy.org/en/latest/topics/practices.html

This example shows how to prepare lists of URLs for distributed crawling by partitioning them into separate files. Each file corresponds to a partition to be processed by a separate spider instance.

```text
http://somedomain.com/urls-to-crawl/spider1/part1.list
http://somedomain.com/urls-to-crawl/spider1/part2.list
http://somedomain.com/urls-to-crawl/spider1/part3.list

```

--------------------------------

### Scrapy Project Configuration Example

Source: https://docs.scrapy.org/en/latest/topics/commands.html

Shows how to define the default settings module within the scrapy.cfg file.

```ini
[settings]
default = myproject.settings
```

--------------------------------

### Get All Descendant Text Nodes

Source: https://docs.scrapy.org/en/latest/topics/selectors.html

Example of using `*::text` to retrieve all text nodes under an element with the ID 'images'. The output may include whitespace.

```python
>>> response.css("#images *::text").getall()
['\n   ',
'Name: My image 1 ',
'\n   ',
'Name: My image 2 ',
'\n   ',
'Name: My image 3 ',
'\n   ',
'Name: My image 4 ',
'\n   ',
'Name: My image 5 ',
'\n  ']
```

--------------------------------

### Example Image File Storage Structure

Source: https://docs.scrapy.org/en/latest/topics/media-pipeline.html

Illustrates the storage structure for full images and generated thumbnails based on IMAGES_THUMBS configuration.

```text
<IMAGES_STORE>/full/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/small/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/big/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
```

--------------------------------

### Get Spider Middleware Instance

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html

Retrieves the runtime instance of a spider middleware by its class. This method requires the crawl engine to be active, typically after engine start.

```python
def get_spider_middleware(self, cls: type[_T]) -> _T | None:
    """Return the run-time instance of a :ref:`spider middleware
    <topics-spider-middleware>` of the specified class or a subclass, or
    ``None`` if none is found.

    .. versionadded:: 2.12

    This method can only be called after the crawl engine has been created,
    e.g. at signals :signal:`engine_started` or :signal:`spider_opened`.
    """
    if not self.engine:
        raise RuntimeError(
            "Crawler.get_spider_middleware() can only be called after the "
            "crawl engine has been created."
        )
    return self._get_component(cls, self.engine.scraper.spidermw.middlewares)
```

--------------------------------

### Generate a Spider with Basic Template

Source: https://docs.scrapy.org/en/latest/topics/commands.html

Creates a new spider named 'example' using the default 'basic' template.

```bash
$ scrapy genspider example example.com
Created spider 'example' using template 'basic'

```

--------------------------------

### Running a Spider Manually with CrawlerProcess

Source: https://docs.scrapy.org/en/latest/news.html

Example of how to run a Scrapy spider from a script using the refactored Crawler API. This involves creating a `CrawlerProcess` instance and starting the crawl.

```python
from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.start()

```

--------------------------------

### Start Asyncio Reactorless Loop

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html

Initiates the asyncio event loop for reactorless operation. Handles different strategies based on `stop_after_crawl` and installs signal handlers if enabled.

```python
loop = self._reactorless_loop
assert loop

if stop_after_crawl:
    self._reactorless_main_task = loop.create_task(self.join())
else:
    self._reactorless_main_task = loop.create_future()
self._stop_after_crawl = stop_after_crawl

try:
    self._run_loop(install_signal_handlers)  # blocking call
except asyncio.CancelledError:
    pass
finally:
    self._close_loop()
```

--------------------------------

### Registering a Custom Scrapy Command

Source: https://docs.scrapy.org/en/latest/topics/commands.html

Example of a setup.py file demonstrating how to register a custom Scrapy command named 'my_command' from an external module.

```python
from setuptools import setup, find_packages

setup(
    name="scrapy-mymodule",
    entry_points={
        "scrapy.commands": [
            "my_command=my_scrapy_module.commands:MyCommand",
        ],
    },
)


```

--------------------------------

### FeedSlot Start Exporting

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/extensions/feedexport.html

Opens the feed file and initializes the exporter. If postprocessing is configured, it wraps the file object.

```python
def start_exporting(self) -> None:
    if not self._fileloaded:
        self.file = self.storage.open(self.spider)
        if "postprocessing" in self.feed_options:
            self.file = cast(
                "IO[bytes]",
                PostProcessingManager(
                    self.feed_options["postprocessing"],
                    self.file,
                    self.feed_options,
                ),
            )
        self.exporter = self._get_exporter(
            file=self.file,
            format_=self.feed_options["format"],
            fields_to_export=self.feed_options["fields"],
            encoding=self.feed_options["encoding"],
            indent=self.feed_options["indent"],
            **self.feed_options["item_export_kwargs"],
        )
        self._fileloaded = True

    if not self._exporting:
        assert self.exporter
        self.exporter.start_exporting()
        self._exporting = True
```

--------------------------------

### Using BeautifulSoup with Scrapy

Source: https://docs.scrapy.org/en/latest/faq.html

This example demonstrates how to integrate BeautifulSoup for parsing HTML content within a Scrapy spider. It uses 'lxml' as the underlying parser for efficiency. Ensure BeautifulSoup and lxml are installed.

```python
from bs4 import BeautifulSoup
import scrapy


class ExampleSpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["example.com"]
    start_urls = ("http://www.example.com/",)

    def parse(self, response):
        # use lxml to get decent HTML parsing speed
        soup = BeautifulSoup(response.text, "lxml")
        yield {"url": response.url, "title": soup.h1.string}

```

--------------------------------

### BaseRedirectMiddleware _engine_started Method

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/redirect.html

Callback for the engine_started signal. It attempts to find a RefererMiddleware instance and logs a warning if none is found, explaining how to enable Referer header handling.

```python
    def _engine_started(self) -> None:
        self._referer_spider_middleware = self.crawler.get_spider_middleware(
            RefererMiddleware
        )
        if self._referer_spider_middleware:
            return
        redirect_cls = global_object_name(self.__class__)
        referer_cls = global_object_name(RefererMiddleware)
        if self.__class__ in {RedirectMiddleware, MetaRefreshMiddleware}:
            replacement = (
                f"replace {redirect_cls} with a subclass that overrides the "
                f"handle_referer() method"
            )
        else:
            replacement = (
                f"or edit {redirect_cls} (if defined in your code base) to "
                f"override the handle_referer() method, or replace "
                f"{redirect_cls} with a subclass that overrides the "
                f"handle_referer() method."
            )
        logger.warning(
            f"{redirect_cls} found no {referer_cls} instance to handle "
            f"Referer header handling, so the Referer header will be removed "
            f"on redirects. To set a Referer header on redirects, enable "
            f"{referer_cls} (or a subclass), or {replacement}.",
        )
```

--------------------------------

### Iterating over all live HtmlResponse objects

Source: https://docs.scrapy.org/en/latest/topics/leaks.html

Use iter_all() to get an iterator for all live HtmlResponse objects. This allows you to process each object, for example, to extract their URLs and identify patterns related to memory leaks.

```python
from scrapy.utils.trackref import iter_all
[r.url for r in iter_all("HtmlResponse")]
```

--------------------------------

### Scrapy HTML Response Setup

Source: https://docs.scrapy.org/en/latest/topics/selectors.html

Sets up an HtmlResponse object with sample HTML content for testing selectors.

```python
>>> from scrapy.http import HtmlResponse
>>> response = HtmlResponse(
...     url="http://example.com",
...     body="""
... <html>
...     <body>
...         <p class="foo bar-baz">First</p>
...         <p class="foo">Second</p>
...         <p class="bar">Third</p>
...         <p>Fourth</p>
...     </body>
... </html>
... """,
...     encoding="utf-8",
... )

```

--------------------------------

### Configure Scrapy Logging

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/log.html

Initializes Scrapy's logging defaults, routes warnings and Twisted logs, and optionally installs a root handler based on settings. Use this early in your project setup.

```python
from scrapy.settings import Settings, _SettingsKey

def configure_logging(
    settings: Settings | dict[_SettingsKey, Any] | None = None,
    install_root_handler: bool = True,
) -> None:
    """
    Initialize logging defaults for Scrapy.

    :param settings: settings used to create and configure a handler for the
        root logger (default: None).
    :type settings: dict, :class:`~scrapy.settings.Settings` object or ``None``

    :param install_root_handler: whether to install root logging handler
        (default: True)
    :type install_root_handler: bool

    This function does:

    - Route warnings and twisted logging through Python standard logging
    - Assign DEBUG and ERROR level to Scrapy and Twisted loggers respectively
    - Route stdout to log if LOG_STDOUT setting is True

    When ``install_root_handler`` is True (default), this function also
    creates a handler for the root logger according to given settings
    (see :ref:`topics-logging-settings`). You can override default options
    using ``settings`` argument. When ``settings`` is empty or None, defaults
    are used.
    """
    if not sys.warnoptions:
        # Route warnings through python logging
        logging.captureWarnings(True)

    observer = twisted_log.PythonLoggingObserver("twisted")
    observer.start()

    dictConfig(DEFAULT_LOGGING)

    if isinstance(settings, dict) or settings is None:
        settings = Settings(settings)

    if settings.getbool("LOG_STDOUT"):
        sys.stdout = StreamLogger(logging.getLogger("stdout"))

    if install_root_handler:
        install_scrapy_root_handler(settings)
```

--------------------------------

### Configure Periodic Log Extension Settings

Source: https://docs.scrapy.org/en/latest/topics/extensions.html

Example configuration for the PeriodicLog extension, showing how to enable specific stats, delta calculations, and timing data.

```python
custom_settings = {
    "LOG_LEVEL": "INFO",
    "PERIODIC_LOG_STATS": {
        "include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
    },
    "PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
    "PERIODIC_LOG_TIMING_ENABLED": True,
    "EXTENSIONS": {
        "scrapy.extensions.periodic_log.PeriodicLog": 0,
    },
}

```

--------------------------------

### Example Items for Export

Source: https://docs.scrapy.org/en/latest/topics/exporters.html

These are example items used in the output examples for Scrapy's built-in Item Exporters.

```python
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
```

--------------------------------

### Install Scrapy in a Virtual Environment on Ubuntu

Source: https://docs.scrapy.org/en/latest/intro/install.html

After installing the necessary non-Python dependencies, install Scrapy within a virtual environment using pip.

```bash
pip install scrapy

```

--------------------------------

### Example Output Directory Structure with Batching

Source: https://docs.scrapy.org/en/latest/topics/feed-exports.html

Illustrates the directory structure and file naming convention when using FEED_EXPORT_BATCH_ITEM_COUNT and URI placeholders.

```text
->projectname
-->dirname
--->1-filename2020-03-28T14-45-08.237134.json
--->2-filename2020-03-28T14-45-09.148903.json
--->3-filename2020-03-28T14-45-10.046092.json
```

--------------------------------

### Verify Installed Reactor

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html

Checks if the currently installed Twisted reactor matches a specified import path. Raises a RuntimeError if no reactor is installed or if the types do not match.

```python
def verify_installed_reactor(reactor_path: str) -> None:
    """Raise :exc:`RuntimeError` if the installed
    :mod:`~twisted.internet.reactor` does not match the specified import
    path or if no reactor is installed."""
    if not is_reactor_installed():
        raise RuntimeError(
            "verify_installed_reactor() called without an installed reactor."
        )

    from twisted.internet import reactor

    expected_reactor_type = load_object(reactor_path)
    reactor_type = type(reactor)
    if not reactor_type == expected_reactor_type:
        raise RuntimeError(
            f"The installed reactor ({global_object_name(reactor_type)}) "
            f"does not match the requested one ({reactor_path})"
        )
```

--------------------------------

### Check if Asyncio Reactor is Installed

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html

Verifies if the installed Twisted reactor is specifically an AsyncioSelectorReactor. Raises a RuntimeError if no reactor is installed, as per Scrapy's current behavior.

```python
def is_asyncio_reactor_installed() -> bool:
    """Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.

    Raise a :exc:`RuntimeError` if no reactor is installed.

    In a future Scrapy version, when Scrapy supports running without a Twisted
    reactor, this function won't be useful for checking if it's possible to use
    asyncio features, so the code that that doesn't directly require a Twisted
    reactor should use :func:`scrapy.utils.asyncio.is_asyncio_available`
    instead of this function.

    .. versionchanged:: 2.13
       In earlier Scrapy versions this function silently installed the default
       reactor if there was no reactor installed. Now it raises an exception to
       prevent silent problems in this case.
    """
    if not is_reactor_installed():
        raise RuntimeError(
            "is_asyncio_reactor_installed() called without an installed reactor."
        )

    from twisted.internet import reactor

    return isinstance(reactor, asyncioreactor.AsyncioSelectorReactor)
```

--------------------------------

### install_reactor

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html

Installs the Twisted reactor with the specified import path, optionally configuring the asyncio event loop if an asyncio reactor is used.

```APIDOC
## install_reactor

### Description
Installs the :mod:`~twisted.internet.reactor` with the specified import path. It also installs the asyncio event loop with the specified import path if the asyncio reactor is enabled.

### Parameters
- **reactor_path** (str) - The import path for the Twisted reactor class (e.g., 'twisted.internet.asyncioreactor.AsyncioSelectorReactor').
- **event_loop_path** (str | None, optional) - The import path for the asyncio event loop class. Defaults to None, which uses the default event loop policy.

### Behavior
- If `reactor_path` points to `AsyncioSelectorReactor`, it configures the asyncio event loop policy (especially for Windows) and installs the asyncio reactor.
- Otherwise, it loads the specified reactor class and installs it using its respective installer function.
```

--------------------------------

### Start Scrapy Engine Asynchronously

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/engine.html

Starts the execution engine asynchronously. It sets the start time, signals engine startup, and begins processing requests if a spider is available.

```python
async def start_async(self, *, _start_request_processing: bool = True) -> None:
    """Start the execution engine.

    .. versionadded:: 2.14
    """
    if self._starting:
        raise RuntimeError("Engine already running")
    self.start_time = time()
    self._starting = True
    await self.signals.send_catch_log_async(signal=signals.engine_started)
    if self._stopping:
        # band-aid until https://github.com/scrapy/scrapy/issues/6916
        return
    if _start_request_processing and self.spider is None:
        # require an opened spider when not run in scrapy shell
        return
    self.running = True
    self._closewait = Deferred()
    if _start_request_processing:
        coro = self._start_request_processing()
        if is_asyncio_available():
            # not wrapping in a Deferred here to avoid https://github.com/twisted/twisted/issues/12470
            # (can happen when this is cancelled, e.g. in test_close_during_start_iteration())
            self._start_request_processing_awaitable = asyncio.ensure_future(coro)
        else:
            self._start_request_processing_awaitable = Deferred.fromCoroutine(coro)
    with contextlib.suppress(asyncio.exceptions.CancelledError):
        await maybe_deferred_to_future(self._closewait)
```

--------------------------------

### DownloaderAwarePriorityQueue Initialization

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pqueues.html

Illustrates the initialization of DownloaderAwarePriorityQueue, including validation for CONCURRENT_REQUESTS_PER_IP and handling of slot_startprios.

```python
def __init__(
        self,
        crawler: Crawler,
        downstream_queue_cls: type[QueueProtocol],
        key: str,
        slot_startprios: dict[str, Iterable[int]] | None = None,
        *,
        start_queue_cls: type[QueueProtocol] | None = None,
    ):
        if crawler.settings.getint("CONCURRENT_REQUESTS_PER_IP") != 0:
            raise ValueError(
                f'"{self.__class__}" does not support CONCURRENT_REQUESTS_PER_IP'
            )

        if slot_startprios and not isinstance(slot_startprios, dict):
            raise ValueError(
                "DownloaderAwarePriorityQueue accepts "
                "``slot_startprios`` as a dict; "
                f"{slot_startprios.__class__!r} instance "
                "is passed. Most likely, it means the state is "
                "created by an incompatible priority queue. "
                "Only a crawl started with the same priority "
                "queue class can be resumed."
            )

        self._downloader_interface: DownloaderInterface = DownloaderInterface(crawler)
        self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls
        self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls
        self.key: str = key
        self.crawler: Crawler = crawler

        self.pqueues: dict[str, ScrapyPriorityQueue] = {}  # slot -> priority queue
        self._last_selected_slot: str | None = None
        if slot_startprios:
            for slot, startprios in slot_startprios.items():
                self.pqueues[slot] = self.pqfactory(slot, startprios)
```

--------------------------------

### Log using a Custom Logger

Source: https://docs.scrapy.org/en/latest/topics/logging.html

Demonstrates how to create and use a custom logger by specifying a name with `logging.getLogger`. This allows for independent configuration of loggers.

```python
import logging

logger = logging.getLogger("mycustomlogger")
logger.warning("This is a warning")

```

--------------------------------

### Start Twisted Reactor with Asyncio Integration

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html

Initializes and runs the Twisted reactor, integrating with asyncio for crawl task completion if configured.

```python
    def _start_twisted(
        self,
        stop_after_crawl: bool,
        install_signal_handlers: bool
    ) -> None:
        from twisted.internet import reactor

        if stop_after_crawl:
            loop = asyncio.get_event_loop()
            join_task = loop.create_task(self.join())
            join_task.add_done_callback(self._stop_reactor)

        self._setup_reactor(install_signal_handlers)
        reactor.run(installSignalHandlers=install_signal_handlers)  # blocking call
```

--------------------------------

### Install Pympler

Source: https://docs.scrapy.org/en/latest/topics/leaks.html

Install the Pympler library, which includes muppy, using pip.

```bash
pip install Pympler

```

--------------------------------

### Verify Installed Asyncio Event Loop

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html

Ensures that the event loop used by the installed Twisted reactor matches the specified import path. Raises a RuntimeError if no reactor is installed or if the loop class mismatches.

```python
def verify_installed_asyncio_event_loop(loop_path: str) -> None:
    """Raise :exc:`RuntimeError` if the even loop of the installed
    :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`
    does not match the specified import path or if no reactor is installed."""
    if not is_reactor_installed():
        raise RuntimeError(
            "verify_installed_asyncio_event_loop() called without an installed reactor."
        )

    from twisted.internet import reactor

    loop_class = load_object(loop_path)
    if isinstance(reactor._asyncioEventloop, loop_class):
        return
    installed = (
        f"{reactor._asyncioEventloop.__class__.__module__}"
        f".{reactor._asyncioEventloop.__class__.__qualname__}"
    )
    raise RuntimeError(
        "Scrapy found an asyncio Twisted reactor already "
        f"installed, and its event loop class ({installed}) does "
        "not match the one specified in the ASYNCIO_EVENT_LOOP "
        f"setting ({global_object_name(loop_class)})
    )
```

--------------------------------

### follow

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/http/response/text.html

Returns a Request instance to follow a link. Accepts arguments similar to Request.__init__.

```APIDOC
## follow

### Description
Returns a :class:`~.Request` instance to follow a link ``url``. It accepts the same arguments as ``Request.__init__()`` method, but ``url`` can be not only an absolute URL, but also a relative URL, a :class:`~scrapy.link.Link` object, a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, or an attribute :class:`~scrapy.Selector`.

See :ref:`response-follow-example` for usage examples.

### Method
`follow(
    url: str | Link | parsel.Selector,
    callback: CallbackT | None = None,
    method: str = "GET",
    headers: Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None = None,
    body: bytes | str | None = None,
    cookies: CookiesT | None = None,
    meta: dict[str, Any] | None = None,
    encoding: str | None = None,
    priority: int = 0,
    dont_filter: bool = False,
    errback: Callable[[Failure], Any] | None = None,
    cb_kwargs: dict[str, Any] | None = None,
    flags: list[str] | None = None,
) -> Request`

### Parameters
#### Path Parameters
None

#### Query Parameters
* **url** (str | Link | parsel.Selector) - Required - The URL to follow. Can be an absolute URL, relative URL, Link object, or a Selector.
* **callback** (CallbackT | None) - Optional - The callback function to use for the request.
* **method** (str) - Optional - The HTTP method to use for the request (default: "GET").
* **headers** (Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None) - Optional - Headers for the request.
* **body** (bytes | str | None) - Optional - Body for the request.
* **cookies** (CookiesT | None) - Optional - Cookies for the request.
* **meta** (dict[str, Any] | None) - Optional - Meta dictionary for the request.
* **encoding** (str | None) - Optional - The encoding to use for the response (defaults to response's encoding).
* **priority** (int) - Optional - Priority of the request (default: 0).
* **dont_filter** (bool) - Optional - Whether to filter the request (default: False).
* **errback** (Callable[[Failure], Any] | None) - Optional - The errback function to use for the request.
* **cb_kwargs** (dict[str, Any] | None) - Optional - Keyword arguments to pass to the callback function.
* **flags** (list[str] | None) - Optional - Flags for the request.

### Request Example
None

### Response
#### Success Response
- **Request** - A Scrapy Request object.

### Response Example
None

ERROR HANDLING:
- Raises ValueError if a SelectorList is passed as the url argument.
```

--------------------------------

### Scrapy Redirect Middleware: Building GET Redirect Request

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/redirect.html

Constructs a new request for redirection using the GET method. It ensures that request body and specific content-related headers are removed for the new GET request.

```python
def _redirect_request_using_get(self, request: Request, response: Response, redirect_url: str) -> Request:
        redirect_request = self._build_redirect_request(
            request,
            response,
            url=redirect_url,
            method="GET",
            body="",
        )
        redirect_request.headers.pop("Content-Type", None)
        redirect_request.headers.pop("Content-Length", None)
        redirect_request.headers.pop("Content-Encoding", None)
        redirect_request.headers.pop("Content-Language", None)
        redirect_request.headers.pop("Content-Location", None)
        return redirect_request
```

--------------------------------

### Loading Default Settings

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html

Illustrates how to load default settings from the Scrapy distribution.

```python
from scrapy.settings import default_settings

def get_default_settings():
    return default_settings
```

--------------------------------

### Getting Global Object Name

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html

Demonstrates using global_object_name to get the string representation of an object.

```python
from scrapy.utils.python import global_object_name

# Example usage:
# class MyClass:
#     pass
# 
# obj = MyClass()
# name = global_object_name(obj)
```

--------------------------------

### Getting Downloader Slot Key

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pqueues.html

Delegates to the downloader to get the slot key for a given request.

```python
def get_slot_key(self, request: Request) -> str:
    return self.downloader.get_slot_key(request)
```

--------------------------------

### ItemLoader with Default Item Class Example

Source: https://docs.scrapy.org/en/latest/_modules/itemloaders.html

Demonstrates using the default_item_class for item instantiation.

```python
from itemloaders import ItemLoader

class MyItemLoader(ItemLoader):
    default_item_class = dict

loader = MyItemLoader()
print(isinstance(loader.item, dict))
```

--------------------------------

### Start Timeout Timer

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/extensions/closespider.html

Starts a delayed call to close the spider after a specified timeout (CLOSESPIDER_TIMEOUT) when the spider is opened.

```python
    def spider_opened(self, spider: Spider) -> None:
        assert self.crawler.engine
        self.task = call_later(
            self.close_on["timeout"], self._close_spider, "closespider_timeout"
        )

```

--------------------------------

### Custom Log Formatter Example

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/logformatter.html

Example of a custom LogFormatter that lowers the severity level for dropped item log messages.

```Python
class PoliteLogFormatter(logformatter.LogFormatter):
    def dropped(self, item, exception, response, spider):
        return {
            'level': logging.INFO, # lowering the level from logging.WARNING
            'msg': "Dropped: %(exception)s" + os.linesep + "%("item")s",
            'args': {
                'exception': exception,
                'item': item,
            }
        }
```

--------------------------------

### JSON Exporter Setup

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/exporters.html

Initializes a JSON exporter with optional indentation and encoding settings. Ensures JSON is properly formatted for readability.

```python
self._kwargs.setdefault("indent", json_indent)
self._kwargs.setdefault("ensure_ascii", not self.encoding)
self.encoder = ScrapyJSONEncoder(**self._kwargs)
self.first_item = True
```

--------------------------------

### XMLFeedSpider Namespace Configuration Example

Source: https://docs.scrapy.org/en/latest/topics/spiders.html

This example demonstrates how to configure namespaces for an XMLFeedSpider, allowing iteration over nodes with specific prefixes and URIs.

```python
class YourSpider(XMLFeedSpider):

    namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
    itertag = 'n:url'
    # ...

```

--------------------------------

### Initialize FTP Files Store

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pipelines/files.html

Initializes the FTPFilesStore with FTP server details and credentials. It validates the URI scheme and extracts connection parameters.

```python
def __init__(self, uri: str):
    if not uri.startswith("ftp://"):
        raise ValueError(f"Incorrect URI scheme in {uri}, expected 'ftp'")
    u = urlparse(uri)
    assert u.port
    assert u.hostname
    self.port: int = u.port
    self.host: str = u.hostname
    self.port = int(u.port or 21)
    assert self.FTP_USERNAME
    assert self.FTP_PASSWORD
    self.username: str = u.username or self.FTP_USERNAME
    self.password: str = u.password or self.FTP_PASSWORD
    self.basedir: str = u.path.rstrip("/")
```

--------------------------------

### Initialize Google Cloud Storage Files Store

Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pipelines/files.html

Initializes the GCSFilesStore with project ID, bucket, and prefix. It also checks for necessary IAM permissions.

```python
def __init__(self, uri: str):
    from google.cloud import storage  # noqa: PLC0415

    client = storage.Client(project=self.GCS_PROJECT_ID)
    bucket, prefix = uri[5:].split("/", 1)
    self.bucket = client.bucket(bucket)
    self.prefix: str = prefix
    permissions = self.bucket.test_iam_permissions(
        ["storage.objects.get", "storage.objects.create"]
    )
    if "storage.objects.get" not in permissions:
        logger.warning(
            "No 'storage.objects.get' permission for GSC bucket %(bucket)s. "
            "Checking if files are up to date will be impossible. Files will be downloaded every time.",
            {"bucket": bucket},
        )
    if "storage.objects.create" not in permissions:
        logger.error(
            "No 'storage.objects.create' permission for GSC bucket %(bucket)s. Saving files will be impossible!",
            {"bucket": bucket},
        )
```

--------------------------------

### XMLFeedSpider Iteration Tag Example

Source: https://docs.scrapy.org/en/latest/topics/spiders.html

This example shows how to define the 'itertag' attribute for an XMLFeedSpider to specify the XML node name to iterate over.

```python
itertag = 'product'

```

--------------------------------

### Enable Scrapy Extensions

Source: https://docs.scrapy.org/en/latest/topics/extensions.html

Add extensions to the EXTENSIONS setting to enable them at startup. Priorities determine the loading order.

```python
EXTENSIONS = {
    "scrapy.extensions.corestats.CoreStats": 500,
    "scrapy.extensions.telnet.TelnetConsole": 500,
}
```

--------------------------------

### Generate Coverage Report

Source: https://docs.scrapy.org/en/latest/contributing.html

Generate a coverage report for the tests. Ensure the `coverage` package is installed (`pip install coverage`) before running.

```bash
coverage report
```
Tokens:
Raw