### Start a New Scrapy Project (Example)
Source: https://docs.scrapy.org/en/latest/topics/commands.html
An example of how to use the `startproject` command to create a project named 'myproject'.
```bash
$ scrapy startproject myproject
```
--------------------------------
### Async Spider Middleware `process_start` Example
Source: https://docs.scrapy.org/en/latest/topics/spider-middleware.html
Implement an asynchronous `process_start` method to iterate over and potentially modify the output of `start()` or an earlier middleware.
```python
async def process_start(self, start):
async for item_or_request in start:
yield item_or_request
```
--------------------------------
### Inspect Scrapy Shell Output
Source: https://docs.scrapy.org/en/latest/intro/tutorial.html
Example of the typical output when starting the Scrapy shell, showing available objects and shortcuts.
```text
[ ... Scrapy log here ... ]
2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler
[s] item {}
[s] request
[s] response <200 https://quotes.toscrape.com/page/1/>
[s] settings
[s] spider
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
```
--------------------------------
### Install form2request
Source: https://docs.scrapy.org/en/latest/topics/request-response.html
Install the form2request library using pip.
```bash
pip install form2request
```
--------------------------------
### Default Start Method Behavior
Source: https://docs.scrapy.org/en/latest/topics/spiders.html
Illustrates the default behavior of the `start` method when using `start_urls`, which is equivalent to yielding requests with `dont_filter=True`.
```python
async def start(self):
for url in self.start_urls:
yield Request(url, dont_filter=True)
```
--------------------------------
### Starting CrawlerProcess with Reactor
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html
Starts the Twisted reactor, adjusts its thread pool size, and installs a DNS resolver. Optionally stops the reactor after all crawlers finish.
```python
from twisted.internet import reactor
if stop_after_crawl:
d = self.join()
# Don't start the reactor if the deferreds are already fired
if d.called:
return
d.addBoth(self._stop_reactor)
self._setup_reactor(install_signal_handlers)
reactor.run(installSignalHandlers=install_signal_handlers) # blocking call
```
--------------------------------
### Install Asyncio Reactor
Source: https://docs.scrapy.org/en/latest/topics/asyncio.html
Installs the AsyncioSelectorReactor for Scrapy. Use this when manually setting up the reactor or if Scrapy complains about a pre-installed reactor.
```python
from twisted.internet.asyncioreactor import install_reactor
install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
```
--------------------------------
### Install and Configure RerpRobotParser
Source: https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
To use the Robotexclusionrulesparser, install it and set ROBOTSTXT_PARSER.
```bash
pip install robotexclusionrulesparser
```
```python
ROBOTSTXT_PARSER = 'scrapy.robotstxt.RerpRobotParser'
```
--------------------------------
### Yielding Items from Start Method
Source: https://docs.scrapy.org/en/latest/topics/spiders.html
Shows how to yield items directly from the `start` method, bypassing request processing for simple data extraction.
```python
async def start(self):
yield {"foo": "bar"}
```
--------------------------------
### Install Scrapy with Pip
Source: https://docs.scrapy.org/en/latest/intro/install.html
Install Scrapy and its dependencies from PyPI using pip. It's strongly recommended to do this within a virtual environment.
```bash
pip install Scrapy
```
--------------------------------
### Install Xcode Command Line Tools on macOS
Source: https://docs.scrapy.org/en/latest/intro/install.html
Run this command in the terminal to install the necessary C compiler and development headers on macOS.
```bash
xcode-select --install
```
--------------------------------
### Start Scrapy Engine (Deprecated)
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/engine.html
Starts the Scrapy engine. This method is deprecated and `start_async()` should be used instead.
```python
def start(
self,
_start_request_processing: bool = True
) -> Deferred[None]: # pragma: no cover
warnings.warn(
"ExecutionEngine.start() is deprecated, use start_async() instead",
ScrapyDeprecationWarning,
stacklevel=2,
)
return deferred_from_coro(
self.start_async(_start_request_processing=_start_request_processing)
)
```
--------------------------------
### Instantiate and Start Crawler Synchronously
Source: https://docs.scrapy.org/en/latest/topics/api.html
Use `crawl` to start the crawler with a spider class and arguments. This method should be called only once and returns a deferred that fires when the crawl is finished.
```python
crawl(_* args: Any_, _** kwargs: Any_) → Generator[Deferred[Any], Any, None][source]
Start the crawler by instantiating its spider class with the given _args_ and _kwargs_ arguments, while setting the execution engine in motion. Should be called only once.
Return a deferred that is fired when the crawl is finished.
```
--------------------------------
### Install Twisted Reactor
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html
Installs the Twisted reactor, optionally with a specific asyncio event loop. It handles different reactor types and suppresses errors if a reactor is already installed.
```python
from twisted.internet import asyncioreactor, error
from scrapy.utils.misc import load_object
def install_reactor(reactor_path: str, event_loop_path: str | None = None) -> None:
"""Installs the :mod:`~twisted.internet.reactor` with the specified
import path. Also installs the asyncio event loop with the specified import
path if the asyncio reactor is enabled"""
reactor_class = load_object(reactor_path)
if reactor_class is asyncioreactor.AsyncioSelectorReactor:
set_asyncio_event_loop_policy()
with suppress(error.ReactorAlreadyInstalledError):
event_loop = set_asyncio_event_loop(event_loop_path)
asyncioreactor.install(eventloop=event_loop)
else:
*module, _ = reactor_path.split(".")
installer_path = [*module, "install"]
installer = load_object(".".join(installer_path))
with suppress(error.ReactorAlreadyInstalledError):
installer()
```
--------------------------------
### Start Crawling with CrawlerProcess
Source: https://docs.scrapy.org/en/latest/topics/api.html
Start a crawler using the CrawlerProcess. This method accepts a Crawler instance, Spider subclass, or spider name. It returns a Deferred that completes when crawling finishes.
```python
process.crawl("my_spider", start_url="http://example.com")
```
--------------------------------
### Install Pre-commit Hooks
Source: https://docs.scrapy.org/en/latest/contributing.html
Run this command in the root of your local Scrapy repository clone to install the pre-commit hooks. This ensures code is formatted and checked before each commit.
```bash
pre-commit install
```
--------------------------------
### Instantiate and Start Crawler Asynchronously
Source: https://docs.scrapy.org/en/latest/topics/api.html
Use `crawl_async` to start the crawler with a spider class and arguments. This method should be called only once and completes when the crawl is finished.
```python
async crawl_async(_* args: Any_, _** kwargs: Any_) → None[source]
Start the crawler by instantiating its spider class with the given _args_ and _kwargs_ arguments, while setting the execution engine in motion. Should be called only once.
Added in version 2.14.
Complete when the crawl is finished.
```
--------------------------------
### Example Component Initialization with Settings
Source: https://docs.scrapy.org/en/latest/topics/components.html
Demonstrates how a component can access crawler settings during initialization. Use this to enable or disable component features based on project settings.
```python
class MyExtension:
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
return cls(settings.getbool("LOG_ENABLED"))
def __init__(self, log_is_enabled=False):
if log_is_enabled:
print("log is enabled!")
```
--------------------------------
### Get Start Queue Class Helper
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/scheduler.html
A helper method to determine the class for the starting queue (disk or memory) based on crawler settings.
```python
def _get_start_queue_cls(
self, crawler: Crawler | None, queue: str
) -> type[BaseQueue] | None:
if crawler is None:
return None
cls = crawler.settings[f"SCHEDULER_START_{queue}_QUEUE"]
if not cls:
return None
return cast("type[BaseQueue]", load_object(cls))
```
--------------------------------
### Spider Yielding Items with Async Start
Source: https://docs.scrapy.org/en/latest/topics/spiders.html
An example of a spider using an asynchronous `start` method to yield requests and processing them into structured items using `MyItem`.
```python
import scrapy
from myproject.items import MyItem
class MySpider(scrapy.Spider):
name = "example.com"
allowed_domains = ["example.com"]
async def start(self):
yield scrapy.Request("http://www.example.com/1.html", self.parse)
yield scrapy.Request("http://www.example.com/2.html", self.parse)
yield scrapy.Request("http://www.example.com/3.html", self.parse)
def parse(self, response):
for h3 in response.xpath("//h3").getall():
yield MyItem(title=h3)
for href in response.xpath("//a/@href").getall():
yield scrapy.Request(response.urljoin(href), self.parse)
```
--------------------------------
### Activating and Configuring Add-ons
Source: https://docs.scrapy.org/en/latest/topics/addons.html
Example of enabling two add-ons in a project's settings.py file. The keys are add-on import paths or class names, and values are their priorities.
```python
ADDONS = {
'path.to.someaddon': 0,
SomeAddonClass: 1,
}
```
--------------------------------
### Importing BaseSettings and Settings
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html
Demonstrates the basic import statements for the core settings classes.
```python
from scrapy.settings import BaseSettings, Settings
```
--------------------------------
### Start Reactor and Signal Handlers
Source: https://docs.scrapy.org/en/latest/topics/api.html
Start the reactor or asyncio event loop and optionally install signal handlers for graceful shutdown. If stop_after_crawl is True, the reactor stops once all crawlers are done.
```python
process.start(stop_after_crawl=True, install_signal_handlers=True)
```
--------------------------------
### Create a New Scrapy Project
Source: https://docs.scrapy.org/en/latest/intro/tutorial.html
Run this command in your terminal to create a new Scrapy project named 'tutorial'. This sets up the basic directory structure for your project.
```bash
scrapy startproject tutorial
```
--------------------------------
### Get All Href Attribute Values
Source: https://docs.scrapy.org/en/latest/topics/selectors.html
Example of using `::attr(href)` to extract all `href` attribute values from `` elements.
```python
>>> response.css("a::attr(href)").getall()
['image1.html',
'image2.html',
'image3.html',
'image4.html',
'image5.html']
```
--------------------------------
### Get Text Content of a Title Element
Source: https://docs.scrapy.org/en/latest/topics/selectors.html
Example of using `::text` to extract the text content from a `` element.
```python
>>> response.css("title::text").get()
'Example website'
```
--------------------------------
### RefererMiddleware Initialization
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/spidermiddlewares/referer.html
Initializes the RefererMiddleware, setting up default policies and loading custom policies from settings.
```python
def __init__(self, settings: BaseSettings | None = None):
self.default_policy: type[ReferrerPolicy] = DefaultReferrerPolicy
self.policies: dict[str, type[ReferrerPolicy]] = {
p.name: p
for p in (
NoReferrerPolicy,
NoReferrerWhenDowngradePolicy,
SameOriginPolicy,
OriginPolicy,
StrictOriginPolicy,
OriginWhenCrossOriginPolicy,
StrictOriginWhenCrossOriginPolicy,
UnsafeUrlPolicy,
DefaultReferrerPolicy,
)
}
# Reference: https://www.w3.org/TR/referrer-policy/#referrer-policy-empty-string
self.policies[""] = NoReferrerWhenDowngradePolicy
if settings is None:
return
setting_policies = settings.getdict("REFERRER_POLICIES")
for policy_name, policy_class_import_path in setting_policies.items():
if policy_class_import_path is None:
del self.policies[policy_name]
else:
self.policies[policy_name] = load_object(policy_class_import_path)
settings_policy = self._load_policy_class(
settings.get("REFERRER_POLICY"), allow_import_path=True
)
assert settings_policy
self.default_policy = settings_policy
```
--------------------------------
### Prepare URL Partitions for Distributed Crawls
Source: https://docs.scrapy.org/en/latest/topics/practices.html
This example shows how to prepare lists of URLs for distributed crawling by partitioning them into separate files. Each file corresponds to a partition to be processed by a separate spider instance.
```text
http://somedomain.com/urls-to-crawl/spider1/part1.list
http://somedomain.com/urls-to-crawl/spider1/part2.list
http://somedomain.com/urls-to-crawl/spider1/part3.list
```
--------------------------------
### Scrapy Project Configuration Example
Source: https://docs.scrapy.org/en/latest/topics/commands.html
Shows how to define the default settings module within the scrapy.cfg file.
```ini
[settings]
default = myproject.settings
```
--------------------------------
### Get All Descendant Text Nodes
Source: https://docs.scrapy.org/en/latest/topics/selectors.html
Example of using `*::text` to retrieve all text nodes under an element with the ID 'images'. The output may include whitespace.
```python
>>> response.css("#images *::text").getall()
['\n ',
'Name: My image 1 ',
'\n ',
'Name: My image 2 ',
'\n ',
'Name: My image 3 ',
'\n ',
'Name: My image 4 ',
'\n ',
'Name: My image 5 ',
'\n ']
```
--------------------------------
### Example Image File Storage Structure
Source: https://docs.scrapy.org/en/latest/topics/media-pipeline.html
Illustrates the storage structure for full images and generated thumbnails based on IMAGES_THUMBS configuration.
```text
<IMAGES_STORE>/full/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/small/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
<IMAGES_STORE>/thumbs/big/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
```
--------------------------------
### Get Spider Middleware Instance
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html
Retrieves the runtime instance of a spider middleware by its class. This method requires the crawl engine to be active, typically after engine start.
```python
def get_spider_middleware(self, cls: type[_T]) -> _T | None:
"""Return the run-time instance of a :ref:`spider middleware
<topics-spider-middleware>` of the specified class or a subclass, or
``None`` if none is found.
.. versionadded:: 2.12
This method can only be called after the crawl engine has been created,
e.g. at signals :signal:`engine_started` or :signal:`spider_opened`.
"""
if not self.engine:
raise RuntimeError(
"Crawler.get_spider_middleware() can only be called after the "
"crawl engine has been created."
)
return self._get_component(cls, self.engine.scraper.spidermw.middlewares)
```
--------------------------------
### Generate a Spider with Basic Template
Source: https://docs.scrapy.org/en/latest/topics/commands.html
Creates a new spider named 'example' using the default 'basic' template.
```bash
$ scrapy genspider example example.com
Created spider 'example' using template 'basic'
```
--------------------------------
### Running a Spider Manually with CrawlerProcess
Source: https://docs.scrapy.org/en/latest/news.html
Example of how to run a Scrapy spider from a script using the refactored Crawler API. This involves creating a `CrawlerProcess` instance and starting the crawl.
```python
from scrapy.crawler import CrawlerProcess
process = CrawlerProcess({
'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})
process.crawl(MySpider)
process.start()
```
--------------------------------
### Start Asyncio Reactorless Loop
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html
Initiates the asyncio event loop for reactorless operation. Handles different strategies based on `stop_after_crawl` and installs signal handlers if enabled.
```python
loop = self._reactorless_loop
assert loop
if stop_after_crawl:
self._reactorless_main_task = loop.create_task(self.join())
else:
self._reactorless_main_task = loop.create_future()
self._stop_after_crawl = stop_after_crawl
try:
self._run_loop(install_signal_handlers) # blocking call
except asyncio.CancelledError:
pass
finally:
self._close_loop()
```
--------------------------------
### Registering a Custom Scrapy Command
Source: https://docs.scrapy.org/en/latest/topics/commands.html
Example of a setup.py file demonstrating how to register a custom Scrapy command named 'my_command' from an external module.
```python
from setuptools import setup, find_packages
setup(
name="scrapy-mymodule",
entry_points={
"scrapy.commands": [
"my_command=my_scrapy_module.commands:MyCommand",
],
},
)
```
--------------------------------
### FeedSlot Start Exporting
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/extensions/feedexport.html
Opens the feed file and initializes the exporter. If postprocessing is configured, it wraps the file object.
```python
def start_exporting(self) -> None:
if not self._fileloaded:
self.file = self.storage.open(self.spider)
if "postprocessing" in self.feed_options:
self.file = cast(
"IO[bytes]",
PostProcessingManager(
self.feed_options["postprocessing"],
self.file,
self.feed_options,
),
)
self.exporter = self._get_exporter(
file=self.file,
format_=self.feed_options["format"],
fields_to_export=self.feed_options["fields"],
encoding=self.feed_options["encoding"],
indent=self.feed_options["indent"],
**self.feed_options["item_export_kwargs"],
)
self._fileloaded = True
if not self._exporting:
assert self.exporter
self.exporter.start_exporting()
self._exporting = True
```
--------------------------------
### Using BeautifulSoup with Scrapy
Source: https://docs.scrapy.org/en/latest/faq.html
This example demonstrates how to integrate BeautifulSoup for parsing HTML content within a Scrapy spider. It uses 'lxml' as the underlying parser for efficiency. Ensure BeautifulSoup and lxml are installed.
```python
from bs4 import BeautifulSoup
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = ("http://www.example.com/",)
def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, "lxml")
yield {"url": response.url, "title": soup.h1.string}
```
--------------------------------
### BaseRedirectMiddleware _engine_started Method
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/redirect.html
Callback for the engine_started signal. It attempts to find a RefererMiddleware instance and logs a warning if none is found, explaining how to enable Referer header handling.
```python
def _engine_started(self) -> None:
self._referer_spider_middleware = self.crawler.get_spider_middleware(
RefererMiddleware
)
if self._referer_spider_middleware:
return
redirect_cls = global_object_name(self.__class__)
referer_cls = global_object_name(RefererMiddleware)
if self.__class__ in {RedirectMiddleware, MetaRefreshMiddleware}:
replacement = (
f"replace {redirect_cls} with a subclass that overrides the "
f"handle_referer() method"
)
else:
replacement = (
f"or edit {redirect_cls} (if defined in your code base) to "
f"override the handle_referer() method, or replace "
f"{redirect_cls} with a subclass that overrides the "
f"handle_referer() method."
)
logger.warning(
f"{redirect_cls} found no {referer_cls} instance to handle "
f"Referer header handling, so the Referer header will be removed "
f"on redirects. To set a Referer header on redirects, enable "
f"{referer_cls} (or a subclass), or {replacement}.",
)
```
--------------------------------
### Iterating over all live HtmlResponse objects
Source: https://docs.scrapy.org/en/latest/topics/leaks.html
Use iter_all() to get an iterator for all live HtmlResponse objects. This allows you to process each object, for example, to extract their URLs and identify patterns related to memory leaks.
```python
from scrapy.utils.trackref import iter_all
[r.url for r in iter_all("HtmlResponse")]
```
--------------------------------
### Scrapy HTML Response Setup
Source: https://docs.scrapy.org/en/latest/topics/selectors.html
Sets up an HtmlResponse object with sample HTML content for testing selectors.
```python
>>> from scrapy.http import HtmlResponse
>>> response = HtmlResponse(
... url="http://example.com",
... body="""
... <html>
... <body>
... <p class="foo bar-baz">First</p>
... <p class="foo">Second</p>
... <p class="bar">Third</p>
... <p>Fourth</p>
... </body>
... </html>
... """,
... encoding="utf-8",
... )
```
--------------------------------
### Configure Scrapy Logging
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/log.html
Initializes Scrapy's logging defaults, routes warnings and Twisted logs, and optionally installs a root handler based on settings. Use this early in your project setup.
```python
from scrapy.settings import Settings, _SettingsKey
def configure_logging(
settings: Settings | dict[_SettingsKey, Any] | None = None,
install_root_handler: bool = True,
) -> None:
"""
Initialize logging defaults for Scrapy.
:param settings: settings used to create and configure a handler for the
root logger (default: None).
:type settings: dict, :class:`~scrapy.settings.Settings` object or ``None``
:param install_root_handler: whether to install root logging handler
(default: True)
:type install_root_handler: bool
This function does:
- Route warnings and twisted logging through Python standard logging
- Assign DEBUG and ERROR level to Scrapy and Twisted loggers respectively
- Route stdout to log if LOG_STDOUT setting is True
When ``install_root_handler`` is True (default), this function also
creates a handler for the root logger according to given settings
(see :ref:`topics-logging-settings`). You can override default options
using ``settings`` argument. When ``settings`` is empty or None, defaults
are used.
"""
if not sys.warnoptions:
# Route warnings through python logging
logging.captureWarnings(True)
observer = twisted_log.PythonLoggingObserver("twisted")
observer.start()
dictConfig(DEFAULT_LOGGING)
if isinstance(settings, dict) or settings is None:
settings = Settings(settings)
if settings.getbool("LOG_STDOUT"):
sys.stdout = StreamLogger(logging.getLogger("stdout"))
if install_root_handler:
install_scrapy_root_handler(settings)
```
--------------------------------
### Configure Periodic Log Extension Settings
Source: https://docs.scrapy.org/en/latest/topics/extensions.html
Example configuration for the PeriodicLog extension, showing how to enable specific stats, delta calculations, and timing data.
```python
custom_settings = {
"LOG_LEVEL": "INFO",
"PERIODIC_LOG_STATS": {
"include": ["downloader/", "scheduler/", "log_count/", "item_scraped_count/"],
},
"PERIODIC_LOG_DELTA": {"include": ["downloader/"]},
"PERIODIC_LOG_TIMING_ENABLED": True,
"EXTENSIONS": {
"scrapy.extensions.periodic_log.PeriodicLog": 0,
},
}
```
--------------------------------
### Example Items for Export
Source: https://docs.scrapy.org/en/latest/topics/exporters.html
These are example items used in the output examples for Scrapy's built-in Item Exporters.
```python
Item(name="Color TV", price="1200")
Item(name="DVD player", price="200")
```
--------------------------------
### Install Scrapy in a Virtual Environment on Ubuntu
Source: https://docs.scrapy.org/en/latest/intro/install.html
After installing the necessary non-Python dependencies, install Scrapy within a virtual environment using pip.
```bash
pip install scrapy
```
--------------------------------
### Example Output Directory Structure with Batching
Source: https://docs.scrapy.org/en/latest/topics/feed-exports.html
Illustrates the directory structure and file naming convention when using FEED_EXPORT_BATCH_ITEM_COUNT and URI placeholders.
```text
->projectname
-->dirname
--->1-filename2020-03-28T14-45-08.237134.json
--->2-filename2020-03-28T14-45-09.148903.json
--->3-filename2020-03-28T14-45-10.046092.json
```
--------------------------------
### Verify Installed Reactor
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html
Checks if the currently installed Twisted reactor matches a specified import path. Raises a RuntimeError if no reactor is installed or if the types do not match.
```python
def verify_installed_reactor(reactor_path: str) -> None:
"""Raise :exc:`RuntimeError` if the installed
:mod:`~twisted.internet.reactor` does not match the specified import
path or if no reactor is installed."""
if not is_reactor_installed():
raise RuntimeError(
"verify_installed_reactor() called without an installed reactor."
)
from twisted.internet import reactor
expected_reactor_type = load_object(reactor_path)
reactor_type = type(reactor)
if not reactor_type == expected_reactor_type:
raise RuntimeError(
f"The installed reactor ({global_object_name(reactor_type)}) "
f"does not match the requested one ({reactor_path})"
)
```
--------------------------------
### Check if Asyncio Reactor is Installed
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html
Verifies if the installed Twisted reactor is specifically an AsyncioSelectorReactor. Raises a RuntimeError if no reactor is installed, as per Scrapy's current behavior.
```python
def is_asyncio_reactor_installed() -> bool:
"""Check whether the installed reactor is :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`.
Raise a :exc:`RuntimeError` if no reactor is installed.
In a future Scrapy version, when Scrapy supports running without a Twisted
reactor, this function won't be useful for checking if it's possible to use
asyncio features, so the code that that doesn't directly require a Twisted
reactor should use :func:`scrapy.utils.asyncio.is_asyncio_available`
instead of this function.
.. versionchanged:: 2.13
In earlier Scrapy versions this function silently installed the default
reactor if there was no reactor installed. Now it raises an exception to
prevent silent problems in this case.
"""
if not is_reactor_installed():
raise RuntimeError(
"is_asyncio_reactor_installed() called without an installed reactor."
)
from twisted.internet import reactor
return isinstance(reactor, asyncioreactor.AsyncioSelectorReactor)
```
--------------------------------
### install_reactor
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html
Installs the Twisted reactor with the specified import path, optionally configuring the asyncio event loop if an asyncio reactor is used.
```APIDOC
## install_reactor
### Description
Installs the :mod:`~twisted.internet.reactor` with the specified import path. It also installs the asyncio event loop with the specified import path if the asyncio reactor is enabled.
### Parameters
- **reactor_path** (str) - The import path for the Twisted reactor class (e.g., 'twisted.internet.asyncioreactor.AsyncioSelectorReactor').
- **event_loop_path** (str | None, optional) - The import path for the asyncio event loop class. Defaults to None, which uses the default event loop policy.
### Behavior
- If `reactor_path` points to `AsyncioSelectorReactor`, it configures the asyncio event loop policy (especially for Windows) and installs the asyncio reactor.
- Otherwise, it loads the specified reactor class and installs it using its respective installer function.
```
--------------------------------
### Start Scrapy Engine Asynchronously
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/core/engine.html
Starts the execution engine asynchronously. It sets the start time, signals engine startup, and begins processing requests if a spider is available.
```python
async def start_async(self, *, _start_request_processing: bool = True) -> None:
"""Start the execution engine.
.. versionadded:: 2.14
"""
if self._starting:
raise RuntimeError("Engine already running")
self.start_time = time()
self._starting = True
await self.signals.send_catch_log_async(signal=signals.engine_started)
if self._stopping:
# band-aid until https://github.com/scrapy/scrapy/issues/6916
return
if _start_request_processing and self.spider is None:
# require an opened spider when not run in scrapy shell
return
self.running = True
self._closewait = Deferred()
if _start_request_processing:
coro = self._start_request_processing()
if is_asyncio_available():
# not wrapping in a Deferred here to avoid https://github.com/twisted/twisted/issues/12470
# (can happen when this is cancelled, e.g. in test_close_during_start_iteration())
self._start_request_processing_awaitable = asyncio.ensure_future(coro)
else:
self._start_request_processing_awaitable = Deferred.fromCoroutine(coro)
with contextlib.suppress(asyncio.exceptions.CancelledError):
await maybe_deferred_to_future(self._closewait)
```
--------------------------------
### DownloaderAwarePriorityQueue Initialization
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pqueues.html
Illustrates the initialization of DownloaderAwarePriorityQueue, including validation for CONCURRENT_REQUESTS_PER_IP and handling of slot_startprios.
```python
def __init__(
self,
crawler: Crawler,
downstream_queue_cls: type[QueueProtocol],
key: str,
slot_startprios: dict[str, Iterable[int]] | None = None,
*,
start_queue_cls: type[QueueProtocol] | None = None,
):
if crawler.settings.getint("CONCURRENT_REQUESTS_PER_IP") != 0:
raise ValueError(
f'"{self.__class__}" does not support CONCURRENT_REQUESTS_PER_IP'
)
if slot_startprios and not isinstance(slot_startprios, dict):
raise ValueError(
"DownloaderAwarePriorityQueue accepts "
"``slot_startprios`` as a dict; "
f"{slot_startprios.__class__!r} instance "
"is passed. Most likely, it means the state is "
"created by an incompatible priority queue. "
"Only a crawl started with the same priority "
"queue class can be resumed."
)
self._downloader_interface: DownloaderInterface = DownloaderInterface(crawler)
self.downstream_queue_cls: type[QueueProtocol] = downstream_queue_cls
self._start_queue_cls: type[QueueProtocol] | None = start_queue_cls
self.key: str = key
self.crawler: Crawler = crawler
self.pqueues: dict[str, ScrapyPriorityQueue] = {} # slot -> priority queue
self._last_selected_slot: str | None = None
if slot_startprios:
for slot, startprios in slot_startprios.items():
self.pqueues[slot] = self.pqfactory(slot, startprios)
```
--------------------------------
### Log using a Custom Logger
Source: https://docs.scrapy.org/en/latest/topics/logging.html
Demonstrates how to create and use a custom logger by specifying a name with `logging.getLogger`. This allows for independent configuration of loggers.
```python
import logging
logger = logging.getLogger("mycustomlogger")
logger.warning("This is a warning")
```
--------------------------------
### Start Twisted Reactor with Asyncio Integration
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/crawler.html
Initializes and runs the Twisted reactor, integrating with asyncio for crawl task completion if configured.
```python
def _start_twisted(
self,
stop_after_crawl: bool,
install_signal_handlers: bool
) -> None:
from twisted.internet import reactor
if stop_after_crawl:
loop = asyncio.get_event_loop()
join_task = loop.create_task(self.join())
join_task.add_done_callback(self._stop_reactor)
self._setup_reactor(install_signal_handlers)
reactor.run(installSignalHandlers=install_signal_handlers) # blocking call
```
--------------------------------
### Install Pympler
Source: https://docs.scrapy.org/en/latest/topics/leaks.html
Install the Pympler library, which includes muppy, using pip.
```bash
pip install Pympler
```
--------------------------------
### Verify Installed Asyncio Event Loop
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/utils/reactor.html
Ensures that the event loop used by the installed Twisted reactor matches the specified import path. Raises a RuntimeError if no reactor is installed or if the loop class mismatches.
```python
def verify_installed_asyncio_event_loop(loop_path: str) -> None:
"""Raise :exc:`RuntimeError` if the even loop of the installed
:class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`
does not match the specified import path or if no reactor is installed."""
if not is_reactor_installed():
raise RuntimeError(
"verify_installed_asyncio_event_loop() called without an installed reactor."
)
from twisted.internet import reactor
loop_class = load_object(loop_path)
if isinstance(reactor._asyncioEventloop, loop_class):
return
installed = (
f"{reactor._asyncioEventloop.__class__.__module__}"
f".{reactor._asyncioEventloop.__class__.__qualname__}"
)
raise RuntimeError(
"Scrapy found an asyncio Twisted reactor already "
f"installed, and its event loop class ({installed}) does "
"not match the one specified in the ASYNCIO_EVENT_LOOP "
f"setting ({global_object_name(loop_class)})
)
```
--------------------------------
### follow
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/http/response/text.html
Returns a Request instance to follow a link. Accepts arguments similar to Request.__init__.
```APIDOC
## follow
### Description
Returns a :class:`~.Request` instance to follow a link ``url``. It accepts the same arguments as ``Request.__init__()`` method, but ``url`` can be not only an absolute URL, but also a relative URL, a :class:`~scrapy.link.Link` object, a :class:`~scrapy.Selector` object for a ``<link>`` or ``<a>`` element, or an attribute :class:`~scrapy.Selector`.
See :ref:`response-follow-example` for usage examples.
### Method
`follow(
url: str | Link | parsel.Selector,
callback: CallbackT | None = None,
method: str = "GET",
headers: Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None = None,
body: bytes | str | None = None,
cookies: CookiesT | None = None,
meta: dict[str, Any] | None = None,
encoding: str | None = None,
priority: int = 0,
dont_filter: bool = False,
errback: Callable[[Failure], Any] | None = None,
cb_kwargs: dict[str, Any] | None = None,
flags: list[str] | None = None,
) -> Request`
### Parameters
#### Path Parameters
None
#### Query Parameters
* **url** (str | Link | parsel.Selector) - Required - The URL to follow. Can be an absolute URL, relative URL, Link object, or a Selector.
* **callback** (CallbackT | None) - Optional - The callback function to use for the request.
* **method** (str) - Optional - The HTTP method to use for the request (default: "GET").
* **headers** (Mapping[AnyStr, Any] | Iterable[tuple[AnyStr, Any]] | None) - Optional - Headers for the request.
* **body** (bytes | str | None) - Optional - Body for the request.
* **cookies** (CookiesT | None) - Optional - Cookies for the request.
* **meta** (dict[str, Any] | None) - Optional - Meta dictionary for the request.
* **encoding** (str | None) - Optional - The encoding to use for the response (defaults to response's encoding).
* **priority** (int) - Optional - Priority of the request (default: 0).
* **dont_filter** (bool) - Optional - Whether to filter the request (default: False).
* **errback** (Callable[[Failure], Any] | None) - Optional - The errback function to use for the request.
* **cb_kwargs** (dict[str, Any] | None) - Optional - Keyword arguments to pass to the callback function.
* **flags** (list[str] | None) - Optional - Flags for the request.
### Request Example
None
### Response
#### Success Response
- **Request** - A Scrapy Request object.
### Response Example
None
ERROR HANDLING:
- Raises ValueError if a SelectorList is passed as the url argument.
```
--------------------------------
### Scrapy Redirect Middleware: Building GET Redirect Request
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/downloadermiddlewares/redirect.html
Constructs a new request for redirection using the GET method. It ensures that request body and specific content-related headers are removed for the new GET request.
```python
def _redirect_request_using_get(self, request: Request, response: Response, redirect_url: str) -> Request:
redirect_request = self._build_redirect_request(
request,
response,
url=redirect_url,
method="GET",
body="",
)
redirect_request.headers.pop("Content-Type", None)
redirect_request.headers.pop("Content-Length", None)
redirect_request.headers.pop("Content-Encoding", None)
redirect_request.headers.pop("Content-Language", None)
redirect_request.headers.pop("Content-Location", None)
return redirect_request
```
--------------------------------
### Loading Default Settings
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html
Illustrates how to load default settings from the Scrapy distribution.
```python
from scrapy.settings import default_settings
def get_default_settings():
return default_settings
```
--------------------------------
### Getting Global Object Name
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/settings.html
Demonstrates using global_object_name to get the string representation of an object.
```python
from scrapy.utils.python import global_object_name
# Example usage:
# class MyClass:
# pass
#
# obj = MyClass()
# name = global_object_name(obj)
```
--------------------------------
### Getting Downloader Slot Key
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pqueues.html
Delegates to the downloader to get the slot key for a given request.
```python
def get_slot_key(self, request: Request) -> str:
return self.downloader.get_slot_key(request)
```
--------------------------------
### ItemLoader with Default Item Class Example
Source: https://docs.scrapy.org/en/latest/_modules/itemloaders.html
Demonstrates using the default_item_class for item instantiation.
```python
from itemloaders import ItemLoader
class MyItemLoader(ItemLoader):
default_item_class = dict
loader = MyItemLoader()
print(isinstance(loader.item, dict))
```
--------------------------------
### Start Timeout Timer
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/extensions/closespider.html
Starts a delayed call to close the spider after a specified timeout (CLOSESPIDER_TIMEOUT) when the spider is opened.
```python
def spider_opened(self, spider: Spider) -> None:
assert self.crawler.engine
self.task = call_later(
self.close_on["timeout"], self._close_spider, "closespider_timeout"
)
```
--------------------------------
### Custom Log Formatter Example
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/logformatter.html
Example of a custom LogFormatter that lowers the severity level for dropped item log messages.
```Python
class PoliteLogFormatter(logformatter.LogFormatter):
def dropped(self, item, exception, response, spider):
return {
'level': logging.INFO, # lowering the level from logging.WARNING
'msg': "Dropped: %(exception)s" + os.linesep + "%("item")s",
'args': {
'exception': exception,
'item': item,
}
}
```
--------------------------------
### JSON Exporter Setup
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/exporters.html
Initializes a JSON exporter with optional indentation and encoding settings. Ensures JSON is properly formatted for readability.
```python
self._kwargs.setdefault("indent", json_indent)
self._kwargs.setdefault("ensure_ascii", not self.encoding)
self.encoder = ScrapyJSONEncoder(**self._kwargs)
self.first_item = True
```
--------------------------------
### XMLFeedSpider Namespace Configuration Example
Source: https://docs.scrapy.org/en/latest/topics/spiders.html
This example demonstrates how to configure namespaces for an XMLFeedSpider, allowing iteration over nodes with specific prefixes and URIs.
```python
class YourSpider(XMLFeedSpider):
namespaces = [('n', 'http://www.sitemaps.org/schemas/sitemap/0.9')]
itertag = 'n:url'
# ...
```
--------------------------------
### Initialize FTP Files Store
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pipelines/files.html
Initializes the FTPFilesStore with FTP server details and credentials. It validates the URI scheme and extracts connection parameters.
```python
def __init__(self, uri: str):
if not uri.startswith("ftp://"):
raise ValueError(f"Incorrect URI scheme in {uri}, expected 'ftp'")
u = urlparse(uri)
assert u.port
assert u.hostname
self.port: int = u.port
self.host: str = u.hostname
self.port = int(u.port or 21)
assert self.FTP_USERNAME
assert self.FTP_PASSWORD
self.username: str = u.username or self.FTP_USERNAME
self.password: str = u.password or self.FTP_PASSWORD
self.basedir: str = u.path.rstrip("/")
```
--------------------------------
### Initialize Google Cloud Storage Files Store
Source: https://docs.scrapy.org/en/latest/_modules/scrapy/pipelines/files.html
Initializes the GCSFilesStore with project ID, bucket, and prefix. It also checks for necessary IAM permissions.
```python
def __init__(self, uri: str):
from google.cloud import storage # noqa: PLC0415
client = storage.Client(project=self.GCS_PROJECT_ID)
bucket, prefix = uri[5:].split("/", 1)
self.bucket = client.bucket(bucket)
self.prefix: str = prefix
permissions = self.bucket.test_iam_permissions(
["storage.objects.get", "storage.objects.create"]
)
if "storage.objects.get" not in permissions:
logger.warning(
"No 'storage.objects.get' permission for GSC bucket %(bucket)s. "
"Checking if files are up to date will be impossible. Files will be downloaded every time.",
{"bucket": bucket},
)
if "storage.objects.create" not in permissions:
logger.error(
"No 'storage.objects.create' permission for GSC bucket %(bucket)s. Saving files will be impossible!",
{"bucket": bucket},
)
```
--------------------------------
### XMLFeedSpider Iteration Tag Example
Source: https://docs.scrapy.org/en/latest/topics/spiders.html
This example shows how to define the 'itertag' attribute for an XMLFeedSpider to specify the XML node name to iterate over.
```python
itertag = 'product'
```
--------------------------------
### Enable Scrapy Extensions
Source: https://docs.scrapy.org/en/latest/topics/extensions.html
Add extensions to the EXTENSIONS setting to enable them at startup. Priorities determine the loading order.
```python
EXTENSIONS = {
"scrapy.extensions.corestats.CoreStats": 500,
"scrapy.extensions.telnet.TelnetConsole": 500,
}
```
--------------------------------
### Generate Coverage Report
Source: https://docs.scrapy.org/en/latest/contributing.html
Generate a coverage report for the tests. Ensure the `coverage` package is installed (`pip install coverage`) before running.
```bash
coverage report
```