### Install Example Dependencies

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/examples.rst

Installs additional libraries required for the example scripts and projects.

```bash
pip install -r requirements/examples.txt
```

--------------------------------

### Install Frontera

Source: https://github.com/scrapinghub/frontera/blob/master/README.md

Install the Frontera library using pip. This command is used for setting up the framework.

```bash
pip install frontera
```

--------------------------------

### Start ZeroMQ Broker

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Initiate the ZeroMQ broker, which is crucial for message communication in the distributed setup. This must be started before other components.

```bash
python -m frontera.contrib.messagebus.zeromq.broker
```

--------------------------------

### Manually Start Frontier Components

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-api.rst

Manually start Frontera components by calling the start() method. This provides fine-grained control over initialization timing, especially when auto_start is disabled.

```python
frontier.start()
```

--------------------------------

### Install Frontera with Options

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/installation.rst

Install Frontera using pip, specifying optional dependencies for specific functionalities like SQL, graphs, or distributed workers.

```bash
pip install frontera[option1,option2,...optionN]
```

--------------------------------

### Install Frontera with Distributed, ZeroMQ, and SQL support

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Install Frontera with the necessary components for distributed mode, ZeroMQ messaging, and SQL database support. This command is for Ubuntu.

```bash
pip install frontera[distributed,zeromq,sql]
```

--------------------------------

### Install Test Dependencies

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/tests.rst

Install pytest and other required libraries for running Frontera tests using pip.

```bash
pip install -r requirements/tests.txt
```

--------------------------------

### Full Backend Test Example

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-tester.rst

A comprehensive example demonstrating how to test different backend implementations with FrontierTester. It sets up a graph, configures frontier settings for a specific backend and test mode, runs the tester, and prints the crawling sequence.

```python
from frontera import FrontierManager, Settings, FrontierTester, graphs


def test_backend(backend):
    # Graph
    graph = graphs.Manager()
    graph.add_site_list(graphs.data.SITE_LIST_02)

    # Frontier
    settings = Settings()
    settings.BACKEND = backend
    settings.TEST_MODE = True
    frontier = FrontierManager.from_settings(settings)

    # Tester
    tester = FrontierTester(frontier, graph)
    tester.run(add_all_pages=True)

    # Show crawling sequence
    print '-'*40
    print frontier.backend.name
    print '-'*40
    for page in tester.sequence:
        print page.url

if __name__ == '__main__':
    test_backend('frontera.contrib.backends.memory.heapq.FIFO')
    test_backend('frontera.contrib.backends.memory.heapq.LIFO')
    test_backend('frontera.contrib.backends.memory.heapq.BFS')
    test_backend('frontera.contrib.backends.memory.heapq.DFS')
```

--------------------------------

### Start DB Worker for Batch Generation

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Command to start the DB worker for batch generation, specifying configuration and partitions.

```bash
# start DB worker only for batch generation
# use single instance for every 10 partitions
$ python -m frontera.worker.db --config [db worker config module] --no-incoming --partitions 0 1
```

--------------------------------

### Start DB Worker

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Start the database worker, which handles batch generation and data storage. This worker is configured with specific parameters for partitions and configuration.

```bash
python -m frontera.worker.db --no-incoming --config config.dbw --partitions 0
```

--------------------------------

### Start Scrapy Spider

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Initiate the Scrapy spider to begin the crawling process. This command assumes you are in the 'general-spider' directory.

```bash
python -m scrapy crawl general
```

--------------------------------

### Start Strategy Workers

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Command to start strategy workers, one process per spider log partition.

```bash
Next, let's start strategy workers, one process per spider log partition: ::

    $ python -m frontera.worker.strategy --config [strategy worker config] --partition-id 0
    $ python -m frontera.worker.strategy --config [strategy worker config] --partition-id 1
    ...
```

--------------------------------

### Start Frontera Strategy Worker

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Launches a Frontera strategy worker process. Ensure you replace '[strategy worker config]' with your actual configuration path and 'N' with the desired partition ID.

```bash
python -m frontera.worker.strategy --config [strategy worker config] --partition-id N
```

--------------------------------

### Run Scrapy Recording Spider

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/examples.rst

Starts a Scrapy spider that follows all links on a site and records the crawling results.

```bash
scrapy crawl recorder
```

--------------------------------

### Initialize FrontierManager with Default Settings

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-api.rst

Initialize FrontierManager without any parameters to use the default settings. This is a convenient way to start with a basic configuration.

```python
from frontera import FrontierManager, Settings
frontier = FrontierManager.from_settings()
```

--------------------------------

### Start Strategy Worker

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Launch the strategy worker component, responsible for managing the crawling strategy. This command requires a configuration file.

```bash
python -m frontera.worker.strategy --config config.sw
```

--------------------------------

### Request Fingerprint Example

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-objects.rst

Demonstrates how to access the generated fingerprint for a Request object. This fingerprint is typically added by the fingerprint middleware.

```python
>>> request.url
'http://thehackernews.com'

>>> request.meta['fingerprint']
'198d99a8b2284701d6c147174cd69a37a7dea90f'
```

--------------------------------

### Integrate Frontera with Requests Library

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/requests-integration.rst

This example demonstrates how to initialize and use RequestsFrontierManager to crawl web pages. It shows adding seed URLs, fetching requests, making HTTP calls with the requests library, extracting links, and updating the frontier with crawled pages and extracted links. Error handling for request exceptions is also included.

```Python
import re

import requests

from urlparse import urljoin

from frontera.contrib.requests.manager import RequestsFrontierManager
from frontera import Settings

SETTINGS = Settings()
SETTINGS.BACKEND = 'frontera.contrib.backends.memory.FIFO'
SETTINGS.LOGGING_MANAGER_ENABLED = True
SETTINGS.LOGGING_BACKEND_ENABLED = True
SETTINGS.MAX_REQUESTS = 100
SETTINGS.MAX_NEXT_REQUESTS = 10

SEEDS = [
    'http://www.imdb.com',
]

LINK_RE = re.compile(r'href="(.*?)"')


def extract_page_links(response):
    return [urljoin(response.url, link) for link in LINK_RE.findall(response.text)]

if __name__ == '__main__':

    frontier = RequestsFrontierManager(SETTINGS)
    frontier.add_seeds([requests.Request(url=url) for url in SEEDS])
    while True:
        next_requests = frontier.get_next_requests()
        if not next_requests:
            break
        for request in next_requests:
                try:
                    response = requests.get(request.url)
                    links = [requests.Request(url=url) for url in extract_page_links(response)]
                    frontier.page_crawled(response=response)
                    frontier.links_extracted(request=request, links=links)
                except requests.RequestException, e:
                    error_code = type(e).__name__
                    frontier.request_error(request, error_code)
```

--------------------------------

### Start DB Worker for Spider Log Processing

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Command to start a DB worker dedicated to spider log processing, without batch generation.

```bash
# Optionally, start next one dedicated to spider log processing.
$ python -m frontera.worker.db --no-batches --config [db worker config module]
```

--------------------------------

### Scrapy Spider Example with Frontera Integration

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

An example of a Scrapy spider that extracts links from responses and creates new requests. This spider is designed to work with Frontera, which will manage the scheduling and queuing of these requests.

```python
from scrapy import Spider
from scrapy.linkextractors import LinkExtractor
from scrapy.http import Request
from scrapy.http.response.html import HtmlResponse

class CommonPageSpider(Spider):

    name = "commonpage"

    def __init__(self, *args, **kwargs):
        super(CommonPageSpider, self).__init__(*args, **kwargs)
        self.le = LinkExtractor()

    def parse(self, response):
        if not isinstance(response, HtmlResponse):
            return
        for link in self.le.extract_links(response):
            r = Request(url=link.url)
            r.meta.update(link_text=link.text)
            yield r
```

--------------------------------

### Configure Recorder Storage Engine

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-recorder.rst

Set the storage engine for the Graph Manager used by the recorder. This example uses an SQLite database.

```python
RECORDER_STORAGE_ENGINE = 'sqlite:///my_record.db'
```

--------------------------------

### Request Domain Information Example

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-objects.rst

Shows how to access domain-related information stored in the Request object's meta field. This data is often added by the domain middleware.

```python
>>> request.url
'http://www.scrapinghub.com'

>>> request.meta['domain']
{
    "name": "scrapinghub.com",
    "netloc": "www.scrapinghub.com",
    "scheme": "http",
    "sld": "scrapinghub",
    "subdomain": "www",
    "tld": "com"
}
```

--------------------------------

### Clone Frontera GitHub Repository

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Clone the Frontera repository from GitHub to access example spider code and other resources.

```bash
git clone https://github.com/scrapinghub/frontera.git
```

--------------------------------

### Backend Test Class with Database Cleanup

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/tests.rst

Example of a test class for a custom backend that requires database file cleanup before and after each test.

```python
class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def setup_backend(self, method):
        self._delete_test_db()

    def teardown_backend(self, method):
        self._delete_test_db()

    def _delete_test_db(self):
        try:
            os.remove('mytestdb.db')
        except OSError:
            pass

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...
```

--------------------------------

### Define a simple site graph

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Represents a site with a starting page 'A' linking to 'B', 'C', and 'D'. Pages without explicit links can be omitted.

```python
site = [
    ('A', ['B', 'C', 'D']),
]
```

--------------------------------

### Basic Backend Test Class

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/tests.rst

Example of a test class for a custom backend, inheriting from BackendTest and creating a new frontier instance for each test method.

```python
class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...
```

--------------------------------

### Test Backend Sequence Sorting

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/tests.rst

Example test class for a backend that sorts pages alphabetically, using BackendSequenceTest to assert the crawling sequence.

```python
class TestAlphabeticSortBackend(backends.BackendSequenceTest):

    backend_class = 'frontera.contrib.backend.abackend.AlphabeticSortBackend'

    SITE_LIST = [
        [
            ('C', []),
            ('B', []),
            ('A', []),
        ],
    ]

    def test_one(self):
        # Check sequence is the expected one
        self.assert_sequence(site_list=self.SITE_LIST,
                             expected_sequence=['A', 'B', 'C'],
                             max_next_requests=0)

    def test_two(self):
        # Get sequence and work with it
        sequence = self.get_sequence(site_list=SITE_LIST,
                            max_next_requests=0)
        assert len(sequence) > 2

    ...
```

--------------------------------

### Auto Throttling and Concurrency Settings

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

Configure auto throttling and concurrency for polite crawling. This includes enabling auto throttling, setting maximum and starting delays, disabling randomized delays, and defining concurrent requests per domain.

```python
# auto throttling
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_DEBUG = False
AUTOTHROTTLE_MAX_DELAY = 3.0
AUTOTHROTTLE_START_DELAY = 0.25     # Any small enough value, it will be adjusted during operation by averaging
                                        # with response latencies.
RANDOMIZE_DOWNLOAD_DELAY = False

# concurrency
CONCURRENT_REQUESTS = 256           # Depends on many factors, and should be determined experimentally
CONCURRENT_REQUESTS_PER_DOMAIN = 10
DOWNLOAD_DELAY = 0.0
```

--------------------------------

### Render Graph to PNG File

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Renders the current state of the graph manager to a PNG file. This visualization requires `pydot` and `Graphviz` to be installed.

```python
>>> g.render(filename='graph.png', label='A simple Graph')
```

--------------------------------

### Run Scrapy Spider with Partition ID

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Starts a Scrapy spider process, assigning it to a specific partition using the SPIDER_PARTITION_ID setting. This is repeated for each partition 'N'. The setting can also be defined in the Scrapy configuration file.

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=1
```

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=2
```

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=N
```

--------------------------------

### Initialize Graph Manager and add site

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Shows the basic steps to create a Graph Manager instance and populate it with a site definition using `add_site`.

```python
from frontera.utils import graphs
g = graphs.Manager()
site = [('A', ['B', 'C', 'D'])]
g.add_site(site)
```

--------------------------------

### Initialize FrontierManager from Settings String

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-api.rst

Initialize FrontierManager using a string path to Frontera settings. Ensure the settings module is importable.

```python
from frontera import FrontierManager
frontier = FrontierManager.from_settings('my_project.frontier.settings')
```

--------------------------------

### Choose a Built-in Backend

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-single.rst

Configure Frontera to use a built-in backend, such as the SQLAlchemy distributed backend.

```python
BACKEND = 'frontera.contrib.backends.sqlalchemy.Distributed'
```

--------------------------------

### Add Seed URLs to Frontera

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Populates Frontera with initial URLs from a specified file. Replace '[your_frontera_config]' with your configuration path and '[path to your seeds file]' with the actual path to your seeds file.

```bash
python -m frontera.utils.add_seeds --config [your_frontera_config] --seeds-file [path to your seeds file]
```

--------------------------------

### Run All Tests

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/tests.rst

Execute all tests in the Frontera project from the root directory using the py.test command.

```bash
py.test
```

--------------------------------

### Get all seed pages in the graph

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Retrieves a list containing only the seed `CrawlPage` objects from the graph manager.

```python
g.seeds
```

--------------------------------

### Run Scrapy Spider with Recording Enabled

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-recorder.rst

Run your Scrapy spider as usual from the command line. Recording is enabled by default.

```bash
scrapy crawl myspider
```

--------------------------------

### Get a specific page by URL

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Fetches a `CrawlPage` object using its URL. Returns `None` if the page does not exist in the manager.

```python
g.get_page('A')
```

```python
g.get_page('F')
```

--------------------------------

### Get all pages in the graph

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Retrieves a list of all `CrawlPage` objects currently in the graph manager. Seed pages are indicated by an asterisk.

```python
g.pages
```

--------------------------------

### Configure Frontera for Single Process

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-single.rst

Set these parameters to indicate that Frontera will run locally. This is essential for single-process mode.

```python
SPIDER_FEED_PARTITIONS = 1
SPIDER_LOG_PARTITIONS = 1
```

--------------------------------

### Initialize FrontierManager from Settings Object

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-api.rst

Initialize FrontierManager using a BaseSettings object instance. Allows programmatic modification of settings before initialization.

```python
from frontera import FrontierManager, Settings
settings = Settings()
settings.MAX_PAGES = 0
frontier = FrontierManager.from_settings(settings)
```

--------------------------------

### Add page and links individually

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Shows an alternative method to build the graph by adding pages and links one by one using `add_page` and `add_link`.

```python
g = graphs.Manager()
a = g.add_page(url='A', is_seed=True)
b = g.add_link(page=a, url='B')
c = g.add_link(page=a, url='C')
d = g.add_link(page=a, url='D')
```

--------------------------------

### Activate MemoryDistributedBackend

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-backends.rst

Set the BACKEND setting to activate the MemoryDistributedBackend. Some backends may require additional configuration.

```python
BACKEND = 'frontera.contrib.backends.memory.MemoryDistributedBackend'
```

--------------------------------

### Configure Frontera Settings via Scrapy

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

Specify the module path for Frontera's settings within the Scrapy settings file. This allows for centralized configuration of Frontera's behavior.

```python
FRONTERA_SETTINGS = 'tutorial.frontera.settings'
```

--------------------------------

### Run Links Follower Script

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/examples.rst

Executes a script that follows all links from a given site using the Requests library.

```bash
python links_follower.py
```

--------------------------------

### Creating a FrontierTester Instance

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-tester.rst

Initializes a FrontierTester with a Graph Manager and a FrontierManager. The graph manager is used for loading fake crawl data, and the frontier manager is created from default settings.

```python
from frontera import FrontierManager, FrontierTester
from frontera.utils import graphs
graph = graphs.Manager('sqlite:///graph.db')  # Crawl fake data loading
frontier = FrontierManager.from_settings()  # Create frontier from default settings
tester = FrontierTester(frontier, graph)
```

--------------------------------

### Add multiple sites to the manager

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Illustrates how to add distinct site graphs to the same Graph Manager instance, resulting in a combined graph of all pages and links.

```python
site1 = [('A1', ['B1', 'C1', 'D1'])]
site2 = [('A2', ['B2', 'C2', 'D2'])]
g = graphs.Manager()
g.add_site(site1)
g.add_site(site2)
```

--------------------------------

### Initialize Graph Manager with SQLite Database

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Initializes the Graph Manager using an in-memory SQLite database. This is the default behavior, but an explicit engine can be provided.

```python
>>> g = graphs.Manager(engine='sqlite:///graph.db')
```

--------------------------------

### Define a site graph with multiple parents

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Illustrates a scenario where page 'C' has multiple referers ('B' and 'D'), showing how shared pages are handled.

```python
site = [
    ('A', ['B', 'C', 'D']),
    ('B', ['C']),
    ('D', ['C']),
]
```

--------------------------------

### Add Seeds to Frontera

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-distributed.rst

Bootstrap the crawling process by adding seed URLs to Frontera. This command uses a specified configuration file and a list of seed URLs.

```bash
python -m frontera.utils.add_seeds --config config.dbw --seeds-file seeds_es_smp.txt
```

--------------------------------

### Fake Crawl Simulation with FrontierManager

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Simulates a crawl process by integrating Frontera's FrontierManager with the Graph Manager. It loads graph data, seeds the frontier, processes requests, and updates the frontier with responses, mimicking a real crawl's lifecycle.

```python
from frontera import FrontierManager, Request, Response
from frontera.utils import graphs

if __name__ == '__main__':
    # Load graph from existing database
    graph = graphs.Manager('sqlite:///graph.db')

    # Create frontier from default settings
    frontier = FrontierManager.from_settings()

    # Create and add seeds
    seeds = [Request(seed.url) for seed in graph.seeds]
    frontier.add_seeds(seeds)

    # Get next requests
    next_requets = frontier.get_next_requests()

    # Crawl pages
    while (next_requests):
        for request in next_requests:

            # Fake page crawling
            crawled_page = graph.get_page(request.url)

            # Create response
            response = Response(url=crawled_page.url, status_code=crawled_page.status)

            # Update Page
            page = frontier.page_crawled(response=response
                                         links=[link.url for link in crawled_page.links])
            # Get next requests
            next_requets = frontier.get_next_requests()
```

--------------------------------

### Settings for Efficient Broad Crawling

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

Configure settings for efficient broad crawling, including disabling disk cache, enabling redirects, disabling cookies, setting download timeout, disabling retries, setting max download size, and adjusting log stats interval.

```python
HTTPCACHE_ENABLED = False   # Turns off disk cache, which has low hit ratio during broad crawls
REDIRECT_ENABLED = True
COOKIES_ENABLED = False
DOWNLOAD_TIMEOUT = 120
RETRY_ENABLED = False   # Retries can be handled by Frontera itself, depending on crawling strategy
DOWNLOAD_MAXSIZE = 10 * 1024 * 1024  # Maximum document size, causes OOM kills if not set
LOGSTATS_INTERVAL = 10  # Print stats every 10 secs to console
```

--------------------------------

### Accessing Settings in a Middleware

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontera-settings.rst

Demonstrates how to access Frontera settings within a custom middleware class. The settings object is available via the manager.

```python
class MyMiddleware(Component):

    @classmethod
    def from_manager(cls, manager):
        manager = crawler.settings
        if settings.TEST_MODE:
            print "test mode is enabled!"
```

--------------------------------

### Activate a Frontier Middleware

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-middlewares.rst

Add middleware class paths to the MIDDLEWARES setting to activate them. Middlewares are processed in the order they appear in the list.

```python
MIDDLEWARES = [
    'frontera.contrib.middlewares.domain.DomainMiddleware',
]
```

--------------------------------

### Configure Recorder Database Reset Options

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-recorder.rst

Options to clear content or drop all tables in the Graph Manager's storage database.

```python
RECORDER_STORAGE_DROP_ALL_TABLES = True
RECORDER_STORAGE_CLEAR_CONTENT = True
```

--------------------------------

### Configure Robots.txt Obedience

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

Set ROBOTSTXT_OBEY to False if you are implementing robots.txt obeying within your crawling strategy. Otherwise, it defaults to True.

```python
ROBOTSTXT_OBEY = False
```

--------------------------------

### Define a site graph using URLs

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Demonstrates using actual URLs as page identifiers in the site graph definition, suitable for real-world crawling scenarios.

```python
site = [
    ('http://example.com', ['http://example.com/anotherpage', 'http://othersite.com']),
]
```

--------------------------------

### Add link to an existing page

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Demonstrates adding a new link ('E') to an already existing page ('D') after the initial site has been added.

```python
site = [('A', ['B', 'C', 'D'])]
g = graphs.Manager()
g.add_site(site)
d = g.get_page('D')
g.add_link(d, 'E')
```

--------------------------------

### Common Frontera Settings

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Defines shared settings for Frontera components, including Kafka configuration, middleware extensions, and partition counts.

```python
from __future__ import absolute_import
from frontera.settings.default_settings import MIDDLEWARES
MAX_NEXT_REQUESTS = 512
SPIDER_FEED_PARTITIONS = 2 # number of spider processes
SPIDER_LOG_PARTITIONS = 2 # worker instances
MIDDLEWARES.extend([
    'frontera.contrib.middlewares.domain.DomainMiddleware',
    'frontera.contrib.middlewares.fingerprint.DomainFingerprintMiddleware'
])

QUEUE_HOSTNAME_PARTITIONING = True
KAFKA_LOCATION = 'localhost:9092' # your Kafka broker host:port
SCORING_TOPIC = 'frontier-scoring'
URL_FINGERPRINT_FUNCTION='frontera.utils.fingerprint.hostname_local_fingerprint'
```

--------------------------------

### Activate Scrapy Recorder Middlewares

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-recorder.rst

Add the CrawlRecorderSpiderMiddleware and CrawlRecorderDownloaderMiddleware to your Scrapy project's settings to enable recording.

```python
SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderDownloaderMiddleware': 1000,
})
```

--------------------------------

### Running a FrontierTester Test

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-tester.rst

Executes the test crawl using the initialized FrontierTester. The tester adds seeds from the graph, requests next pages from the frontier, and simulates page responses.

```python
tester.run()
```

--------------------------------

### Load Graph from Existing Database

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Loads an existing graph from a specified SQLite database file. This allows for persistence of graph data between runs.

```python
>>> graph = graphs.Manager(engine='sqlite:///graph.db')
>>> graph.add_site(('A', []))

>>> another_graph = graphs.Manager(engine='sqlite:///graph.db')
>>> another_graph.pages
[<1:A1*>]
```

--------------------------------

### Scrapy Settings for Frontera Integration

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/cluster-setup.rst

Integrates Frontera with Scrapy by setting Frontera settings module, scheduler, and relevant middlewares.

```python
FRONTERA_SETTINGS = ''  # module path to your Frontera spider config module

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'

SPIDER_MIDDLEWARES = {
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 999,
    'frontera.contrib.scrapy.middlewares.seeds.file.FileSeedLoader': 1,
}
DOWNLOADER_MIDDLEWARES = {
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 999,
}
```

--------------------------------

### Inject Seed URLs

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/quick-start-single.rst

Use this command-line utility to inject seed URLs into the backend's queue if your crawling strategy requires external seeds.

```bash
$ python -m frontera.utils.add_seeds --config [your_frontera_config] --seeds-file [path to your seeds file]
```

--------------------------------

### Creating FrontierTester with Max Next Pages

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-tester.rst

Initializes a FrontierTester, specifying the maximum number of next requests to be retrieved per call to get_next_requests. This can be used to control the test's page retrieval rate.

```python
tester = FrontierTester(frontier, graph, max_next_pages=10)
```

--------------------------------

### Activate Frontera Middlewares and Scheduler in Scrapy

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/scrapy-integration.rst

Configure Scrapy settings to use Frontera's spider middleware, downloader middleware, and scheduler. This is essential for enabling Frontera's crawling logic within a Scrapy project.

```python
SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 1000,
})

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'
```

--------------------------------

### Running a Test with All Graph Pages as Seeds

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-tester.rst

Runs the FrontierTester test while ensuring all pages from the graph are added as initial seeds for the crawl.

```python
tester.run(add_all_pages=True)
```

--------------------------------

### FrontierManager Class Methods

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/frontier-api.rst

This section details the primary methods available on the FrontierManager class for interacting with the Frontera core components.

```APIDOC
## Class Methods

### `from_settings(settings=None)`

**Description**: Initializes the FrontierManager, optionally loading settings from a provided path or object.

**Parameters**:

* **settings** (str or frontera.settings.BaseSettings, optional): A string path to settings or a BaseSettings object. If None, default settings are used.

### `start()`

**Description**: Initiates the startup process for Frontera components, performing necessary initialization operations.

### `stop()`

**Description**: Finalizes Frontera components, performing necessary cleanup operations.

### `add_seeds(seeds)`

**Description**: Adds a list of seed URLs to the frontier to begin the crawling process.

**Parameters**:

* **seeds** (iterable): An iterable of seed URLs or Request objects.

### `get_next_requests()`

**Description**: Retrieves the next batch of requests to be crawled from the frontier.

**Returns**:

* An iterable of Request objects.

### `page_crawled(response, request)`

**Description**: Notifies the frontier that a page has been successfully crawled, processing the response and the original request.

**Parameters**:

* **response** (Response): The response object received from crawling the page.
* **request** (Request): The original request object that led to this response.

### `request_error(request, error)`

**Description**: Reports an error that occurred while trying to process a request.

**Parameters**:

* **request** (Request): The request object that failed.
* **error** (Exception): The exception object representing the error.
```

--------------------------------

### Add Site List to Graph Manager

Source: https://github.com/scrapinghub/frontera/blob/master/docs/source/topics/graph-manager.rst

Adds a list of sites, each with associated URLs, to the graph manager. This is useful for initializing the graph with multiple sites and their outgoing links.

```python
>>> site_list = [
    [('A1', ['B1', 'C1', 'D1'])],
    [('A2', ['B2', 'C2', 'D2'])]
]
>>> g = graphs.Manager()
>>> g.add_site_list(site_list)
```