Frontera

https://frontera.readthedocs.io/en/latest/
Admin
Frontera is a web crawling framework that implements a crawl frontier component for managing and...
Tokens:20,484
Snippets:296
Trust Score:9.6
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install Example Dependencies

Source: https://frontera.readthedocs.io/en/latest/topics/examples.html

Installs necessary libraries for running Frontera examples. Ensure you have pip installed.

```bash
pip install -r requirements/examples.txt
```

--------------------------------

### Start ZeroMQ Broker

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Initiate the ZeroMQ broker, which is essential for message queuing in the distributed setup. This should be started before other components.

```bash
python -m frontera.contrib.messagebus.zeromq.broker
```

--------------------------------

### Install Frontera with Distributed, ZeroMQ, and SQL support

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Install Frontera with the necessary components for distributed crawling, ZeroMQ messaging, and SQL database support.

```bash
pip install frontera[distributed,zeromq,sql]
```

--------------------------------

### Install Test Dependencies

Source: https://frontera.readthedocs.io/en/latest/topics/tests.html

Install pytest and other required libraries for running Frontera tests.

```bash
pip install -r requirements/tests.txt
```

--------------------------------

### Install Frontera with Optional Dependencies

Source: https://frontera.readthedocs.io/en/latest/topics/installation.html

Use this command to install Frontera with specific optional dependencies for features like SQL, Graph Manager, logging, HBase, ZeroMQ, Kafka, distributed workers, S3 seeds, Redis backend, or crawling strategies.

```bash
pip install frontera[option1,option2,...optionN]
```

--------------------------------

### Full example of Frontier Tester with different backends in Python

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-tester.html

Demonstrates a complete usage example of Frontier Tester, simulating crawls with different backend configurations and printing the resulting crawling sequence.

```python
from frontera import FrontierManager, Settings, FrontierTester, graphs


def test_backend(backend):
    # Graph
    graph = graphs.Manager()
    graph.add_site_list(graphs.data.SITE_LIST_02)

    # Frontier
    settings = Settings()
    settings.BACKEND = backend
    settings.TEST_MODE = True
    frontier = FrontierManager.from_settings(settings)

    # Tester
    tester = FrontierTester(frontier, graph)
    tester.run(add_all_pages=True)

    # Show crawling sequence
    print '-'*40
    print frontier.backend.name
    print '-'*40
    for page in tester.sequence:
        print page.url

if __name__ == '__main__':
    test_backend('frontera.contrib.backends.memory.heapq.FIFO')
    test_backend('frontera.contrib.backends.memory.heapq.LIFO')
    test_backend('frontera.contrib.backends.memory.heapq.BFS')
    test_backend('frontera.contrib.backends.memory.heapq.DFS')
```

--------------------------------

### Choose Frontera Backend

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-single.html

Specify the backend to be used by Frontera. This example uses the SQLAlchemy distributed backend.

```ini
BACKEND = 'frontera.contrib.backends.sqlalchemy.Distributed'

```

--------------------------------

### Start Scrapy Spider

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Initiate the Scrapy spider to begin the crawling process. This command should be run from the 'general-spider' directory.

```bash
python -m scrapy crawl general
```

--------------------------------

### Component initialization from manager

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-api.html

Example of a class method for components to initialize themselves using the FrontierManager. It receives the manager instance and can access its settings.

```python
def from_manager(cls, manager):
    return cls(settings=manager.settings)
```

--------------------------------

### Request Object Example

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-objects.html

Demonstrates accessing the URL and meta information, including a generated fingerprint, for a Request object.

```python
>>> request.url
'http://thehackernews.com'

>>> request.meta['fingerprint']
'198d99a8b2284701d6c147174cd69a37a7dea90f'
```

--------------------------------

### Initialize Graph Manager and Add Site

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Demonstrates the basic setup for using the Graph Manager. First, import the Manager class, then create an instance and add a defined site graph to it.

```python
>>> from frontera.utils import graphs
>>> g = graphs.Manager()
>>> site = [('A', ['B', 'C', 'D'])]
>>> g.add_site(site)
```

--------------------------------

### Starting DB Worker

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Command to start the DB worker for batch generation. It specifies the configuration module and the number of partitions.

```bash
# start DB worker only for batch generation
# use single instance for every 10 partitions
$ python -m frontera.worker.db --config [db worker config module] --no-incoming --partitions 0,1

```

--------------------------------

### DomainMiddleware Testing Mode Example

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-middlewares.html

Shows how DomainMiddleware handles testing URLs in TEST_MODE, parsing letter domains and assigning default values for missing components.

```python
>>> request.url
'A1'

>>> request.meta['domain']
{
    "name": "A",
    "netloc": "A",
    "scheme": "-",
    "sld": "-",
    "subdomain": "-",
    "tld": "-"
}
```

--------------------------------

### Start Frontera DB Worker

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Initiates the database worker process for Frontera. Requires a database worker configuration module.

```bash
python -m frontera.worker.db --no-batches --config [db worker config module]
```

--------------------------------

### Load FrontierManager with default settings

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-api.html

Initializes the FrontierManager using default settings. This is a convenient way to start Frontera when no custom configuration is immediately required.

```python
from frontera import FrontierManager, Settings
frontier = FrontierManager.from_settings()
```

--------------------------------

### Run Scrapy Spider

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-single.html

Execute your Scrapy spider using the `scrapy crawl` command. This starts the crawling process integrated with Frontera.

```bash
scrapy crawl myspider

```

--------------------------------

### Integrate Frontera with Requests

Source: https://frontera.readthedocs.io/en/latest/topics/requests-integration.html

This example shows how to initialize and use the RequestsFrontierManager to manage requests and responses using the Requests library. It includes adding seeds, fetching next requests, crawling pages, extracting links, and handling request errors.

```python
import re

import requests

from urlparse import urljoin

from frontera.contrib.requests.manager import RequestsFrontierManager
from frontera import Settings

SETTINGS = Settings()
SETTINGS.BACKEND = 'frontera.contrib.backends.memory.FIFO'
SETTINGS.LOGGING_MANAGER_ENABLED = True
SETTINGS.LOGGING_BACKEND_ENABLED = True
SETTINGS.MAX_REQUESTS = 100
SETTINGS.MAX_NEXT_REQUESTS = 10

SEEDS = [
    'http://www.imdb.com',
]

LINK_RE = re.compile(r'href="(.*?)"')


def extract_page_links(response):
    return [urljoin(response.url, link) for link in LINK_RE.findall(response.text)]

if __name__ == '__main__':

    frontier = RequestsFrontierManager(SETTINGS)
    frontier.add_seeds([requests.Request(url=url) for url in SEEDS])
    while True:
        next_requests = frontier.get_next_requests()
        if not next_requests:
            break
        for request in next_requests:
                try:
                    response = requests.get(request.url)
                    links = [requests.Request(url=url) for url in extract_page_links(response)]
                    frontier.page_crawled(response=response)
                    frontier.links_extracted(request=request, links=links)
                except requests.RequestException, e:
                    error_code = type(e).__name__ 
                    frontier.request_error(request, error_code)

```

--------------------------------

### Start Frontera DB Worker

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Start the database worker responsible for managing crawled data and generating batches. The '--no-incoming' flag is used here.

```bash
python -m frontera.worker.db --no-incoming --config config.dbw --partitions 0
```

--------------------------------

### DomainMiddleware Request Meta Example

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-middlewares.html

Demonstrates the 'domain' field added to a Request's meta attribute by the DomainMiddleware, showing parsed URL components.

```python
>>> request.url
'http://www.scrapinghub.com:8080/this/is/an/url'

>>> request.meta['domain']
{
    "name": "scrapinghub.com",
    "netloc": "www.scrapinghub.com",
    "scheme": "http",
    "sld": "scrapinghub",
    "subdomain": "www",
    "tld": "com"
}
```

--------------------------------

### Run Scrapy Recording Spider

Source: https://frontera.readthedocs.io/en/latest/topics/examples.html

Starts a Scrapy spider for recording crawling results. This command assumes you are in the directory containing the Scrapy project.

```bash
scrapy crawl recorder
```

--------------------------------

### Define a Simple Site Graph

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Represents a site with a starting page 'A' linking to pages 'B', 'C', and 'D'. Use this to define basic site structures.

```python
site = [
    ('A', ['B', 'C', 'D']),
]
```

--------------------------------

### Clone Frontera GitHub Repository

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Clone the Frontera repository from GitHub to access example spider code and other resources.

```bash
git clone https://github.com/scrapinghub/frontera.git
```

--------------------------------

### Start Frontera Strategy Worker

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Launch the strategy worker component, which is responsible for managing the crawling strategy and deciding which URLs to fetch next.

```bash
python -m frontera.worker.strategy --config config.sw
```

--------------------------------

### Start Frontera Strategy Workers

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Launches strategy worker processes, typically one per spider log partition. Requires a strategy worker configuration.

```bash
python -m frontera.worker.strategy --config [strategy worker config] --partition-id 0
```

```bash
python -m frontera.worker.strategy --config [strategy worker config] --partition-id 1
```

```bash
python -m frontera.worker.strategy --config [strategy worker config] --partition-id N
```

--------------------------------

### Simulate Crawl with Frontier Manager and Graph Manager

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

This example demonstrates how to fake a crawl using Frontera's FrontierManager and GraphManager. It loads graph data, creates a frontier, adds seeds, and simulates the crawling process by updating the frontier with responses based on graph data.

```python
from frontera import FrontierManager, Request, Response
from frontera.utils import graphs

if __name__ == '__main__':
    # Load graph from existing database
    graph = graphs.Manager('sqlite:///graph.db')

    # Create frontier from default settings
    frontier = FrontierManager.from_settings()

    # Create and add seeds
    seeds = [Request(seed.url) for seed in graph.seeds]
    frontier.add_seeds(seeds)

    # Get next requests
    next_requets = frontier.get_next_requests()

    # Crawl pages
    while (next_requests):
        for request in next_requests:

            # Fake page crawling
            crawled_page = graph.get_page(request.url)

            # Create response
            response = Response(url=crawled_page.url, status_code=crawled_page.status)

            # Update Page
            page = frontier.page_crawled(response=response
                                         links=[link.url for link in crawled_page.links])
            # Get next requests
            next_requets = frontier.get_next_requests()
```

--------------------------------

### Crawl with Frontera Spider

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Starts a Scrapy spider process, assigning it a specific partition ID. The partition ID can also be set in the Frontera configuration file.

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=1
```

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=2
```

```bash
scrapy crawl [spider] -L INFO -s SPIDER_PARTITION_ID=N
```

--------------------------------

### Add Seeds using Frontera Utility

Source: https://frontera.readthedocs.io/en/latest/topics/custom_crawling_strategy.html

Use the helper application in Frontera to inject seeds into the crawler pipeline. This is typically done when starting a crawl from scratch.

```bash
$ python -m frontera.utils.add_seeds --config ... --seeds-file ...
```

--------------------------------

### Domain Metadata Fingerprinting Example

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-middlewares.html

Illustrates how DomainFingerprintMiddleware adds a 'domain' object, including a 'fingerprint', to a request's metadata. The fingerprint is derived from the URL using a specific domain fingerprinting function.

```python
>>> request.url
'http//www.scrapinghub.com:8080'

>>> request.meta['domain']
{
    "fingerprint": "5bab61eb53176449e25c2c82f172b82cb13ffb9d",
    "name": "scrapinghub.com",
    "netloc": "www.scrapinghub.com",
    "scheme": "http",
    "sld": "scrapinghub",
    "subdomain": "www",
    "tld": "com"
}


```

--------------------------------

### Request Metadata Fingerprinting Example

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-middlewares.html

Demonstrates how UrlFingerprintMiddleware adds a 'fingerprint' to a request's metadata based on its URL. The fingerprint is calculated using a configured function.

```python
>>> request.url
'http//www.scrapinghub.com:8080'

>>> request.meta['fingerprint']
'60d846bc2969e9706829d5f1690f11dafb70ed18'

```

--------------------------------

### Backend Test Class with Database Cleanup

Source: https://frontera.readthedocs.io/en/latest/topics/tests.html

Extend BackendTest to include setup and teardown methods for cleaning a database file before and after each test. This ensures a clean state for tests involving databases.

```python
class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def setup_backend(self, method):
        self._delete_test_db()

    def teardown_backend(self, method):
        self._delete_test_db()

    def _delete_test_db(self):
        try:
            os.remove('mytestdb.db')
        except OSError:
            pass

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...
```

--------------------------------

### Basic Backend Test Class

Source: https://frontera.readthedocs.io/en/latest/topics/tests.html

Define a test class for a Frontera backend using BackendTest. This example shows how to set the backend_class and create a frontier instance for tests.

```python
class TestMyBackend(backends.BackendTest):

    backend_class = 'frontera.contrib.backend.abackend.MyBackend'

    def test_one(self):
        frontier = self.get_frontier()
        ...

    def test_two(self):
        frontier = self.get_frontier()
        ...

    ...
```

--------------------------------

### Test Backend Crawling Sequences

Source: https://frontera.readthedocs.io/en/latest/topics/tests.html

Use BackendSequenceTest to test crawling sequences for a backend. This example demonstrates testing an alphabetic sort backend and asserting the expected page visit sequence.

```python
class TestAlphabeticSortBackend(backends.BackendSequenceTest):

    backend_class = 'frontera.contrib.backend.abackend.AlphabeticSortBackend'

    SITE_LIST = [
        [
            ('C', []),
            ('B', []),
            ('A', []),
        ],
    ]

    def test_one(self):
        # Check sequence is the expected one
        self.assert_sequence(site_list=self.SITE_LIST,
                             expected_sequence=['A', 'B', 'C'],
                             max_next_requests=0)

    def test_two(self):
        # Get sequence and work with it
        sequence = self.get_sequence(site_list=SITE_LIST,
                            max_next_requests=0)
        assert len(sequence) > 2

    ...
```

--------------------------------

### Retrieve Seed Pages from Graph

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Filters and returns only the seed pages from the graph manager. Seed pages are typically the starting points for crawling.

```python
>>> g.seeds
[<1:A*>]
```

--------------------------------

### Create Scrapy Project

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-single.html

Use the `scrapy startproject` command to create a new Scrapy project. This sets up the basic directory structure for your project.

```bash
scrapy startproject tutorial

```

--------------------------------

### Directory Structure for Frontera Settings

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

This illustrates a recommended directory structure for organizing Frontera settings, middlewares, and backends within a project.

```text
my_project/
    frontier/
        __init__.py
        settings.py
        middlewares.py
        backends.py
    ...

```

--------------------------------

### Add Multiple Sites using add_site_list

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Shows how to add multiple sites efficiently by providing them as a list of site definitions to the `add_site_list` method. This is a convenient way to load several site graphs at once.

```python
>>> site_list = [
    [('A1', ['B1', 'C1', 'D1'])],
    [('A2', ['B2', 'C2', 'D2'])],
]
>>> g = graphs.Manager()
>>> g.add_site_list(site_list)
```

--------------------------------

### Activate Memory Backend

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-backends.html

Set the BACKEND setting to 'frontera.contrib.backends.memory.MemoryDistributedBackend' to activate the in-memory backend.

```python
BACKEND = 'frontera.contrib.backends.memory.MemoryDistributedBackend'
```

--------------------------------

### Run All Tests

Source: https://frontera.readthedocs.io/en/latest/topics/tests.html

Execute all tests in the Frontera project from the root directory.

```bash
py.test
```

--------------------------------

### Load FrontierManager from settings path

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-api.html

Initializes the FrontierManager using a string path to project settings. Ensure the path correctly points to your Frontera settings module.

```python
from frontera import FrontierManager
frontier = FrontierManager.from_settings('my_project.frontier.settings')
```

--------------------------------

### Get a Specific Page by URL

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Retrieves a specific page object from the graph manager using its URL. Returns None if the page does not exist in the graph.

```python
>>> g.get_page('A')
<1:A*>
```

```python
>>> g.get_page('F')
None
```

--------------------------------

### Combine add_site with add_link

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Illustrates how `add_site` can be used initially, and then `add_link` can be used to add more links to existing pages. This allows for incremental graph building.

```python
>>> site = [('A', ['B', 'C', 'D'])]
>>> g = graphs.Manager()
>>> g.add_site(site)
>>> d = g.get_page('D')
>>> g.add_link(d, 'E')
```

--------------------------------

### Creating a Frontier Tester instance in Python

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-tester.html

Initializes a FrontierTester instance by creating a Graph Manager for fake crawl data and a FrontierManager from default settings.

```python
>>> from frontera import FrontierManager, FrontierTester
>>> from frontera.utils import graphs
>>> graph = graphs.Manager('sqlite:///graph.db')  # Crawl fake data loading
>>> frontier = FrontierManager.from_settings()  # Create frontier from default settings
>>> tester = FrontierTester(frontier, graph)
```

--------------------------------

### Initialize Graph Manager with SQLite

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Initializes the Graph Manager using an in-memory SQLite database. This is the default storage engine.

```python
>>> g = graphs.Manager(engine='sqlite:///graph.db')
```

--------------------------------

### Load FrontierManager from Settings object

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-api.html

Initializes the FrontierManager using an instance of a BaseSettings object. This allows for programmatic configuration of Frontera settings before initialization.

```python
from frontera import FrontierManager, Settings
settings = Settings()
settings.MAX_PAGES = 0
frontier = FrontierManager.from_settings(settings)
```

--------------------------------

### Configure Scrapy Recorder Storage Engine

Source: https://frontera.readthedocs.io/en/latest/topics/scrapy-recorder.html

Set the `RECORDER_STORAGE_ENGINE` setting to specify the Graph Manager storage engine for crawl recordings. For example, use `sqlite:///my_record.db` for a SQLite database.

```python
RECORDER_STORAGE_ENGINE = 'sqlite:///my_record.db'
```

--------------------------------

### Running Frontier Tester with all pages as seeds in Python

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-tester.html

Runs the Frontier Tester, configuring it to add all available graph pages as initial seeds for the crawl.

```python
>>> tester.run(add_all_pages=True)
```

--------------------------------

### Running a Frontier Tester in Python

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-tester.html

Executes the simulated crawl using the initialized FrontierTester, adding seeds, requesting pages, and faking responses.

```python
>>> tester.run()
```

--------------------------------

### Run Scrapy Spider

Source: https://frontera.readthedocs.io/en/latest/topics/scrapy-recorder.html

Execute your Scrapy spider from the command line as usual. The recorder will be active by default.

```bash
scrapy crawl myspider
```

--------------------------------

### Add Page and Links Individually

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Shows an alternative method to define a graph by adding pages and links one by one using `add_page` and `add_link`. This offers more granular control over graph construction.

```python
>>> g = graphs.Manager()
>>> a = g.add_page(url='A', is_seed=True)
>>> b = g.add_link(page=a, url='B')
>>> c = g.add_link(page=a, url='C')
>>> d = g.add_link(page=a, url='D')
```

--------------------------------

### Activate DomainMiddleware

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-middlewares.html

Add the DomainMiddleware to the MIDDLEWARES setting to activate it. Middlewares are processed in the order they appear in the list.

```python
MIDDLEWARES = [
    'frontera.contrib.middlewares.domain.DomainMiddleware',
]
```

--------------------------------

### Add Seed URLs to Frontera

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-distributed.html

Bootstrap the crawling process by adding seed URLs from a specified file to the Frontera system using the provided configuration.

```bash
python -m frontera.utils.add_seeds --config config.dbw --seeds-file seeds_es_smp.txt
```

--------------------------------

### Load and Add to Graph Manager

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Loads an existing graph from a SQLite database and adds a new site. Changes are committed by default.

```python
>>> graph = graphs.Manager(engine='sqlite:///graph.db')
>>> graph.add_site(('A', []))
```

--------------------------------

### Configure Frontera for Single Process

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-single.html

Set these parameters in your Frontera configuration to enable single-process mode. This is useful for local development and debugging.

```ini
# these two parameters are pointing Frontera that it will run locally

SPIDER_FEED_PARTITIONS = 1
SPIDER_LOG_PARTITIONS = 1


```

--------------------------------

### Inject Seed URLs

Source: https://frontera.readthedocs.io/en/latest/topics/quick-start-single.html

Use the `add_seeds` utility to inject seed URLs into the Frontera backend's queue. This is necessary if your crawling strategy requires external seed injection.

```bash
$ python -m frontera.utils.add_seeds --config [your_frontera_config] --seeds-file [path to your seeds file]

```

--------------------------------

### Accessing Settings in a Middleware

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

Demonstrates how to access Frontera settings within a custom middleware component using the FrontierManager. Ensure settings are passed correctly via class methods.

```python
from frontera.core.components import Component

class MyMiddleware(Component):

    @classmethod
    def from_manager(cls, manager):
        manager = crawler.settings
        if settings.TEST_MODE:
            print "test mode is enabled!"

```

--------------------------------

### Run Links Follower Script

Source: https://frontera.readthedocs.io/en/latest/topics/examples.html

Executes a Python script that follows all links from a given website using the Requests library. Ensure the script is in your current directory.

```python
python links_follower.py
```

--------------------------------

### Initialize Graph Manager with Content Reset

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Initializes the Graph Manager with SQLite, using the `clear_content=True` parameter to reset the database content upon creation.

```python
>>> g = graphs.Manager(engine='sqlite:///graph.db', clear_content=True)
```

--------------------------------

### Activate Scrapy Recorder Middlewares

Source: https://frontera.readthedocs.io/en/latest/topics/scrapy-recorder.html

Add the `CrawlRecorderSpiderMiddleware` and `CrawlRecorderDownloaderMiddleware` to your Scrapy project's settings to enable recording.

```python
SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.recording.CrawlRecorderDownloaderMiddleware': 1000,
})
```

--------------------------------

### Configure Scrapy Recorder Storage Options

Source: https://frontera.readthedocs.io/en/latest/topics/scrapy-recorder.html

Use `RECORDER_STORAGE_DROP_ALL_TABLES` to drop database tables or `RECORDER_STORAGE_CLEAR_CONTENT` to clear table data when resetting the Graph Manager.

```python
RECORDER_STORAGE_DROP_ALL_TABLES = True
```

```python
RECORDER_STORAGE_CLEAR_CONTENT = True
```

--------------------------------

### Add Multiple Sites to Graph Manager

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Demonstrates adding two distinct sites to the same graph manager instance. This is useful for managing multiple related or unrelated site structures within a single graph.

```python
>>> site1 = [('A1', ['B1', 'C1', 'D1'])]
>>> site2 = [('A2', ['B2', 'C2', 'D2'])]

>>> g = graphs.Manager()
>>> g.add_site(site1)
>>> g.add_site(site2)

>>> g.pages
[<1:A1*>, <2:B1>, <3:C1>, <4:D1>, <5:A2*>, <6:B2>, <7:C2>, <8:D2>]

>>> g.seeds
[<1:A1*>, <5:A2*>]
```

--------------------------------

### Kafka SSL Certificate Path Configuration

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

Provides the OS path to the directory containing SSL certificate files (ca-cert.pem, client-cert.pem, client-key.pem) for secure Kafka connections.

```python
KAFKA_CERT_PATH = '/path/to/certificates'
```

--------------------------------

### Activate Frontera Middlewares and Scheduler in Scrapy

Source: https://frontera.readthedocs.io/en/latest/topics/scrapy-integration.html

Configure Scrapy's settings to use Frontera's spider middleware, downloader middleware, and scheduler. This is essential for enabling Frontera's functionality within a Scrapy project.

```python
SPIDER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 1000,
})

DOWNLOADER_MIDDLEWARES.update({
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 1000,
})

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'
```

--------------------------------

### Configuring max_next_pages for Frontier Tester in Python

Source: https://frontera.readthedocs.io/en/latest/topics/frontier-tester.html

Sets the maximum number of pages returned per `get_next_requests` call when initializing the FrontierTester.

```python
>>> tester = FrontierTester(frontier, graph, max_next_pages=10)
```

--------------------------------

### Default Middlewares Configuration

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

This snippet shows the default list of middlewares enabled in Frontera. Middlewares can be customized to extend frontier functionality.

```python
[
    'frontera.contrib.middlewares.fingerprint.UrlFingerprintMiddleware',
]
```

--------------------------------

### Kafka SSL Enable Configuration

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

A boolean flag to enable or disable SSL connections for the Kafka client. Set to True to activate SSL.

```python
KAFKA_ENABLE_SSL = True
```

--------------------------------

### Common Frontera Settings

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Defines shared settings for Frontera components, including middleware extensions and Kafka configuration. Use this as a base for other Frontera configuration modules.

```python
from __future__ import absolute_import
from frontera.settings.default_settings import MIDDLEWARES
MAX_NEXT_REQUESTS = 512
SPIDER_FEED_PARTITIONS = 2 # number of spider processes
SPIDER_LOG_PARTITIONS = 2 # worker instances
MIDDLEWARES.extend([
    'frontera.contrib.middlewares.domain.DomainMiddleware',
    'frontera.contrib.middlewares.fingerprint.DomainFingerprintMiddleware'
])

QUEUE_HOSTNAME_PARTITIONING = True
KAFKA_LOCATION = 'localhost:9092' # your Kafka broker host:port
SCORING_TOPIC = 'frontier-scoring'
URL_FINGERPRINT_FUNCTION='frontera.utils.fingerprint.hostname_local_fingerprint'

```

--------------------------------

### Load Graph and Access Pages

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Loads a graph from a SQLite database and accesses its pages. This demonstrates loading previously saved graph data.

```python
>>> another_graph = graphs.Manager(engine='sqlite:///graph.db')
>>> another_graph.pages
[<1:A1*>]
```

--------------------------------

### SQLAlchemy Backend Models Configuration

Source: https://frontera.readthedocs.io/en/latest/topics/frontera-settings.html

Defines the mapping for SQLAlchemy models used by Frontera backends. This is primarily for customization purposes.

```python
{
    'MetadataModel': 'frontera.contrib.backends.sqlalchemy.models.MetadataModel',
    'StateModel': 'frontera.contrib.backends.sqlalchemy.models.StateModel',
    'QueueModel': 'frontera.contrib.backends.sqlalchemy.models.QueueModel'
}
```

--------------------------------

### Render Graph to PNG

Source: https://frontera.readthedocs.io/en/latest/topics/graph-manager.html

Renders the current graph to a PNG file using pydot and Graphviz. This allows for visual inspection of the graph structure.

```python
>>> g.render(filename='graph.png', label='A simple Graph')
```

--------------------------------

### Spider Frontera Settings

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Configuration for spiders, using a MessageBusBackend and setting Kafka timeouts. It imports settings from the common configuration and disables local mode.

```python
from __future__ import absolute_import
from .common import *

BACKEND = 'frontera.contrib.backends.remote.messagebus.MessageBusBackend'
KAFKA_GET_TIMEOUT = 0.5
LOCAL_MODE = False  # by default Frontera is prepared for single process mode

```

--------------------------------

### Using MessageBusBackend

Source: https://frontera.readthedocs.io/en/latest/topics/message_bus.html

Communicate using the message bus with MessageBusBackend. This class is part of the remote backend implementation.

```python
from frontera.contrib.backends.remote.messagebus import MessageBusBackend

# Assuming 'manager' is an initialized object
message_bus = MessageBusBackend(manager)
```

--------------------------------

### Scrapy Settings for Frontera Integration

Source: https://frontera.readthedocs.io/en/latest/topics/cluster-setup.html

Integrates Frontera with Scrapy by setting the Frontera settings module, scheduler, and spider/downloader middlewares. This configuration should be placed in the Scrapy project's settings module.

```python
FRONTERA_SETTINGS = ''  # module path to your Frontera spider config module

SCHEDULER = 'frontera.contrib.scrapy.schedulers.frontier.FronteraScheduler'

SPIDER_MIDDLEWARES = {
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerSpiderMiddleware': 999,
    'frontera.contrib.scrapy.middlewares.seeds.file.FileSeedLoader': 1,
}
DOWNLOADER_MIDDLEWARES = {
    'frontera.contrib.scrapy.middlewares.schedulers.SchedulerDownloaderMiddleware': 999,
}

```
Tokens:
Raw