Spidermon

https://spidermon.readthedocs.io/en/latest/
Admin
Spidermon is a framework that monitors Scrapy spiders by validating output data, triggering alerts...
Tokens:31,705
Snippets:304
Trust Score:8.3
Update:7 months ago
Show doc for...
Tokens:
Raw
### Install Spidermon with monitoring dependencies

Source: https://spidermon.readthedocs.io/en/latest/_sources/installation.rst.txt

Installs the Spidermon library along with the optional monitoring dependencies required for notification features. This command uses pip to fetch the package from the Python Package Index.

```bash
pip install "spidermon[monitoring]"
```

--------------------------------

### Initialize Scrapy Project and Spider

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Commands to create a new Scrapy project and generate a spider boilerplate for scraping quotes.toscrape.com.

```console
$ scrapy startproject tutorial
$ cd tutorial
$ scrapy genspider quotes quotes.toscrape.com
```

--------------------------------

### Configure Spidermon in Scrapy Settings

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Enables the Spidermon extension and registers monitor suites within the Scrapy settings.py file.

```python
SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}

SPIDERMON_SPIDER_CLOSE_MONITORS = ("tutorial.monitors.SpiderCloseMonitorSuite",)
```

--------------------------------

### Create Scrapy Project and Spider

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Commands to create a new Scrapy project and generate a spider for scraping quotes. This sets up the basic structure for the tutorial.

```bash
$ scrapy startproject tutorial
$ cd tutorial
$ scrapy genspider quotes quotes.toscrape.com

```

--------------------------------

### Define Slack Notification Actions

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Configures a monitor suite to trigger a Slack message upon failure and sets the required credentials in the settings file.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.slack.notifiers import SendSlackMessageSpiderFinished

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendSlackMessageSpiderFinished,
    ]
```

```python
# tutorial/settings.py
SPIDERMON_SLACK_SENDER_TOKEN = "<SLACK_SENDER_TOKEN>"
SPIDERMON_SLACK_SENDER_NAME = "<SLACK_SENDER_NAME>"
SPIDERMON_SLACK_RECIPIENTS = ["@yourself", "#yourprojectchannel"]
```

--------------------------------

### Example of Scraped Item with Validation Errors

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

This example demonstrates the structure of a scraped item when SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS is set to True and validation errors occur. The '_validation' key will contain a dictionary where keys are field names and values are lists of error messages.

```json
{
    "_validation": {"author_url": ["Invalid URL"]},
     "author": "Mark Twain",
     "author_url": "not_a_valid_url",
     "quote": "Never tell the truth to people who are not worthy of it.",
     "tags": ["truth"]
}

```

--------------------------------

### Configure Item Validation Pipeline

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Registers the ItemValidationPipeline in settings and points to the JSON schema file for validation rules.

```python
ITEM_PIPELINES = {
    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800,
}

SPIDERMON_VALIDATION_SCHEMAS = ("./schemas/quote_item.json",)
```

--------------------------------

### Create Spidermon Monitor Suite

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Defines a custom monitor class to validate the number of scraped items and groups it into a suite for execution upon spider closure.

```python
from spidermon import Monitor, MonitorSuite, monitors

@monitors.name("Item count")
class ItemCountMonitor(Monitor):

    @monitors.name("Minimum number of items")
    def test_minimum_number_of_items(self):
        item_extracted = getattr(self.data.stats, "item_scraped_count", 0)
        minimum_threshold = 10

        msg = "Extracted less than {} items".format(minimum_threshold)
        self.assertTrue(item_extracted >= minimum_threshold, msg=msg)

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]
```

--------------------------------

### Example of Item with Validation Errors (JSON)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This JSON example demonstrates the structure of an item after Spidermon has added the '_validation' field due to schema mismatches. The '_validation' field contains a dictionary where keys are field names and values are lists of validation error messages.

```json
{
    "_validation": {"author_url": ["Invalid URL"]},
     "author": "Mark Twain",
     "author_url": "not_a_valid_url",
     "quote": "Never tell the truth to people who are not worthy of it.",
     "tags": ["truth"]
}
```

--------------------------------

### Configure Spidermon Monitor Suite in Scrapy

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Registers a monitor suite to be executed when a spider closes. This setting is defined in the Scrapy settings.py file.

```python
SPIDERMON_SPIDER_CLOSE_MONITORS = ("tutorial.monitors.SpiderCloseMonitorSuite",)
```

--------------------------------

### Install Slack Client Dependency

Source: https://spidermon.readthedocs.io/en/latest/_sources/actions/slack-action.rst.txt

Installs the required slackclient library version 2.0 or higher to enable Slack integration.

```shell
$ pip install "slackclient>=2.0"
```

--------------------------------

### Implement Item Validation Monitor

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Creates a custom monitor that checks spider statistics for validation errors and fails the suite if errors are detected.

```python
from spidermon.contrib.monitors.mixins import StatsMonitorMixin

@monitors.name("Item validation")
class ItemValidationMonitor(Monitor, StatsMonitorMixin):

    @monitors.name("No item validation errors")
    def test_no_item_validation_errors(self):
        validation_errors = getattr(self.stats, "spidermon/validation/fields/errors", 0)
        self.assertEqual(
            validation_errors,
            0,
            msg="Found validation errors in {} fields".format(validation_errors),
        )
```

--------------------------------

### Enable Spidermon in Scrapy Settings

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Configuration snippet for Scrapy's settings.py file to enable Spidermon and register its extension. This is necessary for Spidermon to function within the Scrapy project.

```python
SPIDERMON_ENABLED = True

EXTENSIONS = {
    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
}


```

--------------------------------

### Define Telegram Notification Actions

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Configures a monitor suite to trigger a Telegram message upon failure and sets the required bot token and recipient IDs in the settings file.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.telegram.notifiers import (
    SendTelegramMessageSpiderFinished,
)

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendTelegramMessageSpiderFinished,
    ]
```

```python
# tutorial/settings.py
SPIDERMON_TELEGRAM_SENDER_TOKEN = "<TELEGRAM_SENDER_TOKEN>"
SPIDERMON_TELEGRAM_RECIPIENTS = ["chatid", "groupid", "@channelname"]
```

--------------------------------

### Example Field Coverage Statistics Output

Source: https://spidermon.readthedocs.io/en/latest/_sources/settings.rst.txt

Represents the generated statistics dictionary when SPIDERMON_ADD_FIELD_COVERAGE is enabled. It maps field paths to their respective scrape counts and coverage ratios.

```python
{
    "spidermon_item_scraped_count/dict": 2,
    "spidermon_item_scraped_count/dict/field_1": 2,
    "spidermon_item_scraped_count/dict/field_1/nested_field_1_1": 2,
    "spidermon_item_scraped_count/dict/field_1/nested_field_1_2": 1,
    "spidermon_item_scraped_count/dict/field_2": 1,
    "spidermon_field_coverage/dict/field_1": 1,
    "spidermon_field_coverage/dict/field_1/nested_field_1_1": 1,
    "spidermon_field_coverage/dict/field_1/nested_field_1_2": 0.5,
    "spidermon_item_scraped_count/dict/field_2": 0.5
}
```

--------------------------------

### Define Scrapy Spider Logic

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

A basic Scrapy spider implementation that extracts quotes, authors, and tags from the target website and follows pagination links.

```python
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["http://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css(".quote"):
            yield {
                "quote": quote.css(".text::text").get(),
                "author": quote.css(".author::text").get(),
                "author_url": response.urljoin(
                    quote.css(".author a::attr(href)").get()
                ),
                "tags": quote.css(".tag *::text").getall(),
            }

        yield scrapy.Request(
            response.urljoin(response.css(".next a::attr(href)").get())
        )
```

--------------------------------

### Configure Discord Notifications for Spidermon

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Integrates the SendDiscordMessageSpiderFinished action into a MonitorSuite and configures the required webhook URL in settings. This allows automated alerts to be sent to a Discord channel when a spider finishes or fails.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.discord.notifiers import SendDiscordMessageSpiderFinished

class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendDiscordMessageSpiderFinished,
    ]
```

```python
# tutorial/settings.py
SPIDERMON_DISCORD_WEBHOOK_URL = "<DISCORD_WEBHOOK_URL>"
```

--------------------------------

### Configure Telegram Settings in settings.py (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code snippet demonstrates how to set up Telegram notification credentials in your `settings.py` file for Spidermon. It requires a sender token and a list of recipients (chat IDs, group IDs, or channel names).

```python
# tutorial/settings.py
(...)
SPIDERMON_TELEGRAM_SENDER_TOKEN = "<TELEGRAM_SENDER_TOKEN>"
SPIDERMON_TELEGRAM_RECIPIENTS = ["chatid", "groupid", "@channelname"]
```

--------------------------------

### Configure Slack Settings in settings.py (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code snippet shows how to configure Slack credentials within your `settings.py` file for Spidermon. You need to provide a sender token, sender name, and specify recipients for the Slack notifications.

```python
# tutorial/settings.py
(...)
SPIDERMON_SLACK_SENDER_TOKEN = "<SLACK_SENDER_TOKEN>"
SPIDERMON_SLACK_SENDER_NAME = "<SLACK_SENDER_NAME>"
SPIDERMON_SLACK_RECIPIENTS = ["@yourself", "#yourprojectchannel"]
```

--------------------------------

### Declare Multiple Expression Monitors with Tests

Source: https://spidermon.readthedocs.io/en/latest/_sources/expression-monitors.rst.txt

This example demonstrates declaring multiple expression monitors, each with its own set of tests. It includes monitors for checking spider name, crawler existence, and finish reason using defined expressions.

```python
[
    {
        "name": "DumbChecksMonitor",
        "description": "My expression monitor",
        "tests": [
            {
                "name": "test_spider_name",
                "description": "Test spider name",
                "expression": "spider.name == 'httpbin'",
            },
            {
                "name": "test_crawler_exists",
                "description": "Test Crawler exists",
                "expression": "crawler is not None"
            }
        ],
    },
    {
        "name": "FinishedOkMonitor",
        "description": "My expression monitor 2",
        "tests": [
            {
                "name": "test_finish_reason",
                "description": "Test finish reason",
                "expression": 'stats["finish_reason"] == "finished"',
            }
        ],
    }
]
```

--------------------------------

### Spidermon Monitor for Item Count

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Python code defining a Spidermon monitor suite and a monitor class. The `ItemCountMonitor` checks if the spider extracted at least 10 items, and `SpiderCloseMonitorSuite` groups this monitor.

```python
# tutorial/monitors.py
from spidermon import Monitor, MonitorSuite, monitors


@monitors.name("Item count")
class ItemCountMonitor(Monitor):

    @monitors.name("Minimum number of items")
    def test_minimum_number_of_items(self):
        item_extracted = getattr(self.data.stats, "item_scraped_count", 0)
        minimum_threshold = 10

        msg = "Extracted less than {} items".format(minimum_threshold)
        self.assertTrue(item_extracted >= minimum_threshold, msg=msg)


class SpiderCloseMonitorSuite(MonitorSuite):

    monitors = [
        ItemCountMonitor,
    ]


```

--------------------------------

### Configure Discord Webhook URL in Scrapy Settings

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Sets the Discord webhook URL in the project settings to enable notifications for monitor failures. Requires the SPIDERMON_DISCORD_WEBHOOK_URL variable.

```python
SPIDERMON_DISCORD_WEBHOOK_URL = "<DISCORD_WEBHOOK_URL>"
```

--------------------------------

### Configure list field coverage depth in Spidermon

Source: https://spidermon.readthedocs.io/en/latest/settings.html

Examples showing how the SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS setting affects the output statistics of scraped items. The examples demonstrate the difference in output granularity when the setting is configured to 0, 1, or 2.

```json
[
    {
        "field_1": null,
        "field_2": [{"nested_field1": "value", "nested_field2": "value"}],
    },
    {
        "field_1": "value",
        "field_2": [
            {"nested_field2": "value", "nested_field3": {"deeper_field1": "value"}}
        ],
    },
    {
        "field_1": "value",
        "field_2": [
            {
                "nested_field2": "value",
                "nested_field4": [
                    {"deeper_field41": "value"},
                    {"deeper_field41": "value"},
                ],
            }
        ],
    },
]
```

```json
{
    "item_scraped_count": 3,
    "spidermon_item_scraped_count": 3,
    "spidermon_item_scraped_count/dict": 3,
    "spidermon_item_scraped_count/dict/field_1": 3,
    "spidermon_item_scraped_count/dict/field_2": 3
}
```

```json
{
    "item_scraped_count": 3,
    "spidermon_item_scraped_count": 3,
    "spidermon_item_scraped_count/dict": 3,
    "spidermon_item_scraped_count/dict/field_1": 3,
    "spidermon_item_scraped_count/dict/field_2": 3,
    "spidermon_item_scraped_count/dict/field_2/_items": 3,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field1": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field2": 3,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field3": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field3/deeper_field1": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field4": 1
}
```

```json
{
    "item_scraped_count": 3,
    "spidermon_item_scraped_count": 3,
    "spidermon_item_scraped_count/dict": 3,
    "spidermon_item_scraped_count/dict/field_1": 3,
    "spidermon_item_scraped_count/dict/field_2": 3,
    "spidermon_item_scraped_count/dict/field_2/_items": 3,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field1": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field2": 3,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field3": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field3/deeper_field1": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field4": 1,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field4/_items": 2,
    "spidermon_item_scraped_count/dict/field_2/_items/nested_field4/_items/deeper_field41": 2
}
```

--------------------------------

### Define Scrapy Item and Spider Implementation

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

Defines a Scrapy Item class to structure scraped data and updates the spider to yield these items. This ensures consistent data extraction for validation.

```python
# tutorial/items.py
import scrapy

class QuoteItem(scrapy.Item):
    quote = scrapy.Field()
    author = scrapy.Field()
    author_url = scrapy.Field()
    tags = scrapy.Field()

# tutorial/spiders/quotes.py
import scrapy
from tutorial.items import QuoteItem

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["http://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css(".quote"):
            item = QuoteItem(
                quote=quote.css(".text::text").get(),
                author=quote.css(".author::text").get(),
                author_url=response.urljoin(quote.css(".author a::attr(href)").get()),
                tags=quote.css(".tag *::text").getall(),
            )
            yield item

        yield scrapy.Request(
            response.urljoin(response.css(".next a::attr(href)").get())
        )
```

--------------------------------

### Configure Validation Settings

Source: https://spidermon.readthedocs.io/en/latest/_sources/item-validation.rst.txt

Examples of configuring validation error field paths and schema locations within Scrapy settings.

```python
# settings.py
SPIDERMON_VALIDATION_ERRORS_FIELD = "top_level.second_level._validation"

# settings.py
SPIDERMON_VALIDATION_SCHEMAS = [
    "/path/to/schema.json",
    "s3://bucket/schema.json",
    "https://example.com/schema.json",
]

# settings.py
from tutorial.items import DummyItem, OtherItem

SPIDERMON_VALIDATION_SCHEMAS = {
    DummyItem: "/path/to/dummyitem_schema.json",
    OtherItem: "/path/to/otheritem_schema.json",
}
```

--------------------------------

### Validate Field Errors and Missing Fields in SpiderMon

Source: https://spidermon.readthedocs.io/en/latest/_sources/item-validation.rst.txt

Examples demonstrating how to check for missing required fields using a list of names and how to validate error percentages for specific fields. Note that percentage methods require a ratio input (e.g., 0.15 for 15%).

```python
# checks that each of field2 and field3 is missing in no more than 10 items
self.check_missing_required_fields(field_names=["field2", "field3"], allowed_count=10)

# checks that field2 has errors in no more than 15% of items
self.check_field_errors_percent(field_name="field2", allowed_percent=0.15)

# checks that no errors is present in any fields
self.check_field_errors_percent()
```

--------------------------------

### Example JSON Schema for Item Validation

Source: https://spidermon.readthedocs.io/en/latest/item-validation.html

An example of a JSON Schema used to define the structure and types of fields for an item. This schema specifies required fields, data types, and patterns for validation.

```json
{
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "properties": {
    "quote": {
      "type": "string"
    },
    "author": {
      "type": "string"
    },
    "author_url": {
      "type": "string",
      "pattern": ""
    },
    "tags": {
      "type": "array",
      "items": {
        "type":"string"
      }
    }
  },
  "required": [
    "quote",
    "author",
    "author_url"
  ]
}

```

--------------------------------

### Configure SendSmtpEmail Action in Spidermon

Source: https://spidermon.readthedocs.io/en/latest/actions/email-action.html

Example of configuring the SendSmtpEmail action within a Spidermon MonitorSuite. This action sends email reports using SMTP when monitors fail.

```python
from spidermon.contrib.actions.email.smtp import SendSmtpEmail


class DummyMonitorSuite(MonitorSuite):
    monitors = [
        DummyMonitor,
    ]

    monitors_failed_actions = [
        SendSmtpEmail,
    ]

```

--------------------------------

### Enable Basic Spider Close Monitor Suite (Scrapy)

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

This configuration enables the default SpiderCloseMonitorSuite in Spidermon for Scrapy projects. This suite automatically includes several common monitors that run after the spider has finished execution. Ensure Spidermon is installed and configured before enabling this setting.

```python
SPIDERMON_SPIDER_CLOSE_MONITORS = (
    'spidermon.contrib.scrapy.monitors.SpiderCloseMonitorSuite',
)
```

--------------------------------

### Define Item Field Coverage Structure

Source: https://spidermon.readthedocs.io/en/latest/_sources/settings.rst.txt

Example of nested item structures used to calculate field coverage and scrape counts. This demonstrates how Spidermon processes complex dictionaries to generate statistical metrics.

```python
[
    {
        "field_1": {
            "nested_field_1_1": "value",
            "nested_field_1_2": "value",
        },
    },
    {
        "field_1": {
            "nested_field_1_1": "value",
        },
        "field_2": "value",
    },
]
```

--------------------------------

### Create a Periodic Monitor Suite

Source: https://spidermon.readthedocs.io/en/latest/_sources/monitors.rst.txt

This Python code demonstrates how to create a Spidermon Monitor Suite for periodic execution. It includes a custom monitor to check the number of errors and specifies the custom action to be taken if the monitor fails. The monitor suite is named and configured to use a list of monitors and a list of actions.

```python
from tutorial.actions import CloseSpiderAction
from spidermon.core.monitors import Monitor, MonitorSuite, StatsMonitorMixin
from spidermon import monitors


@monitors.name("Periodic job stats monitor")
class PeriodicJobStatsMonitor(Monitor, StatsMonitorMixin):

    @monitors.name("Maximum number of errors reached")
    def test_number_of_errors(self):
        accepted_num_errors = 20
        num_errors = self.data.stats.get("log_count/ERROR", 0)

        msg = "The job has exceeded the maximum number of errors"
        self.assertLessEqual(num_errors, accepted_num_errors, msg=msg)


class PeriodicMonitorSuite(MonitorSuite):
    monitors = [PeriodicJobStatsMonitor]
    monitors_failed_actions = [CloseSpiderAction]
```

--------------------------------

### Configure Spidermon Monitor Suites

Source: https://spidermon.readthedocs.io/en/latest/_sources/monitors.rst.txt

This Python code demonstrates how to define a `MonitorSuite` in Spidermon, which groups monitors and specifies actions to be executed at different stages of a spider's lifecycle. It also shows how to configure these suites in `settings.py` for spider open and close events.

```python
# monitors.py
from spidermon.core.suites import MonitorSuite


# Monitor definition above...
class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        # (your monitors)
    ]

    monitors_finished_actions = [
        # actions to execute when suite finishes its execution
    ]

    monitors_failed_actions = [
        # actions to execute when suite finishes its execution with a failed monitor
    ]
```

```python
# settings.py
SPIDERMON_SPIDER_OPEN_MONITORS = (
    # list of monitor suites to be executed when the spider starts
)

SPIDERMON_SPIDER_CLOSE_MONITORS = (
    # list of monitor suites to be executed when the spider finishes
)
```

--------------------------------

### Create a Custom Stat Monitor with BaseStatMonitor

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

Demonstrates how to inherit from BaseStatMonitor to validate a specific job statistic against a project setting threshold. It defines the target statistic, the setting key, and the comparison operator.

```python
class MyCustomStatMonitor(BaseStatMonitor):
    stat_name = "numerical_job_statistic"
    threshold_setting = "CUSTOM_STAT_THRESHOLD"
    assert_type = ">="
```

--------------------------------

### Configure Job Tags in Monitor Suite

Source: https://spidermon.readthedocs.io/en/latest/actions/job-tags-action.html

Demonstrates how to integrate AddJobTags and RemoveJobTags into a MonitorSuite to manage job tags based on whether the monitors pass or fail.

```python
# monitors.py
from spidermon.contrib.actions.jobs.tags import AddJobTags, RemoveJobTags

class DummyMonitorSuite(MonitorSuite):
    monitors = [
        DummyMonitor,
    ]

    monitors_passed_actions = [
        RemoveJobTags,
    ]

    monitors_failed_actions = [
        AddJobTags,
    ]
```

```python
# settings.py
SPIDERMON_JOB_TAGS_TO_ADD = [
    "failed",
]
SPIDERMON_JOB_TAGS_TO_REMOVE = [
    "running",
]
```

--------------------------------

### Create File Report Action with Jinja2 Template (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/actions/file-report-action.rst.txt

This Python code demonstrates how to configure the `CreateFileReport` action in Spidermon. It shows how to define monitors, specify actions to run after monitors finish, and set up report generation settings like the template, context, and filename.

```python
# monitors.py
from spidermon.contrib.actions.reports.files import CreateFileReport


class DummyMonitorSuite(MonitorSuite):
    monitors = [
        DummyMonitor,
    ]

    monitors_finished_actions = [
        CreateFileReport,
    ]

```

```python
# settings.py
SPIDERMON_REPORT_TEMPLATE = "reports/email/monitors/result.jinja"
SPIDERMON_REPORT_CONTEXT = {"report_title": "Spidermon File Report"}
SPIDERMON_REPORT_FILENAME = "my_report.html"

```

--------------------------------

### Implement Custom Job Tag Actions

Source: https://spidermon.readthedocs.io/en/latest/actions/job-tags-action.html

Shows how to create custom action classes that inherit from AddJobTags to use specific settings for different monitoring results.

```python
# monitors.py
from spidermon.contrib.actions.jobs.tags import AddJobTags

class AddJobTagsPassed(AddJobTags):
    tag_settings = "TAG_TO_ADD_WHEN_PASS"

class AddJobTagsFailed(AddJobTags):
    tag_settings = "TAG_TO_ADD_WHEN_FAIL"

class DummyMonitorSuite(MonitorSuite):
    monitors = [
        DummyMonitor,
    ]

    monitors_passed_actions = [
        AddJobTagsPassed,
    ]

    monitors_failed_actions = [
        AddJobTagsFailed,
    ]
```

```python
# settings.py
TAG_TO_ADD_WHEN_PASS = [
    "passed",
]
TAG_TO_ADD_WHEN_FAIL = [
    "failed",
]
```

--------------------------------

### Define a custom Monitor class

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

Demonstrates how to create a monitor by inheriting from spidermon.Monitor. It uses unittest-style assertions to validate spider statistics, such as checking if a minimum number of items were scraped.

```python
from spidermon import Monitor, monitors

@monitors.name("Item count")
class ItemCountMonitor(Monitor):

    @monitors.name("Minimum items extracted")
    def test_minimum_number_of_items_extracted(self):
        minimum_threshold = 100
        item_extracted = getattr(self.data.stats, "item_scraped_count", 0)
        self.assertFalse(
            item_extracted < minimum_threshold,
            msg="Extracted less than {} items".format(minimum_threshold),
        )
```

--------------------------------

### Implement Dynamic Thresholds in Monitors

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

Shows how to override the get_threshold method to calculate a dynamic threshold based on other job statistics. This allows for complex validation logic, such as percentage-based thresholds.

```python
class MyCustomStatMonitor(BaseStatMonitor):
    stat_name = "log_count/ERROR"
    assert_type = "<"

    def get_threshold(self):
        item_scraped_count = self.stats.get("item_scraped_count")
        return item_scraped_count * 0.01
```

--------------------------------

### Scrapy Spider Code for Quotes

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

A Python script defining a Scrapy spider that scrapes quotes, authors, author URLs, and tags from quotes.toscrape.com. It also handles pagination by following the 'next' link.

```python
# tutorial/spiders/quotes.py
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = ["http://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css(".quote"):
            yield {
                "quote": quote.css(".text::text").get(),
                "author": quote.css(".author::text").get(),
                "author_url": response.urljoin(
                    quote.css(".author a::attr(href)").get()
                ),
                "tags": quote.css(".tag *::text").getall(),
            }

        yield scrapy.Request(
            response.urljoin(response.css(".next a::attr(href)").get())
        )

```

--------------------------------

### Configure Telegram Notification Action in Spidermon (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code configures Spidermon to send a Telegram message when a monitor fails. It involves importing `SendTelegramMessageSpiderFinished` and including it in the `monitors_failed_actions` list.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.telegram.notifiers import (
    SendTelegramMessageSpiderFinished,
)

# (...your monitors code...)


class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendTelegramMessageSpiderFinished,
    ]
```

--------------------------------

### Configure Discord Notification Action in Spidermon (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code configures Spidermon to send a Discord message when a monitor fails. It requires importing `SendDiscordMessageSpiderFinished` and adding it to the `monitors_failed_actions` list in your monitor suite.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.discord.notifiers import SendDiscordMessageSpiderFinished

# (...your monitors code...)


class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendDiscordMessageSpiderFinished,
    ]
```

--------------------------------

### Periodic Monitor Configuration

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

Setting up monitors that run at specific intervals during the spider's execution lifecycle.

```APIDOC
## POST /settings/SPIDERMON_PERIODIC_MONITORS

### Description
Configures a monitor suite to run periodically at a specified time interval (in seconds) while the spider is running.

### Method
POST

### Request Body
- **SPIDERMON_PERIODIC_MONITORS** (dict) - Required - A dictionary mapping the MonitorSuite class path to the execution interval in seconds.

### Request Example
{
  "SPIDERMON_PERIODIC_MONITORS": {
    "tutorial.monitors.PeriodicMonitorSuite": 60
  }
}
```

--------------------------------

### Implement Item Validation with JSON Schema

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

Defines Scrapy items, configures the validation pipeline, and creates a JSON schema to enforce data structure. This ensures that scraped items meet specific formatting requirements before processing.

```python
# tutorial/items.py
import scrapy

class QuoteItem(scrapy.Item):
    quote = scrapy.Field()
    author = scrapy.Field()
    author_url = scrapy.Field()
    tags = scrapy.Field()
```

```json
{
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "properties": {
    "quote": { "type": "string" },
    "author": { "type": "string" },
    "author_url": { "type": "string", "pattern": "" },
    "tags": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["quote", "author", "author_url"]
}
```

```python
# tutorial/settings.py
ITEM_PIPELINES = {
    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800,
}

SPIDERMON_VALIDATION_SCHEMAS = ("./schemas/quote_item.json",)
```

--------------------------------

### Configure Slack Notification Action in Spidermon (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code configures Spidermon to send a Slack message when a monitor fails. It requires importing the `SendSlackMessageSpiderFinished` action and adding it to the `monitors_failed_actions` list in your monitor suite.

```python
# tutorial/monitors.py
from spidermon.contrib.actions.slack.notifiers import SendSlackMessageSpiderFinished

# (...your monitors code...)


class SpiderCloseMonitorSuite(MonitorSuite):
    monitors = [
        ItemCountMonitor,
    ]

    monitors_failed_actions = [
        SendSlackMessageSpiderFinished,
    ]
```

--------------------------------

### Spidermon Monitor Failure Log Output

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This console output shows the logs generated by Spidermon when a monitor condition fails. It includes details about the failed monitor, the assertion error, and a summary of monitor and action statuses.

```console
$ scrapy crawl quotes
(...)
INFO: [Spidermon] -------------------- MONITORS --------------------
INFO: [Spidermon] Item count/Minimum number of items... FAIL
INFO: [Spidermon] --------------------------------------------------
ERROR: [Spidermon]
====================================================================
FAIL: Item count/Minimum number of items
--------------------------------------------------------------------
Traceback (most recent call last):
  File "/tutorial/monitors.py",
    line 17, in test_minimum_number_of_items
    item_extracted >= minimum_threshold, msg=msg
AssertionError: False is not true : Extracted less than 10 items
INFO: [Spidermon] 1 monitor in 0.001s
INFO: [Spidermon] FAILED (failures=1)
INFO: [Spidermon] ---------------- FINISHED ACTIONS ----------------
INFO: [Spidermon] --------------------------------------------------
INFO: [Spidermon] 0 actions in 0.000s
INFO: [Spidermon] OK
INFO: [Spidermon] ----------------- PASSED ACTIONS -----------------
INFO: [Spidermon] --------------------------------------------------
INFO: [Spidermon] 0 actions in 0.000s
INFO: [Spidermon] OK
INFO: [Spidermon] ----------------- FAILED ACTIONS -----------------
INFO: [Spidermon] --------------------------------------------------
INFO: [Spidermon] 0 actions in 0.000s
INFO: [Spidermon] OK
```

--------------------------------

### Configure Spidermon to Add Validation Errors to Items

Source: https://spidermon.readthedocs.io/en/latest/getting-started.html

This setting in Spidermon's settings.py determines if validation errors found during scraping are appended to the scraped item. This is useful for debugging and understanding why an item might be failing validation. It requires no external dependencies beyond Spidermon itself.

```python
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True
```

--------------------------------

### Configure Spidermon to Add Validation Errors to Items (Python)

Source: https://spidermon.readthedocs.io/en/latest/_sources/getting-started.rst.txt

This Python code snippet shows how to enable the SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS setting in Spidermon's settings.py file. When set to True, Spidermon will add a '_validation' field to items that fail schema validation, indicating the specific validation errors.

```python
# tutorial/settings.py
SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True
```

--------------------------------

### Configure Monitor Behavior for Missing Stats

Source: https://spidermon.readthedocs.io/en/latest/monitors.html

Illustrates how to use the fail_if_stat_missing attribute to prevent a monitor from failing when a specific statistic is absent from the job data.

```python
class MyCustomStatMonitor(BaseStatMonitor):
    stat_name = "numerical_job_statistic"
    threshold_setting = "CUSTOM_STAT_THRESHOLD"
    assert_type = ">="
    fail_if_stat_missing = False
```

--------------------------------

### Configure Slack Credentials in Spidermon

Source: https://spidermon.readthedocs.io/en/latest/_sources/actions/slack-action.rst.txt

Defines the necessary Slack authentication tokens, bot name, and recipient channels within the project's settings.py file.

```python
# settings.py
SPIDERMON_SLACK_SENDER_TOKEN = "<SLACK_SENDER_TOKEN>"
SPIDERMON_SLACK_SENDER_NAME = "<SLACK_SENDER_NAME>"
SPIDERMON_SLACK_RECIPIENTS = ["@yourself", "#yourprojectchannel"]
```

--------------------------------

### Validation Error Output Format

Source: https://spidermon.readthedocs.io/en/latest/_sources/item-validation.rst.txt

Example of an item structure containing validation errors when SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS is enabled.

```javascript
{
    '_validation': {'author_url': ['Invalid URL']},
    'author': 'C.S. Lewis',
    'author_url': 'invalid_url',
    'quote': 'Some day you will be old enough to start reading fairy tales again.',
    'tags': ['age', 'fairytales', 'growing-up']
}
```

--------------------------------

### Configure Field Coverage with None Values

Source: https://spidermon.readthedocs.io/en/latest/_sources/settings.rst.txt

Demonstrates item data containing None values and the resulting statistics when SPIDERMON_FIELD_COVERAGE_SKIP_NONE is enabled to ignore empty fields.

```python
items = [
    {"field_1": None, "field_2": "value"},
    {"field_1": "value", "field_2": "value"},
]

stats = {
    "spidermon_item_scraped_count/dict": 2,
    "spidermon_item_scraped_count/dict/field_1": 1  # Ignored None value
}
```

--------------------------------

### Configure Sentry DSN in settings.py

Source: https://spidermon.readthedocs.io/en/latest/actions/sentry-action.html

This code snippet shows how to configure the Sentry Data Source Name (DSN), project name, and environment type in your Django settings.py file for the Spidermon Sentry action. Ensure you replace placeholder values with your actual Sentry credentials.

```python
# settings.py
SPIDERMON_SENTRY_DSN = "<SENTRY_DSN_URL>"
SPIDERMON_SENTRY_PROJECT_NAME = "<PROJECT_NAME>"
SPIDERMON_SENTRY_ENVIRONMENT_TYPE = "<ENVIRONMENT_TYPE>"

```

--------------------------------

### Configure Scraped Item Statistics Depth

Source: https://spidermon.readthedocs.io/en/latest/_sources/settings.rst.txt

Demonstrates how different integer settings affect the granularity and depth of the item statistics reported by Spidermon.

```python
"spidermon_item_scraped_count/dict": 2
"spidermon_item_scraped_count/dict/field1": 2
"spidermon_item_scraped_count/dict/field2": 2
"spidermon_item_scraped_count/dict/field3": 1
"spidermon_item_scraped_count/dict/field4": 1
```
Tokens:
Raw