OCDS Kingfisher Collect

https://kingfisher-collect.readthedocs.io/en/latest/
Admin
Kingfisher Collect is a tool built on the Scrapy framework for downloading OCDS data, storing it on...
Tokens:392,906
Snippets:1,511
Trust Score:8.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Kingfisher Collect with psycopg2

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Installs Kingfisher Collect, specifically addressing potential psycopg2 errors by installing psycopg2-binary.

```bash
pip install psycopg2-binary -r requirements.txt
```

--------------------------------

### Install Kingfisher Collect

Source: https://kingfisher-collect.readthedocs.io/en/latest/local

Basic command to navigate to the Kingfisher Collect directory after installation. Assumes prerequisites like Git and Python are met.

```shell
cd
```

--------------------------------

### Install Kingfisher Collect Locally

Source: https://kingfisher-collect.readthedocs.io/en/latest/index

Instructions on how to install Kingfisher Collect on your local machine to download OCDS data.

```python
pip install kingfisher-collect
```

--------------------------------

### Install Kingfisher Collect

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Clones the Kingfisher Collect repository, changes the directory, and installs dependencies. Requires Git and Python 3.10+.

```bash
git clone https://github.com/open-contracting/kingfisher-collect.git
cd kingfisher-collect
pip install -r requirements.txt
```

--------------------------------

### Bulk Download Command Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/spiders/nepal

Example of using curl to download the HTTP headers for the bulk download URL, useful for checking server responses and availability.

```bash
curl -I https://ppip.gov.np/bulk-download/2019
```

--------------------------------

### Download a Sample of Data

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Collects only a sample of the available data by setting the 'sample' argument with a desired count.

```bash
scrapy crawl spider_name -a sample=10
```

--------------------------------

### Scrapy Item Pipeline Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/pipelines

This snippet demonstrates the basic setup for an item pipeline in Scrapy, including importing necessary modules and handling signals. It shows how to use libraries like ijson for efficient JSON parsing and flattentool for data transformation.

```python
importjson
importos
importpkgutil
importtempfile
importwarnings

importijson
importjsonpointer
fromflattentoolimport unflatten
fromflattentool.exceptionsimport FlattenToolWarning
fromscrapy.exceptionsimport DropItem, NotSupported

fromkingfisher_scrapy.itemsimport File, FileItem, PluckedItem
fromkingfisher_scrapy.utilimport transcode
```

--------------------------------

### CompressedFileSpider Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/base_spiders/compressed_file_spider

Example of how to inherit from CompressedFileSpider and define a start_requests method to fetch a ZIP archive.

```python
from kingfisher_scrapy.base_spiders import CompressedFileSpider
from kingfisher_scrapy.util import components

class MySpider(CompressedFileSpider):
    name = 'my_spider'

    # CompressedFileSpider
    data_type = 'release_package'

    def start_requests(self):
        yield self.build_request('https://example.com/api/packages.zip', formatter=components(-1))
```

--------------------------------

### Contributing Guides

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/base_spiders/periodic_spider

Guides for contributing to the Kingfisher Collect project. This includes information on downloader middlewares, spider middlewares, item pipelines, extensions, utilities, and exceptions.

```en
Downloader Middlewares
Spider Middlewares
Item Pipelines
Extensions (Sentry Logging, Pluck, Files Store, Kingfisher Process API, Database Store, Item Count)
Utilities
Exceptions
```

--------------------------------

### Select processing steps

Source: https://kingfisher-collect.readthedocs.io/en/latest/kingfisher_process

Examples of selecting specific processing steps in Kingfisher Process. These steps can include structural checks and creating compiled releases.

```bash
steps=check,compile
```

```bash
steps=check
```

```bash
steps=compile
```

```bash
steps=
```

--------------------------------

### List Available Spiders

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Lists all available spiders within the Kingfisher Collect project. This helps in identifying which data sources can be crawled.

```bash
scrapy list
```

--------------------------------

### LinksSpider Implementation Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/links_spider

This example demonstrates how to inherit from the LinksSpider class to create a custom spider. It shows setting essential class attributes like `data_type` and `formatter`, and implementing the `start_requests` method to initiate the scraping process.

```python
import scrapy

from kingfisher_scrapy.base_spiders import LinksSpider

class MySpider(LinksSpider):
    name = 'my_spider'

    # SimpleSpider
    data_type = 'release_package'

    # LinksSpider
    formatter = staticmethod(parameters('page'))

    def start_requests(self):
        yield scrapy.Request('https://example.com/api/packages.json', meta={'file_name': 'page-1.json'})

```

--------------------------------

### Contributing to Kingfisher Collect

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/index_spider

Guides for contributing to the Kingfisher Collect project, covering various aspects of spider development, middlewares, pipelines, and extensions.

```en
Contributing
Base Spider Classes
Downloader Middlewares
Spider Middlewares
Item Pipelines
Extensions
Utilities
Exceptions
```

--------------------------------

### CompressedFileSpider Usage Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/compressed_file_spider

This example shows how to create a custom spider by inheriting from `CompressedFileSpider`. It sets the `data_type` to 'release_package' and defines the `start_requests` method to request a ZIP file from a given URL, using `components(-1)` as the formatter.

```python
from kingfisher_scrapy.base_spiders import CompressedFileSpider
from kingfisher_scrapy.util import components

class MySpider(CompressedFileSpider):
    name = 'my_spider'

    # CompressedFileSpider
    data_type = 'release_package'

    def start_requests(self):
        yield self.build_request('https://example.com/api/packages.zip', formatter=components(-1))

```

--------------------------------

### Kingfisher Collect Contribution Guide

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/spiders/albania_public_procurement_commission_historical

Provides a comprehensive guide for contributing to the Kingfisher Collect project. It covers various aspects of development, including base spider classes, downloader middlewares, spider middlewares, item pipelines, extensions, utilities, and exception handling.

```markdown
- [Base Spider Classes](https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/index.html)
- [Downloader Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/downloadermiddlewares.html)
- [Spider Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/spidermiddlewares.html)
- [Item Pipelines](https://kingfisher-collect.readthedocs.io/en/latest/contributing/pipelines.html)
- [Extensions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/index.html)
- [Utilities](https://kingfisher-collect.readthedocs.io/en/latest/contributing/util.html)
- [Exceptions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/exceptions.html)
```

--------------------------------

### Install Kingfisher Collect Requirements

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/scrapyd.rst

Installs Kingfisher Collect's requirements on a remote server where Scrapyd is installed. This ensures the environment is set up correctly for running Kingfisher Collect spiders.

```bash
curl -O https://raw.githubusercontent.com/open-contracting/kingfisher-collect/main/requirements.txt
pip install -r requirements.txt
```

--------------------------------

### LinksSpider Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/base_spiders/links_spider

Example of how to use the LinksSpider class to collect data from an API. It demonstrates setting the data_type, formatter, and defining start_requests.

```python
import scrapy

from kingfisher_scrapy.base_spiders import LinksSpider

class MySpider(LinksSpider):
    name = 'my_spider'

    # SimpleSpider
    data_type = 'release_package'

    # LinksSpider
    formatter = staticmethod(parameters('page'))

    def start_requests(self):
        yield scrapy.Request('https://example.com/api/packages.json', meta={'file_name': 'page-1.json'})
```

--------------------------------

### Apply Query String Filters

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Applies filters to the data collection by passing query string parameters to the spider.

```bash
scrapy crawl spider_name -a qs:parameter1=value1 -a qs:parameter2=value2
```

--------------------------------

### Crawl Data with a Specific Spider

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Initiates a data crawl using a specified spider. Replace 'spider_name' with the actual name of the spider.

```bash
scrapy crawl spider_name
```

--------------------------------

### Configure Files Store

Source: https://kingfisher-collect.readthedocs.io/en/latest/local

Example of how to configure the directory where Kingfisher Collect stores files by modifying the FILES_STORE variable in settings.py. Supports relative or absolute paths.

```python
FILES_STORE = '/home/user/path'
```

--------------------------------

### Get Crawl Start Time

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/base_spiders/base_spider

Retrieves the formatted start time of the crawl, prioritizing the `crawl_time` attribute or falling back to the crawler's stats.

```python
def get_start_time(self, date_format):
"""Return the formatted start time of the crawl."""
    date = self.crawl_time if self.crawl_time else self.crawler.stats.get_value("start_time")
    return date.strftime(date_format)
```

--------------------------------

### Scrapy Crawl Log - No Data Downloaded

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Example log output from a Scrapy crawl indicating that no data was downloaded. This message helps users identify when a crawl did not produce the expected OCDS files.

```bash
2023-01-05 09:14:43 [scrapy.core.engine] INFO: Closing spider (finished)
2023-01-05 09:14:43 [argentina_vialidad] INFO: +---------------- DATA DIRECTORY ----------------+
2023-01-05 09:14:43 [argentina_vialidad] INFO: |                                                |
2023-01-05 09:14:43 [argentina_vialidad] INFO: | Something went wrong. No data was downloaded.  |
2023-01-05 09:14:43 [argentina_vialidad] INFO: |                                                |
2023-01-05 09:14:43 [argentina_vialidad] INFO: +------------------------------------------------+
```

--------------------------------

### Scrapy Crawl Log - Data Directory

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Example log output from a Scrapy crawl indicating the successful download of OCDS files and the path to the crawl directory within the 'data' directory. This helps users locate their downloaded data.

```bash
2023-01-05 09:12:14 [scrapy.core.engine] INFO: Closing spider (finished)
2023-01-05 09:12:14 [argentina_vialidad] INFO: +-------------------------- DATA DIRECTORY --------------------------+
2023-01-05 09:12:14 [argentina_vialidad] INFO: |                                                                    |
2023-01-05 09:12:14 [argentina_vialidad] INFO: | The data is available at: data/argentina_vialidad/20230105_121201  |
2023-01-05 09:12:14 [argentina_vialidad] INFO: |                                                                    |
2023-01-05 09:12:14 [argentina_vialidad] INFO: +--------------------------------------------------------------------+
2023-01-05 09:12:14 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
   {'downloader/request_bytes': 267,
   'downloader/request_count': 1,
   'downloader/request_method_count/GET': 1,
   ...
```

--------------------------------

### ParaguayHacienda Spider Start Requests

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/spiders/paraguay_hacienda

Initiates the data collection process by making the first request to the API to get the list of process IDs. It uses `build_request` with a formatter for pagination and sets meta information for the first request.

```python
defstart_requests(self):
    # Paraguay Hacienda has a service that return all the ids that we need to get the releases packages
    # so we first iterate over this list that is paginated
    yield self.build_request(
        f"{self.list_url_prefix}1",
        formatter=parameters("page"),
        meta={
            "meta": True,
            "first": True,
        },
        # send duplicate requests when the token expired and in the continuation of requests_backlog saved.
        dont_filter=True,
    )
```

--------------------------------

### DatabaseStore Class Methods

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/index

Documentation for the DatabaseStore class, covering initialization, lifecycle methods, table management, data retrieval, formatting, execution, and table naming.

```APIDOC
DatabaseStore:
  from_crawler(crawler)
    Class method to create a DatabaseStore instance from a crawler.

  spider_opened(spider)
    Called when the spider is opened.

  spider_closed(spider, reason)
    Called when the spider is closed.

  create_table(table_name, columns)
    Creates a database table with the specified name and columns.

  yield_items_from_directory(directory)
    Yields items from all JSON files in the given directory.

  format(item)
    Formats an item for database storage.

  execute(query, params=None)
    Executes a SQL query.

  get_table_name(spider)
    Returns the database table name for a given spider.
```

--------------------------------

### DatabaseStore Extension Initialization

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/extensions/database_store

Initializes the DatabaseStore extension with the database URL and file store directory. It connects to the PostgreSQL database and sets up the necessary cursor for database operations.

```python
importos
importwarnings
fromdatetimeimport datetime

importijson
importpsycopg2.sql
fromocdskit.combineimport merge
fromocdskit.exceptionsimport MergeErrorWarning
fromscrapyimport signals
fromscrapy.exceptionsimport NotConfigured

fromkingfisher_scrapyimport util
fromkingfisher_scrapy.extensions.files_storeimport FilesStore




classDatabaseStore:
    """
    If the ``DATABASE_URL`` Scrapy setting and the ``crawl_time`` spider argument are set, the OCDS data is stored in a
    PostgreSQL database, incrementally.

    This extension stores data in the "data" column of a table named after the spider, or the ``table_name`` spider
    argument (if set). When the spider is opened, if the table doesn't exist, it is created. The spider's ``from_date``
    attribute is then set, in order of precedence, to: the ``from_date`` spider argument (unless equal to the spider's
    ``default_from_date`` class attribute); the maximum value of the ``date`` field of the stored data (if any); the
    spider's ``default_from_date`` class attribute (if set).

    When the spider is closed, this extension reads the data written by the FilesStore extension to the crawl directory
    that matches the ``crawl_time`` spider argument. If the ``compile_releases`` spider argument is set, it creates
    compiled releases, using individual releases. Then, it recreates the table, and inserts either the compiled
    releases if the ``compile_releases`` spider argument is set, the individual releases in release packages (if the
    spider returns releases), or the compiled releases in record packages (if the spider returns records).

    .. warning::

       If the ``compile_releases`` spider argument is set, spiders that return records without embedded releases are
       not supported. If it isn't set, then spiders that return records without compiled releases are not supported.

    To perform incremental updates, the OCDS data in the crawl directory must not be deleted between crawls.
    """

    def__init__(self, database_url, files_store_directory):
        self.database_url = database_url
        self.files_store_directory = files_store_directory

        self.connection = None
        self.cursor = None

```

--------------------------------

### Running a Kingfisher Collect Spider

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/index

Provides a command-line instruction for running a Kingfisher Collect spider and redirecting its output to a log file for debugging purposes.

```bash
=debug.log
```

--------------------------------

### Scrapy Log Output - No Data Downloaded Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/local

An example of Scrapy log output when no data was downloaded during a crawl. This indicates potential issues with the data source or the scraping process.

```log
2023-01-0509:14:43[scrapy.core.engine](finished)
2023-01-0509:14:43[argentina_vialidad]2023-01-0509:14:43[argentina_vialidad]||
2023-01-0509:14:43[argentina_vialidad]||
2023-01-0509:14:43[argentina_vialidad]||
2023-01-0509:14:43[argentina_vialidad]
```

--------------------------------

### Kingfisher Collect - Command-line Interface

Source: https://kingfisher-collect.readthedocs.io/en/latest/index

Details on using the command-line interface for Kingfisher Collect.

```markdown
- [Command-line interface](https://kingfisher-collect.readthedocs.io/en/latest/cli.html)
```

--------------------------------

### Scrapy Log Output Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/local

An example of the log output from a Scrapy crawl, showing request counts, bytes downloaded, and other statistics. This helps in understanding the progress and potential issues during data collection.

```log
2023-01-0509:12:14[scrapy.core.engine](finished)
2023-01-0509:12:14[argentina_vialidad]2023-01-0509:12:14[argentina_vialidad]||
2023-01-0509:12:14[argentina_vialidad]||
2023-01-0509:12:14[argentina_vialidad]||
2023-01-0509:12:14[argentina_vialidad]2023-01-0509:12:14[scrapy.statscollectors]{'downloader/request_bytes':267,
'downloader/request_count':1,
'downloader/request_method_count/GET':1,
...
```

--------------------------------

### DatabaseStore Methods

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/index

Provides documentation for various methods within the DatabaseStore class, including yielding items, formatting data, executing queries, and retrieving table names.

```APIDOC
DatabaseStore.yield_items_from_directory()
  - Yields items from a specified directory.

DatabaseStore.format()
  - Formats data for storage.

DatabaseStore.execute()
  - Executes a database query.

DatabaseStore.get_table_name()
  - Retrieves the name of the database table.
```

--------------------------------

### Accessing Data Source Information

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/index

This section guides users on how to learn about a data source's access methods. It involves reading API or bulk download documentation, navigating the API using tools like `curl`, and inspecting responses to locate OCDS data and pagination information. It also notes when to inform the data support manager about missing documentation or incorrect JSON data structure.

```bash
curl <api_endpoint>
```

--------------------------------

### Sentry Logging Initialization

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/extensions/sentry_logging

This snippet shows the basic import and initialization of the Sentry SDK within the Kingfisher Collect framework. It demonstrates how to import necessary modules and handle potential configuration errors.

```python
import sentry_sdk
from scrapy.exceptions import NotConfigured

```

--------------------------------

### Client Methods

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/kingfisher_process_api_v2

Methods for the Kingfisher Process API client, including signaling exchange readiness and resetting the client.

```APIDOC
Client.exchange_ready()
  - Signals that the exchange is ready.

Client.reset()
  - Resets the client.
```

--------------------------------

### Pluck Extension Initialization from Crawler

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/extensions/pluck

The `from_crawler` class method initializes the Pluck extension using a Scrapy crawler. It retrieves settings for the pluck directory and max bytes, connects signals, and returns an instance of the extension.

```python
@classmethod
def from_crawler(cls, crawler):
    directory = crawler.settings["KINGFISHER_PLUCK_PATH"]
    max_bytes = crawler.settings["KINGFISHER_PLUCK_MAX_BYTES"]

    extension = cls(directory=directory, max_bytes=max_bytes)
    crawler.signals.connect(extension.item_scraped, signal=signals.item_scraped)
    crawler.signals.connect(extension.spider_closed, signal=signals.spider_closed)
    if max_bytes:
        crawler.signals.connect(extension.bytes_received, signal=signals.bytes_received)

    return extension
```

--------------------------------

### Add a note to the collection

Source: https://kingfisher-collect.readthedocs.io/en/latest/kingfisher_process

Example of adding a note to the collection_note table in Kingfisher Process to track provenance.

```python
note='Started by NAME.'
```

--------------------------------

### Scrapy Download Warnings and Settings

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/logs.rst

Explanation of Scrapy warnings related to download size, data loss, and retries, along with relevant settings.

```APIDOC
WARNING: Got data loss in https:…. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests
 Issued by Scrapy if the ``Content-Length`` header doesn't match the bytes received, after which Scrapy retries the request. If you don't trust the ``Content-Length`` header, set to ``False`` either the `DOWNLOAD_FAIL_ON_DATALOSS <https://docs.scrapy.org/en/latest/topics/settings.html#download-fail-on-dataloss>`__ key of the spider's `custom_settings <https://docs.scrapy.org/en/latest/topics/settings.html#settings-per-spider>`__ attribute, or the `download_fail_on_dataloss <https://docs.scrapy.org/en/latest/topics/request-response.html#std-reqmeta-download_fail_on_dataloss>`__ key of the request's ``meta`` attribute.
WARNING: Expected response size (987654321) larger than download warn size (123456789) in request <GET https:…>.
  Issued based on the `DOWNLOAD_WARNSIZE <https://docs.scrapy.org/en/latest/topics/settings.html#download-warnsize>`__ setting, ``download_warnsize`` spider attribute or ``download_warnsize`` Request.meta key. Can be ignored.
WARNING: Received more bytes than download warn size (123456789) in request <GET https:…>.
  Issued based on the `DOWNLOAD_WARNSIZE <https://docs.scrapy.org/en/latest/topics/settings.html#download-warnsize>`__ setting, ``download_warnsize`` spider attribute or ``download_warnsize`` Request.meta key. Can be ignored.
```

--------------------------------

### Configure Files Store

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Sets a custom directory for storing collected data by modifying the FILES_STORE variable in the settings file.

```python
FILES_STORE = '/home/user/path'
```

--------------------------------

### Select Kingfisher Process Steps

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/kingfisher_process.rst

Demonstrates how to select specific processing steps in Kingfisher Process using the 'steps' spider argument. This allows control over running structural checks and creating compiled releases.

```bash
scrapy crawl spider_name -a steps=check,compile
scrapy crawl spider_name -a steps=check
 scrapy crawl spider_name -a steps=compile
 scrapy crawl spider_name -a steps=
```

--------------------------------

### Add Note to Collection

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/kingfisher_process.rst

Example of how to add a note to the collection_note table in Kingfisher Process using a Scrapy spider argument.

```bash
scrapy crawl spider_name -a note='Started by NAME.'
```

--------------------------------

### Database Store Extension

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/index

Documentation for the DatabaseStore extension, which handles storing scraped data in a database. It includes methods for initialization, spider lifecycle events, and table creation.

```APIDOC
DatabaseStore
  - Stores scraped data in a database.

  from_crawler(crawler)
    - Class method to create an instance of DatabaseStore from a Scrapy crawler.
    - Parameters:
      - crawler: The Scrapy crawler object.

  spider_opened(spider)
    - Called when a spider is opened.
    - Parameters:
      - spider: The spider object.

  spider_closed(spider)
    - Called when a spider is closed.
    - Parameters:
      - spider: The spider object.

  create_table(table_name, columns)
    - Creates a database table.
    - Parameters:
      - table_name: The name of the table to create.
      - columns: A dictionary defining the table columns and their types.
```

--------------------------------

### Contributing Guidelines

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/spiders/chile_compra_api_releases

Guidelines for contributing to the Kingfisher Collect project. This section covers various aspects of development, including middlewares, pipelines, extensions, and utilities.

```en
contributing/downloadermiddlewares.html
contributing/spidermiddlewares.html
contributing/pipelines.html
contributing/util.html
contributing/exceptions.html
```

--------------------------------

### Run a Spider

Source: https://kingfisher-collect.readthedocs.io/en/latest/local

Command to start a data crawl for a specific spider. Replace 'spider_name' with the actual name of the spider.

```shell
scrapy crawl spider_name
```

--------------------------------

### BigFileSpider Class Methods

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/big_file_spider

Provides documentation for key class methods of BigFileSpider, including the class method from_crawler for initializing the spider and the parse method for processing responses. It also mentions the resize_package attribute.

```APIDOC
BigFileSpider:
  __init__(*args, **kwargs)
    Collect data from a source that publishes very large packages.
    Each package is split into smaller packages, each containing 100 releases or records.
    Users can then process the files without using an iterative parser and without having memory issues.

  resize_package = True

  @classmethod
  from_crawler(_crawler_, *args, **kwargs)
    Class method to create a spider instance from a crawler.

  parse(_response_)
    Default method to handle responses from requests. This method is expected to parse the response and yield items.
```

--------------------------------

### Apply Path Filters

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Applies filters to the data collection by appending path components to the URL, useful for APIs with path-based parameters.

```bash
scrapy crawl spider_name -a path=key1/value1/key2/value2/value3
```

--------------------------------

### Filter Data by Date

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Filters the data collected by a spider based on a specified date range using 'from_date' and 'until_date' arguments.

```bash
scrapy crawl colombia -a from_date=2015-01-01
```

--------------------------------

### OCDS Kingfisher Collect Documentation Overview

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/spiders/europe_eu_digiwhist

Provides an overview of the OCDS Kingfisher Collect documentation, including sections on downloading data, spiders, logging, CLI, integration, and contributing.

```APIDOC
OCDS Kingfisher Collect Documentation:
  - Download data to your computer
  - Download data to a remote server
  - Spiders
  - Log files
  - Command-line interface
  - Integrate with Kingfisher Process
  - Contributing
```

--------------------------------

### Use Kingfisher Collect in Google Colaboratory

Source: https://kingfisher-collect.readthedocs.io/en/latest/index

Step-by-step guide for using Kingfisher Collect within a Google Colaboratory environment.

```python
# The documentation provides an interactive guide:
# https://colab.research.google.com/drive/1bN5C9h5YLUF9YpIprAeOMZcXJ5Tr6iaO
```

--------------------------------

### Kingfisher Collect - Contributing

Source: https://kingfisher-collect.readthedocs.io/en/latest/index

Guidelines for contributing to the Kingfisher Collect project, covering various aspects like middlewares, pipelines, and extensions.

```markdown
- [Contributing](https://kingfisher-collect.readthedocs.io/en/latest/contributing/index.html)
- [Downloader Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/downloadermiddlewares.html)
- [Spider Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/spidermiddlewares.html)
- [Item Pipelines](https://kingfisher-collect.readthedocs.io/en/latest/contributing/pipelines.html)
- [Extensions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/index.html)
  - [Sentry Logging](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/sentry_logging.html)
  - [Pluck](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/pluck.html)
  - [Files Store](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/files_store.html)
  - [Database Store](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/database_store.html)
  - [Item Count](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/item_count.html)
- [Utilities](https://kingfisher-collect.readthedocs.io/en/latest/contributing/util.html)
- [Exceptions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/exceptions.html)
```

--------------------------------

### Download Data to Remote Server with Scrapyd

Source: https://kingfisher-collect.readthedocs.io/en/latest/index

Guide for using Kingfisher Collect with Scrapyd to download data to a remote server.

```python
# Refer to the full documentation for Scrapyd integration details.
# Example command structure (conceptual):
# scrapyd-deploy <project_name>
```

--------------------------------

### Kingfisher Process API v2

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/index

Documentation for the Kingfisher Process API v2, detailing its functionalities and usage for integrating with Kingfisher Collect.

```APIDOC
Kingfisher Process API v2:
  This section details the API for interacting with Kingfisher Process, enabling integration with Kingfisher Collect.

  Endpoints:
    - /api/v2/collect/...

  Functionalities:
    - Data submission
    - Status checks
    - Configuration management

  Refer to the specific API documentation for detailed endpoint definitions, request/response formats, and authentication methods.
```

--------------------------------

### KingfisherProcessAPI2 Spider Opened

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/kingfisher_process_api_v2

Method called when a spider is opened, likely for initialization or connection setup with Kingfisher Process.

```APIDOC
KingfisherProcessAPI2.spider_opened(_spider_)
  - Called when a spider is opened.
```

--------------------------------

### Incremental Data Collection with Crawl Time Override

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Collects incremental data into the same directory by overriding the crawl time using the 'crawl_time' argument.

```bash
scrapy crawl spider_name -a from_date=2020-10-15 -a until_date=2020-10-31 -a crawl_time=2020-10-14T12:34:56
```

--------------------------------

### BigFileSpider Usage Example

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/big_file_spider

Demonstrates how to inherit from BigFileSpider and implement the start_requests method to process large package files. It shows how to build a request for a package file and use the components formatter.

```python
from kingfisher_scrapy.base_spiders import BigFileSpider
from kingfisher_scrapy.util import components

class MySpider(BigFileSpider):
    name = 'my_spider'

    def start_requests(self):
        yield self.build_request('https://example.com/api/package.json', formatter=components(-1))
```

--------------------------------

### Keep Collection Open for Kingfisher Process

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Sets the 'keep_collection_open' argument to 'true' when integrating with Kingfisher Process, preventing the collection from closing after the crawl.

```bash
scrapy crawl spider_name -a keep_collection_open=true
```

--------------------------------

### DatabaseStore from_crawler Class Method

Source: https://kingfisher-collect.readthedocs.io/en/latest/_modules/kingfisher_scrapy/extensions/database_store

Factory method to create an instance of the DatabaseStore extension. It retrieves the database URL and file store directory from Scrapy settings and connects the extension to spider signals.

```python
    @classmethod
    deffrom_crawler(cls, crawler):
        database_url = crawler.settings["DATABASE_URL"]
        directory = crawler.settings["FILES_STORE"]

        if not database_url:
            raise NotConfigured("DATABASE_URL is not set.")
        if not directory:
            raise NotConfigured("FILES_STORE is not set.")

        extension = cls(database_url, directory)
        crawler.signals.connect(extension.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(extension.spider_closed, signal=signals.spider_closed)

        return extension

```

--------------------------------

### Incremental Data Collection with Date Range

Source: https://kingfisher-collect.readthedocs.io/en/latest/_sources/local.rst

Collects data incrementally within a specified date range. Useful for updating existing datasets.

```bash
scrapy crawl spider_name -a from_date=2020-10-15 -a until_date=2020-10-31
```

--------------------------------

### Kingfisher Collect Contributing Guide

Source: https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/item_count

This section outlines how to contribute to Kingfisher Collect, covering base spider classes, downloader and spider middlewares, item pipelines, extensions, and utilities. It provides links to specific documentation for each area.

```markdown
## Contributing

* [Base Spider Classes](https://kingfisher-collect.readthedocs.io/en/latest/contributing/base_spiders/index.html)
* [Downloader Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/downloadermiddlewares.html)
* [Spider Middlewares](https://kingfisher-collect.readthedocs.io/en/latest/contributing/spidermiddlewares.html)
* [Item Pipelines](https://kingfisher-collect.readthedocs.io/en/latest/contributing/pipelines.html)
* [Extensions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/extensions/index.html)
* [Utilities](https://kingfisher-collect.readthedocs.io/en/latest/contributing/util.html)
* [Exceptions](https://kingfisher-collect.readthedocs.io/en/latest/contributing/exceptions.html)
```
Tokens:
Raw