OCDS Kingfisher Process

https://kingfisher-process.readthedocs.io/en/latest/
Admin
Kingfisher Process stores and pre-processes OCDS data, receiving it from Kingfisher Collect or the...
Tokens:38,691
Snippets:155
Trust Score:8.3
Update:1 year ago
Show doc for...
Tokens:
Raw
### Setup Git Pre-commit Hook

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Installs the git pre-commit hook, which helps automate code quality checks before commits are made. This ensures that code adheres to project standards.

```bash
pre-commit install
```

--------------------------------

### Database Connection Example

Source: https://kingfisher-process.readthedocs.io/en/latest/contributing/index

An example of how to set the DATABASE_URL environment variable for connecting to a PostgreSQL database. This is used to configure the application's database connection.

```shell
DATABASE_URL=postgresql://user@host/dbname
```

--------------------------------

### Install Development Dependencies

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Installs the necessary development dependencies for the Kingfisher Process project using pip. This command reads the list of dependencies from the 'requirements_dev.txt' file.

```bash
pip install -r requirements_dev.txt
```

--------------------------------

### Run Kingfisher Process Server (API)

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Starts the development server for the Kingfisher Process project, making the API accessible. This command is used for local development and testing.

```bash
./manage.py runserver
```

--------------------------------

### Consume Messages (Async)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Starts an asynchronous RabbitMQ consumer client. This function initializes and starts the consumer with specified arguments and default RabbitMQ settings.

```python
def consume(*args, **kwargs):
    client = AsyncConsumer(*args, **kwargs, **YAPW_KWARGS)
    client.start()
```

--------------------------------

### Kingfisher Process Utilities

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

API documentation for utility functions within the Kingfisher Process project. These functions cover various operations like wrapping data, walking through structures, getting publisher information, consuming data, applying decorators, managing notes and steps, and retrieving extensions.

```APIDOC
wrap()
  Wraps data.
walk()
  Walks through a structure.
get_publisher()
  Gets publisher information.
consume()
  Consumes data.
decorator()
  Applies a decorator.
get_or_create()
  Gets or creates an entity.
create_note()
  Creates a note.
create_step()
  Creates a step.
delete_step()
  Deletes a step.
create_warnings_note()
  Creates a note for warnings.
create_logger_note()
  Creates a note for the logger.
get_extensions()
  Gets available extensions.
```

--------------------------------

### Get Publisher Client (Blocking)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Provides a context manager for obtaining a blocking RabbitMQ client. Ensures the client is closed properly after use, even if errors occur.

```python
@contextmanager
def get_publisher():
    client = Blocking(**YAPW_KWARGS)
    try:
        yield client
    finally:
        client.close()
```

--------------------------------

### Get or Create Model Instance

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Retrieves or creates a model instance based on provided data. It calculates an MD5 hash of the data to check for existing records, ensuring data integrity and preventing duplicates. Uses atomic transactions for database operations.

```python
def get_or_create(model, data):
    """Get or create a Data or PackageData instance."""
    hash_md5 = hashlib.md5(  # noqa: S324 # non-cryptographic
        json.dumps(data, separators=(',', ':'), sort_keys=True, use_decimal=True).encode("utf-8")
    ).hexdigest()

    try:
        # Another transaction is needed here, otherwise a parent transaction catches the integrity error.
        with transaction.atomic():
```

--------------------------------

### Kingfisher Process Models

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

API documentation for models in the Kingfisher Process project, including Default and Collection models. The Collection model includes methods for transforming data, cleaning fields, and getting upgraded collections.

```APIDOC
Default
  Represents the default model.
Collection
  Represents a collection.
  Transform
    Represents a transformation within a collection.
  clean_fields()
    Cleans the fields of a collection.
  get_upgraded_collection()
    Gets the upgraded version of the collection.
```

--------------------------------

### Kingfisher Process CLI Help

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Displays the main help message for the Kingfisher Process command-line interface.

```bash
./manage.py --help
```

--------------------------------

### Compiler Worker

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Starts compilation and routes messages to the release or record compiler. For release packages, compilation starts at most once after all files are loaded and the collection is closed. Performs no work if 'compile' is not in the collection's steps.

```bash
./manage.py compiler
```

--------------------------------

### API Documentation Access

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/reference/index.rst

Provides instructions on how to view the API's documentation in development by running the server and accessing specific URLs for Swagger UI or Redoc.

```APIDOC
API Documentation:
  Access Swagger UI at http://127.0.0.1:8000/api/schema/swagger-ui/
  Access Redoc at http://127.0.0.1:8000/api/schema/redoc/
```

--------------------------------

### Create Kingfisher Process Database

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Creates a new PostgreSQL database named 'kingfisher_process'. It assumes the current user has the necessary privileges to create databases without requiring a password.

```bash
createdb kingfisher_process
```

--------------------------------

### Get Collection Status

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Retrieves the status of a root collection and its children.

```bash
./manage.py collectionstatus collection_id
```

--------------------------------

### Run Finisher Worker with Debug Logging

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Starts the finisher worker and enables debug logging by setting the LOG_LEVEL environment variable.

```bash
env LOG_LEVEL=DEBUG ./manage.py finisher
```

--------------------------------

### API Documentation Links

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Provides links to access the API documentation in different formats. Users can choose between Redoc and Swagger UI for interactive API exploration.

```APIDOC
API (Redoc): https://kingfisher-process.readthedocs.io/en/latest/reference/redoc.html
API (Swagger UI): https://kingfisher-process.readthedocs.io/en/latest/reference/swagger-ui.html
```

--------------------------------

### Get Package Extensions

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Retrieves a set of extensions from a package dictionary. It also ensures that the 'lots' extension implies the 'submissionTerms' extension.

```python
defget_extensions(package):
    extensions = set()

    package_extensions = package.get("extensions")
    if isinstance(package_extensions, list):
        extensions = {extension for extension in package_extensions if isinstance(extension, str)}

    if EXTENSION_URL.format("lots") in extensions:
        extensions.add(EXTENSION_URL.format("submissionTerms"))

    return frozenset(extensions)
```

--------------------------------

### Run Kingfisher Process Tests

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Executes the test suite for the Kingfisher Process project. This command helps ensure the project's functionality and stability.

```bash
./manage.py test
```

--------------------------------

### API Reference Links

Source: https://kingfisher-process.readthedocs.io/en/latest/index

Provides links to the API documentation in different formats, including Redoc and Swagger UI, for detailed API information.

```APIDOC
API (Redoc): https://kingfisher-process.readthedocs.io/en/latest/reference/redoc.html
API (Swagger UI): https://kingfisher-process.readthedocs.io/en/latest/reference/swagger-ui.html
```

--------------------------------

### Add Files to Open Collection

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Loads data into an open root collection asynchronously.

```bash
./manage.py addfiles collection_id path [path ...]
```

--------------------------------

### Walk Paths for Files

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Iterates through a list of paths, yielding individual files. If a path is a directory, it recursively walks through it to find all files, excluding hidden files (those starting with '.').

```python
def walk(paths):
    for path in paths:
        if os.path.isfile(path):
            yield path
        else:
            for root, _, files in os.walk(path):
                for name in files:
                    if not name.startswith("."):
                        yield os.path.join(root, name)
```

--------------------------------

### Get JSON Data for Releases in a Collection

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/querying-data.rst

Fetches the raw OCDS JSON data for compiled releases within a specific collection. Joins the 'data' table with 'compiled_release' to filter by collection_id.

```sql
SELECT
    data
FROM
    data -- raw OCDS JSON data is stored as jsonb blobs in the `data` column of the `data` table
JOIN
    compiled_release ON data.id = compiled_release.data_id -- join to the `compiled_release` table to filter data from a specific collection
WHERE
    collection_id = 584
LIMIT 3;
```

--------------------------------

### OCDS Kingfisher Process API Reference

Source: https://kingfisher-process.readthedocs.io/en/latest/index

Provides an overview of the OCDS Kingfisher Process API, including utilities, models, loader, scrapyd, and command-line interface modules.

```APIDOC
API Reference:
  - Utilities (process.util)
  - Models (process.models)
  - Loader (process.processors.loader)
  - Scrapyd (process.scrapyd)
  - Command-line interface (process.cli)
```

--------------------------------

### Optimistic Locking for Compilation Status

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Illustrates optimistic locking to update a 'Collection' object's 'compilation_started' status. It ensures that the update only occurs if the record has not yet started compilation, preventing duplicate processing.

```python
updated = Collection.objects.filter(pk=collection.pk, compilation_started=False).update(compilation_started=True)
if not updated:
    return
```

--------------------------------

### API Documentation Access

Source: https://kingfisher-process.readthedocs.io/en/latest/reference/index

Provides URLs to access the API documentation in Redoc and Swagger UI formats. Also includes instructions for running the development server to view the API schema.

```APIDOC
API Endpoints:
  - Redoc: http://127.0.0.1:8000/api/schema/redoc/
  - Swagger UI: http://127.0.0.1:8000/api/schema/swagger-ui/

Usage:
  Run the development server and open the provided URLs to view the API documentation.

Related Information:
  - Kingfisher Collect documentation
  - KINGFISHER_PROCESS_URL in Data Registry
```

--------------------------------

### Kingfisher Process API Endpoints

Source: https://kingfisher-process.readthedocs.io/en/latest/reference/swagger-ui

This section details the available API endpoints for managing collections and retrieving schema information. It covers POST, DELETE, and GET requests for various collection-related operations and schema access.

```APIDOC
OpenAPI Specification Version: 3.0

Collections API:
  POST /api/collections/
    Description: Creates a new collection.
    Request Body: (Schema: CreateCollection)
    Response: (Schema: Collection)

  DELETE /api/collections/{id}/
    Description: Deletes a collection by its ID.
    Parameters:
      - name: id
        in: path
        required: true
        schema: (type: string)
    Response: 204 No Content

  POST /api/collections/{id}/close/
    Description: Closes a collection by its ID.
    Parameters:
      - name: id
        in: path
        required: true
        schema: (type: string)
    Request Body: (Schema: CloseCollection)
    Response: (Schema: Collection)

  GET /api/collections/{id}/metadata/
    Description: Retrieves metadata for a specific collection.
    Parameters:
      - name: id
        in: path
        required: true
        schema: (type: string)
    Response: (Schema: CollectionMetadata)

  GET /api/collections/{id}/notes/
    Description: Retrieves notes for a specific collection.
    Parameters:
      - name: id
        in: path
        required: true
        schema: (type: string)
    Response: (Schema: CollectionNotes)

  GET /api/collections/{id}/tree/
    Description: Retrieves the tree structure for a specific collection.
    Parameters:
      - name: id
        in: path
        required: true
        schema: (type: string)
    Response: (Schema: Tree)

Schema API:
  GET /api/schema/
    Description: Retrieves the API schema.
    Response: (Schema: OpenAPI)

Schemas:
  BlankEnum: An enumeration with no values.
  CloseCollection: Schema for closing a collection.
  CreateCollection: Schema for creating a collection.
  TransformTypeEnum: An enumeration for transform types.
  Tree: Represents the tree structure of a collection.
```

--------------------------------

### Create Collections

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/processors/loader

Creates root, upgraded, and compiled collections based on source ID and data version. It supports planning upgrade, compilation, and schema checks, and allows for optional sample data, Scrapyd job IDs, notes, and force validation. Returns a tuple of the root, upgraded, and compiled collections.

```python
import copy
import logging

from process.forms import CollectionForm, CollectionNote, CollectionNoteForm
from process.models import Collection

logger = logging.getLogger(__name__)

def create_collections(
    # Identification
    source_id,
    data_version,
    *,
    sample=False,
    # Steps
    upgrade=False,
    compile=False,  # noqa: A002 # consistency
    check=False,
    # Other
    scrapyd_job="",
    note="",
    force=False,
) -> tuple[Collection, Collection, Collection]:
    """
    Create the root collection, derived collections and notes.

    :param str source_id: collection source
    :param str data_version: data version in ISO format
    :param boolean sample: is this sample only
    :param boolean upgrade: whether to plan collection upgrade
    :param boolean compile: whether to plan collection compile
    :param boolean check: whether to plan schema-based checks
    :param str scrapyd_job: Scrapyd job ID
    :param str note: text description
    :param boolean force: skip validation of the source_id against the Scrapyd project
    :returns: the root collection, upgraded collection and compiled_collection
    """
    data = {
        "source_id": source_id,
        "data_version": data_version,
        "sample": sample,
        "scrapyd_job": scrapyd_job,
        "force": force,
    }

    steps = []
    if check:
        steps.append("check")
    if upgrade:
        steps.append("upgrade")
    elif compile:
        steps.append("compile")

    collection = _create_collection(data, note, steps=steps)

    upgraded_collection = None
    if upgrade:
        # main -> upgrade -> compile / main -> upgrade
        upgrade_steps = ["compile"] if compile else []
        upgraded_collection = _create_collection(
            data, note, steps=upgrade_steps, parent=collection, transform_type=Collection.Transform.UPGRADE_10_11
        )

    compiled_collection = None
    if compile:
        # main -> upgrade -> compile / main -> compile
        base_collection = upgraded_collection if upgrade else collection
        compiled_collection = _create_collection(
            data, note, parent=base_collection, transform_type=Collection.Transform.COMPILE_RELEASES
        )

    return collection, upgraded_collection, compiled_collection

def _create_collection(data, note, **kwargs):
    collection_data = copy.deepcopy(data)
    collection_data.update(kwargs)
    # If steps is empty, Django attempts to save it as NULL, but the column has a NOT NULL constraint.
    if "steps" in collection_data and not collection_data["steps"]:
        collection_data.pop("steps")

    form = CollectionForm(collection_data)

    if form.is_valid():
        collection = form.save()
        if note:
            _save_note(collection, note)
        return collection

    raise ValueError(form.error_messages)


def _save_note(collection, note):
    form = CollectionNoteForm({"collection": collection, "note": note, "code": CollectionNote.Level.INFO})

    if form.is_valid():
        return form.save()

    raise ValueError(form.error_messages)
```

--------------------------------

### API Documentation Links

Source: https://kingfisher-process.readthedocs.io/en/latest/contributing/index

Provides links to the API documentation, accessible via Redoc and Swagger UI, for understanding the project's API endpoints and structure.

```APIDOC
API (Redoc): https://kingfisher-process.readthedocs.io/en/latest/reference/redoc.html
API (Swagger UI): https://kingfisher-process.readthedocs.io/en/latest/reference/swagger-ui.html
```

--------------------------------

### Run Database Migrations

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/contributing/index.rst

Applies database migrations to set up the schema for the Kingfisher Process project. This command is part of the Django management utility.

```bash
./manage.py migrate
```

--------------------------------

### Message Routing - Addfiles Command

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/reference/index.rst

Details the message routing for the 'addfiles' command, specifying that it publishes 'loader' messages for each collection file and creates 'LOAD' events.

```APIDOC
Message Routing - addfiles command:
  Consumer routing keys (input): N/A
  Publisher routing keys (output): 'loader' for each collection file
  Processing step: Create 'LOAD' for each collection file
```

--------------------------------

### Get JSON Data from a Collection

Source: https://kingfisher-process.readthedocs.io/en/latest/querying-data

This SQL query retrieves the full JSON data for compiled releases within a specific collection. It joins the `data` table with the `compiled_release` table to filter by `collection_id` and limits the results. To retrieve releases or records, join to the `release` or `record` tables instead.

```sql
SELECT
data
FROM
data
JOIN
compiled_release ON data.id=compiled_release.data_id
WHERE
collection_id=584
LIMIT 3;
```

--------------------------------

### API Reference (Redoc)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/processors/loader

Provides an interactive API documentation interface using Redoc, allowing users to explore and test API endpoints.

```APIDOC
API Reference (Redoc):
  Access the API documentation via the Redoc interface for detailed endpoint information, request/response schemas, and interactive testing.
```

--------------------------------

### Retrieve Scrapyd Spider Names

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/scrapyd

Fetches the names of all spiders deployed in the configured Scrapyd project. It makes an HTTP GET request to the Scrapyd API's '/listspiders.json' endpoint. The function requires the Scrapyd URL and project name to be set in Django settings and returns a list of spider names.

```python
fromurllib.parseimport urljoin

importrequests
fromdjango.confimport settings




[docs][](https://kingfisher-process.readthedocs.io/en/latest/api/index.html#process.scrapyd.spiders)
defspiders() -> list[str]:
"""Return the names of the spiders in the Scrapyd project."""
    # https://scrapyd.readthedocs.io/en/stable/api.html#listspiders-json
    url = urljoin(settings.SCRAPYD["url"], "/listspiders.json")
    response = requests.get(url, params={"project": settings.SCRAPYD["project"]}, timeout=10)
    response.raise_for_status()
    return response.json()["spiders"]
```

--------------------------------

### User Notes in Kingfisher Process

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/database.rst

Describes user-created collections and how to determine the creator. Also includes notes related to spider closure reasons and statistics in Kingfisher Collect.

```text
- A user-provided note
- Determine who created the collection.
```

```text
Spider close reason: {reason}
Check the reason for closing the spider <https://kingfisher-collect.readthedocs.io/en/latest/logs.html#check-the-reason-for-closing-the-spider>__.
```

```text
Spider stats
Check the crawl statistics <https://kingfisher-collect.readthedocs.io/en/latest/logs.html#check-the-crawl-statistics>__ (in the ``data`` column).
```

--------------------------------

### Process API Reference

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/cli

API documentation for the Kingfisher Process project, providing details on available endpoints and their functionalities. Links to Redoc and Swagger UI are provided for interactive exploration.

```APIDOC
API (Redoc): https://kingfisher-process.readthedocs.io/en/latest/reference/redoc.html
API (Swagger UI): https://kingfisher-process.readthedocs.io/en/latest/reference/swagger-ui.html
API reference: https://kingfisher-process.readthedocs.io/en/latest/api/index.html
process.cli.CollectionCommand:
  add_arguments(self, parser)
    Add default arguments to the command.
    Parameters:
      parser: The argument parser.
  add_collection_arguments(self, parser)
    Add arguments specific to this command.
    Parameters:
      parser: The argument parser.
  handle(self, *args, **options)
    Get the collection.
    Handles the core logic of the command, including fetching the collection and calling handle_collection.
    Parameters:
      args: Positional arguments.
      options: Keyword arguments passed to the command.
    Raises:
      CommandError: If the collection does not exist.
  handle_collection(self, collection, *args, **options)
    Run the command.
    This method must be implemented by subclasses to define the specific collection processing logic.
    Parameters:
      collection: The Collection object.
      args: Positional arguments.
      options: Keyword arguments passed to the command.
    Raises:
      NotImplementedError: If the method is not implemented by a subclass.
```

--------------------------------

### Load Data into Collection

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Loads data into a collection asynchronously. Supports specifying source, retrieval time, notes, and options for forcing, upgrading, checking, and keeping the collection open.

```bash
./manage.py load [OPTIONS] PATH [PATH ...]

-s SOURCE, --source SOURCE
                      the source from which the files were retrieved (append '_local' if not sourced from Scrapy)
-t TIME, --time TIME  the time at which the files were retrieved in 'YYYY-MM-DD HH:MM:SS' format (defaults to the earliest file modification time)
--sample              whether the files represent a sample from the source
-n NOTE, --note NOTE  a note to add to the collection
-f, --force           use the provided --source value, regardless of whether it is recognized
-u, --upgrade         upgrade the collection to the latest OCDS version
-c, --compile         create compiled releases from the collection
-e, --check           run structural checks on the collection
-k, --keep-open       keep collection open for future file additions
```

--------------------------------

### API Reference (Redoc)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Provides an interactive API documentation interface using Redoc, allowing users to explore and test API endpoints.

```APIDOC
API Reference (Redoc):
  Explore and test API endpoints interactively.
```

--------------------------------

### Default Model API

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for the Default model.

```APIDOC
class process.models.Default
```

--------------------------------

### Kingfisher Process Loader Functions

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for functions within the Kingfisher Process loader module, used for handling files and creating collections.

```APIDOC
Loader:
  file_or_directory(path: str)
    Handles a file or directory path.
  create_collection_file(path: str, collection_id: str)
    Creates a collection file at the specified path.
  create_collections(path: str)
    Creates collections from the specified path.
```

--------------------------------

### Scrapyd Configuration and Spider Listing

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Provides functions to interact with Scrapyd. The `configured()` function checks if a connection to Scrapyd is set up, returning a boolean. The `spiders()` function retrieves a list of all spider names available in the Scrapyd project.

```APIDOC
process.scrapyd.configured()
  - Return whether the connection to Scrapyd is configured.
  - Return type: bool

process.scrapyd.spiders()
  - Return the names of the spiders in the Scrapyd project.
  - Return type: list[str]
```

--------------------------------

### Message Routing - Load Command

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/reference/index.rst

Details the message routing for the 'load' command, specifying that it publishes 'loader' messages for each collection file and creates 'LOAD' events.

```APIDOC
Message Routing - load command:
  Consumer routing keys (input): N/A
  Publisher routing keys (output): 'loader' for each collection file
  Processing step: Create 'LOAD' for each collection file
```

--------------------------------

### File Worker

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/cli.rst

Creates releases, records, and compiled releases.

```bash
./manage.py file_worker
```

--------------------------------

### Environment Variables

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/reference/index.rst

Lists and describes the environment variables used to configure the Kingfisher Process application, including logging, RabbitMQ connections, Scrapyd integration, file storage, and feature toggles.

```APIDOC
Environment Variables:
  LOG_LEVEL: The log level of the root logger.
  RABBIT_URL: The connection string for RabbitMQ.
  RABBIT_EXCHANGE_NAME: The name of the RabbitMQ exchange, following the pattern 'kingfisher_process_{service}_{environment}'.
  SCRAPYD_URL: The base URL of Scrapyd.
  SCRAPYD_PROJECT: The project within Scrapyd.
  KINGFISHER_COLLECT_FILES_STORE: The directory from which to read files written by Kingfisher Collect.
  ENABLE_CHECKER: Whether to enable the 'checker' worker.
```

--------------------------------

### Find Compiled Collections by Source

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/querying-data.rst

Retrieves a list of compiled release collections downloaded with a specific spider (source_id) from the 'collection' table. Filters for collections with compiled releases and orders by newest.

```sql
SELECT
    *
FROM
    collection -- the `collection` table contains a list of all collections in the database
WHERE
    source_id = 'canada_montreal' -- filter by collections from the 'canada_montreal' source.
AND
    cached_compiled_releases_count > 0 -- filter by collections containing compiled releases
ORDER BY
    id DESC; -- collection ids are sequential, order by newest first
```

--------------------------------

### Process Utilities API

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

This section details the API for various utility functions within the process.util module. It covers functions for string formatting, file system operations, data management, and logging.

```APIDOC
process.util.wrap(_string_)
  Formats a long string as a help message.
  Parameters:
    _string_: The string to format.
  Returns: The formatted string.
```

```APIDOC
process.util.walk(_paths_)
  Traverses file paths. Specific functionality not detailed.
  Parameters:
    _paths_: Path(s) to traverse.
```

```APIDOC
process.util.get_publisher()
  Retrieves publisher information. Specific return value not detailed.
```

```APIDOC
process.util.consume(_* args_, _** kwargs_)
  Consumes arguments. Specific functionality not detailed.
```

```APIDOC
process.util.decorator(_decode_ , _callback_ , _state_ , _channel_ , _method_ , _properties_ , _body_)
  A decorator function that manages database connections and message acknowledgments.
  It closes database connections opened by the callback. If the callback raises an exception, it shuts down the client in the main thread without acknowledgment. For certain exceptions, it assumes duplicate message delivery, logs an error, and negatively acknowledges the message.
  Parameters:
    _decode_: Decoding information.
    _callback_: The callback function to execute.
    _state_: The current state.
    _channel_: The communication channel.
    _method_: The method being called.
    _properties_: Message properties.
    _body_: The message body.
```

```APIDOC
process.util.get_or_create(_model_ , _data_)
  Gets or creates a Data or PackageData instance.
  Parameters:
    _model_: The model to use for getting or creating.
    _data_: The data to use for creation if the instance does not exist.
```

```APIDOC
process.util.create_note(_collection_ , _code_ , _note_ , _** kwargs_)
  Creates a note within a collection.
  Parameters:
    _collection_: The collection to add the note to.
    _code_: A code associated with the note.
    _note_: The content of the note.
    _** kwargs_: Additional keyword arguments.
```

```APIDOC
process.util.create_step(_name_ , _collection_id_ , _** kwargs_)
  Creates a step associated with a collection.
  Parameters:
    _name_: The name of the step.
    _collection_id_: The ID of the collection.
    _** kwargs_: Additional keyword arguments.
```

```APIDOC
process.util.delete_step(_* args_, _** kwargs_)
  Deletes a named step and its associated callbacks. It only runs finish callbacks if the deletion is successful or if the error is expected.
  Parameters:
    _* args_: Positional arguments.
    _** kwargs_: Keyword arguments.
```

```APIDOC
process.util.create_warnings_note(_collection_ , _category_)
  Creates a note for warnings within a collection, categorized by a given category.
  Parameters:
    _collection_: The collection to add the warning note to.
    _category_: The category of the warnings.
```

```APIDOC
process.util.create_logger_note(_collection_ , _name_)
  Creates a note for a logger within a collection.
  Parameters:
    _collection_: The collection to add the logger note to.
    _name_: The name of the logger.
```

```APIDOC
process.util.get_extensions(_package_)
  Retrieves extensions for a given package.
  Parameters:
    _package_: The package to get extensions for.
```

--------------------------------

### Process Utilities API

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/api/index.rst

Provides access to utility functions within the process.util module. This includes various helper functions used across the project.

```APIDOC
process.util:
  Exposes utility functions for the Kingfisher Process project.
  Members include various helper functions for common tasks.
```

--------------------------------

### Process CLI API

Source: https://kingfisher-process.readthedocs.io/en/latest/_sources/api/index.rst

API reference for the command-line interface module of the Kingfisher Process project. It details all members and undocumented members.

```APIDOC
process.cli:
  Contains the command-line interface for the Kingfisher Process.
  Members: All public attributes and methods, including undocumented ones.
```

--------------------------------

### Kingfisher Process CLI - CollectionCommand

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for the CollectionCommand in the Kingfisher Process command-line interface, including argument handling and execution.

```APIDOC
CLI:
  CollectionCommand:
    add_arguments(parser)
      Adds arguments to the collection command parser.
    add_collection_arguments(parser)
      Adds specific arguments for collections.
    handle()
      Handles the collection command execution.
    handle_collection()
      Handles the execution of a specific collection.
```

--------------------------------

### Kingfisher Loader Functions

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Provides functions for loading and processing data within the Kingfisher framework. This includes file path validation, collection file creation, and the creation of root and derived collections.

```APIDOC
file_or_directory(_path_)
    Check whether the path exists. Raise an exception if not.

create_collection_file(_collection_ , _filename =None_, _url =None_)
    Create file for a collection and steps for this file.
    Parameters:
      collection (_Collection_): collection
      filename (_str_): path to file data
    Returns:
      created collection file (_CollectionFile_)
    Raises:
      InvalidFormError: if there is a validation error

create_collections(_source_id_ , _data_version_ , _*_ , _sample =False_, _upgrade =False_, _compile =False_, _check =False_, _scrapyd_job =''_, _note =''_, _force =False_)
    Create the root collection, derived collections and notes.
    Parameters:
      source_id (_str_): collection source
      data_version (_str_): data version in ISO format
      sample (_boolean_): is this sample only
      upgrade (_boolean_): whether to plan collection upgrade
      compile (_boolean_): whether to plan collection compile
      check (_boolean_): whether to plan schema-based checks
      scrapyd_job (_str_): Scrapyd job ID
      note (_str_): text description
      force (_boolean_): skip validation of the source_id against the Scrapyd project
    Returns:
      the root collection, upgraded collection and compiled_collection (tuple[_Collection_, _Collection_, _Collection_])
```

--------------------------------

### Kingfisher Process Scrapyd Integration

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for functions related to Scrapyd integration in the Kingfisher Process library.

```APIDOC
Scrapyd:
  configured() -> bool
    Checks if Scrapyd is configured.
  spiders() -> list
    Returns a list of available spiders from Scrapyd.
```

--------------------------------

### API Reference (Swagger UI)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/processors/loader

Offers an interactive API documentation experience through Swagger UI, enabling users to visualize and interact with the API.

```APIDOC
API Reference (Swagger UI):
  Explore the API using the Swagger UI, which provides a user-friendly interface for understanding API endpoints, parameters, and responses.
```

--------------------------------

### Kingfisher Process Models

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for various data models used within the Kingfisher Process library, including Collection, CollectionNote, CollectionFile, ProcessingStep, Data, PackageData, Release, Record, CompiledRelease, ReleaseCheck, and RecordCheck.

```APIDOC
Collection:
  get_compiled_collection()
    Returns the compiled collection.
  get_root_parent()
    Returns the root parent of the collection.

CollectionNote:
  Level: Enum for collection note levels.

CollectionFile:
  Represents a file within a collection.

ProcessingStep:
  Name: Enum for processing step names.

Data:
  Base class for data entities.

PackageData:
  Represents data associated with a package.

Release:
  Represents a release.

Record:
  Represents a record.

CompiledRelease:
  Represents a compiled release.

ReleaseCheck:
  Represents a check performed on a release.

RecordCheck:
  Represents a check performed on a record.
```

--------------------------------

### Kingfisher Process CLI Commands

Source: https://kingfisher-process.readthedocs.io/en/latest/cli

This section details the various commands available in the Kingfisher Process command-line interface. Each command serves a specific purpose in managing and processing OCDS data.

```bash
load [-s SOURCE] [-t TIME] [--sample] [-n NOTE] [-f] [-u] [-c] [-e] [-k]

  Load data into a collection, asynchronously.

  Options:
    -s SOURCE, --source SOURCE      the source from which the files were retrieved (append ‘_local’ if not sourced from Scrapy)
    -t TIME, --time TIME            the time at which the files were retrieved in ‘YYYY-MM-DD HH:MM:SS’ format (defaults to the earliest file modification time)
    --sample                        whether the files represent a sample from the source
    -n NOTE, --note NOTE            a note to add to the collection
    -f, --force                     use the provided –source value, regardless of whether it is recognized
    -u, --upgrade                   upgrade the collection to the latest OCDS version
    -c, --compile                   create compiled releases from the collection
    -e, --check                     run structural checks on the collection
    -k, --keep-open                 keep collection open for future file additions

  Note:
    If the files are arrays of packages, only the first package’s metadata is saved. In other words, it is assumed that all packages have the same metadata.
```

```bash
addfiles [path]

  Load data into an **open** root collection, asynchronously.
```

```bash
closecollection

  Close an **open** root collection and its derived collections, if any.
```

```bash
addchecks

  Add processing steps to check data, if unchecked.
```

```bash
cancelcollection

  Cancel all processing of a collection.

  Note:
    For performance, the finisher worker picks one message for each collection, and ignores the rest. It requeues the message until the collection is completed. If the collection can never be completed, cancel the collection to stop the requeueing.
```

```bash
deletecollection

  Delete a collection and its ancestors.
  Rows in the `package_data` and `data` tables are not deleted. Use [deleteorphan](https://kingfisher-process.readthedocs.io/en/latest/cli.html#cli-deleteorphan) instead.
```

```bash
collectionstatus

  Get the status of a root collection and its children.
```

```bash
deleteorphan

  Delete rows from the data and package_data tables that relate to no collections.
```

--------------------------------

### Kingfisher Process CLI CollectionCommand

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for the CollectionCommand class in the Kingfisher Process CLI. This class handles adding default and collection-specific arguments to a parser and processing the collection.

```APIDOC
CollectionCommand(_stdout =None_, _stderr =None_, _no_color =False_, _force_color =False_)
  Adds default arguments to the command.
  add_arguments(_parser_)
    Add default arguments to the command.
  add_collection_arguments(_parser_)
    Add arguments specific to this command.
  handle(_* args_, _** options_)
    Get the collection.
  handle_collection(_collection_ , _* args_, _** options_)
    Run the command.
```

--------------------------------

### ProcessingStep Model API

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for the ProcessingStep model, which represents a step in the lifecycle of a collection file.

```APIDOC
class process.models.ProcessingStep(*args, **kwargs)
    A step in the lifecycle of collection file.

    class Name(*values)
```

--------------------------------

### API Reference (Swagger UI)

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/util

Provides an interactive API documentation interface using Swagger UI, allowing users to explore and test API endpoints.

```APIDOC
API Reference (Swagger UI):
  Explore and test API endpoints interactively.
```

--------------------------------

### Environment Variables

Source: https://kingfisher-process.readthedocs.io/en/latest/reference/index

Lists and describes environment variables used for configuring the Kingfisher Process. This includes settings for logging, message queues, external services, and feature flags.

```APIDOC
Environment Variables:

LOG_LEVEL:
  Description: The log level of the root logger.

RABBIT_URL:
  Description: The connection string for RabbitMQ.
  Example: See pika documentation for URL parameter examples.

RABBIT_EXCHANGE_NAME:
  Description: The name of the RabbitMQ exchange.
  Pattern: kingfisher_process_{service}_{environment}
  Example: kingfisher_process_data_registry_production

SCRAPYD_URL:
  Description: The base URL of Scrapyd.
  Example: http://localhost:6800

SCRAPYD_PROJECT:
  Description: The project within Scrapyd.

KINGFISHER_COLLECT_FILES_STORE:
  Description: The directory from which to read files written by Kingfisher Collect.
  Note: If Kingfisher Collect and Kingfisher Process share a filesystem, this value should be the same for both.

ENABLE_CHECKER:
  Description: Whether to enable the 'checker' worker.

Configuration Notes:
  - It is recommended to set REQUESTS_POOL_MAXSIZE to 20 for the connection pool used by ocdsextensionregistry.
  - This value is equivalent to the prefetch_count for RabbitMQ consumers.
  - Refer to OCP's approach to Django settings for more details.
```

--------------------------------

### Data Model API

Source: https://kingfisher-process.readthedocs.io/en/latest/api/index

Documentation for the Data model, which represents the contents of a release, record, or compiled release.

```APIDOC
class process.models.Data(*args, **kwargs)
    The contents of a release, record or compiled release.
```

--------------------------------

### Create Collection File

Source: https://kingfisher-process.readthedocs.io/en/latest/_modules/process/processors/loader

Creates a CollectionFile object for a given collection, optionally specifying a filename or URL. It also creates a processing step for loading the file. Raises InvalidFormError if the provided data is invalid.

```python
from process.exceptions import InvalidFormError
from process.forms import CollectionFileForm
from process.models import CollectionFile, ProcessingStep
from process.util import create_step

def create_collection_file(collection, filename=None, url=None) -> CollectionFile:
    """
    Create file for a collection and steps for this file.

    :param Collection collection: collection
    :param str filename: path to file data
    :returns: created collection file
    :raises InvalidFormError: if there is a validation error
    """
    form = CollectionFileForm({"collection": collection, "filename": filename, "url": url})

    if form.is_valid():
        collection_file = form.save()
        create_step(ProcessingStep.Name.LOAD, collection.pk, collection_file=collection_file)
        return collection_file

    raise InvalidFormError(form.error_messages)
```
Tokens:
Raw