### Docker Installation for ingest-file

Source: https://openaleph.org/docs/lib/ingest-file/setup

Installs the ingest-file Docker image from GitHub Container Registry. This is the recommended method for out-of-the-box usage.

```shell
docker pull ghcr.io/openaleph/ingest-file
```

--------------------------------

### Manual Installation Dependencies (Debian/Ubuntu)

Source: https://openaleph.org/docs/lib/ingest-file/setup

Installs system dependencies required for running ingest-file locally without Docker on Debian-like systems. This is useful for rapid development and testing.

```shell
git clone https://github.com/openaleph/ingest-file
cd ingest-file
./contrib/install_deb.sh
pip install .
```

--------------------------------

### OpenAleph Initialization Command

Source: https://openaleph.org/docs/dev-admin-guide/102/docker

This command initializes or upgrades the OpenAleph application, including database migrations and Elasticsearch index setup. It is typically run within an application container after the services have started and Elasticsearch is healthy.

```bash
docker compose run --rm worker aleph upgrade

```

--------------------------------

### Verify OpenAleph Installation

Source: https://openaleph.org/docs/user-guide/104/install

Verifies the OpenAleph CLI installation by displaying its help information. This confirms that the package has been installed correctly and is accessible from the command line.

```bash
openaleph --help
```

--------------------------------

### Install openaleph-procrastinate with Poetry

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/setup

Adds the openaleph-procrastinate package as a dependency to a project managed by Poetry. This ensures that all necessary dependencies are managed within the project's environment.

```bash
poetry add openaleph-procrastinate
```

--------------------------------

### Install openaleph-procrastinate with pip

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/setup

Installs the openaleph-procrastinate package and its required dependencies using pip. This is a standard Python package installation method.

```bash
pip install openaleph-procrastinate
```

--------------------------------

### Initialize OpenAleph Procrastinate Database

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/setup

Sets up the necessary database schema and tables for the openaleph-procrastinate application. This command should be run after configuring the database connection via environment variables.

```bash
opal-procrastinate init-db
```

--------------------------------

### Run ingest-file Locally

Source: https://openaleph.org/docs/lib/ingest-file/usage

This command shows how to run the ingest-file tool directly on your local machine, assuming it is installed and configured.

```bash
ingestors ...
```

--------------------------------

### Install OpenAleph Package

Source: https://openaleph.org/docs/user-guide/104/install

Installs the OpenAleph Python package using pip. Ensure you have Python 3.6+ and pip installed. It's recommended to use a virtual environment.

```bash
pip install openaleph
```

--------------------------------

### Development Setup - Shell

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Provides minimal configuration for local development, enabling debug mode, single-user operation, and a local database connection.

```shell
# Minimal for local development
ALEPH_SECRET_KEY=dev-secret-not-for-production
ALEPH_DEBUG=true
ALEPH_SINGLE_USER=true
OPENALEPH_DB_URI=postgresql://aleph:aleph@localhost/aleph

```

--------------------------------

### Install openaleph-search Package

Source: https://openaleph.org/docs/lib/openaleph-search

Installs the openaleph-search Python package using pip. This is the primary method to get the library for use in your projects.

```bash
pip install openaleph-search

```

--------------------------------

### FollowTheMoney PostgreSQL Store Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Configures the FollowTheMoney (FTM) data store URI to use a PostgreSQL database. This is recommended for distributed production setups.

```env
FTM_STORE_URI=postgresql://user:password@host/database
```

--------------------------------

### Minimal Production Setup - Shell

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Sets up essential configuration for a minimal production environment, including secret keys, database URIs, application details, email server settings, and admin user identification.

```shell
# Required
ALEPH_SECRET_KEY=your-secret-key-here
OPENALEPH_DB_URI=postgresql://user:pass@db:5432/aleph

# Instance details
ALEPH_APP_TITLE=My Investigation Platform
ALEPH_UI_URL=https://aleph.example.com

# Email
ALEPH_MAIL_FROM=noreply@example.com
ALEPH_MAIL_HOST=smtp.example.com
ALEPH_MAIL_USERNAME=smtp-user
ALEPH_MAIL_PASSWORD=smtp-pass

# Admin users
ALEPH_ADMINS=admin@example.com

```

--------------------------------

### Install openaleph-procrastinate Dependency

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/howto

Instructions for adding the 'openaleph-procrastinate' library to your project using pip or poetry.

```shell
pip install openaleph-procrastinate

```

```shell
poetry add openaleph-procrastinate

```

--------------------------------

### Search Query Examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/parser

Illustrative examples of common search query patterns using the OpenALEPH CLI.

```APIDOC
## Search Query Examples

### Basic search with filters
```bash
openaleph-search search query-string "jane doe" \
  --args "filter:schema=Person&filter:countries=us"
```

### Faceted search
```bash
openaleph-search search query-string "darc" \
  --args "facet=schema&facet=countries&facet_size:schema=50"
```

### Date range with highlighting
```bash
openaleph-search search query-string "investigation" \
  --args "filter:gte:date_created=2020-01-01&filter:lte:date_created=2022-12-31&highlight=true"
```

### Pagination and sorting
```bash
openaleph-search search query-string "transaction" \
  --args "offset=100&limit=50&sort=date_created:desc"
```

### Significant terms analysis
```bash
openaleph-search search query-string "offshore" \
  --args "facet_significant=names&facet_significant_text=content&facet_significant_text_size=10"
```

### Complex query
```bash
openaleph-search search query-string "properties.keywords:corruption" \
  --args "filter:schema=Person&filter:schema=Company&filter:countries=us&exclude:schema=Page&facet=dataset&facet=schema&highlight=true&highlight_count=5&sort=_score:desc&limit=100"
```

### Performance-optimized query
Doesn't return entity properties in the payload:
```bash
openaleph-search search query-string "bank" \
  --args "dehydrate=true&limit=1000&facet=schema&facet_values:schema=false"
```
```

--------------------------------

### Minimal OpenAleph Environment Configuration

Source: https://openaleph.org/docs/dev-admin-guide/102/docker

This file sets essential environment variables for OpenAleph, including a secret key, single-user mode flag, and database URIs for PostgreSQL. It ensures basic connectivity and security for a local setup. Other configurations have default values.

```env
ALEPH_SECRET_KEY=random-secret-string
ALEPH_SINGLE_USER=true  # no user management at all for quick local set up
# best practice to set db uris explicitly:
OPENALEPH_DB_URI=postgresql://aleph:aleph@postgres
FTM_FRAGMENTS_URI=postgresql://aleph:aleph@postgres
PROCRASTINATE_DB_URI=postgresql://aleph:aleph@postgres

```

--------------------------------

### High-Availability Setup - Shell

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Enables high-availability features by configuring database connection pooling, index replication, and integrating with monitoring services like Prometheus and Sentry.

```shell
# Database connection pooling
ALEPH_SQLALCHEMY_POOL_SIZE=20
ALEPH_SQLALCHEMY_POOL_TIMEOUT=60

# Index replication
ALEPH_INDEX_REPLICAS=2

# Enable monitoring
PROMETHEUS_ENABLED=true
SENTRY_DSN=https://xxx@sentry.io/xxx
SENTRY_ENVIRONMENT=production

```

--------------------------------

### Separate Task Database URI Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Allows configuring a separate database URI specifically for procrastinate task data, distinct from the main application data.

```env
OPENALEPH_PROCRASTINATE_DB_URI=postgresql://user:password@host/database
```

--------------------------------

### FollowTheMoney SQLite Store Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Sets the FollowTheMoney (FTM) data store URI to a local SQLite database file. This is the default configuration for simple or local use cases.

```env
sqlite:///followthemoney.store
```

--------------------------------

### Optimization Tip: Use Dehydration with Highlighting

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Combine highlighting with dehydration for potential performance gains. This example enables both features.

```bash
--args "highlight=true&dehydrate=true"
```

--------------------------------

### Task Queue App Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Specifies the Python path to the procrastinate.App instance used by ingest-file for managing distributed tasks.

```env
PROCRASTINATE_APP=ingestors.tasks.app
```

--------------------------------

### Run ingest-file with Docker

Source: https://openaleph.org/docs/lib/ingest-file/usage

This command demonstrates how to execute the ingest-file tool within a Docker container. Ensure necessary volumes are mounted and a PostgreSQL container is attached if required by your setup.

```bash
docker run -it ghcr.io/openaleph/ingest-file ingestors ...
```

--------------------------------

### Docker Compose Configuration for OpenAleph Services

Source: https://openaleph.org/docs/dev-admin-guide/102/docker

This YAML file defines the services required to run OpenAleph, including databases, caching, processing workers, API, and UI. It specifies image versions, volumes, environment variables, and inter-service dependencies. This configuration is intended as a starting point for local development and deployment.

```yaml
services:
  postgres:
    image: postgres:latest
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: aleph
      POSTGRES_PASSWORD: aleph
      POSTGRES_DATABASE: aleph

  elasticsearch:
    image: ghcr.io/openaleph/elasticsearch:latest
    hostname: elasticsearch
    environment:
      - discovery.type=single-node
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data

  redis:
    image: redis:alpine
    command: [ "redis-server", "--save", "3600", "10" ]

  ingest:
    image: ghcr.io/openaleph/ingest-file:latest
    command: procrastinate worker -q ingest
    tmpfs:
      - /tmp:mode=777
    volumes:
      - archive-data:/data
    depends_on:
      - postgres
      - redis
    restart: on-failure
    env_file:
      - aleph.env

  analyze:
    image: ghcr.io/openaleph/ftm-analyze:latest
    command: procrastinate worker -q analyze
    tmpfs:
      - /tmp:mode=777
    depends_on:
      - postgres
      - redis
    restart: on-failure
    env_file:
      - aleph.env

  worker:
    image: ghcr.io/openaleph/openaleph:latest
    command: procrastinate worker -q openaleph
    restart: on-failure
    depends_on:
      - postgres
      - elasticsearch
      - redis
      - ingest
    tmpfs:
      - /tmp
    volumes:
      - archive-data:/data
    env_file:
      - aleph.env

  api:
    image: ghcr.io/openaleph/openaleph:latest
    expose:
      - 8000
    depends_on:
      - postgres
      - elasticsearch
      - redis
      - worker
      - ingest
    tmpfs:
      - /tmp
    volumes:
      - archive-data:/data
    env_file:
      - aleph.env

  ui:
    image: ghcr.io/openaleph/aleph-ui:latest
    depends_on:
      - api
    ports:
      - "8080:8080"

volumes:
  archive-data: {}
  postgres-data: {}
  elasticsearch-data: {}

```

--------------------------------

### Update Docker Services and Configuration

Source: https://openaleph.org/docs/dev-admin-guide/103/upgrade-3x

This section describes updating the Docker setup to include new service definitions and configurations for OpenAleph. It references an example `docker-compose.yml` file and provides specific environment variable settings for database and task queue configurations.

```yaml
# Example docker-compose.yml (reference)
# ...
# FTM_FRAGMENTS_URI and PROCRASTINATE_DB_URI should be set to OPENALEPH_DB_URI
# if using a single database for application data, FollowTheMoney data,
# and the task queue.

```

--------------------------------

### S3-like Storage Archive Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Configures ingest-file to use S3-like object storage for the file archive. Requires specifying the bucket name, endpoint URL (if not AWS), and AWS credentials.

```env
ARCHIVE_TYPE=s3
ARCHIVE_BUCKET=data
ARCHIVE_ENDPOINT_URL=https://my.storage.org  # if not using AWS
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
```

--------------------------------

### OAuth/OIDC Setup - Shell

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Configures OpenAleph to use OAuth/OIDC for authentication, disabling password-based login and specifying client credentials and identity provider details.

```shell
# Enable OAuth
ALEPH_OAUTH=true
ALEPH_OAUTH_HANDLER=oidc
ALEPH_OAUTH_KEY=your-client-id
ALEPH_OAUTH_SECRET=your-client-secret
ALEPH_OAUTH_METADATA_URL=https://idp.example.com/.well-known/openid-configuration
ALEPH_OAUTH_ADMIN_GROUP=aleph-admins

# Disable password login
ALEPH_PASSWORD_LOGIN=false

```

--------------------------------

### Optimization Tip: Reduce Snippets

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Improve query performance by reducing the number of highlighted snippets per document. This example sets 'highlight_count' to 2.

```bash
--args "highlight=true&highlight_count=2"
```

--------------------------------

### Create a Task with OpenAleph Procrastinate

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/howto

Example of creating a task file ('tasks.py') that imports OpenAleph dependencies, sets up the application, and registers a geocode task. This task processes entities and puts them into a writer.

```python
from openaleph_procrastinate.app import make_app
from openaleph_procrastinate.model import DatasetJob
from openaleph_procrastinate.tasks import task

from ftm_geocode.geocode import geocode_proxy
from ftm_geocode.settings import Settings

settings = Settings()
app = make_app(__loader__.name)

@task(app=app)
def geocode(job: DatasetJob):
    with job.get_writer() as bulk:
        for proxy in geocode_proxy(settings.geocoders, job.get_entities()):
            bulk.put(proxy)

```

--------------------------------

### Post-Filters Example

Source: https://openaleph.org/docs/lib/openaleph-search/aggregations

Demonstrates how facets can exclude their own filters to reveal alternative options, allowing users to explore different subsets of data.

```bash
# Dataset facet shows ALL datasets, not just filtered ones
openaleph-search search query-string "company" \
  --args "filter:dataset=collection1&filter:dataset=collection2&facet=dataset"
```

--------------------------------

### Configure Number of Snippets

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Control the number of highlighted snippets displayed per document using the 'highlight_count' parameter. Setting it to 0 returns the full highlighted text. This example requests 5 snippets.

```bash
openaleph-search search query-string "investigation" \
  --args "highlight=true&highlight_count=5"
```

--------------------------------

### Defer Tasks using OpenAleph CLI

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/howto

Command line example using 'opal-procrastinate' to defer tasks. It pipes entities from a source (using 'ftmq') to the defer command, specifying the queue and task.

```shell
ftmq -i https://data.ftm.store/de_lobbyregister/entities.ftm.json -s Address | opal-procrastinate defer-entities -q ftm-geocode -t ftm_geocode.tasks.geocode -d de_lobbyregister

```

--------------------------------

### Query String Syntax Examples (Shell)

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Demonstrates various ways to construct query strings using Lucene syntax for field-specific searches, phrase searches, fuzzy matching, range queries, boosting, and boolean logic.

```shell
# Field-specific search
openaleph-search search query-string "name:john AND countries:us"

# Phrase search
openaleph-search search query-string '"money laundering"'

# Fuzzy search
openaleph-search search query-string "john~0.8"

# Range search
openaleph-search search query-string "date:[2020 TO 2022]"

# Boosting
openaleph-search search query-string "company^2 person"

# Boolean operators
openaleph-search search query-string "john AND smith OR doe"
```

--------------------------------

### OpenAleph CLI Global Options and Environment Variables

Source: https://openaleph.org/docs/user-guide/104/cli

Demonstrates the global options for OpenAleph CLI commands and how to set host and API key using environment variables for convenience. These settings apply to all subsequent commands.

```bash
openaleph --host <URL> --api-key <KEY> [--retries N] <command> [options]

```

```bash
export OPAL_HOST=https://your-aleph-instance.org
export OPAL_API_KEY=your_api_key_here

```

--------------------------------

### Start Procrastinate Workers

Source: https://openaleph.org/docs/lib/ingest-file/usage

This command starts one or more procrastinate workers to handle ingestion tasks. The `-q ingest` flag specifies the queue to listen on, and `--concurrency` sets the number of parallel worker processes.

```bash
procrastinate worker -q ingest --concurrency 8
```

--------------------------------

### Query Parameters (`--args`) Examples (Shell)

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Illustrates the use of the `--args` parameter for specifying query parameters like filters, exclusions, range filters, empty field checks, facets, pagination, sorting, highlighting, and dehydration.

```shell
# Filters
filter:schema=Person
filter:countries=us
filter:dataset=mydata

# Multiple values (OR logic)
filter:schema=Person&filter:schema=Company

# Exclusions
exclude:schema=Page

# Range filters
filter:gte:date_created=2020-01-01
filter:lte:date_created=2022-12-31

# Empty field check
empty:birth_date=true

# Facets
facet=schema&facet=countries
facet_size:schema=50
facet_total:dataset=true

# Pagination
offset=100
limit=50

# Sorting
sort=label:asc
sort=_score:desc

# Highlighting
highlight=true
highlight_count=5

# Performance
dehydrate=true
```

--------------------------------

### Redis Cache Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Configures the Redis connection URL for the runtime cache. If not set, an in-memory cache is used, which lacks persistence.

```env
REDIS_URL=redis://localhost
```

--------------------------------

### OpenAleph Database URI Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Sets the database connection URI for OpenAleph, which may be used by the task queue backend. It can be a PostgreSQL connection string.

```env
OPENALEPH_DB_URI=postgresql://user:password@host/database
```

--------------------------------

### Example Usage for Priority Queuing

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/settings

This bash script demonstrates how to set up priority queues for a specific dataset using environment variables. It exports custom queue names for ingest and analyze services and then launches procrastinate workers with these specific queues, ensuring tasks are processed with the intended priority. The `--one-shot` flag ensures workers stop after completing their assigned tasks.

```bash
# ingest service
export OPENALEPH_INGEST_QUEUE=ingest-prio-dataset
export OPENALEPH_ANALYZE_QUEUE=analyze-prio-dataset
ingestors ingest -d prio_dataset ./documents
procrastinate worker -q ingest-prio-dataset --one-shot  # stop worker after complete

# analyze service
procrastinate worker -q analyze-prio-dataset --one-shot  # stop worker after complete

```

--------------------------------

### Local File Archive Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Configures ingest-file to use a local directory for storing source files. The `ARCHIVE_PATH` specifies the directory, and `ARCHIVE_TYPE` must be set to 'file'.

```env
ARCHIVE_TYPE=file
ARCHIVE_PATH=./data
```

--------------------------------

### Tesseract OCR Data Path Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Sets the environment variable for Tesseract OCR's data path, which is necessary for ingest-file to locate language data files.

```shell
TESSDATA_PREFIX=/usr/share/tesseract-ocr/5/tessdata
```

--------------------------------

### Aggregating and Ingesting Data with `curl`, `ftm`, and OpenAleph CLI

Source: https://openaleph.org/docs/user-guide/104/cli

Provides a bash script example that downloads data using `curl`, ingests it using `ftm store write`, iterates through it with `ftm store iterate`, streams it to OpenAleph using `openaleph write-entities`, and finally cleans up the local dataset. This showcases a complete data pipeline.

```bash
curl -o us_ofac.json hhttps://data.opensanctions.org/datasets/latest/us_sanctions/entities.ftm.json
cat us_ofac.json | ftm store write -d us_ofac
ftm store iterate -d us_ofac | openaleph write-entities -f us_ofac
ftm store delete -d us_ofac

```

--------------------------------

### Configure OpenAleph Environment Variables

Source: https://openaleph.org/docs/user-guide/104/install

Sets environment variables for the OpenAleph CLI to specify the API key and host of the OpenAleph instance. These can be added to your shell's rc file or exported directly.

```bash
export OPAL_API_KEY=your_api_key
export OPAL_HOST=https://your-aleph-instance.org
```

--------------------------------

### Dynamic String Configuration Loading

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Dynamically loads string configuration values from environment variables. Any environment variable prefixed with `ALEPH_STRING_CONFIG_PREFIX` will have its value loaded as a setting attribute, excluding the prefix itself. This allows for flexible configuration without modifying code.

```Shell
export ALEPH_STRING_CONFIG_PREFIX="CUSTOM_"
export CUSTOM_MY_SETTING="value"
```

--------------------------------

### Debug Mode Configuration

Source: https://openaleph.org/docs/lib/ingest-file/setup

Enables debug mode, which utilizes an in-memory store for the task queue. This mode is suitable for local development, testing, or quick one-shot usage, as task data will not persist.

```env
DEBUG=1
```

--------------------------------

### Enable Basic Highlighting

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Enable highlighting for search results by setting the 'highlight' parameter to 'true'. This is the most basic way to activate the feature.

```bash
openaleph-search search query-string "corruption" --args "highlight=true"
```

--------------------------------

### Get File References for Job Entities using Python

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/model

Generates file references for entities associated with the current job. It iterates through entities, extracts content hashes, and yields EntityFileReference objects. The example code demonstrates processing these references using local file paths or file handlers.

```python
from typing import Generator, Any
from openaleph_procrastinate.entity import EntityProxy, EntityFileReference

class JobModel:
    def __init__(self, dataset: str, payload: dict[str, Any] | None = None):
        self.dataset = dataset
        self.payload = payload if payload is not None else {}

    def get_entities(self) -> Generator[EntityProxy, None, None]:
        # Placeholder for get_entities implementation
        # This should yield EntityProxy objects as per the documentation
        if "entities" in self.payload:
            for data in self.payload["entities"]:
                yield EntityProxy(data)

    def get_file_references(self) -> Generator[EntityFileReference, None, None]:
        """
        Get file references per entity from this job

        Example:
            ```python
            # process temporary file paths
            for reference in job.get_file_references():
                with reference.get_local_path() as path:
                    subprocess.run(["command", "-i", str(path)])
                # temporary path will be cleaned up when leaving context

            # process temporary file handlers
            for reference in job.get_file_references():
                with reference.open() as handler:
                    do_something(handler.read())
                # temporary path will be cleaned up when leaving context
            ```

        Yields:
            The file references
        """
        for entity in self.get_entities():
            for content_hash in entity.get("contentHash", quiet=True):
                yield EntityFileReference(
                    dataset=self.dataset,
                    entity=entity,
                    content_hash=content_hash
                )

# Dummy classes for demonstration purposes
class EntityProxy:
    def __init__(self, data: dict[str, Any]):
        self._data = data

    def get(self, key: str, quiet: bool = False) -> Any:
        return self._data.get(key, None)

class EntityFileReference:
    def __init__(self, dataset: str, entity: EntityProxy, content_hash: str):
        self.dataset = dataset
        self.entity = entity
        self.content_hash = content_hash

    def get_local_path(self) -> Any: # Placeholder for ContextManager
        pass

    def open(self) -> Any: # Placeholder for ContextManager
        pass

```

--------------------------------

### Entity Writer Context Manager for FollowTheMoney Datasets (Python)

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/helpers

Provides a context manager to get a BulkLoader for a given dataset, useful for efficiently writing FollowTheMoney entities. The loader is automatically flushed upon exiting the context. It relies on `get_fragments` and `settings.fragments_uri` for store configuration.

```python
@contextmanager
def entity_writer(dataset: str) -> Generator[BulkLoader, None, None]:
    """
    Get the `ftmq.store.fragments.BulkLoader` for the given `dataset`. The
    writer is flushed when leaving the context.
    """
    store = get_fragments(
        dataset, origin=OPAL_ORIGIN, database_uri=settings.fragments_uri
    )
    loader = store.bulk()
    try:
        yield loader
    finally:
        loader.flush()

```

--------------------------------

### Investigative Journalism Example (OpenAleph CLI)

Source: https://openaleph.org/docs/lib/openaleph-search/significant_terms

This command provides an example of using significant terms and text analysis for investigative journalism. It searches for 'minister', filters by 'countries=us', and applies significant analysis to both names and content fields.

```bash
# Key terms in corruption investigation
openaleph-search search query-string "minister" \
  --args "filter:countries=us&facet_significant=names&facet_significant_text=content"
```

--------------------------------

### Get Entities

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/model

Retrieves the entities stored within the job's payload.

```APIDOC
## GET /websites/openaleph/model/get_entities

### Description
Get the entities from the payload.

### Method
GET

### Endpoint
/websites/openaleph/model/get_entities

### Parameters
None

### Request Example
```json
{
  "job_id": "some_job_id"
}
```

### Response
#### Success Response (200)
* **Generator[EntityProxy, None, None]** - A generator yielding EntityProxy objects.

#### Response Example
```json
[
  {
    "id": "entity1",
    "data": "..."
  },
  {
    "id": "entity2",
    "data": "..."
  }
]
```
```

--------------------------------

### Piping OpenAleph Entity Streams with `ftm`

Source: https://openaleph.org/docs/user-guide/104/cli

Illustrates how to pipe entity data streamed from OpenAleph into the `ftm` (FollowtheMoney) CLI tool for further processing, such as exporting to Excel. This showcases integrating OpenAleph with other command-line data tools.

```bash
openaleph stream-entities -f FOREIGN_ID -s Company | ftm export-excel -o entities.xlsx

```

--------------------------------

### Get File References

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/model

Yields file references associated with each entity in the job. This is useful for processing files linked to entities.

```APIDOC
## GET /websites/openaleph/model/get_file_references

### Description
Get file references per entity from this job.

### Method
GET

### Endpoint
/websites/openaleph/model/get_file_references

### Parameters
None

### Request Example
```json
{
  "job_id": "some_job_id"
}
```

### Response
#### Success Response (200)
* **Generator[EntityFileReference, None, None]** - A generator yielding EntityFileReference objects.

#### Response Example
```json
[
  {
    "dataset": "dataset_name",
    "entity": {
      "id": "entity1",
      "data": "..."
    },
    "content_hash": "some_hash"
  }
]
```
```

--------------------------------

### Downloading Collection Data with OpenAleph CLI (`fetchdir`)

Source: https://openaleph.org/docs/user-guide/104/cli

Details the `fetchdir` command for downloading all entities from a collection or a specific entity into a local folder structure. It includes options for specifying the foreign ID, entity ID, output path, and overwriting existing files.

```bash
openaleph fetchdir -f <foreign-id> [-e <entity-id>] [-p <path>] [--overwrite]

```

--------------------------------

### Get Writer

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/model

Provides a context manager for obtaining a writer for the dataset associated with the current job. This is used for bulk loading data.

```APIDOC
## GET /websites/openaleph/model/get_writer

### Description
Get the writer for the dataset of the current job.

### Method
GET

### Endpoint
/websites/openaleph/model/get_writer

### Parameters
None

### Request Example
```json
{
  "job_id": "some_job_id"
}
```

### Response
#### Success Response (200)
* **ContextManager[BulkLoader]** - A context manager that yields a BulkLoader instance.

#### Response Example
```json
{
  "message": "Writer obtained successfully"
}
```
```

--------------------------------

### Raw Output Format (JSON)

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Example of the default raw JSON output from Elasticsearch, including timing and hits information.

```json
{
  "took": 15,
  "hits": {
    "total": {"value": 1234},
    "hits": [...] 
  }
}
```