OpenAleph

https://openaleph.org/docs
Admin
OpenAleph is an open source platform for storing, searching, and analyzing large volumes of...
Tokens:184,611
Snippets:1,625
Trust Score:6.2
Update:1 week ago
Show doc for...
Tokens:
Raw
### Quick Start: Clone, start services, install, migrate, and run Aleph

Source: https://openaleph.org/docs/dev-admin-guide/104

Run these commands in order to set up a local Aleph development environment. Requires Python 3.12+, Node.js 20+, Docker, and Git. After starting, the UI is at http://localhost:3000 and the API at http://localhost:5000.

```bash
# Clone the repository
git clone https://github.com/openaleph/openaleph.git
cd openaleph

# Start services (postgres, elasticsearch, redis, ingest, analyze)
make services

# Install Python dependencies
make dev

# Run database migrations
make upgrade

# In one terminal - run the API
make api

# In another terminal - run the worker
make worker

# In another terminal - run the UI
make ui

```

--------------------------------

### Development setup: local environment variables

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Starts a minimal local development environment with debug mode, a single-user login bypass, and a local PostgreSQL database. ALEPH_SINGLE_USER=true logs you in automatically without needing the admin email or password from the production setup.

```properties
# Minimal for local development
ALEPH_SECRET_KEY=dev-secret-not-for-production
ALEPH_DEBUG=true
ALEPH_SINGLE_USER=true
OPENALEPH_DB_URI=postgresql://aleph:aleph@localhost/aleph
```

--------------------------------

### Priority queuing setup example

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/settings

Shell commands to set environment variables and run workers for a priority queuing setup for a specific dataset. The ingest service uses ingest-prio-dataset and analyze-prio-dataset queues, with one-shot workers that stop after completion.

```bash
# ingest service
export OPENALEPH_INGEST_QUEUE=ingest-prio-dataset
export OPENALEPH_ANALYZE_QUEUE=analyze-prio-dataset
ingestors ingest -d prio_dataset ./documents
procrastinate worker -q ingest-prio-dataset --one-shot  # stop worker after complete

# analyze service
procrastinate worker -q analyze-prio-dataset --one-shot  # stop worker after complete
```

--------------------------------

### Install openaleph-search

Source: https://openaleph.org/docs/lib/openaleph-search

Install the openaleph-search package from PyPI.

```bash
pip install openaleph-search
```

--------------------------------

### Format entities examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of formatting entities from a file and from stdin.

```bash
# Format entities from file
openaleph-search format-entities -d companies -i companies.ijson -o actions.json

# From stdin
cat entities.ijson | openaleph-search format-entities -d mydata > actions.json

```

--------------------------------

### Install openaleph-procrastinate with pip

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/setup

Installs the package and its dependencies using pip.

```bash
pip install openaleph-procrastinate
```

--------------------------------

### Dump actions examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of exporting an entire dataset, exporting a specific schema, and exporting to stdout.

```bash
# Export entire dataset
openaleph-search dump-actions --args "filter:dataset=companies" -o backup.json

# Export specific schema
openaleph-search dump-actions \
  --args "filter:dataset=mydata&filter:schema=Person" \
  -o persons.json

# Export to stdout
openaleph-search dump-actions --args "filter:dataset=test"

```

--------------------------------

### Analyze command examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of using the analyze command with various options: analyzing from stdin, getting tokens only, specifying field and schema, reading from file, and analyzing different fields.

```bash
# Analyze text from stdin
echo "The quick brown fox" | openaleph-search analyze --field content

# Get tokens only
echo "John Smith & Associates" | openaleph-search analyze --tokens-only

# Analyze name field
echo "María José García" | openaleph-search analyze --field name --schema Person

# From file
openaleph-search analyze --field content -i document.txt

# Different analyzers by field
echo "john@example.com" | openaleph-search analyze --field properties.email

```

--------------------------------

### Start Docker services with make services

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Starts PostgreSQL, Elasticsearch, Redis, ingest-file, and ftm-analyze services using Docker Compose.

```bash
make services

```

--------------------------------

### Initialize the database

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/setup

Runs the initial database setup for openaleph-procrastinate.

```bash
openaleph-procrastinate init-db
```

--------------------------------

### Index entities examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of indexing entities from a file, from stdin, and with debug output enabled.

```bash
# Index from file
openaleph-search index-entities -d companies -i companies.ijson

# Index from stdin
cat entities.ijson | openaleph-search index-entities -d mydata

# With debug output
OPENALEPH_SEARCH_INDEXER_DEBUG=1 openaleph-search index-entities \
  -d test -i tests/fixtures/samples.ijson

```

--------------------------------

### Install runtime-only dependencies with make install

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Installs only runtime dependencies, suitable for production.

```bash
make install

```

--------------------------------

### Install system dependencies via apt

Source: https://openaleph.org/docs/lib/ingest-file/setup

Runs the provided script to install system dependencies needed for local development/testing without Docker.

```bash
./contrib/install_deb.sh

```

--------------------------------

### Verify OpenAleph CLI installation

Source: https://openaleph.org/docs/user-guide/104/install

Displays help information for the OpenAleph CLI to confirm the installation.

```bash
openaleph --help
```

--------------------------------

### Install Node dependencies for the frontend

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Installs frontend dependencies using either make dev-ui or npm install.

```bash
make dev-ui
# or
cd ui && npm install

```

--------------------------------

### Install openaleph CLI locally

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Installs the openaleph CLI package without dependencies.

```bash
pip install --no-cache-dir --no-deps .

```

--------------------------------

### Install Python dependencies with make dev

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Installs both runtime and development dependencies for the backend.

```bash
make dev

```

--------------------------------

### Basic highlighting example

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Runs a search for 'money laundering' with highlighting enabled.

```bash
openaleph-search search query-string "money laundering" --args "highlight=true"

```

--------------------------------

### Name keys example

Source: https://openaleph.org/docs/lib/openaleph-search/matching

Shows how names with the same tokens in different order produce the same key.

```text
"John A. Smith Jr." → "jjrsmith"
"Smith John Jr. A." → "jjrsmith"

```

--------------------------------

### Basic Facet Query (facet=FIELD)

Source: https://openaleph.org/docs/lib/openaleph-search/reference/query

Sets up a basic facet query. Use --args with the format facet=FIELD. The single facet example is followed by a multi-facet example with three fields. With an AND filter on a faceted field, the filter moves to post_filter so facet values remain unfiltered.

```bash
--args "facet=schema"
```

```bash
--args "facet=schema&facet=group.countries&facet=dataset"
```

--------------------------------

### openaleph-search search query-string examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of query-string searches: basic search, filters, multiple filters, facets, date range filter, pagination, and saving to file. The --args parameter accepts URL-like arguments used in OpenAleph API calls.

```bash
# Basic search
openaleph-search search query-string "corruption"

# With filters
openaleph-search search query-string "darc" --args "filter:schema=Company"

# Multiple filters
openaleph-search search query-string "person" \
  --args "filter:schema=Person&filter:group.countries=us&highlight=true"

# With facets
openaleph-search search query-string "bank" \
  --args "facet=schema&facet=group.countries&facet_size:schema=20"

# Date range filter
openaleph-search search query-string "transaction" \
  --args "filter:gte:context.created_at=2020-01-01&filter:lte:context.created_at=2022-12-31"

# Pagination
openaleph-search search query-string "investigation" \
  --args "offset=20&limit=50"

# Save to file
openaleph-search search query-string "evidence" -o results.json
```

--------------------------------

### Install openaleph package with pip

Source: https://openaleph.org/docs/user-guide/104/install

Installs the openaleph package using pip. Requires Python 3.6 or higher and pip. Recommended to use a virtual environment.

```bash
pip install openaleph
```

--------------------------------

### analyze command example

Source: https://openaleph.org/docs/lib/ftm-analyze/reference/cli

Example invocation of the analyze command reading from an S3 URI and writing to a PostgreSQL database, with --resolve-mentions and --annotate enabled.

```bash
ftm-analyze analyze \
    -i s3://data/entities.ftm.json \
    -o postgresql:///ftm \
    --resolve-mentions \
    --annotate
```

--------------------------------

### Example .openalephignore patterns

Source: https://openaleph.org/docs/user-guide/104/cli

Example .openalephignore file with glob patterns to skip hidden files, common junk, temporary directories, and log files. Patterns match relative paths; a trailing slash matches directories only.

```text
# Skip hidden files
.*

# Common junk
.DS_Store
Thumbs.db

# Temporary directories
tmp/
build/

# Log files
*.log

```

--------------------------------

### More snippets example

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Runs a search for 'investigation' with highlighting enabled and sets highlight_count to 5.

```bash
openaleph-search search query-string "investigation" \
  --args "highlight=true&highlight_count=5"

```

--------------------------------

### Name parts example

Source: https://openaleph.org/docs/lib/openaleph-search/matching

Shows how a name is split into individual components for partial matching.

```text
"John Smith & Associates" → ["john", "smith", "associates"]

```

--------------------------------

### Initialize the database with make upgrade

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Runs database migrations to set up the schema.

```bash
make upgrade

```

--------------------------------

### Cardinality aggregation: get total count of distinct values

Source: https://openaleph.org/docs/lib/openaleph-search/aggregations

Gets the total count of distinct values using `facet_total` with a facet. The example searches for 'investigation' and facets on `group.countries` with total count enabled.

```bash
openaleph-search search query-string "investigation" \
  --args "facet=group.countries&facet_total:group.countries=true"

```

--------------------------------

### Run the UI development server

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Starts the UI development server at http://localhost:3000 with hot module replacement, proxy to backend, and source maps.

```bash
make ui
# or
cd ui && npm start

```

--------------------------------

### get_file_references()

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/model

Method to get file references per entity from this job. The source text only provides an example, so details are limited.

```APIDOC
## get_file_references()

### Description
Method to get file references per entity from this job. The source text only provides an example, so details are limited.

### Method
instance method

### Parameters
None

### Response
Returns file references per entity from this job.
```

--------------------------------

### Install ftm-analyze with pip

Source: https://openaleph.org/docs/lib/ftm-analyze

Install the ftm-analyze package using pip.

```bash
pip install ftm-analyze

```

--------------------------------

### Create and activate a Python virtual environment

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Creates a virtual environment, activates it, and installs dependencies with make dev.

```bash
python -m venv .venv
source .venv/bin/activate
make dev

```

--------------------------------

### Install ingest-file Python package

Source: https://openaleph.org/docs/lib/ingest-file/setup

Installs the ingest-file Python package from the cloned repository.

```bash
pip install .

```

--------------------------------

### Indexing workflow: format, index, and verify

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Format entities from an ijson file, index the resulting actions, and verify the indexed data with a search query.

```bash
# 1. Format entities
openaleph-search format-entities -d companies -i data.ijson -o actions.json

# 2. Index actions
openaleph-search index-actions -i actions.json

# 3. Verify indexed data
openaleph-search search query-string "*" --args "filter:dataset=companies&limit=10"
```

--------------------------------

### Start worker processes

Source: https://openaleph.org/docs/lib/ingest-file/usage

Start workers to process the ingest queue with a specified concurrency.

```bash
procrastinate worker -q ingest --concurrency 8
```

--------------------------------

### With filters example

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Runs a search for 'corruption' with highlighting enabled and applies filters for schema and country.

```bash
openaleph-search search query-string "corruption" \
  --args "filter:schema=Document&filter:group.countries=us&highlight=true"

```

--------------------------------

### Full text highlighting example

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Runs a search for 'evidence' with highlighting enabled and highlight_count set to 0, which likely means full text highlighting.

```bash
openaleph-search search query-string "evidence" \
  --args "highlight=true&highlight_count=0"

```

--------------------------------

### analyze-text command example

Source: https://openaleph.org/docs/lib/ftm-analyze/reference/cli

Example invocation of analyze-text that pipes a text string via stdin and analyzes it.

```bash
echo "Angela Merkel met Emmanuel Macron in Paris." | ftm-analyze analyze-text
```

--------------------------------

### Common query parameters for --args

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of common query parameters including filters, exclusions, range filters, facets, pagination, sorting, highlighting, and performance options.

```text
# Filters
filter:schema=Person
filter:group.countries=us
filter:dataset=mydata

# Multiple values (OR logic)
filter:schema=Person&filter:schema=Company

# Exclusions
exclude:schema=Page

# Range filters
filter:gte:context.created_at=2020-01-01
filter:lte:context.created_at=2022-12-31

# Empty field check
empty:properties.birthDate=true

# Facets
facet=schema&facet=group.countries
facet_size:schema=50
facet_total:dataset=true

# Pagination
offset=100
limit=50

# Sorting
sort=label:asc
sort=_score:desc

# Highlighting
highlight=true
highlight_count=5

# Performance
dehydrate=true

```

--------------------------------

### Install openaleph-procrastinate with pip or poetry

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/howto

Install the openaleph-procrastinate package as a dependency. Use the pip command for pip-based environments and the poetry command for Poetry-managed projects.

```bash
pip install openaleph-procrastinate
```

```bash
poetry add openaleph-procrastinate
```

--------------------------------

### Create .env file with OpenALeph search settings

Source: https://openaleph.org/docs/lib/openaleph-search/reference/settings

Create a `.env` file in your project directory with the specified environment variables. Settings are automatically loaded from `.env` files.

```plaintext
# .env
OPENALEPH_SEARCH_URI=http://localhost:9200
OPENALEPH_SEARCH_INDEX_PREFIX=myproject
OPENALEPH_SEARCH_INDEX_SHARDS=5
OPENALEPH_SEARCH_INDEXER_CONCURRENCY=4

```

--------------------------------

### openaleph-search search body examples

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Examples of body searches: from file and from stdin. The -i option specifies the JSON query body (default: stdin).

```bash
# From file
openaleph-search search body -i custom_query.json -o results.json

# From stdin
cat query.json | openaleph-search search body
```

--------------------------------

### Optimization tips

Source: https://openaleph.org/docs/lib/openaleph-search/highlighting

Three separate command-line arguments for performance optimization: reducing snippets, limiting analyzed text, and using dehydration with highlighting.

```bash
# Reduce snippets
--args "highlight=true&highlight_count=2"

# Limit analyzed text
--args "highlight=true&max_highlight_analyzed_offset=500000"

# Use dehydration with highlighting
--args "highlight=true&dehydrate=true"

```

--------------------------------

### defer.index example

Source: https://openaleph.org/docs/lib/openaleph-procrastinate/reference/defer

Example of a task that explicitly defers the next stage (indexing) using the defer module. The deferral is controlled by the OPENALEPH_INDEX_DEFER environment variable.

```python
from openaleph_procrastinate import defer

@task(app=app)
def analyze(job: DatasetJob) -> None:
    result = analyze_entities(job.load_entities())
    # defer to index stage
    defer.index(app, job.dataset, result)

```

--------------------------------

### Search workflow: query, match, and highlight

Source: https://openaleph.org/docs/lib/openaleph-search/reference/cli

Run a sequence of search commands: find companies with a query string and facets, get similar entities, and search with highlighting.

```bash
# 1. Find companies
openaleph-search search query-string "offshore" \
  --args "filter:schema=Company&facet=group.countries"

# 2. Get similar entities
openaleph-search search match "Acme Corporation"

# 3. Search with highlighting
openaleph-search search query-string "investigation" \
  --args "highlight=true&highlight_count=3"
```

--------------------------------

### Production setup: minimal environment variables

Source: https://openaleph.org/docs/dev-admin-guide/102/configuration

Defines the required and common instance settings for a production deployment, including the secret key, database URI, app title, UI URL, mail server credentials, and the initial admin email address. Set ALEPH_ADMINS to the email of the first admin user; without it the instance cannot be administered.

```properties
# Required
ALEPH_SECRET_KEY=your-secret-key-here
OPENALEPH_DB_URI=postgresql://user:pass@db:5432/aleph

# Instance details
ALEPH_APP_TITLE=My Investigation Platform
ALEPH_UI_URL=https://aleph.example.com

# Email
ALEPH_MAIL_FROM=noreply@example.com
ALEPH_MAIL_HOST=smtp.example.com
ALEPH_MAIL_USERNAME=smtp-user
ALEPH_MAIL_PASSWORD=smtp-pass

# Admin users
ALEPH_ADMINS=admin@example.com
```

--------------------------------

### Run the API server with make api

Source: https://openaleph.org/docs/dev-admin-guide/104/setup

Starts the API server at http://localhost:5000 with auto-reload, debug mode, and Flask debugger.

```bash
make api

```
Tokens:
Raw