Juditha

https://docs.investigraph.dev/lib/juditha
Admin
Juditha is a fast in-process lookup service for canonical names backed by Tantivy, designed to...
Tokens:12,008
Snippets:149
Trust Score:6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install juditha via pip

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Install the juditha package from PyPI, which includes dependencies like tantivy, rigour, FollowTheMoney, and anystore.

```bash
pip install juditha
```

--------------------------------

### Reset the store

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

To start over, delete the store directory and reload the data.

```bash
rm -rf "$JUDITHA_URI"
juditha load-names -i names.txt
juditha build
```

--------------------------------

### Import models from juditha.model

Source: https://docs.investigraph.dev/lib/juditha/usage/python

juditha.model exposes the data classes you get back from the API. All inherit from pydantic.BaseModel.

```python
from juditha.model import Doc, Result, Mention

```

--------------------------------

### Get a Store with get_store()

Source: https://docs.investigraph.dev/lib/juditha/usage/python

get_store resolves the URI at call time and caches one Store per resolved URI. plyvel allows only one open handle per LevelDB path, so this cache is effectively a per-URI singleton.

```python
from juditha import get_store

store = get_store()                  # uses settings.uri (env var or default)
store = get_store("/var/lib/juditha") # explicit path

```

--------------------------------

### CLI: Slop example

Source: https://docs.investigraph.dev/lib/juditha/extras/percolate

Shows the effect of `--slop 1` on matching a name with an intervening token. Without slop, no hit; with slop, the full span is matched.

```bash
echo "Jane M. Doe was here." > /tmp/doc.txt

juditha percolate -i /tmp/doc.txt              # no hit
juditha percolate -i /tmp/doc.txt --slop 1
# {"text":"Jane M. Doe","start":0,"end":11,"schema":"LegalEntity"}

```

--------------------------------

### Python: Extract mentions from text

Source: https://docs.investigraph.dev/lib/juditha/extras/extract

Uses the Python API to extract mentions from a text. `store.extract` returns a list of `Mention` objects with `text`, `start`, `end`, and `schema_` attributes. The `schema_` attribute is the FollowTheMoney schema label.

```python
from juditha import get_store

store = get_store()
text = "The European Parliament met today, and the European Council convened later."
mentions = store.extract(text)
for m in mentions:
    print(m.text, m.start, m.end, m.schema_)
# European Parliament 4 23 PublicBody
# European Council    44 60 PublicBody

```

--------------------------------

### Load, Build, and Query Workflow

Source: https://docs.investigraph.dev/lib/juditha/usage/cli

Standard sequence for ingesting data, building indexes, and querying. First run load commands, then build, and finally query commands.

```bash
# Ingest
juditha load-names    -i names.txt
juditha load-entities -i entities.ftm.json
juditha load-dataset  -i https://data.ftm.store/eu_authorities/index.json

# Materialise
juditha build

# Query
juditha lookup "European Parliament"
juditha extract   -i document.txt
juditha percolate -i document.txt
```

--------------------------------

### Load a complete dataset

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Loads a dataset config that points at entity resources, fetches entities.ftm.json (and optionally names.txt), and streams them into the aggregator. Then builds.

```bash
juditha load-dataset -i https://data.ftm.store/eu_authorities/index.json
juditha build

```

--------------------------------

### Load names via CLI

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Load names into the juditha database by piping input from stdin.

```bash
printf "Jane Doe\nAlice Smith\nACME Holdings" | juditha load-names
```

--------------------------------

### Load a complete catalog

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Loads a catalog config that points at multiple datasets, walking them all. Can pull in gigabytes of data; load times scale linearly with entity count.

```bash
juditha load-catalog -i https://data.ftm.store/investigraph/catalog.json
juditha build

```

--------------------------------

### Load entities from Python with juditha.io

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Loads entities from Python using the `io` helpers. Each `load_*` helper accepts an optional `sync=True` flag that runs `store.build()` immediately; otherwise build once at the end.

```python
from juditha import get_store
from juditha import io

store = get_store()

# Pick the shape you have:
io.load_proxies("entities.ftm.json", store)
io.load_dataset("https://data.ftm.store/eu_authorities/index.json", store)
io.load_catalog("https://data.ftm.store/investigraph/catalog.json", store)

store.build()

```

--------------------------------

### Store methods: search, extract, percolate

Source: https://docs.investigraph.dev/lib/juditha/usage/python

These are the methods you will use most. search is the best-match search, same engine as juditha.lookup. extract and percolate both return list[Mention].

```python
# Best-match search, same engine as juditha.lookup
result = store.search(query, threshold=None, limit=None, schemata=None)

# Aho-Corasick extraction over fulltext
mentions = store.extract("Some text mentioning Jane Doe.")

# Percolation: reverse search of the names index against the text
mentions = store.percolate("Some text mentioning Jane Doe.", slop=0)

```

--------------------------------

### Pipe and S3 Input URIs

Source: https://docs.investigraph.dev/lib/juditha/usage/cli

Shows how to pipe data from standard input or fetch from an S3 bucket. Commands accept `-i` for input and `-o` for output, with URIs handled by anystore.

```bash
# Pipe two sources
cat names1.txt names2.txt | juditha load-names

# Pull from S3
juditha load-names -i s3://my-bucket/known-orgs.txt
```

--------------------------------

### Import lookup and get_store

Source: https://docs.investigraph.dev/lib/juditha/usage/python

The package re-exports two helpers: lookup and get_store. lookup is a memoised top-level convenience for one query, one best match; get_store returns the cached Store object for fine-grained access.

```python
from juditha import lookup, get_store

```

--------------------------------

### Build the index via CLI

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Build the tantivy index and Aho-Corasick extractor. This must be rerun whenever new data is loaded.

```bash
juditha build
```

--------------------------------

### Run benchmark with percolate block limit

Source: https://docs.investigraph.dev/lib/juditha/benchmark

Runs the benchmark script with a custom JUDITHA_PERCOLATE_BLOCK_LIMIT environment variable set to 100000.

```bash
JUDITHA_PERCOLATE_BLOCK_LIMIT=100000 .venv/bin/python contrib/benchmark/run.py
```

--------------------------------

### CLI: Basic percolate command

Source: https://docs.investigraph.dev/lib/juditha/extras/percolate

Runs the percolate command on a document file, outputting a JSON object with the matched text, offsets, and schema.

```bash
echo "The Jane Doe meeting was at 3pm." > /tmp/doc.txt
juditha percolate -i /tmp/doc.txt
# {"text":"Jane Doe","start":4,"end":12,"schema":"LegalEntity"}

```

--------------------------------

### Load entities from file, stream, or URL

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Loads entities from a file, stdin, or URL. After loading, you must run `juditha build` to build the aggregator.

```bash
juditha load-entities -i entities.ftm.json
cat entities.ftm.json | juditha load-entities
juditha load-entities -i https://data.ftm.store/eu_authorities/entities.ftm.json

```

--------------------------------

### Run benchmark with default names and text

Source: https://docs.investigraph.dev/lib/juditha/benchmark

Runs the benchmark script with default names and text. The script times the import separately, warms the Aho-Corasick automaton, and prints the symmetric difference of unique-name sets.

```bash
.venv/bin/python contrib/benchmark/run.py                                # default names + text
```

--------------------------------

### Load names from a file

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

Use the juditha load-names command to load names from a local file.

```bash
juditha load-names -i names.txt
```

--------------------------------

### Run benchmark with different text

Source: https://docs.investigraph.dev/lib/juditha/benchmark

Runs the benchmark script with a different input text file (vsbericht_2024.txt).

```bash
.venv/bin/python contrib/benchmark/run.py vsbericht_2024.txt             # different text
```

--------------------------------

### Set Storage URI with Environment Variable

Source: https://docs.investigraph.dev/lib/juditha/usage/cli

Overrides the default storage location (`juditha.db` in the current working directory) with an alternative path.

```bash
export JUDITHA_URI=/var/lib/juditha
```

--------------------------------

### Perform lookups via Python

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Use the juditha Python package to perform lookups. The result is a juditha.model.Result object.

```python
from juditha import lookup

# Hits at the default threshold (0.97)
res = lookup("Jane Doe")
assert res is not None
assert "Jane Doe" in res.names

# Same canonical key, different surface form
assert lookup("doe, jane") is not None

# Below threshold returns None
assert lookup("Jane Dae") is None
assert lookup("Jane Dae", threshold=0.5) is not None
```

--------------------------------

### Configure JUDITHA_URI for persistent storage

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Set the JUDITHA_URI environment variable to specify a custom directory for the database state.

```bash
export JUDITHA_URI=/var/lib/juditha
juditha build
juditha lookup "Jane Doe"
```

--------------------------------

### CLI: Pipe from a document store

Source: https://docs.investigraph.dev/lib/juditha/extras/extract

Pipes a document from a URL into `juditha extract` and writes the mention list to a file. The `-o` flag writes to a file, URL, or stdout, one JSON object per line.

```bash
# Pipe straight from a document store
curl -s https://example.org/report.txt | juditha extract -o mentions.json

```

--------------------------------

### Python: Basic percolate call

Source: https://docs.investigraph.dev/lib/juditha/extras/percolate

Uses the `store.percolate` method to find mentions in a text. The output is a list of `Mention` objects.

```python
from juditha import get_store

store = get_store()
mentions = store.percolate("The Jane Doe meeting was at 3pm.")
# [Mention(text='Jane Doe', start=4, end=12, schema_='LegalEntity')]

```

--------------------------------

### Push entities one at a time

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Pushes entities one at a time programmatically. `put` buffers up to 10,000 entries and auto-flushes; `flush()` is only needed at the end of a batch (or before `build`).

```python
from ftmq.util import make_entity
from juditha import get_store

store = get_store()
entity = make_entity({
    "id": "j",
    "schema": "Person",
    "properties": {
        "name": ["Jane Doe"],
        "nationality": ["mt"],
        "alias": ["Jani"],
    },
})
store.aggregator.put(entity)
store.aggregator.flush()
store.build()

```

--------------------------------

### Load names using Python

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

Use the juditha.io module to load names from a file or stdin within a Python script.

```python
from juditha import get_store
from juditha import io

store = get_store()
io.load_names("names.txt", store)
# or pipe-style from stdin:
io.load_names("-", store)

store.build()
```

--------------------------------

### Writing into the store

Source: https://docs.investigraph.dev/lib/juditha/usage/python

Stream FTM entities into the aggregator or push individual entities, then rebuild the searchable index and extractor. store.build() deletes and recreates the tantivy index, then iterates the aggregator once feeding both tantivy and the Aho-Corasick extractor.

```python
from juditha import get_store
from juditha import io

store = get_store()

# Either: stream FTM entities into the aggregator
io.load_proxies("entities.ftm.json", store)

# ...or push individual entities
store.aggregator.put(some_entity_proxy)
store.aggregator.flush()

# Then rebuild the searchable index + extractor
store.build()

```

--------------------------------

### Python: Slop parameter

Source: https://docs.investigraph.dev/lib/juditha/extras/percolate

Demonstrates the `slop` parameter in the Python API, allowing intervening tokens between name parts.

```python
mentions = store.percolate("Jane M. Doe was here.", slop=1)
# [Mention(text='Jane M. Doe', start=0, end=11, schema_='LegalEntity')]

```

--------------------------------

### Perform lookups via CLI

Source: https://docs.investigraph.dev/lib/juditha/quickstart

Execute lookups against the index. The default fuzzy threshold is 0.97, which can be adjusted using the --threshold flag.

```bash
juditha lookup "Jane Doe"
juditha lookup "jane doe"         # case-insensitive, matches
juditha lookup "doe, jane"        # order-independent, matches
juditha lookup "Jane Dae"         # one-character typo, requires lower threshold
juditha lookup "Jane Dae" --threshold 0.5
```

--------------------------------

### Use lookup() with optional parameters

Source: https://docs.investigraph.dev/lib/juditha/usage/python

lookup is a memoised convenience function wrapped in lru_cache(100_000), so repeated queries with the same args are O(1). It returns Result | None. The schemata parameter must be a tuple.

```python
from juditha import lookup

res = lookup(
    "Jane Doe",
    threshold=0.95,           # optional, defaults to settings.fuzzy_threshold
    uri=None,                 # optional, defaults to settings.uri
    schemata=("Person",),     # optional FollowTheMoney schema narrowing, must be a tuple
)

```

--------------------------------

### Load names from remote URLs

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

The path argument supports local paths, HTTP, and S3 URLs.

```bash
juditha load-names -i https://example.org/known-orgs.txt
juditha load-names -i s3://my-bucket/persons-of-interest.txt
```

--------------------------------

### Build aggregator

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Builds the aggregator after loading entities. Required after any load operation.

```bash
juditha build

```

--------------------------------

### Name list format

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

The input format is a flat list of names, one per line, with no header.

```text
Jane Doe
Alice Smith
ACME Holdings GmbH
European Parliament
European Central Bank

```

--------------------------------

### Load names from stdin

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

Pipe data from other commands or files into juditha load-names using the default stdin input.

```bash
printf "Jane Doe\nAlice Smith\n" | juditha load-names
cat names1.txt names2.txt | juditha load-names
```

--------------------------------

### Run benchmark with different corpus

Source: https://docs.investigraph.dev/lib/juditha/benchmark

Runs the benchmark script with a different name corpus (peps.txt).

```bash
.venv/bin/python contrib/benchmark/run.py --names peps.txt               # different corpus
```

--------------------------------

### CLI: Extract names from a file

Source: https://docs.investigraph.dev/lib/juditha/extras/extract

Runs `juditha extract` on a text file, outputting JSON mentions to stdout. The `-i` flag reads from a file, URL, or stdin.

```bash
echo "The European Parliament met today." > /tmp/doc.txt
juditha extract -i /tmp/doc.txt
# {"text":"European Parliament","start":4,"end":23,"schema":"PublicBody"}

```

--------------------------------

### Inspect aggregator clusters

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Inspects the aggregator, which clusters by `name_key` (canonical, order-independent form). Each line is one cluster as JSON with the union of names, aliases, countries, and schemata.

```bash
juditha iterate -o docs.jsonl

```

--------------------------------

### Shutting down: store.close()

Source: https://docs.investigraph.dev/lib/juditha/usage/python

In one-shot scripts or tests that switch URIs, call store.close() to flush pending writes, drain tantivy merges, and close the LevelDB handle.

```python
store = get_store("/tmp/jtest")
# ... do work ...
store.close()

```

--------------------------------

### Stream names from curl

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

Stream a list of names directly from a URL using curl.

```bash
curl -s https://example.org/list.txt | juditha load-names
```

--------------------------------

### Load names with custom schema

Source: https://docs.investigraph.dev/lib/juditha/load-data/names

Specify a custom FTM schema for loaded names using the schema argument.

```python
io.load_names("companies.txt", store, schema="Company")
io.load_names("people.txt",    store, schema="Person")
```

--------------------------------

### Result object attributes

Source: https://docs.investigraph.dev/lib/juditha/usage/python

Result extends Doc with query, score, took (ms), caption (best display name via rigour's pick_name), and common_schema (FollowTheMoney schema reduction). The key is order-independent canonical key.

```python
res.key            # "doe jane" – the order-independent canonical key
res.names          # {"Jane Doe"}
res.aliases        # set of alternate surface forms
res.countries      # ISO country codes derived from the FTM entity
res.schemata       # FTM schemata that contributed to this cluster
res.score          # similarity in [0, 1]
res.caption        # human-readable display name
res.common_schema  # e.g. "Person", "Organization", "LegalEntity"

```

--------------------------------

### Entity shape: newline-delimited JSON

Source: https://docs.investigraph.dev/lib/juditha/load-data/entities

Newline-delimited JSON lines, each representing one FollowTheMoney entity (proxy) with schema, name, and optional alias/country properties. Use this shape when loading entities from a registry, sanctions list, or dataset that carries metadata to filter on at query time.

```json
{"id": "ep-1", "schema": "PublicBody", "properties": {"name": ["European Parliament"], "country": ["eu"]}}
{"id": "ec-1", "schema": "PublicBody", "properties": {"name": ["European Council"], "alias": ["EU Council"]}}
{"id": "jane-1", "schema": "Person", "properties": {"name": ["Jane Doe"], "nationality": ["mt"], "alias": ["Jani"]}}

```
Tokens:
Raw