### Install juditha via pip Source: https://docs.investigraph.dev/lib/juditha/quickstart Install the juditha package from PyPI, which includes dependencies like tantivy, rigour, FollowTheMoney, and anystore. ```bash pip install juditha ``` -------------------------------- ### Reset the store Source: https://docs.investigraph.dev/lib/juditha/load-data/names To start over, delete the store directory and reload the data. ```bash rm -rf "$JUDITHA_URI" juditha load-names -i names.txt juditha build ``` -------------------------------- ### Import models from juditha.model Source: https://docs.investigraph.dev/lib/juditha/usage/python juditha.model exposes the data classes you get back from the API. All inherit from pydantic.BaseModel. ```python from juditha.model import Doc, Result, Mention ``` -------------------------------- ### Get a Store with get_store() Source: https://docs.investigraph.dev/lib/juditha/usage/python get_store resolves the URI at call time and caches one Store per resolved URI. plyvel allows only one open handle per LevelDB path, so this cache is effectively a per-URI singleton. ```python from juditha import get_store store = get_store() # uses settings.uri (env var or default) store = get_store("/var/lib/juditha") # explicit path ``` -------------------------------- ### CLI: Slop example Source: https://docs.investigraph.dev/lib/juditha/extras/percolate Shows the effect of `--slop 1` on matching a name with an intervening token. Without slop, no hit; with slop, the full span is matched. ```bash echo "Jane M. Doe was here." > /tmp/doc.txt juditha percolate -i /tmp/doc.txt # no hit juditha percolate -i /tmp/doc.txt --slop 1 # {"text":"Jane M. Doe","start":0,"end":11,"schema":"LegalEntity"} ``` -------------------------------- ### Python: Extract mentions from text Source: https://docs.investigraph.dev/lib/juditha/extras/extract Uses the Python API to extract mentions from a text. `store.extract` returns a list of `Mention` objects with `text`, `start`, `end`, and `schema_` attributes. The `schema_` attribute is the FollowTheMoney schema label. ```python from juditha import get_store store = get_store() text = "The European Parliament met today, and the European Council convened later." mentions = store.extract(text) for m in mentions: print(m.text, m.start, m.end, m.schema_) # European Parliament 4 23 PublicBody # European Council 44 60 PublicBody ``` -------------------------------- ### Load, Build, and Query Workflow Source: https://docs.investigraph.dev/lib/juditha/usage/cli Standard sequence for ingesting data, building indexes, and querying. First run load commands, then build, and finally query commands. ```bash # Ingest juditha load-names -i names.txt juditha load-entities -i entities.ftm.json juditha load-dataset -i https://data.ftm.store/eu_authorities/index.json # Materialise juditha build # Query juditha lookup "European Parliament" juditha extract -i document.txt juditha percolate -i document.txt ``` -------------------------------- ### Load a complete dataset Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Loads a dataset config that points at entity resources, fetches entities.ftm.json (and optionally names.txt), and streams them into the aggregator. Then builds. ```bash juditha load-dataset -i https://data.ftm.store/eu_authorities/index.json juditha build ``` -------------------------------- ### Load names via CLI Source: https://docs.investigraph.dev/lib/juditha/quickstart Load names into the juditha database by piping input from stdin. ```bash printf "Jane Doe\nAlice Smith\nACME Holdings" | juditha load-names ``` -------------------------------- ### Load a complete catalog Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Loads a catalog config that points at multiple datasets, walking them all. Can pull in gigabytes of data; load times scale linearly with entity count. ```bash juditha load-catalog -i https://data.ftm.store/investigraph/catalog.json juditha build ``` -------------------------------- ### Load entities from Python with juditha.io Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Loads entities from Python using the `io` helpers. Each `load_*` helper accepts an optional `sync=True` flag that runs `store.build()` immediately; otherwise build once at the end. ```python from juditha import get_store from juditha import io store = get_store() # Pick the shape you have: io.load_proxies("entities.ftm.json", store) io.load_dataset("https://data.ftm.store/eu_authorities/index.json", store) io.load_catalog("https://data.ftm.store/investigraph/catalog.json", store) store.build() ``` -------------------------------- ### Store methods: search, extract, percolate Source: https://docs.investigraph.dev/lib/juditha/usage/python These are the methods you will use most. search is the best-match search, same engine as juditha.lookup. extract and percolate both return list[Mention]. ```python # Best-match search, same engine as juditha.lookup result = store.search(query, threshold=None, limit=None, schemata=None) # Aho-Corasick extraction over fulltext mentions = store.extract("Some text mentioning Jane Doe.") # Percolation: reverse search of the names index against the text mentions = store.percolate("Some text mentioning Jane Doe.", slop=0) ``` -------------------------------- ### Pipe and S3 Input URIs Source: https://docs.investigraph.dev/lib/juditha/usage/cli Shows how to pipe data from standard input or fetch from an S3 bucket. Commands accept `-i` for input and `-o` for output, with URIs handled by anystore. ```bash # Pipe two sources cat names1.txt names2.txt | juditha load-names # Pull from S3 juditha load-names -i s3://my-bucket/known-orgs.txt ``` -------------------------------- ### Import lookup and get_store Source: https://docs.investigraph.dev/lib/juditha/usage/python The package re-exports two helpers: lookup and get_store. lookup is a memoised top-level convenience for one query, one best match; get_store returns the cached Store object for fine-grained access. ```python from juditha import lookup, get_store ``` -------------------------------- ### Build the index via CLI Source: https://docs.investigraph.dev/lib/juditha/quickstart Build the tantivy index and Aho-Corasick extractor. This must be rerun whenever new data is loaded. ```bash juditha build ``` -------------------------------- ### Run benchmark with percolate block limit Source: https://docs.investigraph.dev/lib/juditha/benchmark Runs the benchmark script with a custom JUDITHA_PERCOLATE_BLOCK_LIMIT environment variable set to 100000. ```bash JUDITHA_PERCOLATE_BLOCK_LIMIT=100000 .venv/bin/python contrib/benchmark/run.py ``` -------------------------------- ### CLI: Basic percolate command Source: https://docs.investigraph.dev/lib/juditha/extras/percolate Runs the percolate command on a document file, outputting a JSON object with the matched text, offsets, and schema. ```bash echo "The Jane Doe meeting was at 3pm." > /tmp/doc.txt juditha percolate -i /tmp/doc.txt # {"text":"Jane Doe","start":4,"end":12,"schema":"LegalEntity"} ``` -------------------------------- ### Load entities from file, stream, or URL Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Loads entities from a file, stdin, or URL. After loading, you must run `juditha build` to build the aggregator. ```bash juditha load-entities -i entities.ftm.json cat entities.ftm.json | juditha load-entities juditha load-entities -i https://data.ftm.store/eu_authorities/entities.ftm.json ``` -------------------------------- ### Run benchmark with default names and text Source: https://docs.investigraph.dev/lib/juditha/benchmark Runs the benchmark script with default names and text. The script times the import separately, warms the Aho-Corasick automaton, and prints the symmetric difference of unique-name sets. ```bash .venv/bin/python contrib/benchmark/run.py # default names + text ``` -------------------------------- ### Load names from a file Source: https://docs.investigraph.dev/lib/juditha/load-data/names Use the juditha load-names command to load names from a local file. ```bash juditha load-names -i names.txt ``` -------------------------------- ### Run benchmark with different text Source: https://docs.investigraph.dev/lib/juditha/benchmark Runs the benchmark script with a different input text file (vsbericht_2024.txt). ```bash .venv/bin/python contrib/benchmark/run.py vsbericht_2024.txt # different text ``` -------------------------------- ### Set Storage URI with Environment Variable Source: https://docs.investigraph.dev/lib/juditha/usage/cli Overrides the default storage location (`juditha.db` in the current working directory) with an alternative path. ```bash export JUDITHA_URI=/var/lib/juditha ``` -------------------------------- ### Perform lookups via Python Source: https://docs.investigraph.dev/lib/juditha/quickstart Use the juditha Python package to perform lookups. The result is a juditha.model.Result object. ```python from juditha import lookup # Hits at the default threshold (0.97) res = lookup("Jane Doe") assert res is not None assert "Jane Doe" in res.names # Same canonical key, different surface form assert lookup("doe, jane") is not None # Below threshold returns None assert lookup("Jane Dae") is None assert lookup("Jane Dae", threshold=0.5) is not None ``` -------------------------------- ### Configure JUDITHA_URI for persistent storage Source: https://docs.investigraph.dev/lib/juditha/quickstart Set the JUDITHA_URI environment variable to specify a custom directory for the database state. ```bash export JUDITHA_URI=/var/lib/juditha juditha build juditha lookup "Jane Doe" ``` -------------------------------- ### CLI: Pipe from a document store Source: https://docs.investigraph.dev/lib/juditha/extras/extract Pipes a document from a URL into `juditha extract` and writes the mention list to a file. The `-o` flag writes to a file, URL, or stdout, one JSON object per line. ```bash # Pipe straight from a document store curl -s https://example.org/report.txt | juditha extract -o mentions.json ``` -------------------------------- ### Python: Basic percolate call Source: https://docs.investigraph.dev/lib/juditha/extras/percolate Uses the `store.percolate` method to find mentions in a text. The output is a list of `Mention` objects. ```python from juditha import get_store store = get_store() mentions = store.percolate("The Jane Doe meeting was at 3pm.") # [Mention(text='Jane Doe', start=4, end=12, schema_='LegalEntity')] ``` -------------------------------- ### Push entities one at a time Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Pushes entities one at a time programmatically. `put` buffers up to 10,000 entries and auto-flushes; `flush()` is only needed at the end of a batch (or before `build`). ```python from ftmq.util import make_entity from juditha import get_store store = get_store() entity = make_entity({ "id": "j", "schema": "Person", "properties": { "name": ["Jane Doe"], "nationality": ["mt"], "alias": ["Jani"], }, }) store.aggregator.put(entity) store.aggregator.flush() store.build() ``` -------------------------------- ### Load names using Python Source: https://docs.investigraph.dev/lib/juditha/load-data/names Use the juditha.io module to load names from a file or stdin within a Python script. ```python from juditha import get_store from juditha import io store = get_store() io.load_names("names.txt", store) # or pipe-style from stdin: io.load_names("-", store) store.build() ``` -------------------------------- ### Writing into the store Source: https://docs.investigraph.dev/lib/juditha/usage/python Stream FTM entities into the aggregator or push individual entities, then rebuild the searchable index and extractor. store.build() deletes and recreates the tantivy index, then iterates the aggregator once feeding both tantivy and the Aho-Corasick extractor. ```python from juditha import get_store from juditha import io store = get_store() # Either: stream FTM entities into the aggregator io.load_proxies("entities.ftm.json", store) # ...or push individual entities store.aggregator.put(some_entity_proxy) store.aggregator.flush() # Then rebuild the searchable index + extractor store.build() ``` -------------------------------- ### Python: Slop parameter Source: https://docs.investigraph.dev/lib/juditha/extras/percolate Demonstrates the `slop` parameter in the Python API, allowing intervening tokens between name parts. ```python mentions = store.percolate("Jane M. Doe was here.", slop=1) # [Mention(text='Jane M. Doe', start=0, end=11, schema_='LegalEntity')] ``` -------------------------------- ### Perform lookups via CLI Source: https://docs.investigraph.dev/lib/juditha/quickstart Execute lookups against the index. The default fuzzy threshold is 0.97, which can be adjusted using the --threshold flag. ```bash juditha lookup "Jane Doe" juditha lookup "jane doe" # case-insensitive, matches juditha lookup "doe, jane" # order-independent, matches juditha lookup "Jane Dae" # one-character typo, requires lower threshold juditha lookup "Jane Dae" --threshold 0.5 ``` -------------------------------- ### Use lookup() with optional parameters Source: https://docs.investigraph.dev/lib/juditha/usage/python lookup is a memoised convenience function wrapped in lru_cache(100_000), so repeated queries with the same args are O(1). It returns Result | None. The schemata parameter must be a tuple. ```python from juditha import lookup res = lookup( "Jane Doe", threshold=0.95, # optional, defaults to settings.fuzzy_threshold uri=None, # optional, defaults to settings.uri schemata=("Person",), # optional FollowTheMoney schema narrowing, must be a tuple ) ``` -------------------------------- ### Load names from remote URLs Source: https://docs.investigraph.dev/lib/juditha/load-data/names The path argument supports local paths, HTTP, and S3 URLs. ```bash juditha load-names -i https://example.org/known-orgs.txt juditha load-names -i s3://my-bucket/persons-of-interest.txt ``` -------------------------------- ### Build aggregator Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Builds the aggregator after loading entities. Required after any load operation. ```bash juditha build ``` -------------------------------- ### Name list format Source: https://docs.investigraph.dev/lib/juditha/load-data/names The input format is a flat list of names, one per line, with no header. ```text Jane Doe Alice Smith ACME Holdings GmbH European Parliament European Central Bank ``` -------------------------------- ### Load names from stdin Source: https://docs.investigraph.dev/lib/juditha/load-data/names Pipe data from other commands or files into juditha load-names using the default stdin input. ```bash printf "Jane Doe\nAlice Smith\n" | juditha load-names cat names1.txt names2.txt | juditha load-names ``` -------------------------------- ### Run benchmark with different corpus Source: https://docs.investigraph.dev/lib/juditha/benchmark Runs the benchmark script with a different name corpus (peps.txt). ```bash .venv/bin/python contrib/benchmark/run.py --names peps.txt # different corpus ``` -------------------------------- ### CLI: Extract names from a file Source: https://docs.investigraph.dev/lib/juditha/extras/extract Runs `juditha extract` on a text file, outputting JSON mentions to stdout. The `-i` flag reads from a file, URL, or stdin. ```bash echo "The European Parliament met today." > /tmp/doc.txt juditha extract -i /tmp/doc.txt # {"text":"European Parliament","start":4,"end":23,"schema":"PublicBody"} ``` -------------------------------- ### Inspect aggregator clusters Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Inspects the aggregator, which clusters by `name_key` (canonical, order-independent form). Each line is one cluster as JSON with the union of names, aliases, countries, and schemata. ```bash juditha iterate -o docs.jsonl ``` -------------------------------- ### Shutting down: store.close() Source: https://docs.investigraph.dev/lib/juditha/usage/python In one-shot scripts or tests that switch URIs, call store.close() to flush pending writes, drain tantivy merges, and close the LevelDB handle. ```python store = get_store("/tmp/jtest") # ... do work ... store.close() ``` -------------------------------- ### Stream names from curl Source: https://docs.investigraph.dev/lib/juditha/load-data/names Stream a list of names directly from a URL using curl. ```bash curl -s https://example.org/list.txt | juditha load-names ``` -------------------------------- ### Load names with custom schema Source: https://docs.investigraph.dev/lib/juditha/load-data/names Specify a custom FTM schema for loaded names using the schema argument. ```python io.load_names("companies.txt", store, schema="Company") io.load_names("people.txt", store, schema="Person") ``` -------------------------------- ### Result object attributes Source: https://docs.investigraph.dev/lib/juditha/usage/python Result extends Doc with query, score, took (ms), caption (best display name via rigour's pick_name), and common_schema (FollowTheMoney schema reduction). The key is order-independent canonical key. ```python res.key # "doe jane" – the order-independent canonical key res.names # {"Jane Doe"} res.aliases # set of alternate surface forms res.countries # ISO country codes derived from the FTM entity res.schemata # FTM schemata that contributed to this cluster res.score # similarity in [0, 1] res.caption # human-readable display name res.common_schema # e.g. "Person", "Organization", "LegalEntity" ``` -------------------------------- ### Entity shape: newline-delimited JSON Source: https://docs.investigraph.dev/lib/juditha/load-data/entities Newline-delimited JSON lines, each representing one FollowTheMoney entity (proxy) with schema, name, and optional alias/country properties. Use this shape when loading entities from a registry, sanctions list, or dataset that carries metadata to filter on at query time. ```json {"id": "ep-1", "schema": "PublicBody", "properties": {"name": ["European Parliament"], "country": ["eu"]}} {"id": "ec-1", "schema": "PublicBody", "properties": {"name": ["European Council"], "alias": ["EU Council"]}} {"id": "jane-1", "schema": "Person", "properties": {"name": ["Jane Doe"], "nationality": ["mt"], "alias": ["Jani"]}} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.