### Install followthemoney and verify CLI

Source: https://followthemoney.tech/docs/cli

Install the package via pip and check the help menu to confirm installation.

```bash
pip install followthemoney
ftm --help
```

--------------------------------

### Install followthemoney-store with SQLite

Source: https://followthemoney.tech/docs/cli

Install the store package with default SQLite support.

```bash
pip install followthemoney-store
```

--------------------------------

### Install followthemoney-store with PostgreSQL

Source: https://followthemoney.tech/docs/cli

Install the store package with PostgreSQL support and configure the database URI.

```bash
pip install followthemoney-store[postgresql]
export FTM_STORE_URI=postgresql://localhost/followthemoney
```

--------------------------------

### Query Examples

Source: https://followthemoney.tech/docs/metadata

Various examples of dataset query structures.

```json
"us_ofac_sdn"
```

```json
["us_ofac_sdn", "eu_fsf", "gb_hmt_sanctions"]
```

```json
"#list.sanction"
```

```json
{"and": ["#issuer.eu", "#list.sanction"]}
```

```json
{"and": [
    {"or": ["#issuer.west", "#list.sanction", "#list.debarment"]},
    {"not": "lt_fiu_sanctions"},
    {"not": "#issuer.ru"}
]}
```

```json
{"and": [
    "#list.sanction",
    {"not": {"or": ["lt_fiu_sanctions", "ru_fedsfm"]}}
]}
```

--------------------------------

### Install TypeScript/JavaScript library

Source: https://followthemoney.tech/libraries

Use npm to install the FollowTheMoney package for TypeScript and JavaScript environments.

```bash
npm install @opensanctions/followthemoney
```

--------------------------------

### Advanced Property Mapping Hacks

Source: https://followthemoney.tech/docs/mappings

Examples for handling multiple columns, joining strings, setting constants, and formatting dates.

```yaml
# Setting multiple values for a property:
properties:
  name:
    columns:
    - person_name
    - maiden_name

# Merging values ad-hoc:
properties:
  name:
    columns:
    - first_name
    - last_name
    join: " "

# Setting a constant value:
properties:
  country:
    literal: "SS"

# Defining a date format:
properties:
  birthDate:
    column: dob
    format: "%d.%m.%Y"
```

--------------------------------

### Install ICU dependencies

Source: https://followthemoney.tech/docs/cli

Required for transliteration and fuzzy name comparison features.

```bash
apt install libicu-dev
pip install pyicu
```

--------------------------------

### Entity Key Collision Example

Source: https://followthemoney.tech/docs/mappings

An example of an insufficient key configuration that leads to entity collisions when names are not unique.

```yaml
entities:
  person:
    schema: Person
    keys:
      - FirstName
      - LastName
    properties:
      firstName:
        column: FirstName
      lastName:
        column: LastName
      birthDate:
        column: DoB
```

--------------------------------

### Example entity format

Source: https://followthemoney.tech/docs/cli

The standard newline-delimited JSON format for entities.

```json
{"id": "person-1", "schema": "Person", "properties": {"name": ["Jane Doe"]}}
```

--------------------------------

### Complex Mapping Configuration

Source: https://followthemoney.tech/docs/mappings

Example of mapping multiple CSV sources to Company entities with specific property assignments.

```yaml
- csv_url: http://assets.data.occrp.org/tools/aleph/fixtures/md_companies/licensed.csv
  entities:
    company:
      schema: Company
      keys:
        - Company_IDNO
        - Company_Name
      properties:
        sector:
          column: Denumire
- csv_url: http://assets.data.occrp.org/tools/aleph/fixtures/md_companies/unlicensed.csv
  entities:
    company:
      schema: Company
      keys:
        - Company_IDNO
        - Company_Name
      properties:
        sector:
          column: Denumire
        caemCode:
          column: Cod_CAEM
```

--------------------------------

### Define SQL database mapping

Source: https://followthemoney.tech/docs/mappings

Configures a PostgreSQL database connection with table aliases and join conditions. Ensure required Python drivers for the specific database backend are installed.

```yaml
za_cipc:
  queries:
    - database: postgresql://localhost/cipc
      tables:
        - table: za_cipc_companies
          alias: companies
        - table: za_cipc_directors
          alias: directors
      joins:
        - left: companies.regno
          right: directors.company_regno
```

--------------------------------

### Get entity caption

Source: https://followthemoney.tech/python/statements

Calculates the user-facing label for the entity based on schema caption properties.

```python
@property
def caption(self) -> str:
    """The user-facing label to be used for this entity. This checks a list
    of properties defined by the schema (caption) and returns the first
    available value. If no caption is available, return the schema label.

    This implementation prefers statements where the language property is that
    of the preferred system language."""
    if self._caption is None:
        for prop_ in self.schema.caption:
            stmts = self._statements.get(prop_)
            if stmts is None:
                continue
            prop = self.schema.properties[prop_]
            if prop.type == registry.name and len(stmts) > 1:
                values = [LangStr(s.value, lang=s.lang) for s in stmts]
                name = pick_lang_name(values)
                if name is not None:
                    self._caption = name
                    return self._caption

            for stmt in sorted(stmts):
                self._caption = stmt.value
                return self._caption
        if self._caption is None:
            self._caption = self.schema.label
    return self._caption
```

--------------------------------

### Get Entity Length

Source: https://followthemoney.tech/python/statements

Returns the count of statements plus one.

```python
def __len__(self) -> int:
    return len(list(self._iter_stmt())) + 1
```

--------------------------------

### Interact with Schemas and Entities

Source: https://followthemoney.tech/python

Demonstrates loading the model, creating entity proxies, assigning properties, and serializing data.

```python
# Load the standard instance of the model
from followthemoney import model

## Schema metadata
# Access a schema metadata object
schema = model.get('Person')

# Access a property metadata object
prop = schema.get('birthDate')

## Working with entities and entity proxies
# Next, let's instantiate a proxy object for a new Person entity:
entity = model.make_entity(schema)

# First, you'll want to assign an ID to the entity. You can do this directly:
entity.id = 'john-smith'

# Or you can use a hashing function to make a safe ID:
entity.make_id('John Smith', '1979')

# Now, let's assign this entity a birthDate property (see above):
entity.add(prop, '1979-08-23')

# You can also assign properties by name:
entity.add('firstName', 'John')
entity.add('lastName', 'Smith')
entity.add('name', 'John Smith')

# Adding a property value will perform some data normalisation and validation:
entity.add('nationality', 'Atlantis')
assert not entity.has('nationality')
entity.add('nationality', 'Germani', fuzzy=True)
assert 'de' == entity.first('nationality')
enttiy.add('nationality', '阿拉伯聯合大公國')
assert 'ae' in entity.get('nationality')

# Lets make a second entity, this time for a passport:
passport_entity = model.make_entity('Passport')
passport_entity.make_id(entity.id, 'C716818')
passport_entity.add('number', 'C716818')

# Entities can link to other entities like this:
passport_entity.add('holder', entity)
# Which is the same as:
passport_entity.add('holder', entity.id)

# Finally, you can turn the contents of the entity proxy into a plain dictionary
# that is suitable for JSON serialization or storage in a database:
data = entity.to_dict()
assert data.get('id') == entity.id

# If you want to turn this back into an entity proxy:
entity2 = model.get_proxy(data)
assert entity2 == entity
```

--------------------------------

### Execute mapping via command-line

Source: https://followthemoney.tech/docs/mappings

Run the mapping file using the ftm command-line tool to output JSON entities.

```bash
ftm map brexitonians.yml
```

--------------------------------

### Aggregate datasets using the store CLI

Source: https://followthemoney.tech/docs/cli

Perform a write-then-iterate workflow to aggregate entities using the store command.

```bash
cat us_ofac.ijson | ftm store write -d us_ofac
ftm store iterate -d us_ofac > aggregated.ijson
ftm store delete -d us_ofac
```

--------------------------------

### Accessing and using property types via registry

Source: https://followthemoney.tech/python/registry

Demonstrates cleaning values, comparing entities, and iterating over matchable types using the registry singleton.

```python
from followthemoney.types import registry

# Clean a value
phone = registry.phone.clean("+1 (415) 555-0100")

# Compare two values
score = registry.name.compare("Jane Doe", "J. Doe")

# Iterate all types that support comparison
for t in registry.matchable:
    print(t.name, t.label)
```

--------------------------------

### Configure Java Maven dependency

Source: https://followthemoney.tech/libraries

Add the followthemoney dependency to your project's pom.xml file.

```xml
<dependency>
    <groupId>tech.followthemoney</groupId>
    <artifactId>followthemoney</artifactId>
    <version>4.10.1</version>
</dependency>
```

--------------------------------

### Aggregate entities using Python

Source: https://followthemoney.tech/docs/fragments

Using the ftmstore library to bulk load fragments and iterate over combined entities or partials.

```python
import os
from ftmstore import get_dataset
# Assume a function that will emit fragments:
from myapp.data import generate_fragments

# If no `database_uri` is given, ftmstore will read connection from
# $FTM_STORE_URI, or create a file called `followthemoney.sqlite` in
# the current directory.
database_uri = os.environ.get('DATABASE_URI')
dataset = get_dataset('myapp_dataset', database_uri=database_uri)
bulk = dataset.bulk()
for idx, proxy in enumerate(generate_fragments()):
    bulk.put(proxy, fragment=idx)
bulk.flush()

# This will print the number of combined entities (ie. DISTINCT id):
print(len(dataset))

# This will return combined entities:
for entity in dataset.iterate():
    print(entity.caption)

# You could also iterate the underlying fragments:
for proxy in dataset.partials():
    print(proxy)

# Note: `dataset.partials()` returns `EntityProxy` objects. The method
# `dataset.fragments()` would return raw Python dictionaries instead.

# All three methods also support the `entity_id` filter, which can also be
# shortened to `get`:
entity = dataset.get(entity_id)
```

--------------------------------

### Execute a mapping from a YAML file

Source: https://followthemoney.tech/docs/cli

Downloads a mapping file and executes it to generate an ijson stream of entities.

```bash
curl -o md_companies.yml https://raw.githubusercontent.com/alephdata/aleph/main/mappings/md_companies.yml
ftm map md_companies.yml > moldova.ijson
```

--------------------------------

### Perform Neo4J bulk import

Source: https://followthemoney.tech/docs/cli

Generate CSV files and a shell script for high-performance Neo4J bulk loading.

```bash
cat us_ofac.ijson | ftm export-neo4j-bulk -o neo4j_import/ -e iban -e address
```

--------------------------------

### Sign an Ownership entity with a namespace

Source: https://followthemoney.tech/docs/namespace

Demonstrates the transformation of an Ownership entity's bare IDs into signed IDs using a specific namespace.

```json
{
  "id": "owner-ship-1",
  "schema": "Ownership",
  "properties": {
    "owner": ["person-1"],
    "asset": ["company-1"]
  }
}
```

```json
{
  "id": "owner-ship-1.a40a29300ac6bb79dd2f911e77bbda7a3b502126",
  "schema": "Ownership",
  "properties": {
    "owner": ["person-1.c2c1…"],
    "asset": ["company-1.8b3f…"]
  }
}
```

--------------------------------

### Aggregate entities using CLI

Source: https://followthemoney.tech/docs/fragments

Commands to map CSV data to fragments, store them in a database, and iterate over merged entities.

```bash
# Generate entities from a CSV file and a mapping:
cat company-registry.csv | ftm map-csv mapping_file.yml > fragments.ijson

# Write the fragments to a table `company_registry`:
cat fragments.ijson | ftm store write -d company_registry

# List the tables in the store:
ftm store list

# Output merged entities:
ftm store iterate -d company_registry
```

--------------------------------

### Pick a name from a set

Source: https://followthemoney.tech/explorer/types/name

Selects the most plausible user-facing name from a sequence of name strings.

```python
def pick(self, values: Sequence[str]) -> str | None:
    """From a set of names, pick the most plausible user-facing one."""
    return pick_name(list(values))
```

--------------------------------

### Export graph data to GEXF

Source: https://followthemoney.tech/docs/cli

Create GEXF files for quantitative analysis in tools like Gephi.

```bash
cat us_ofac.ijson | ftm validate | ftm export-gexf -e iban -o ofac.gexf
```

--------------------------------

### Pull and run Docker container

Source: https://followthemoney.tech/libraries

Commands to pull the latest FollowTheMoney Docker image and run it interactively.

```bash
docker pull ghcr.io/opensanctions/followthemoney:latest
docker run --rm -t -i ghcr.io/opensanctions/followthemoney:latest
```

--------------------------------

### Normalize phone numbers with country hints

Source: https://followthemoney.tech/explorer/types/phone

Demonstrates cleaning phone numbers and adding them to entities using the registry and entity proxy.

```python
from followthemoney import model
from followthemoney.types import registry

registry.phone.clean("020 8366 1177")               # None
registry.phone.clean("020 8366 1177", format="gb")  # '+442083661177'

entity = model.make_entity("Company")
entity.add("phone", "020 8366 1177", format="gb")
```

--------------------------------

### Explore schema model with ftm ref

Source: https://followthemoney.tech/docs/cli

Browse schema definitions, properties, and types. Outputs tables for terminals or JSON when piped.

```bash
ftm ref                       # model overview + the subcommand index
ftm ref schemata              # every schema, with matchable/abstract flags
ftm ref schemata --matchable  # only schemata that can be matched
ftm ref schema Person         # one schema, with all inherited properties
ftm ref types                 # every property type
ftm ref type country          # one type, including its enum values
ftm ref prop Person:nationality   # one property's full definition
```

--------------------------------

### Read and process FtM entities in Java

Source: https://followthemoney.tech/libraries

Demonstrates loading the default model and iterating through entity data from a JSON file.

```java
package tech.followthemoney;

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import tech.followthemoney.entity.ValueEntity;
import tech.followthemoney.exc.SchemaException;
import tech.followthemoney.model.Model;

public class FtMLibraryDemo {
    public static void main(String[] args) {
        String path = "entities.ftm.json";
        try {
            BufferedReader reader = new BufferedReader(new FileReader(path));
            Model model = Model.loadDefault();
            ObjectMapper mapper = new ObjectMapper();
            String line;
            while ((line = reader.readLine()) != null) {
                JsonNode node = mapper.readTree(line);
                ValueEntity entity = ValueEntity.fromJson(model, node);
                System.out.println("Processing entity: " + entity.getId());
                System.out.println(" -> Entity name: " + entity.getCaption());
                System.out.println(" -> Entity type: " + entity.getSchema().getLabel());
            }

        } catch (IOException | SchemaException e) {
            e.printStackTrace();
        } 
    }
}
```

--------------------------------

### followthemoney.compare.compare(left, right, weights=COMPARE_WEIGHTS)

Source: https://followthemoney.tech/python/lib

Compares two entity proxies and returns a float representing the match score based on provided weights.

```APIDOC
## followthemoney.compare.compare

### Description
Compares two entities and returns a match score as a float. If the entities have identical checksums and contain properties, it returns a perfect match score of 1.0.

### Signature
`compare(left: EntityProxy, right: EntityProxy, weights: Weights = COMPARE_WEIGHTS) -> float`

### Parameters
- **left** (EntityProxy) - Required - The first entity to compare.
- **right** (EntityProxy) - Required - The second entity to compare.
- **weights** (Weights) - Optional - The weighting configuration used for scoring, defaults to COMPARE_WEIGHTS.

### Returns
- **float** - The calculated match score.
```

--------------------------------

### Compare two entities using followthemoney.compare

Source: https://followthemoney.tech/python/lib

Calculates a match score between two EntityProxy objects based on provided weights. Returns a float representing the similarity score.

```python
def compare(
    left: EntityProxy,
    right: EntityProxy,
    weights: Weights = COMPARE_WEIGHTS,
) -> float:
    """Compare two entities and return a match score."""
    if left.checksum == right.checksum:
        # Check if there is any data at all (ie any basis for making a decision),
        # if so, return a perfect match. This avoids marking two empty entities
        # as matching. Bit ambiguous, but practical.
        if len(left.properties) > 0 and len(right.properties) > 0:
            return 1.0
    scores = compare_scores(left, right)
    return _compare(scores, weights)
```

--------------------------------

### Load and iterate a dataset catalog

Source: https://followthemoney.tech/python/dataset

Load a catalog from a YAML file and iterate through the resources of a specific dataset.

```python
from followthemoney.dataset import DataCatalog, Dataset

catalog = DataCatalog.from_path(Dataset, "catalog.yml")
ofac = catalog.require("us_ofac_sdn")
for resource in ofac.resources:
    print(resource.name, resource.url)
```

--------------------------------

### Apply filters to source data

Source: https://followthemoney.tech/docs/mappings

Demonstrates filtering imported data using equality conditions. Complex filtering logic should be handled during data cleaning before mapping execution.

```yaml
gb_parliament_57:
  queries:
    - csv_url: http://bit.ly/uk-mps-csv
      filters:
        group: 'Conservative'
      filters_not:
        gender: 'male'
      entities:
        member:
          schema: Person
          keys:
            - id
          properties:
            name:
              column: name
```

--------------------------------

### Create Statement instance from dictionary

Source: https://followthemoney.tech/python/statements

Instantiates a Statement object using a dictionary of statement attributes.

```python
@classmethod
def from_dict(cls, data: StatementDict) -> "Statement":
    return cls(
        entity_id=data["entity_id"],
        prop=data["prop"],
        schema=data["schema"],
        value=data["value"],
        dataset=data["dataset"],
        lang=data.get("lang", None),
        original_value=data.get("original_value", None),
        first_seen=data.get("first_seen", None),
        external=data.get("external", False),
        id=data.get("id", None),
        canonical_id=data.get("canonical_id", None),
        last_seen=data.get("last_seen", None),
        origin=data.get("origin", None),
    )
```

--------------------------------

### Process statement-based workflows

Source: https://followthemoney.tech/docs/cli

Convert between entity streams and statement tables for provenance-aware data handling.

```bash
# Entities → statements (CSV, attributed to dataset 'md_companies')
cat moldova.ijson | ftm statements -d md_companies > statements.csv

# Statements → entities, reconstructing full records
ftm aggregate-statements -i statements.csv > rebuilt.ijson
```

--------------------------------

### Model and Schema Access

Source: https://followthemoney.tech/python

Accessing schema and property metadata from the model singleton.

```APIDOC
## model.get(name)

### Description
Retrieves a schema metadata object by its name.

### Parameters
- **name** (string) - Required - The name of the schema (e.g., 'Person').

## schema.get(name)

### Description
Retrieves a property metadata object from a schema.

### Parameters
- **name** (string) - Required - The name of the property (e.g., 'birthDate').
```

--------------------------------

### Create Statement instance from database row

Source: https://followthemoney.tech/python/statements

Instantiates a Statement object from a database row object.

```python
@classmethod
def from_db_row(cls, row: Row[Any]) -> "Statement":
    return cls(
        id=row.id,
        canonical_id=row.canonical_id,
        entity_id=row.entity_id,
        prop=row.prop,
        schema=row.schema,
        value=row.value,
        dataset=row.dataset,
        lang=row.lang,
        original_value=row.original_value,
        first_seen=datetime_iso(row.first_seen),
        external=row.external,
        last_seen=datetime_iso(row.last_seen),
        origin=row.origin,
    )
```

--------------------------------

### Initialize StatementEntity

Source: https://followthemoney.tech/python/statements

Defines the StatementEntity class which links entities to their source datasets.

```python
class StatementEntity(EntityProxy):
    """An entity object that can link to a set of datasets that it is sourced from."""

    __slots__ = (
        "_caption",
        "_statements",
        "dataset",
        "extra_referents",
        "id",
        "last_change",
        "schema",
    )

    def __init__(
        self,
        dataset: Dataset,
        data: dict[str, Any],
        cleaned: bool = True,
    ) -> None:
        data = dict(data or {})
        schema = Model.instance().get(data.pop("schema", None))
        if schema is None:
            raise InvalidData(gettext("No schema for entity."))
        self.schema = schema
```

--------------------------------

### Standard Column Mapping

Source: https://followthemoney.tech/docs/mappings

Basic configuration for mapping a single column to a property.

```yaml
properties:
  name:
    column: person_name
```

--------------------------------

### followthemoney.types.Registry.get(name)

Source: https://followthemoney.tech/python/registry

Retrieves a specific property type object by its name.

```APIDOC
## get(name)

### Description
Retrieves the property type object associated with the provided name. This can also be accessed via attribute access on the registry instance (e.g., registry.phone).

### Parameters
- **name** (string) - Required - The name of the property type to retrieve.
```

--------------------------------

### Aggregate Entities from Multiple Files

Source: https://followthemoney.tech/docs/statements

Map multiple CSV files to statements, sort them, and aggregate them into a single entity file.

```bash
# Map several source files to FtM. Some of the source files may emit copies of
# the same entity.
ftm map-csv mapping.yml -i source1.csv | ftm statements -o source1.json
ftm map-csv mapping.yml -i source2.csv | ftm statements -o source2.json
ftm map-csv mapping.yml -i source3.csv | ftm statements -o source3.json

# Invoke a normal UNIX sort:
sort -o combined.json source1.json source2.json source3.json 

# Now, all statements representing one entity are grouped and can be turned
# into FtM entities:
ftm aggregate-statements -i combined.json -o entities.ftm.json
```

--------------------------------

### Statement Cloning and Key Generation

Source: https://followthemoney.tech/python/statements

Methods for creating deep copies of statements and generating unique keys for data identification.

```python
def clone(
    self: Self,
    *,
    entity_id: str | None = None,
    prop: str | None = None,
    schema: str | None = None,
    value: str | None = None,
    dataset: str | None = None,
    lang: str | None | object = UNSET,
    original_value: str | None | object = UNSET,
    first_seen: str | None | object = UNSET,
    external: bool | None = None,
    canonical_id: str | None = None,
    last_seen: str | None | object = UNSET,
    origin: str | None | object = UNSET,
) -> "Statement":
    """Make a deep copy of the given statement."""
    lang = lang if is_not_unset(lang) else self._lang
    ov = original_value if is_not_unset(original_value) else self.original_value
    first_seen = first_seen if is_not_unset(first_seen) else self.first_seen
    last_seen = last_seen if is_not_unset(last_seen) else self.last_seen
    origin = origin if is_not_unset(origin) else self.origin
    if external is None:
        external = self._external
    if canonical_id is None and self._entity_id != self.canonical_id:
        canonical_id = self.canonical_id

    # Decide if the statement ID can be kept the same:
    stmt_id = self.id
    if entity_id is not None and entity_id != self.entity_id:
        stmt_id = None
    if prop is not None and prop != self._prop:
        stmt_id = None
    if schema is not None and schema != self._schema:
        stmt_id = None
    if value is not None and value != self._value:
        stmt_id = None
    if dataset is not None and dataset != self._dataset:
        stmt_id = None
    if external != self._external:
        stmt_id = None
    if lang != self._lang:
        stmt_id = None
    return Statement(
        id=stmt_id,
        entity_id=entity_id or self._entity_id,
        prop=prop or self._prop,
        schema=schema or self._schema,
        value=value or self._value,
        dataset=dataset or self._dataset,
        lang=lang,
        original_value=ov,
        first_seen=first_seen,
        external=external,
        canonical_id=canonical_id,
        last_seen=last_seen,
        origin=origin,
    )
```

```python
def generate_key(self) -> str | None:
    return self.make_key(
        self._dataset,
        self._entity_id,
        self._prop,
        self._value,
        self._external,
        lang=self._lang,
    )
```

--------------------------------

### pick(values)

Source: https://followthemoney.tech/explorer/types/name

Selects the most plausible user-facing name from a set of names.

```APIDOC
## pick(values)

### Description
From a set of names, pick the most plausible user-facing one.

### Parameters
- **values** (Sequence[str]) - Required - A list of name strings.

### Returns
- **str | None** - The most plausible name or None.
```

--------------------------------

### Caption Method

Source: https://followthemoney.tech/explorer/types/country

Converts a country code into its corresponding short name.

```python
def caption(self, value: str, format: str | None = None) -> str:
    """Turn country codes into short names."""
    territory = get_territory(value)
    if territory is not None:
        return territory.name
    return value
```

--------------------------------

### Convert and Aggregate Statements

Source: https://followthemoney.tech/docs/statements

Convert entities to statement streams and perform the inverse aggregation operation.

```bash
# Convert "traditional" FtM entities to a statement stream
cat entities.ftm.json | ftm statements --format json -o statements.json
#. The inverse operation:
ftm aggregate-statements -i statements.json -o entities.ftm.json
```

--------------------------------

### Sign entity IDs with a namespace

Source: https://followthemoney.tech/docs/cli

Applies an HMAC signature to entity IDs in a stream using a specified namespace key.

```bash
cat entities.ijson | ftm sign -s md_companies > signed.ijson
```

--------------------------------

### apply(proxy, shallow=False)

Source: https://followthemoney.tech/docs/namespace

Rewrites an entity proxy so that all IDs mentioned are limited to the current namespace.

```APIDOC
## apply(proxy, shallow=False)

### Description
Rewrite an entity proxy so all IDs mentioned are limited to the namespace.

### Parameters
- **proxy** (object) - Required - The entity proxy to rewrite.
- **shallow** (bool) - Optional - Whether to perform a shallow rewrite.
```

--------------------------------

### compare(left, right)

Source: https://followthemoney.tech/explorer/types/url

Compares two URL strings and returns a similarity score, ignoring fragments and performing hard normalization.

```APIDOC
## compare(left, right)

### Description
Compare two URLs and return a float indicating how similar they are. This ignores fragments and performs hard URL normalisation.

### Parameters
- **left** (str) - Required - The first URL to compare.
- **right** (str) - Required - The second URL to compare.

### Returns
- **float** - A similarity score between the two URLs.
```

--------------------------------

### Parse Compact Query String

Source: https://followthemoney.tech/docs/metadata

Convert a compact string query into a format suitable for evaluation.

```python
from followthemoney.dataset import parse_query, evaluate_query

query = parse_query("(#issuer.west|#list.sanction)-lt_fiu-#issuer.ru")
results = evaluate_query(catalog, query)
```

--------------------------------

### Entity Proxy Operations

Source: https://followthemoney.tech/python

Methods for creating, identifying, and populating entity proxies.

```APIDOC
## model.make_entity(schema)

### Description
Instantiates a new entity proxy object based on the provided schema.

## entity.make_id(*args)

### Description
Generates a safe ID for the entity based on provided arguments.

## entity.add(prop, value, fuzzy=False)

### Description
Assigns a property value to the entity, performing normalization and validation.

## entity.to_dict()

### Description
Converts the entity proxy into a plain dictionary suitable for JSON serialization.

## model.get_proxy(data)

### Description
Reconstructs an entity proxy from a dictionary representation.
```

--------------------------------

### Export entities to RDF

Source: https://followthemoney.tech/docs/cli

Convert entity streams into NTriples format.

```bash
cat us_ofac.ijson | ftm validate | ftm export-rdf > ofac.nt
```

--------------------------------

### Statement Magic Methods

Source: https://followthemoney.tech/python/statements

Standard Python magic methods for hashing, representation, equality, and comparison of Statement objects.

```python
def __hash__(self) -> int:
    if self.id is None:
        warnings.warn(
            "Hashing a statement without an ID results in undefined behaviour",
            RuntimeWarning,
        )
    return hash(self.id)
```

```python
def __repr__(self) -> str:
    return f"<Statement({self._entity_id!r}, {self._prop!r}, {self._value!r})>"
```

```python
def __eq__(self, other: Any) -> bool:
    return not self.id != other.id
```

```python
def __lt__(self, other: Any) -> bool:
    self_key = (self._prop != BASE_ID, self.id or "")
    other_key = (other._prop != BASE_ID, other.id or "")
    return self_key < other_key
```

--------------------------------

### clone(*, entity_id=None, prop=None, schema=None, value=None, dataset=None, lang=UNSET, original_value=UNSET, first_seen=UNSET, external=None, canonical_id=None, last_seen=UNSET, origin=UNSET)

Source: https://followthemoney.tech/python/statements

Creates a deep copy of the current statement, allowing for optional overrides of specific attributes.

```APIDOC
## clone

### Description
Creates a deep copy of the given statement. If any parameters are provided, they override the values from the original statement.

### Parameters
- **entity_id** (str, optional) - The ID of the entity.
- **prop** (str, optional) - The property name.
- **schema** (str, optional) - The schema name.
- **value** (str, optional) - The property value.
- **dataset** (str, optional) - The dataset name.
- **lang** (str, optional) - The language of the value.
- **original_value** (str, optional) - The original value.
- **first_seen** (str, optional) - Timestamp of first observation.
- **external** (bool, optional) - Whether the statement is external.
- **canonical_id** (str, optional) - The canonical entity ID.
- **last_seen** (str, optional) - Timestamp of last observation.
- **origin** (str, optional) - The origin of the statement.
```

--------------------------------

### Dataset Query Grammar

Source: https://followthemoney.tech/docs/metadata

The recursive structure for defining dataset queries.

```text
DatasetQuery = str | list[DatasetQuery]
             | {"or": list[DatasetQuery]}
             | {"and": list[DatasetQuery]}
             | {"not": DatasetQuery}
```

--------------------------------

### Filter a catalog using a query

Source: https://followthemoney.tech/python/dataset

Parse a query string and evaluate it against a catalog to retrieve matching datasets.

```python
from followthemoney.dataset import evaluate_query, parse_query

query = parse_query("#list.sanction & #issuer.eu")
matched = evaluate_query(catalog, query)
```

--------------------------------

### Manage fragment origins via CLI

Source: https://followthemoney.tech/docs/fragments

Commands to write fragments with specific origins and selectively delete them from a dataset.

```bash
cat us_ofac.ijson | ftm store write -d sanctions -o us_ofac
cat eu_eeas.ijson | ftm store write -d sanctions -o eu_eeas

# Will now have entities from both source files:
ftm store iterate -d sanctions | wc -l

# Delete all fragments from the second file:
ftm store delete -d sanctions -o eu_eeas

# Only one source file is left:
ftm store iterate -d sanctions | wc -l
```

--------------------------------

### StatementEntity Class Implementation

Source: https://followthemoney.tech/python/statements

Core methods and properties for managing entity statements, including iteration, dataset tracking, and statement addition.

```python
        self._caption: str | None = None
        """A pre-computed label for this entity."""

        self.extra_referents: set[str] = set(data.pop("referents", []))
        """The IDs of all entities which are included in this canonical entity."""

        self.last_change: str | None = data.get("last_change", None)
        """The last time this entity was changed."""

        self.dataset = dataset
        """The default dataset for new statements."""

        self.id: str | None = data.pop("id", None)
        self._statements: dict[str, set[Statement]] = {}

        properties = data.pop("properties", None)
        if isinstance(properties, Mapping):
            for key, value in properties.items():
                self.add(key, value, cleaned=cleaned, quiet=True)

        for stmt_data in data.pop("statements", []):
            stmt = Statement.from_dict(stmt_data)
            if self.id is not None:
                stmt.canonical_id = self.id
            self.add_statement(stmt)

    @property
    def _properties(self) -> dict[str, list[str]]:  # type: ignore
        return {p: [s.value for s in v] for p, v in self._statements.items()}

    def _iter_stmt(self) -> Generator[Statement, None, None]:
        for stmts in self._statements.values():
            for stmt in stmts:
                if stmt.entity_id is None and self.id is not None:
                    stmt = stmt.clone(entity_id=self.id)
                if stmt.id is None:
                    stmt.id = stmt.generate_key()
                yield stmt

    @property
    def statements(self) -> Generator[Statement, None, None]:
        """Return all statements for this entity, with extra ID statement."""
        ids: list[str] = []
        last_seen: set[str] = set()
        first_seen: set[str] = set()
        for stmt in self._iter_stmt():
            yield stmt
            if stmt.id is not None:
                ids.append(stmt.id)
            if stmt.last_seen is not None:
                last_seen.add(stmt.last_seen)
            if stmt.first_seen is not None:
                first_seen.add(stmt.first_seen)
        if self.id is not None:
            digest = sha1(self.schema.name.encode(HASH_ENCODING))
            for id in sorted(ids):
                digest.update(id.encode(HASH_ENCODING))
            checksum = digest.hexdigest()
            # This is to make the last_change value stable across
            # serialisation:
            first = self.last_change or min(first_seen, default=None)
            yield Statement(
                canonical_id=self.id,
                entity_id=self.id,
                prop=BASE_ID,
                schema=self.schema.name,
                value=checksum,
                dataset=self.dataset.name,
                first_seen=first,
                last_seen=max(last_seen, default=None),
            )

    @property
    def first_seen(self) -> str | None:
        seen = (s.first_seen for s in self._iter_stmt() if s.first_seen is not None)
        return min(seen, default=None)

    @property
    def last_seen(self) -> str | None:
        seen = (s.last_seen for s in self._iter_stmt() if s.last_seen is not None)
        return max(seen, default=None)

    @property
    def datasets(self) -> set[str]:
        datasets: set[str] = set()
        for stmt in self._iter_stmt():
            datasets.add(stmt.dataset)
        return datasets

    @property
    def referents(self) -> set[str]:
        referents: set[str] = set(self.extra_referents)
        for stmt in self._iter_stmt():
            if stmt.entity_id is not None and stmt.entity_id != self.id:
                referents.add(stmt.entity_id)
        return referents

    @property
    def key_prefix(self) -> str | None:
        return self.dataset.name

    @key_prefix.setter
    def key_prefix(self, dataset: str | None) -> None:
        raise NotImplementedError()

    def add_statement(self, stmt: Statement) -> None:
        schema = self.schema
        if schema.name != stmt.schema and not schema.is_a(stmt.schema):
            try:
                self.schema = schema.model.common_schema(schema, stmt.schema)
            except InvalidData as exc:
                raise InvalidData(f"{self.id}: {exc}") from exc

        if stmt.prop == BASE_ID:
            if stmt.first_seen is not None:
                # The last_change attribute describes the latest checksum change
                # of any emitted component of the entity, which is stored in the BASE
                # field.
                if self.last_change is None:
                    self.last_change = stmt.first_seen
                else:
                    self.last_change = max(self.last_change, stmt.first_seen)
        else:
            self._caption = None
            if stmt.prop not in self._statements:
                self._statements[stmt.prop] = set()
            self._statements[stmt.prop].add(stmt)
```

--------------------------------

### Retrieve values by type

Source: https://followthemoney.tech/python/statements

Returns a list of values associated with a specific property type.

```python
def get_type_values(
    self, type_: PropertyType, matchable: bool = False
) -> list[str]:
    combined: set[str] = set()
    for stmt in self.get_type_statements(type_, matchable=matchable):
        combined.add(stmt.value)
    return list(combined)
```

--------------------------------

### Export entities to tabular formats

Source: https://followthemoney.tech/docs/cli

Convert entity streams into Excel workbooks or directories of CSV files.

```bash
cat us_ofac.ijson | ftm validate | ftm export-excel -o OFAC.xlsx
```

```bash
cat us_ofac.ijson | ftm validate | ftm export-csv -o OFAC/
```

--------------------------------

### Retrieve statements by type

Source: https://followthemoney.tech/python/statements

Returns a list of Statement objects filtered by property type and matchability.

```python
def get_type_statements(
    self, type_: PropertyType, matchable: bool = False
) -> list[Statement]:
    combined: list[Statement] = []
    for prop_name, statements in self._statements.items():
        prop = self.schema.properties[prop_name]
        # Used in performance-critical code paths:
        if prop.type is not type_:
            continue
        if matchable and not prop.matchable:
            continue
        combined.extend(statements)
    return combined
```

--------------------------------

### followthemoney.types.Registry.get_types(names)

Source: https://followthemoney.tech/python/registry

Retrieves a list of property type objects based on a collection of names.

```APIDOC
## get_types(names)

### Description
Returns a list of property type objects corresponding to the provided set of names.

### Parameters
- **names** (list) - Required - A list of property type names to retrieve.
```

--------------------------------

### Serialize Entity to Context Dictionary

Source: https://followthemoney.tech/python/statements

Converts the entity into a dictionary representation suitable for context, including metadata like referents, datasets, and temporal information.

```python
def to_context_dict(self) -> dict[str, Any]:
    """Return a dictionary representation of the entity for context."""
    data: dict[str, Any] = {
        "id": self.id,
        "caption": self.caption,
        "schema": self.schema.name,
    }
    referents: set[str | None] = set(self.extra_referents)
    datasets = set(self.datasets)
    origins: set[str] = set()
    first_seen = None
    last_seen = None
    for stmts in self._statements.values():
        for stmt in stmts:
            if stmt.first_seen is not None:
                if first_seen is None or stmt.first_seen < first_seen:
                    first_seen = stmt.first_seen
            if stmt.last_seen is not None:
                if last_seen is None or stmt.last_seen > last_seen:
                    last_seen = stmt.last_seen
            if stmt.entity_id is not None and stmt.entity_id != self.id:
                referents.add(stmt.entity_id)
            datasets.add(stmt.dataset)
            if stmt.origin is not None:
                origins.add(stmt.origin)

    data["referents"] = list(referents)
    data["datasets"] = [d for d in datasets if d != Dataset.UNDEFINED]
    if origins:
        data["origin"] = list(origins)

    if first_seen is not None:
        data["first_seen"] = first_seen
    if last_seen is not None:
        data["last_seen"] = last_seen
    if self.last_change is not None:
        data["last_change"] = self.last_change
    return data
```

--------------------------------

### Serialize Entity to Dictionary

Source: https://followthemoney.tech/python/statements

Returns a full dictionary representation of the entity, including its properties.

```python
def to_dict(self) -> dict[str, Any]:
    data = self.to_context_dict()
    data["properties"] = self.properties
    return data
```

--------------------------------

### Implement caption method for NumberType

Source: https://followthemoney.tech/explorer/types/number

This method formats a numeric string for display purposes, optionally applying a format string and appending a unit if present.

```python
def caption(self, value: str, format: str | None = None) -> str:
    """Return a caption for the number. This is used for display purposes.

    Args:
        value (str): The string to format.
        format (Optional[str]): An optional format string to use for formatting the number.

    Returns:
        str: The formatted number string, possibly with a unit.
    """
    number, unit = self.parse(value)
    if number is None:
        return value
    try:
        fnumber = float(number)
    except ValueError:
        return value
    if format is not None:
        number = format.format(fnumber)
    elif fnumber.is_integer():
        number = self._INT_FMT.format(int(fnumber))
    else:
        number = self._FLOAT_FMT.format(fnumber)
    if unit is not None:
        return f"{number} {unit}"
    return number
```

--------------------------------

### Compare Sets Method

Source: https://followthemoney.tech/explorer/types/country

Compares two sets of country values and returns a score based on overlap.

```python
def compare_sets(
    self,
    left: Sequence[str],
    right: Sequence[str],
    func: Callable[[Sequence[float]], float] = max,
) -> float:
    """Compare two sets of values and select the highest-scored result."""
    overlap = territories_intersect(left, right)
    return 1.0 if len(overlap) else 0.0
```

--------------------------------

### Format entity streams for humans

Source: https://followthemoney.tech/docs/cli

Indents entity JSON for terminal inspection. Do not use this output as input for other ftm commands.

```bash
head -n 5 moldova.ijson | ftm pretty
```

--------------------------------

### Export entities to Cypher for Neo4J

Source: https://followthemoney.tech/docs/cli

Generate Cypher scripts for loading data into Neo4J, including options for reifying property types into nodes.

```bash
cat us_ofac.ijson | ftm export-cypher | cypher-shell -u user -p password
```

```bash
cat us_ofac.ijson | ftm export-cypher -e name -e iban -e address
```

```cypher
MATCH ()-[r:ANCESTORS]-() DELETE r;
MATCH ()-[r:PARENT]-() DELETE r;
MATCH (n:Page) DETACH DELETE n;
// Delete reified value nodes that connect to only one entity:
MATCH (n:name)    WHERE size((n)--()) <= 1 DETACH DELETE (n);
MATCH (n:email)   WHERE size((n)--()) <= 1 DETACH DELETE (n);
MATCH (n:address) WHERE size((n)--()) <= 1 DETACH DELETE (n);
```

```cypher
MATCH (n) DETACH DELETE n;
```

--------------------------------

### Create Entity from Dictionary

Source: https://followthemoney.tech/python/statements

Factory methods for creating a StatementEntity instance from a dictionary or raw data.

```python
@classmethod
def from_dict(
    cls,
    data: dict[str, Any],
    cleaned: bool = True,
    default_dataset: Dataset | None = None,
) -> Self:
    # Exists only for backwards compatibility.
    dataset = default_dataset or UndefinedDataset
    return cls(dataset, data, cleaned=cleaned)
```

```python
@classmethod
def from_data(
    cls,
    dataset: Dataset,
    data: dict[str, Any],
    cleaned: bool = True,
) -> Self:
    return cls(dataset, data, cleaned=cleaned)
```

--------------------------------

### Event Entity Key Configuration

Source: https://followthemoney.tech/docs/mappings

Configuration for linking entities via an event schema using a combination of participant keys.

```yaml
entities:
  company:
    schema: Company
    keys:
      - company_name
  owner:
    schema: Person
    keys:
      - owner_name
  ownership:
    schema: Ownership
    keys:
      - company_name
      - owner_name
```

--------------------------------

### Compare two URLs

Source: https://followthemoney.tech/explorer/types/url

The compare method returns a similarity score between two URLs, ignoring fragments and performing normalization.

```python
def compare(self, left: str, right: str) -> float:
    """Compare two URLs and return a float indicating how similar they are. This ignores
    fragments and peforms hard URL normalisation."""
    return compare_urls(left, right)
```

--------------------------------

### parse(value, decimal=DECIMAL, separator=SEPARATOR)

Source: https://followthemoney.tech/explorer/types/number

Parses a string into a numeric value and a unit. The numeric value is aligned with the provided decimal and separator settings, and the unit is stripped of whitespace.

```APIDOC
## parse(value, decimal=DECIMAL, separator=SEPARATOR)

### Description
Parses a number into a numeric value and a unit. The numeric value is aligned with the decimal and separator settings. The unit is stripped of whitespace and returned as a string. If no unit or number is found, returns (None, None).

### Parameters
- **value** (str) - Required - The string to parse.
- **decimal** (str) - Optional - The character used as the decimal separator. Defaults to DECIMAL.
- **separator** (str) - Optional - The character used to separate thousands, lakhs, or crores. Defaults to SEPARATOR.

### Returns
- **tuple[str | None, str | None]** - A tuple of (number, unit), where number is a string and unit is a string or None.
```

--------------------------------

### Iterate over properties

Source: https://followthemoney.tech/python/statements

Returns a list of Property objects present in the entity.

```python
def iterprops(self) -> list[Property]:
    return [self.schema.properties[p] for p in self._statements]
```

--------------------------------

### caption(value, format=None)

Source: https://followthemoney.tech/explorer/types/number

Formats a numerical string for display purposes, optionally applying a specific format string and preserving units.

```APIDOC
## Method: caption(value, format=None)

### Description
Returns a caption for the number. This is used for display purposes.

### Parameters
- **value** (str) - Required - The string to format.
- **format** (Optional[str]) - Optional - An optional format string to use for formatting the number.

### Returns
- **str** - The formatted number string, possibly with a unit.
```

--------------------------------

### Dump schema model

Source: https://followthemoney.tech/docs/cli

Exports the full FtM schema definition to a JSON file.

```bash
ftm dump-model -o model.json
```