### Install Entity Matching Model Library

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Install the EMM library using pip. This is the recommended way to get started with the package.

```shell
pip install emm
```

--------------------------------

### Setup Environment Script

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Executes a setup script to ensure the Python environment can import all necessary modules for documentation generation.

```bash
$ source setup.sh
```

--------------------------------

### Install Sphinx and Dependencies

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Installs Sphinx, the Read the Docs theme, and nbsphinx for Jupyter Notebook integration.

```bash
pip install -U Sphinx
pip install -U sphinx-rtd-theme
conda install -c conda-forge nbsphinx
```

--------------------------------

### Import Spark Configuration Example

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/spark.md

This import statement shows how to access the predefined Spark configuration dictionary from the emm.parameters module.

```python
from emm.parameters import SPARK_CONFIG_EXAMPLE
```

--------------------------------

### Install Entity Matching Model with Additional Dependencies

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Install the EMM library with additional dependencies for Spark, development, and testing. Use this command if you plan to use Spark or contribute to the project.

```shell
pip install "emm[spark,dev,test]"
```

--------------------------------

### Clone and Install Entity Matching Model from Source

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Clone the EMM repository from GitHub and install it in editable mode using pip. This is useful for development or when you need the latest code.

```shell
git clone https://github.com/ing-bank/EntityMatchingModel.git
pip install -e EntityMatchingModel/
```

--------------------------------

### Spark Configuration Example

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/spark.md

This dictionary contains recommended Spark settings for the driver and executors, optimized for matching large datasets. Adjust 'spark.driver.memoryOverhead' to '32G' if memory issues arise. Ensure 'spark.sql.adaptive.enabled' is set to 'false' for partition control.

```python
SPARK_CONFIG_EXAMPLE = {
    "spark.driver.memory": "25G",
    # default overhead = driverMemory * 0.10, with minimum of 384, in MiB unless otherwise specified
    "spark.driver.memoryOverhead": "10G",  # try "32G" if you face memory issues
    # 'spark.driver.cores': '1',  # default: 1
    # Amount of memory that can be occupied by the objects created via the Py4J bridge during a Spark operation,
    # above it spills over to the disk.
    "spark.python.worker.memory": "4G",  # default: 512m
    "spark.executor.memory": "30G",  # default 1G, 30G necessary for scoring
    # unlimited size object accepted by driver in collect() from workers (default 1G).
    # needed to collect large tfidf matrices between workers and driver.
    "spark.driver.maxResultSize": 0,
    "spark.rpc.message.maxSize": 1024,  # 1024mb message transfer size
    # In Spark 3.2+ adaptive shuffling/partitioning is enabled by default.
    # it is important to disable this to keep full control over the partitions and their consistency
    "spark.sql.adaptive.enabled": "false",
    # checkpoint directory are not cleaned up by default, and that leads to waste of HDFS space:
    "spark.cleaner.referenceTracking.cleanCheckpoints": "true",
}
```

--------------------------------

### Generate Example Data for Entity Matching

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Generate example ground-truth company names and their noised versions for testing the Entity Matching Model. This function helps in creating sample datasets for quick runs.

```python
from emm import PandasEntityMatching
from emm.data.create_data import create_example_noised_names

# generate example ground-truth names and matching noised names, with typos and missing words.
ground_truth, noised_names = create_example_noised_names(random_seed=42)
train_names, test_names = noised_names[:5000], noised_names[5000:]
```

--------------------------------

### Create Spark Session with Configuration

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Configures and creates a Spark session with specific memory, max result size, and shuffle partitions settings, optimized for small datasets used in examples.

```python
# create spark session
conf = {
"spark.driver.memory": "4G",
"spark.driver.memoryOverhead": "4G",
"spark.driver.maxResultSize": "1G",
"spark.executor.memory": "4G",
"spark.executor.memoryOverhead": "4G",
"spark.sql.shuffle.partitions": 1,  # because in examples we use very small datasets
}

conf = [(k, v) for k, v in conf.items()]
config = SparkConf().setAll(conf)

spark_session = SparkSession.builder.appName("Spark EMM Example").config(conf=config)
spark = spark_session.getOrCreate()
```

--------------------------------

### Customize IO Functions for Serialization

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/persistence.md

Allows customization of the reader and writer functions used for model serialization, defaulting to `joblib` with compression. This example shows how to use `pickle`.

```python
io = emm.helper.io.IOFunc()
io.writer = pickle.dump
io.reader = pickle.load
```

--------------------------------

### Import Entity Matching Model Package

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Import the EMM package into your Python script. This is the first step after installation to start using the library's functionalities.

```python
import emm
```

--------------------------------

### Configure Candidate Generators and Entity Matching Parameters

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/parameters.md

Defines a list of indexer configurations (cosine similarity, sorted neighboring indexing) and overall entity matching parameters. This setup is used to initialize the PandasEntityMatching model and prepare indexers based on ground truth data.

```python
# three example name-pair candidate generators:
# word-based and character-based cosine similarity, and sorted neighbouring indexing
indexers = [
    {
        "type": "cosine_similarity",
        "tokenizer": "words",        # word-based cosine similarity
        "ngram": 1,                  # 1-gram tokens only
        "num_candidates": 10,        # max 10 candidates per name-to-match
        "cos_sim_lower_bound": 0.,   # lower bound on cosine similarity
    },
    {
        "type": "cosine_similarity",
        "tokenizer": "characters",   # character-based cosine similarity
        "ngram": 2,                  # 2-gram character tokens only
        "num_candidates": 5,         # max 5 candidates per name-to-match
        "cos_sim_lower_bound": 0.2,  # lower bound on cosine similarity
    },
    {
        "type": "sni",
        "window_length": 3,          # sorted neighbouring indexing window of size 3.
    },
]
em_params = {
    "name_col": "Name",                     # important to set both index and name columns
    "entity_id_col": "Index",
    "indexers": indexers,
    "carry_on_cols": [],                    # names of columns in the GT and names-to-match dataframes passed on by the indexers. GT columns get prefix 'gt_'.
    "supervised_on": False,                 # no initial supervised model to select best candidates right now
    "name_only": True,                      # only consider name information for matching, e.g. not "country" info
    "without_rank_features": False,         # add rank-based features for improved probability of match
    "with_legal_entity_forms_match": True,  # add feature that indicates match of legal entity forms (eg. ltd != co)
    "aggregation_layer": False,             # whether to use name aggregation (turned off by default).
}
# initialize the entity matcher
p = PandasEntityMatching(em_params)
# prepare the indexers based on the ground truth names: e.g. fit the tfidf matrix of the first indexer.
p.fit(ground_truth)

# pandas dataframe with name-pair candidates, made by the indexers. all names have been preprocessed.
candidates_pd = p.transform(test_names)
candidates_pd.head()
```

--------------------------------

### Configure Cosine Similarity Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Defines a list of indexer configurations for entity matching. This example includes word-based and character-based cosine similarity indexers, and a sorted neighboring indexing (SNI) indexer.

```python
# example indexers
indexers = [
        {
            'type': 'cosine_similarity',
            'tokenizer': 'words',           # word-based cosine similarity
            'ngram': 1,
            'num_candidates': 5,            # max 5 candidates per name-to-match
            'cos_sim_lower_bound': 0.2,     # lower bound on cosine similarity
        },
        {
            'type': 'cosine_similarity',
            'tokenizer': 'characters',      # 2character-based cosine similarity
            'ngram': 2,
            'num_candidates': 5,
            'cos_sim_lower_bound': 0.2,
            'blocking_func': first_character
        },
        {'type': 'sni', 'window_length': 3}  # sorted neighbouring indexing window of size 3.
]
```

--------------------------------

### Initialize PandasEntityMatching with Cosine Similarity and SNI Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Initializes the PandasEntityMatching class with two types of indexers: character-based cosine similarity and sorted neighboring indexing. This setup is used for generating name-pair candidates for entity matching.

```python
indexers = [
  {
      'type': 'cosine_similarity',
      'tokenizer': 'characters',   # character-based cosine similarity. alternative: 'words'
      'ngram': 2,                  # 2-character tokens only
      'num_candidates': 5,         # max 5 candidates per name-to-match
      'cos_sim_lower_bound': 0.2,  # lower bound on cosine similarity
  },
  {'type': 'sni', 'window_length': 3}  # sorted neighbouring indexing window of size 3.
]
em_params = {
  'name_only': True,         # only consider name information for matching
  'entity_id_col': 'Index',  # important to set both index and name columns to pick up
  'name_col': 'Name',
  'indexers': indexers,
  'supervised_on': False,    # no supervided model (yet) to select best candidates
  'with_legal_entity_forms_match': True,   # add feature that indicates match of legal entity forms (e.g. ltd != co)
}

# 1. initialize the entity matcher
p = PandasEntityMatching(em_params)
```

--------------------------------

### Get Discrimination Threshold Curve Parameters

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Calculates parameters for discrimination threshold curves based on the best candidate matches. These parameters are essential for optimizing the model's decision threshold to meet specific precision or recall requirements.

```python
# get discrimination threshold curves for best candidates
curves = get_threshold_curves_parameters(best_candidates)
```

--------------------------------

### Initialize Entity Matching Parameters

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Sets up parameters for the PandasEntityMatching model, including indexer configurations and options for supervised learning and aggregation.

```python
# example indexers
indexers = [
    {
        'type': 'cosine_similarity',
        'tokenizer': 'words',           # word-based cosine similarity
        'ngram': 1,
        'num_candidates': 5,            # max 5 candidates per name-to-match
        'cos_sim_lower_bound': 0.2,     # lower bound on cosine similarity
    },
]

em_params = {
    'name_only': True,          # only consider name information for matching
    'entity_id_col': 'Index',   # important to set index and name columns
    'name_col': 'Name',
    'indexers': [indexers[0]],
    'supervised_on': True,       # without specifying a model, this option add an untrained supervised model  
    'return_sm_features': True,  # when calling transform, return the features used by the supervised model
    'without_rank_features': False,
    'with_legal_entity_forms_match': True,  # add feature with match of legal entity forms, e.g. ltd != co
    'aggregation_layer': True,    # aggregation layer, the aggregation of names on an account level
    'aggregation_method': 'mean_score', # aggregation method
}
p = PandasEntityMatching(em_params)
```

--------------------------------

### Prepare Supervised Model for Name Matching

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Prepares a supervised model for name matching using provided data. This includes creating training data, training both a general supervised model (sem) and a name-only model (sem_nm), and saving them to disk.

```python
from emm.supervised_model.base_supervised_model import train_test_model
from emm.helper.io import save_file
from emm.data import create_training_data

df, vocabulary = create_training_data()
sem, _= train_test_model(df, vocabulary, name_only=False)
save_file("sem.pkl", sem)
sem_nm, _ = train_test_model(df, vocabulary, name_only=True)
save_file("sem_nm.pkl", sem_nm)
```

--------------------------------

### Initialize and Use PandasEntityMatching

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/index.md

Demonstrates the initialization, fitting, and transformation process for the PandasEntityMatching class. This includes fitting the indexers, training a supervised classifier, and scoring test names.

```python
indexers = [
  {
      'type': 'cosine_similarity',
      'tokenizer': 'characters',   # character-based cosine similarity. alternative: 'words'
      'ngram': 2,                  # 2-character tokens only
      'num_candidates': 5,         # max 5 candidates per name-to-match
      'cos_sim_lower_bound': 0.2,  # lower bound on cosine similarity
  },
  {'type': 'sni', 'window_length': 3}  # sorted neighbouring indexing window of size 3.
]
em_params = {
  'name_only': True,         # only consider name information for matching
  'entity_id_col': 'Index',  # important to set both index and name columns to pick up
  'name_col': 'Name',
  'indexers': indexers,
  'supervised_on': False,    # no supervided model (yet) to select best candidates
  'with_legal_entity_forms_match': True,   # add feature that indicates match of legal entity forms (e.g. ltd != co)
}
# 1. initialize the entity matcher
p = PandasEntityMatching(em_params)

# 2. fitting: prepare the indexers based on the ground truth names, eg. fit the tfidf matrix of the first indexer.
p.fit(ground_truth)

# 3. create and fit a supervised model for the PandasEntityMatching object, to pick the best match (this takes a while)
#    input is "positive" names column 'Name' that are all supposed to match to the ground truth,
#    and an id column 'Index' to check with candidate name-pairs are matching and which not.
#    A fraction of these names may be turned into negative names (= no match to the ground truth).
#    (internally, candidate name-pairs are automatically generated, these are the input to the classification)
p.fit_classifier(train_names, create_negative_sample_fraction=0.5)

# 4. scoring: generate pandas dataframe of all name-pair candidates.
#    The classifier-based probability of match is provided in the column 'nm_score'.
#    Note: can also call p.transform() without training the classifier first.
candidates_scored_pd = p.transform(test_names)

# 5. scoring: for each name-to-match, select the best ground-truth candidate.
best_candidates = candidates_scored_pd[candidates_scored_pd.best_match]
best_candidates.head()
```

--------------------------------

### Create a New Sphinx Project

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Generates the basic structure and configuration files for a new Sphinx documentation project.

```bash
sphinx-quickstart
```

--------------------------------

### Build HTML Documentation

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Compiles the Sphinx documentation into HTML format, typically stored in the docs/build/html/ directory.

```bash
make html
```

--------------------------------

### Plot NM Score Histogram for Best Matches in Negative Data

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Generates a histogram of 'nm_score' for the best matches found in the negative dataset. Note that the model was trained without negative names, so this shows performance on unseen negative examples.

```python
# note: we have trained without negative names!
resn2[resn2.best_match]['nm_score'].hist(bins=40, log=True, alpha=0.5)
```

--------------------------------

### Prepare Test Data and Generate Candidates

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Splits data into positive and negative test sets and generates candidate matches using the entity matching model. It then concatenates these candidates and flags them with a 'correct' boolean based on ground truth entity IDs.

```python
positive_test = positive_noised_pd[2267:]
negative_test = negative_pd[:len(positive_test)]

candidates_pos = p.transform(positive_test)
candidates_neg = p.transform(negative_test)
candidates_pos['positive_set'] = True
candidates_neg['positive_set'] = False
candidates = pd.concat([candidates_pos, candidates_neg])
candidates['correct'] = (candidates['gt_entity_id'] == candidates['entity_id'])
```

--------------------------------

### Initialize Pandas and Spark Candidate Selection Objects

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Initialize the Candidate Selection (indexing) components for Pandas and Spark. These are responsible for generating name-pair candidates efficiently.

```python
from emm.indexing import pandas_candidate_selection, spark_candidate_selection
p_cs = pandas_candidate_selection.PandasCandidateSelectionTransformer(indexers=[])
s_cs = spark_candidate_selection.SparkCandidateSelectionEstimator(indexers=[])
```

--------------------------------

### Initialize Pandas and Spark Preprocessor Objects

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Initialize the Preprocessor components for Pandas and Spark. These are used for cleaning and standardizing input names and their legal entity forms.

```python
import emm.preprocessing
p_pr = emm.preprocessing.PandasPreprocessor()
s_pr = emm.preprocessing.SparkPreprocessor()
```

--------------------------------

### Import Core Libraries

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Imports the necessary libraries for the entity matching model and plotting.

```python
import emm
import matplotlib
```

--------------------------------

### Import Libraries

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Imports essential Python libraries for data manipulation, entity matching, and threshold analysis.

```python
import numpy as np
import pandas as pd
from emm import PandasEntityMatching, resources
from emm.data.create_data import pandas_create_noised_data
from emm.threshold.threshold_decision import get_threshold_curves_parameters
import warnings
warnings.filterwarnings("ignore")
```

--------------------------------

### Create Sample Ground Truth DataFrame

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Creates a PySpark DataFrame with sample company names to serve as ground truth for entity matching.

```python
gt = spark.createDataFrame([
    (1, 'John Smith LLC'),
    (2, 'ING LLC'),
    (3, 'John Doe LLC'),
    (4, 'Zhe Sun G.M.B.H'),
    (5, 'Random GMBH'),
], ['id', 'name'])
gt.show(10, False)
```

--------------------------------

### Instantiate Pandas and Spark EntityMatching Objects

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Instantiate the main EntityMatching objects for Pandas and Spark using default settings. These objects serve as the entry point for the entity matching process.

```python
from emm import PandasEntityMatching, SparkEntityMatching
p = PandasEntityMatching()
s = SparkEntityMatching()
```

--------------------------------

### Initialize Pandas and Spark Supervised Model Objects

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Initialize the Supervised Model components for Pandas and Spark. These are used for classifying name-pair candidates to improve accuracy.

```python
import emm.supervised_model
p_sm = emm.supervised_model.PandasSupervisedLayerTransformer(supervised_models={})
s_sm = emm.supervised_model.SparkSupervisedLayerEstimator(supervised_models={})
```

--------------------------------

### Supervised Name Matching with Spark

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Initializes and fits a SparkEntityMatching model with a supervised component, then transforms a DataFrame to find matches. Ensure 'sem_nm.pkl' is available in the specified directory.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'characters',
        'ngram': 2,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': True,
    'supervised_model_filename': 'sem_nm.pkl',
    'supervised_model_dir': '.',
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Display Training Name Pairs

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Shows the first few rows of the generated training name pairs, which consist of pairs of names and their matching status.

```python
name_pairs.head()
```

--------------------------------

### Initialize PandasEntityMatching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/fitting.md

Initializes the PandasEntityMatching model. This is a prerequisite for using other model functions.

```python
from emm import PandasEntityMatching
model = PandasEntityMatching()
```

--------------------------------

### Load IPython Extensions

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Loads necessary IPython extensions for interactive notebooks.

```python
%matplotlib inline
%load_ext autoreload
%autoreload 2
```

--------------------------------

### Display First Few Matching Results

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Shows the first few rows of the transformation result, including predicted entity IDs and features.

```python
resp.head()
```

--------------------------------

### Alternative: Fit Classifier with Pre-created Pairs

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Provides an alternative method to fit the classifier using pre-created training name pairs.

```python
# alternatively one can fit the classifier using:
#p.fit_classifier(train_name_pairs=name_pairs)
```

--------------------------------

### Import Pandas and Helper Modules

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Imports Pandas for data manipulation and specific modules for creating data, blocking functions, and threshold calculations. Also suppresses warnings.

```python
import pandas as pd
from emm import PandasEntityMatching, resources
from emm.data.create_data import pandas_create_noised_data
from emm.helper.blocking_functions import first as first_character
from emm.threshold.threshold_decision import get_threshold_curves_parameters
import warnings
warnings.filterwarnings("ignore")
```

--------------------------------

### Initialize Pandas and Spark Entity Aggregation Objects

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Initialize the Entity Aggregation components for Pandas and Spark. These are optional and used for matching groups of company names to a ground truth.

```python
import emm.aggregation
p_ag = emm.aggregation.PandasEntityAggregation(score_col='nm_score')
s_ag = emm.aggregation.SparkEntityAggregation(score_col='nm_score')
```

--------------------------------

### Prepare Test Data for Aggregation Scoring

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Adds 'account' and 'counterparty_account_count_distinct' columns to the test datasets, which are required for scoring name aggregations. Dummy values are used as each name belongs to a single account and is used once.

```python
# For aggregation of name-scores, need to have:
# an 'account' column: which indicated which names belong together
# and a frequency column, here call 'counterparty_account_count_distinct',
# which indicates how frequently each name occurs.

# Below we add these column with dummy values. 
# Each name belongs to a single account and is used just once.
positive_test_set['account'] = range(len(positive_test_set))
positive_test_set['account'] = positive_test_set['account'].astype(str)
positive_test_set['counterparty_account_count_distinct'] = 1

negative_test_set['account'] = range(len(negative_test_set))
negative_test_set['account'] += 10000
negative_test_set['account'] = negative_test_set['account'].astype(str)
negative_test_set['counterparty_account_count_distinct'] = 1
```

--------------------------------

### Import Pandas and EntityMatching with warnings

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Imports the Pandas library for data manipulation and the PandasEntityMatching class. It also suppresses warnings.

```python
import pandas as pd
from emm import PandasEntityMatching
import warnings
warnings.filterwarnings("ignore")
```

--------------------------------

### Save and Load Spark Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/persistence.md

Saves a Spark-based entity matching model to a directory and loads it back. Ensure the `emm` library is imported.

```python
s.save("spark_entity_matching_model")
from emm import SparkEntityMatching
s2 = SparkEntityMatching.load("spark_entity_matching_model")
```

--------------------------------

### Import Spark and Entity Matching Classes

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Imports necessary classes from PySpark and the entity matching model library, and suppresses warnings.

```python
from pyspark import SparkConf
from pyspark.sql import SparkSession
from emm import SparkEntityMatching

import warnings
warnings.filterwarnings("ignore")
```

--------------------------------

### Instantiate and Fit PandasEntityMatching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Instantiates a PandasEntityMatching model configured for name-only matching with specific preprocessing and indexing settings. It then fits the model's indexers to the ground truth dataset.

```python
# instantiate a matching model
nm = PandasEntityMatching({
    'name_only': True,
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'words',
        'ngram': 1,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': True,
    'supervised_model_filename': 'sem_nm.pkl',
    'supervised_model_dir': '.',
})

# matching of names is done against the ground-truth dataset (gt).
# for this we need to fit our indexers to the ground-truth.
nm.fit(gt)
```

--------------------------------

### Name Matching with Supervised Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Initializes and fits a PandasEntityMatching model with a supervised learning component. Use this when you have labeled data for training.

```python
nm = PandasEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'characters',
        'ngram': 2,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': True,
    'supervised_model_filename': 'sem_nm.pkl',
    'supervised_model_dir': '.',
})
pm.fit(gt)
res = nm.transform(pd.DataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Pandas and Spark Sorted Neighbourhood Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Instantiates Pandas and Spark Sorted Neighbourhood Indexers for efficient name matching. The window length parameter controls the search window size.

```python
from emm import indexing
p_sni = indexing.PandasSortedNeighbourhoodIndexer(window_length=5)
s_sni = indexing.SparkSortedNeighbourhoodIndexer(window_length=3)
```

--------------------------------

### Load Trained Entity Matching Model with Parameter Override

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Loads a previously saved entity matching model from a pickle file. Allows overriding specific parameters during loading, such as the name and entity ID columns, to adapt the model to different data schemas.

```python
# change of column names
nm = PandasEntityMatching.load("trained_em.pickle", 
                               override_parameters={'name_col': 'name', 'entity_id_col': 'index'})
```

--------------------------------

### Create Training Name Pairs

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Generates name pairs from the positive noised dataset that will be used to train the supervised classifier. Only a subset of the data is used here.

```python
# in more detail: internally the supervised model is trained on the follow name-pairs
name_pairs = p.create_training_name_pairs(positive_noised_pd[:2267])
```

--------------------------------

### Name Matching with Word Cosine Similarity

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Uses basic preprocessing and word tokenization with cosine similarity for name matching. Suitable for general name matching where word order and exact matches are important.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'words',
        'ngram': 1,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': False,
})
pm.fit(gt)
```

```python
res = nm.transform(spark.createDataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'), # this will not be matched due to mispellings
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Save Trained Model with Thresholds

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Saves the entity matching model, including the newly calculated discrimination thresholds, to a pickle file. This allows the model to be loaded later with the optimized thresholds already applied.

```python
nm.save('trained_em_with_thresholds.pickle')
```

--------------------------------

### Configure and Serialize Name-Only Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Initializes and serializes a Spark Entity Matching model configured for name-only matching, using a preprocessor, cosine similarity indexing, and a pre-trained supervised model.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'words',
        'ngram': 1,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': True,
    'supervised_model_filename': 'sem_nm.pkl',
    'supervised_model_dir': '.',
})
nm.write().overwrite().save("serialized_em_nm.pkl")
```

--------------------------------

### Autogenerate RST Files

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Runs a script to scan Python files and generate .rst files containing documentation from docstrings.

```bash
./autogenerate.sh
```

--------------------------------

### Display Second Matching Results

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Shows the second set of transformation results, including predicted entity IDs and correctness flags.

```python
resp2
```

--------------------------------

### Clean Sphinx Build

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/README.rst

Removes old build artifacts to ensure a clean generation of the documentation.

```bash
make clean
```

--------------------------------

### Pandas and Spark Cosine Similarity Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/pipeline.md

Instantiates Pandas and Spark Cosine Similarity Indexers for name matching. The Pandas version uses word-based tokenization, while the Spark version uses character-based tokenization with blocking.

```python
from emm import indexing
from emm.helper.blocking_functions import first
p_cossim = indexing.PandasCosSimIndexer(tokenizer='words', ngram=1, num_candidates=10, cos_sim_lower_bound=0.2)
s_cossim = indexing.SparkCosSimIndexer(tokenizer='characters', ngram=2, blocking_func=first, cos_sim_lower_bound=0.5)
```

--------------------------------

### Save Trained Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Saves the trained entity matching model to a pickle file for later use. Ensure the model object 'p' is already trained.

```python
p.save('trained_em.pickle')
```

--------------------------------

### Load Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Loads the previously saved PandasEntityMatching model from the 'am_curves.pkl' file.

```python
am = PandasEntityMatching.load('am_curves.pkl')
```

--------------------------------

### Save and Load Pandas Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/persistence.md

Saves a pandas-based entity matching model to a file and loads it back. Ensure the `emm` library is imported.

```python
p.save("pandas_entity_matching_model.pkl")
from emm import PandasEntityMatching
p2 = PandasEntityMatching.load("pandas_entity_matching_model.pkl")
```

--------------------------------

### Load and Use Serialized Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Loads a pre-trained PandasEntityMatching model from a file. This is efficient for reusing trained models without refitting, especially on large datasets.

```python
nm = PandasEntityMatching.load("serialized_em_nm.pkl")
res = nm.transform(pd.DataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOE LLC'),
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Initialize PandasEntityMatching Object

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Initializes the PandasEntityMatching object with specific parameters for name-only matching, including entity ID and name columns, indexers, and enabling supervised learning.

```python
em_params = {
    'name_only': True,          # only consider name information for matching
    'entity_id_col': 'Index',   # important to set index and name columns
    'name_col': 'Name',
    'indexers': [indexers[0]],
    'supervised_on': True,       # without specifying a model, this option add an untrained supervided model  
    'return_sm_features': True,  # when calling transform, return the features used by the supervised model
    'without_rank_features': False,
    'with_legal_entity_forms_match': False,  # add feature with match of legal entity forms, e.g. ltd != co
}
p = PandasEntityMatching(em_params)
```

--------------------------------

### Compare Input and Output Record Counts

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Compares the number of records in the input noised dataset with the number of records in the transformation output.

```python
# approximately ~3 candidates per name to match.
len(positive_noised_pd), len(resp)
```

--------------------------------

### Calculate Discrimination Threshold for All Names (Min 80% Precision)

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Calculates the discrimination threshold for all names (positive and negative), requiring a minimum precision of 80%. This provides a balance between precision and recall across the entire dataset.

```python
# discrimination threshold for positive and negative names, with minimum precision of 80%
threshold2 = nm.calc_threshold(agg_name='non_aggregated', type_name='all', metric_name='precision', min_value=0.80)
```

--------------------------------

### Load Spark Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Loads a previously saved SparkEntityMatching model from disk. The loaded model can be used directly for transformations without needing to be refitted.

```python
nm2 = SparkEntityMatching.load('name_matching_spark_model')
```

--------------------------------

### Create Ground Truth DataFrame

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Creates a Pandas DataFrame to serve as the ground truth dataset for entity matching. This DataFrame contains IDs and names.

```python
gt = pd.DataFrame([
    (1, 'John Smith LLC'),
    (2, 'ING LLC'),
    (3, 'John Doe LLC'),
    (4, 'Zhe Sun G.M.B.H'),
    (5, 'Random GMBH'),
], columns=['id', 'name'])
display(gt)
```

--------------------------------

### Calculate Discrimination Threshold

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Calculates a specific discrimination threshold for positive names, aiming for a minimum precision of 95%, using the loaded model and specifying the aggregation method.

```python
# discrimination threshold for positive names only, with minimum precision of 95%
threshold1 = am.calc_threshold(agg_name="mean_score", type_name='positive', metric_name='precision', min_value=0.95)
```

--------------------------------

### Calculate Discrimination Threshold for Positive Names (Min 95% Precision)

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Calculates the discrimination threshold for positive names only, ensuring a minimum precision of 95%. This helps in setting a cutoff score that prioritizes high accuracy for identifying true positive matches.

```python
# discrimination threshold for positive names only, with minimum precision of 95%
threshold1 = nm.calc_threshold(agg_name='non_aggregated', type_name='positive', metric_name='precision', min_value=0.95)
```

--------------------------------

### Name Matching with Sorted Neighbourhood Indexing (SNI)

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Utilizes Sorted Neighbourhood Indexing (SNI) for name matching. This method is effective for finding similar records by sorting and comparing within a defined window.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'uid_col': 'uid',
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'sni', 'window_length': 3},
    ],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (10, 'Jo S'),
    (11, 'InG. LLC'),
    (12, 'Jon DOEE LLC'),
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Display Data Set Sizes

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Prints the number of records in the ground truth, positive noised, and negative datasets.

```python
len(ground_truth), len(positive_noised_pd), len(negative_pd)
```

--------------------------------

### Display Ground Truth Data

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/03-entity-matching-training-pandas-version.ipynb

Shows the first few rows of the ground truth dataset to inspect the names.

```python
# have a look at the names in the ground truth
ground_truth
```

--------------------------------

### Name Matching with Multiple Cosine Similarity Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Combines word and character n-gram cosine similarity indexers for more robust name matching. This approach leverages both word-level and character-level similarities.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'cosine_similarity', 'tokenizer': 'words', 'ngram': 1, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
        {'type': 'cosine_similarity', 'tokenizer': 'characters', 'ngram': 2, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
    ],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Cosine Similarity with Word and N-gram Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Combines word and 2-gram character cosine similarity indexers for comprehensive name matching. This approach leverages both word-level and character-level similarities. Ensure 'gt' is fitted before transforming.

```python
nm = PandasEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'cosine_similarity', 'tokenizer': 'words', 'ngram': 1, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
        {'type': 'cosine_similarity', 'tokenizer': 'characters', 'ngram': 2, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
    ],
    'supervised_on': False,
})
nm.fit(gt)
res = nm.transform(pd.DataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Generate Candidates

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Generates candidate matches for both positive and negative test sets using the fitted entity matching model. This operation can be time-consuming.

```python
# this can take some time.
candidates_pos = p.transform(positive_test_set)
candidates_neg = p.transform(negative_test_set)
candidates_neg['positive_set'] = False
candidates_pos['positive_set'] = True
```

--------------------------------

### Name Matching with SNI and Custom Preprocessor (Reverse Name)

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Applies Sorted Neighbourhood Indexing (SNI) with a custom preprocessing function that reverses names. This demonstrates how to customize name transformation for SNI.

```python
reverse_name = lambda x: x[::-1]
nm = SparkEntityMatching({
    'name_only': True,
    'uid_col': 'uid',
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'sni', 'window_length': 3, 'mapping_func': reverse_name},
    ],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (11, 'a InG. LLC'),
    (12, 'ING. LLC ZZZ'),
    (13, 'John Smith LLC'),
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Create Noised Data

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Generates synthetic noisy name data for training and testing entity matching models, based on Dutch Chamber of Commerce data.

```python
# create noised names, based on Dutch chamber of commerce data
ground_truth, _, positive_noised_pd, negative_pd = pandas_create_noised_data(random_seed=42)
train_set, positive_test_set = positive_noised_pd[:2267], positive_noised_pd[2267:]
negative_test_set = negative_pd[:len(positive_test_set)]
```

--------------------------------

### Inspect Threshold Curves Keys

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Displays the keys available within the calculated threshold curves dictionary, indicating the different types of threshold information generated.

```python
# aggregation here
curves['threshold_curves'].keys()
```

--------------------------------

### Name Matching with Character N-gram Cosine Similarity

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Employs basic preprocessing and 2-character n-gram tokenization with cosine similarity. This is useful for matching names with variations in spelling or character sequences.

```python
nm = SparkEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'characters',
        'ngram': 2,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),  # it will not be matched due to mispellings
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Name Matching with Multiple Indexers

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Configures and uses a PandasEntityMatching model with a combination of different indexer types for robust name matching. Suitable for complex matching scenarios.

```python
nm = PandasEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'cosine_similarity', 'tokenizer': 'words', 'ngram': 1, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
        {'type': 'cosine_similarity', 'tokenizer': 'characters', 'ngram': 2, 'num_candidates': 5, 'cos_sim_lower_bound': 0.2},
        {'type': 'sni', 'window_length': 3},
    ],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(pd.DataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'),
    (14, 'Z'),  # this will be matched only by SNI
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Sorted Neighbourhood Indexing (SNI)

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Performs name matching using Sorted Neighbourhood Indexing with a specified window length. This method is efficient for large datasets by sorting and comparing records within a window. Ensure 'gt' is fitted before transforming.

```python
nm = PandasEntityMatching({
    'name_only': True,
    'uid_col': 'uid',
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'sni', 'window_length': 3},
    ],
    'supervised_on': False,
})
nm.fit(gt)
res = nm.transform(pd.DataFrame([
    (10, 'Jo S'),
    (11, 'InG. LLC'),
    (12, 'Jon DOEE LLC'),
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Fit Classifier for Supervised Learning

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/docs/sphinx/source/parameters.md

Trains the classifier using positive name pairs and optionally generates negative samples. Set supervised_on=True.

```python
p.fit_classifier(train_positive_names_to_match=train_names, create_negative_sample_fraction=0.5,
                 drop_duplicate_candidates=True, extra_features=None)
```

--------------------------------

### Save Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Saves the configured PandasEntityMatching model, including the updated parameters, to a pickle file named 'am_curves.pkl'.

```python
p.save('am_curves.pkl')
```

--------------------------------

### Cosine Similarity with Word Tokenization

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/01-entity-matching-pandas-version.ipynb

Performs name matching using word tokenization and cosine similarity. This is useful for finding similar names where word order and exact spelling might vary slightly. Ensure 'gt' is fitted before transforming.

```python
nm = PandasEntityMatching({
    'name_only': True,
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [{
        'type': 'cosine_similarity',
        'tokenizer': 'words',
        'ngram': 1,
        'num_candidates': 5,
        'cos_sim_lower_bound': 0.2,
    }],
    'supervised_on': False,
})
nm.fit(gt)
res = nm.transform(pd.DataFrame([
    (10, 'John Smith'),
    (11, 'I.n.G. LLC'),
    (12, 'Jon DOEE LLC'), # this will not be matched due to mispellings
], columns=['id', 'name']))
display(res)
```

--------------------------------

### Name Matching with Blocking Function

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Uses a blocking function with cosine similarity to generate candidate pairs. Only pairs with the same blocking function output are considered, improving efficiency.

```python
first_character = lambda x: x[0] if len(x) > 0 else '?'

nm = SparkEntityMatching({
    'name_only': True,
    'uid_col': 'uid',
    'entity_id_col': 'id',
    'name_col': 'name',
    'preprocessor': 'preprocess_merge_abbr',
    'indexers': [
        {'type': 'cosine_similarity', 'tokenizer': 'characters', 'ngram': 1, 'blocking_func': first_character},
    ],
    'supervised_on': False,
})
pm.fit(gt)
res = nm.transform(spark.createDataFrame([
    (10, '!notING'),  # it will not be matched due to different value of blocking function (first character)
    (11, 'ING'),
], ['id', 'name']))
res.show(10, False)
```

--------------------------------

### Fit Entity Matching Model with Ground Truth

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/README.md

Fits the PandasEntityMatching model using ground truth data. This step prepares the indexers, such as fitting the TF-IDF matrix for the cosine similarity indexer.

```python
# 2. fitting: prepare the indexers based on the ground truth names, eg. fit the tfidf matrix of the first indexer.
p.fit(ground_truth)
```

--------------------------------

### Save Spark Entity Matching Model

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/02-entity-matching-spark-version.ipynb

Saves the trained SparkEntityMatching model to disk. This allows for later loading and reuse without retraining, which is beneficial for large datasets.

```python
nm.save('name_matching_spark_model')
```

--------------------------------

### Calculate Discrimination Threshold

Source: https://github.com/ing-bank/entitymatchingmodel/blob/main/notebooks/04-entity-matching-aggregation-pandas-version.ipynb

Calculates the discrimination threshold for positive and negative names, ensuring a minimum precision of 80%. This is useful for tuning the confidence level of matching results.

```python
threshold2 = am.calc_threshold(agg_name="mean_score", type_name='all', metric_name='precision', min_value=0.80)
```