### Install pytest

Source: https://recordlinkage.readthedocs.io/en/latest/contributing.html

Install the pytest framework for running tests. This is a prerequisite for testing the package.

```bash
pip install pytest

```

--------------------------------

### Install RecordLinkage with All Dependencies

Source: https://recordlinkage.readthedocs.io/en/latest/installation.html

Install the Python Record Linkage Toolkit along with all recommended and optional dependencies. Use this command if you need the full suite of features.

```bash
pip install recordlinkage['all']
```

--------------------------------

### Full Record Linkage Workflow Example

Source: https://recordlinkage.readthedocs.io/en/latest/guides/link_two_dataframes.html

Demonstrates a complete record linkage process including loading data, indexing, comparison, and classification. This example shows how to perform record linkage from start to finish.

```python
import recordlinkage
from recordlinkage.datasets import load_febrl4

dfA, dfB = load_febrl4()

# Indexation step
indexer = recordlinkage.Index()
indexer.block("given_name")
candidate_links = indexer.index(dfA, dfB)

# Comparison step
compare_cl = recordlinkage.Compare()

compare_cl.exact("given_name", "given_name", label="given_name")
compare_cl.string(
    "surname", "surname", method="jarowinkler", threshold=0.85, label="surname"
)
compare_cl.exact("date_of_birth", "date_of_birth", label="date_of_birth")
compare_cl.exact("suburb", "suburb", label="suburb")
compare_cl.exact("state", "state", label="state")
compare_cl.string("address_1", "address_1", threshold=0.85, label="address_1")

features = compare_cl.compute(candidate_links, dfA, dfB)

# Classification step
matches = features[features.sum(axis=1) > 3]
print(len(matches))
```

--------------------------------

### Full Deduplication Workflow Example

Source: https://recordlinkage.readthedocs.io/en/latest/guides/data_deduplication.html

A complete example demonstrating the entire deduplication process: loading data, indexing candidate links, comparing attributes, and classifying matches. This script provides a runnable end-to-end solution.

```python
import recordlinkage
from recordlinkage.datasets import load_febrl1

dfA = load_febrl1()

# Indexation step
indexer = recordlinkage.Index()
indexer.block(left_on="given_name")
candidate_links = indexer.index(dfA)

# Comparison step
compare_cl = recordlinkage.Compare()

compare_cl.exact("given_name", "given_name", label="given_name")
compare_cl.string(
    "surname", "surname", method="jarowinkler", threshold=0.85, label="surname"
)
compare_cl.exact("date_of_birth", "date_of_birth", label="date_of_birth")
compare_cl.exact("suburb", "suburb", label="suburb")
compare_cl.exact("state", "state", label="state")
compare_cl.string("address_1", "address_1", threshold=0.85, label="address_1")

features = compare_cl.compute(candidate_links, dfA)

# Classification step
matches = features[features.sum(axis=1) > 3]
print(len(matches))
```

--------------------------------

### Install Airspeed Velocity (asv)

Source: https://recordlinkage.readthedocs.io/en/latest/contributing.html

Install Airspeed Velocity, a tool for monitoring performance changes. This is crucial for maintaining performance standards in the toolkit.

```bash
pip install asv

```

--------------------------------

### Example: Read Links from Annotation File

Source: https://recordlinkage.readthedocs.io/en/latest/annotation.html

Demonstrates reading links from a specified annotation file and printing them. Ensure the file path is correct.

```python
> annotation = read_annotation_file("result.json")
> print(annotation.links)
```

--------------------------------

### Custom Index Example

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Illustrates how to create a custom indexing algorithm by subclassing BaseIndexAlgorithm and implementing the _link_index and optionally _dedup_index methods.

```APIDOC
## Example: Creating a Custom Index

### Description
This example demonstrates how to define a new indexing algorithm by inheriting from `recordlinkage.base.BaseIndexAlgorithm` and overriding the necessary methods.

### Usage

```python
class CustomIndex(BaseIndexAlgorithm):

    def _link_index(self, df_a, df_b):
        # Custom index logic for linking two datasets
        # df_a and df_b are pandas.Series objects (or tuples of Series)
        # Return a pandas.MultiIndex of record pairs
        return ...

    def _dedup_index(self, df_a):
        # Custom index logic for duplicate detection within a single dataset (optional)
        # df_a is a pandas.Series object (or tuple of Series)
        # Return a pandas.MultiIndex of record pairs
        return ...

# Instantiate and use the custom index
custom_index = CustomIndex()
result_index = custom_index.index(dataframe_a, dataframe_b) # For linking
# or
# result_index = custom_index.index(dataframe_a) # For deduplication
```
```

--------------------------------

### Install RecordLinkage with Pip

Source: https://recordlinkage.readthedocs.io/en/latest/installation.html

Install the Python Record Linkage Toolkit using pip. This is the standard method for adding the package to your Python environment.

```bash
pip install recordlinkage
```

--------------------------------

### Custom Indexing Algorithm: First Letter 'W'

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Implements and uses a custom indexing algorithm to find record pairs where the given name starts with 'W'. This example requires importing BaseIndexAlgorithm and pandas.

```python
import recordlinkage
from recordlinkage.datasets import load_febrl4

df_a, df_b = load_febrl4()

from recordlinkage.base import BaseIndexAlgorithm

class FirstLetterWIndex(BaseIndexAlgorithm):
    """Custom class for indexing"""

    def _link_index(self, df_a, df_b):
        """Make pairs with given names starting with the letter 'w'."""

        # Select records with names starting with a w.
        name_a_w = df_a[df_a['given_name'].str.startswith('w') == True]
        name_b_w = df_b[df_b['given_name'].str.startswith('w') == True]

        # Make a product of the two numpy arrays
        return pandas.MultiIndex.from_product(
            [name_a_w.index.values, name_b_w.index.values],
            names=[df_a.index.name, df_b.index.name]
        )

indexer = FirstLetterWIndex()
candidate_pairs = indexer.index(df_a, df_b)

print ('Returns a', type(candidate_pairs).__name__)
print ('Number of candidate record pairs starting with the letter w:', len(candidate_pairs))
```

--------------------------------

### Add a Full Index Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

This example shows how to add a 'full' index algorithm to an `Index` object. A full index creates all possible pairs between records, which can be computationally intensive but ensures no potential matches are missed.

```python
from recordlinkage.index import Full

indexer = recordlinkage.Index()
indexer.add(Full())
```

--------------------------------

### Initialize Compare Object and Add Comparison Methods

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Initializes the `recordlinkage.Compare` class and adds various comparison methods for different data types. This is a high-level usage example for comparing attributes between two datasets.

```python
comp = recordlinkage.Compare()

comp.string('first_name', 'name', method='jarowinkler')
comp.string('lastname', 'lastname', method='jarowinkler')
comp.exact('dateofbirth', 'dob')
comp.exact('sex', 'sex')
comp.string('address', 'address', method='levenshtein')
comp.exact('place', 'place')
comp.numeric('income', 'income')
```

--------------------------------

### Define Record Comparisons

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

Set up comparison methods for various attributes. This example includes string comparisons with specific methods and thresholds, exact matches, and handling of missing values.

```python
compare = recordlinkage.Compare()

compare.string('name', 'name', method='jarowinkler', threshold=0.85)
compare.exact('sex', 'gender')
compare.exact('dob', 'date_of_birth')
compare.string('streetname', 'streetname', method='damerau_levenshtein', threshold=0.7)
compare.exact('place', 'placename')
compare.exact('haircolor', 'haircolor', missing_value=9)

```

--------------------------------

### Unsupervised Learning with ECM Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

An example of unsupervised learning using the ECM algorithm. This algorithm does not require pre-defined training data.

```python
ecm = recordlinkage.BernoulliEMClassifier()
ecm.fit_predict(compare_vectors)
```

--------------------------------

### Add a Sorted Neighbourhood Index Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

This example shows how to add a 'SortedNeighbourhood' index algorithm. This method is efficient for large datasets by sorting records and then only comparing records within a defined window, reducing the number of comparisons.

```python
from recordlinkage.index import SortedNeighbourhood

indexer = recordlinkage.Index()
indexer.add(SortedNeighbourhood())
```

--------------------------------

### Combine Multiple Blocking Strategies

Source: https://recordlinkage.readthedocs.io/en/latest/performance.html

To avoid excluding too many links when blocking, you can combine multiple blocking strategies. This example adds a second blocking condition based on age.

```python
indexer = recordlinkage.Index()
indexer.block(left_on=['first_name', 'surname'],
              right_on=['name', 'surname'])
indexer.block(left_on=['first_name', 'age'],
              right_on=['name', 'age'])
pairs = indexer.index(dfA, dfB)
```

--------------------------------

### Supervised Learning with Logistic Regression

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

An example of supervised learning using Logistic Regression. Requires training data (golden data) to fit the model.

```python
true_linkage = pandas.Series(YOUR_GOLDEN_DATA, index=pandas.MultiIndex(YOUR_MULTI_INDEX))

logrg = recordlinkage.LogisticRegressionClassifier()
logrg.fit(compare_vectors[true_linkage.index], true_linkage)

logrg.predict(compare_vectors)
```

--------------------------------

### Scikit-learn Random Forest Classifier Adapter

Source: https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html

Example of creating a custom classifier using Scikit-learn's RandomForestClassifier wrapped by RecordLinkage's SKLearnClassifier. This is useful for classifying record pairs with a Random Forest model.

```python
from sklearn.ensemble import RandomForestClassifier

from recordlinkage.base import BaseClassifier
from recordlinkage.adapters import SKLearnClassifier
from recordlinkage.datasets import binary_vectors

class RandomForest(SKLearnClassifier, BaseClassifier):

    def __init__(*args, **kwargs):
        super(self, RandomForest).__init__()

        # set the kernel
        kernel = RandomForestClassifier(*args, **kwargs)


# make a sample dataset
features, links = binary_vectors(10000, 2000, return_links=True)

# initialise the random forest
cl = RandomForest(n_estimators=20)
cl.fit(features, links)

# predict the matches
cl.predict(...)

```

--------------------------------

### Keras Neural Network Classifier Adapter

Source: https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html

Example of creating a custom classifier using Keras models wrapped by RecordLinkage's KerasAdapter. This allows classification of record pairs using a neural network built with Keras.

```python
from tensorflow.keras import layers
from recordlinkage.base import BaseClassifier
from recordlinkage.adapters import KerasAdapter

class NNClassifier(KerasAdapter, BaseClassifier):
    """Neural network classifier."""
    def __init__(self):
        super(NNClassifier, self).__init__()

        model = tf.keras.Sequential()
        model.add(layers.Dense(16, input_dim=8, activation='relu'))
        model.add(layers.Dense(8, activation='relu'))
        model.add(layers.Dense(1, activation='sigmoid'))
        model.compile(
            optimizer=tf.train.AdamOptimizer(0.001),
            loss='binary_crossentropy',
            metrics=['accuracy']
        )

        self.kernel = model

# initialise the model
cl = NNClassifier()
# fit the model to the data
cl.fit(X_train, links_true)
# predict the class of the data
cl.predict(X_pred)

```

--------------------------------

### Initialize and Configure Comparison

Source: https://recordlinkage.readthedocs.io/en/latest/guides/data_deduplication.html

Instantiate the Compare class and add various comparison methods (exact, string with threshold) for different attributes. This sets up how records will be compared.

```python
compare_cl = recordlinkage.Compare()
compare_cl.exact("given_name", "given_name", label="given_name")
compare_cl.string(
    "surname", "surname", method="jarowinkler", threshold=0.85, label="surname"
)
compare_cl.exact("date_of_birth", "date_of_birth", label="date_of_birth")
compare_cl.exact("suburb", "suburb", label="suburb")
compare_cl.exact("state", "state", label="state")
compare_cl.string("address_1", "address_1", threshold=0.85, label="address_1")
features = compare_cl.compute(candidate_links, dfA)
```

--------------------------------

### Importing the Block Index Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Demonstrates how to import a specific indexing algorithm, such as Block, from the recordlinkage.index submodule.

```python
from recordlinkage.index import Block
```

--------------------------------

### Create and Execute a Block Index

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

This snippet demonstrates how to initialize an `Index` object, add a blocking strategy based on specific columns, and then execute the indexing process on two datasets. It's useful for creating candidate link pairs by matching records on common attributes.

```python
indexer = recordlinkage.Index()
indexer.block(left_on='first_name', right_on='givenname')
indexer.index(census_data_1980, census_data_1990)
```

--------------------------------

### Compute Comparison Features

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Computes the similarity features for a given set of pairs and dataframes. The resulting features can then be analyzed, for example, by calculating their mean.

```python
features = comparer.compute(pairs, dfA, dfB)
features.mean()
```

--------------------------------

### Initialize and Configure Comparison

Source: https://recordlinkage.readthedocs.io/en/latest/guides/link_two_dataframes.html

Initializes a `Compare` object and adds various comparison methods (exact and string) for specified attributes. This sets up the comparison logic for candidate record pairs.

```python
compare_cl = recordlinkage.Compare()
compare_cl.exact("given_name", "given_name", label="given_name")
compare_cl.string(
    "surname", "surname", method="jarowinkler", threshold=0.85, label="surname"
)
compare_cl.exact("date_of_birth", "date_of_birth", label="date_of_birth")
compare_cl.exact("suburb", "suburb", label="suburb")
compare_cl.exact("state", "state", label="state")
compare_cl.string("address_1", "address_1", threshold=0.85, label="address_1")
```

--------------------------------

### Run Performance Tests (Version Range)

Source: https://recordlinkage.readthedocs.io/en/latest/contributing.html

Run performance tests for all versions of the package since a specific tag (e.g., v0.6.0) up to the master branch. This provides a historical performance overview.

```bash
asv run --skip-existing-commits v0.6.0..master

```

--------------------------------

### Block Indexing with recordlinkage

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Demonstrates how to use the Block indexer to create candidate pairs based on shared values in specified columns.

```python
import recordlinkage as rl
from recordlinkage.datasets import load_febrl4
from recordlinkage.index import Block

df_a, df_b = load_febrl4()

indexer = rl.Index()
indexer.add(Block('given_name', 'given_name'))
indexer.add(Block('surname', 'surname'))
indexer.index(df_a, df_b)
```

--------------------------------

### Run Performance Tests (Current Version)

Source: https://recordlinkage.readthedocs.io/en/latest/contributing.html

Execute performance tests for the current version of the package from the repository root. This helps identify performance regressions.

```bash
asv run

```

--------------------------------

### Custom Indexing Algorithm with Parameterized Letter

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Defines a custom index class where the starting letter for filtering is a parameter initialized during object creation. This allows for flexible filtering based on the first letter of the given name.

```python
class FirstLetterIndex(BaseIndexAlgorithm):
    """Custom class for indexing"""

    def __init__(self, letter):
        super(FirstLetterIndex, self).__init__()

        # the letter to save
        self.letter = letter

    def _link_index(self, df_a, df_b):
        """Make record pairs that agree on the first letter of the given name."""

        # Select records with names starting with a 'letter'.
        a_startswith_w = df_a[df_a['given_name'].str.startswith(self.letter) == True]
        b_startswith_w = df_b[df_b['given_name'].str.startswith(self.letter) == True]

        # Make a product of the two numpy arrays
        return pandas.MultiIndex.from_product(
            [a_startswith_w.index.values, b_startswith_w.index.values],
            names=[df_a.index.name, df_b.index.name]
        )
```

--------------------------------

### Block Indexing using shorthand method

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Shows an equivalent way to perform block indexing using the shorthand 'block' method.

```python
import recordlinkage as rl
from recordlinkage.datasets import load_febrl4

df_a, df_b = load_febrl4()

indexer = rl.Index()
indexer.block('given_name', 'given_name')
indexer.block('surname', 'surname')
index.index(df_a, df_b)
```

--------------------------------

### recordlinkage.describe_option

Source: https://recordlinkage.readthedocs.io/en/latest/ref-misc.html

Prints the description for one or more registered options. If called without arguments, it lists all registered options.

```APIDOC
## recordlinkage.describe_option

### Description
Prints the description for one or more registered options.
Call with not arguments to get a listing for all registered options.
The available options with its descriptions: 

classification.return_typestr
    
The format of the classification result. The value ‘index’ returns the classification result as a pandas.MultiIndex. The MultiIndex contains the predicted matching record pairs. The value ‘series’ returns a pandas.Series with zeros (distinct) and ones (matches). The argument value ‘array’ will return a numpy.ndarray with zeros and ones. [default: index] [currently: index] 

indexing.pairsstr
    
Specify the format how record pairs are stored. By default, record pairs generated by the toolkit are returned in a pandas.MultiIndex object (‘multiindex’ option).
Valid values: ‘multiindex’ [default: multiindex] [currently: multiindex] 

### Parameters
#### Path Parameters
- **pat** (_str_) – Regexp which should match a single option. Note: partial matches are supported for convenience, but unless you use the full option name (e.g. x.y.z.option_name), your code may break in future versions if new options with similar names are introduced.
- **__print_desc =False** (_bool_) –

### Returns
- **None** 
```

--------------------------------

### Train and Predict with K-means Clustering

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Initializes and trains a K-means classifier, then immediately predicts the cluster assignments for the provided comparison vectors. This is an unsupervised learning method.

```python
kmeans = rl.KMeansClassifier()
result_kmeans = kmeans.fit_predict(krebs_X)
len(result_kmeans)
```

--------------------------------

### Create Candidate Links using Blocking

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

Generate candidate record pairs using the blocking index method on the 'surname' attribute to reduce computational load. Only pairs with matching surnames will be considered.

```python
indexer = recordlinkage.Index()
indexer.block('surname')
candidate_links = indexer.index(df_a, df_b)

```

--------------------------------

### Import RecordLinkage and Load Febrl Dataset

Source: https://recordlinkage.readthedocs.io/en/latest/guides/data_deduplication.html

Imports the recordlinkage library and its datasets submodule. Loads the Febrl dataset, which is suitable for testing deduplication techniques. The loaded data is a pandas DataFrame.

```python
import recordlinkage
from recordlinkage.datasets import load_febrl1
```

--------------------------------

### Generating Block Index

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Shows how to create candidate record pairs that agree on specified variables using the Block algorithm. This is an effective method for reducing the comparison space.

```python
indexer = recordlinkage.BlockIndex(on='first_name')
indexer.index(census_data_1980, census_data_1990)
```

--------------------------------

### Instantiate and Use a Custom Index Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

After defining a custom indexing algorithm, instantiate the class and call its `index()` method to generate record pairs. This method can link two DataFrames or find duplicates within a single DataFrame.

```python
custom_index = CustomIndex():
custom_index.index()
```

--------------------------------

### High-Level Compare Usage

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Instantiate the Compare class and add comparison methods like string, exact, date, and numeric. Then, compute the comparisons on the provided pairs and dataframes.

```python
import recordlinkage as rl

comparer = rl.Compare()
comparer.string('name_a', 'name_b', method='jarowinkler', threshold=0.85, label='name')
comparer.exact('sex', 'gender', label='gender')
comparer.date('dob', 'date_of_birth', label='date')
comparer.string('str_name', 'streetname', method='damerau_levenshtein', threshold=0.7, label='streetname')
comparer.exact('place', 'placename', label='placename')
comparer.numeric('income', 'income', method='gauss', offset=3, scale=3, missing_value=0.5, label='income')
comparer.compute(pairs, dfA, dfB)
```

--------------------------------

### Import Preprocessing Functions

Source: https://recordlinkage.readthedocs.io/en/latest/ref-preprocessing.html

Import the necessary cleaning and phonetic encoding functions from the recordlinkage.preprocessing submodule.

```python
from recordlinkage.preprocessing import clean, phonetic

```

--------------------------------

### Import recordlinkage and load Febrl datasets

Source: https://recordlinkage.readthedocs.io/en/latest/guides/link_two_dataframes.html

Imports the necessary recordlinkage module and loads the Febrl datasets 4A and 4B. These datasets are returned as pandas DataFrames.

```python
import recordlinkage
from recordlinkage.datasets import load_febrl4
```

```python
dfA, dfB = load_febrl4()
dfA
```

--------------------------------

### Generating Sorted Neighbourhood Index

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Demonstrates the use of the SortedNeighbourhood algorithm to generate candidate pairs based on agreement on a sorting key and records within a defined neighbourhood. This is useful for handling minor spelling variations.

```python
indexer = recordlinkage.SortedNeighbourhoodIndex(left_on='first_name', right_on='first_name', window=5)
indexer.index(census_data_1980, census_data_1990)
```

--------------------------------

### Low-Level Compare Usage

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Instantiate the Compare class with a list of comparison objects (e.g., String, Exact, Date, Numeric). Then, compute the comparisons on the provided pairs and dataframes.

```python
import recordlinkage as rl
from recordlinkage.compare import Exact, String, Numeric, Date

comparer = rl.Compare([
    String('name_a', 'name_b', method='jarowinkler', threshold=0.85, label='name'),
    Exact('sex', 'gender', label='gender'),
    Date('dob', 'date_of_birth', label='date'),
    String('str_name', 'streetname', method='damerau_levenshtein', threshold=0.7, label='streetname'),
    Exact('place', 'placename', label='placename'),
    Numeric('income', 'income', method='gauss', offset=3, scale=3, missing_value=0.5, label='income'),
])
comparer.compute(pairs, dfA, dfB)
```

--------------------------------

### Initialize and Train Naive Bayes Classifier

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Initializes a NaiveBayesClassifier with a specified binarization threshold and trains it using the golden pairs.

```python
nb = rl.NaiveBayesClassifier(binarize=0.3)
nb.fit(golden_pairs, golden_matches_index)
```

--------------------------------

### Full Indexing for Large Files

Source: https://recordlinkage.readthedocs.io/en/latest/performance.html

This method handles large input files by splitting them into smaller blocks, computing record pairs for each block, and then iterating over these blocks. While usable, SQL databases might offer better performance for very large datasets.

```python
import recordlinkage
import numpy

cl = recordlinkage.index.Full()

for dfB_subset in numpy.split(dfB):

    # a subset of record pairs
    pairs_subset = cl.index(dfA, dfB_subset)

    # Your analysis on pairs_subset here

```

--------------------------------

### Add a Random Index Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

This snippet demonstrates adding a 'Random' index algorithm. This method is useful when a full comparison is not feasible due to dataset size, as it randomly samples pairs of records to compare.

```python
from recordlinkage.index import Random

indexer = recordlinkage.Index()
indexer.add(Random())
```

--------------------------------

### learn

Source: https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html

Deprecated method, use 'fit_predict' instead.

```APIDOC
## learn

### Description
[DEPRECATED] Use ‘fit_predict’.
```

--------------------------------

### Generate Annotation File for Linking

Source: https://recordlinkage.readthedocs.io/en/latest/annotation.html

Use this snippet to create an annotation file for record linkage tasks. It requires loading two datasets, defining an indexing strategy, and then calling `write_annotation_file` with appropriate parameters.

```python
import recordlinkage as rl
from recordlinkage.index import Block
from recordlinkage.datasets import load_febrl4

df_a, df_b = load_febrl4()

blocker = Block("surname", "surname")
pairs = blocker.index(df_a, df_b)

rl.write_annotation_file(
    "annotation_demo_linking.json",
    pairs[0:50],
    df_a,
    df_b,
    dataset_a_name="Febrl4 A",
    dataset_b_name="Febrl4 B"
)

```

--------------------------------

### recordlinkage.datasets.load_febrl4

Source: https://recordlinkage.readthedocs.io/en/latest/ref-datasets.html

Loads the FEBRL 4 dataset, consisting of 10000 records split into two files (dataset4a.csv and dataset4b.csv) for testing linkage procedures. It can optionally return the true links.

```APIDOC
## load_febrl4

### Description
Loads the FEBRL 4 dataset, which consists of 10000 records (5000 originals and 5000 duplicates). The originals and duplicates are split into two separate files (dataset4a.csv and dataset4b.csv). This function can optionally return the true links.

### Parameters
#### Query Parameters
- **return_links** (bool) - Optional - When True, the function returns also the true links.

### Returns
- **(pandas.DataFrame, pandas.DataFrame)** - A `pandas.DataFrame` with Febrl dataset4a.csv and a pandas dataframe with Febrl dataset4b.csv. When return_links is True, the function returns also the true links.
```

--------------------------------

### recordlinkage.datasets.load_febrl3

Source: https://recordlinkage.readthedocs.io/en/latest/ref-datasets.html

Loads the FEBRL 3 dataset, which contains 5000 records with duplicates. It can optionally return the true links between records.

```APIDOC
## load_febrl3

### Description
Loads the FEBRL 3 dataset, which contains 5000 records (2000 originals and 3000 duplicates). It can optionally return the true links between records.

### Parameters
#### Query Parameters
- **return_links** (bool) - Optional - When True, the function returns also the true links.

### Returns
- **pandas.DataFrame** - A `pandas.DataFrame` with Febrl dataset3.csv. When return_links is True, the function returns also the true links. The true links are all links in the lower triangular part of the matrix.
```

--------------------------------

### recordlinkage.datasets.load_febrl2

Source: https://recordlinkage.readthedocs.io/en/latest/ref-datasets.html

Loads the second FEBRL dataset, containing 5000 records with a distribution of duplicates. This function can optionally return the true links.

```APIDOC
## load_febrl2

### Description
Loads the second FEBRL dataset, containing 5000 records with a distribution of duplicates. This function can optionally return the true links.

### Method
load_febrl2

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **return_links** (_bool_) - Optional - When True, the function returns also the true links.

### Returns
_pandas.DataFrame_ – A `pandas.DataFrame` with Febrl dataset2.csv. When return_links is True, the function returns also the true links. The true links are all links in the lower triangular part of the matrix.

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### Supervised Classifier Methods

Source: https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html

Provides documentation for the core methods of supervised classifiers, including training, prediction, and probability computation.

```APIDOC
## fit_predict

### Description
Train the classifier and predict the class of the record pairs.

### Method
fit_predict

### Parameters
#### Path Parameters
- **comparison_vectors** (pandas.DataFrame) - The comparison vectors.
- **match_index** (pandas.MultiIndex) - The true matches.

#### Query Parameters
- **return_type** (str) - Deprecated. Use recordlinkage.options instead. Use the option recordlinkage.set_option(‘classification.return_type’, ‘index’) instead.

### Returns
- **pandas.Series** - A pandas Series with the labels 1 (for the matches) and 0 (for the non-matches).

## learn

### Description
[DEPRECATED] Use ‘fit_predict’.

### Method
learn

## predict

### Description
Predict the class of the record pairs. Classify a set of record pairs based on their comparison vectors into matches, non-matches and possible matches. The classifier has to be trained to call this method.

### Method
predict

### Parameters
#### Path Parameters
- **comparison_vectors** (pandas.DataFrame) - Dataframe with comparison vectors.

#### Query Parameters
- **return_type** (str) - Deprecated. Use recordlinkage.options instead. Use the option recordlinkage.set_option(‘classification.return_type’, ‘index’) instead.

### Returns
- **pandas.Series** - A pandas Series with the labels 1 (for the matches) and 0 (for the non-matches).

## prob

### Description
Compute the probabilities for each record pair. For each pair of records, estimate the probability of being a match.

### Method
prob

### Parameters
#### Path Parameters
- **comparison_vectors** (pandas.DataFrame) - The dataframe with comparison vectors.

#### Query Parameters
- **return_type** (str) - Deprecated. (default ‘series’)

### Returns
- **pandas.Series or numpy.ndarray** - The probability of being a match for each record pair.
```

--------------------------------

### Import Record Linkage and Pandas

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

Import the necessary libraries for record linkage and data manipulation.

```python
import recordlinkage
import pandas

```

--------------------------------

### Generate Full Index for Record Pairs

Source: https://recordlinkage.readthedocs.io/en/latest/guides/data_deduplication.html

Initializes a recordlinkage Indexer and uses the full() method to generate all possible unique pairs of records from a single DataFrame (dfA). This is a common first step in deduplication.

```python
indexer = recordlinkage.Index()
indexer.full()
candidate_links = indexer.index(dfA)
```

--------------------------------

### Load Datasets into Pandas DataFrames

Source: https://recordlinkage.readthedocs.io/en/latest/about.html

Load your datasets into pandas DataFrames for processing. Replace YOUR_FIRST_DATASET and YOUR_SECOND_DATASET with your actual data.

```python
df_a = pandas.DataFrame(YOUR_FIRST_DATASET)
df_b = pandas.DataFrame(YOUR_SECOND_DATASET)

```

--------------------------------

### Exact Comparison

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Compares attributes of record pairs for exact matches. This is a shortcut for using the Exact comparison algorithm.

```python
from recordlinkage.compare import Exact

indexer = recordlinkage.Compare()
indexer.add(Exact())
```

--------------------------------

### Run Package Tests

Source: https://recordlinkage.readthedocs.io/en/latest/contributing.html

Execute all tests for the record linkage package using pytest. Ensure all tests pass before submitting contributions.

```bash
python -m pytest tests/

```

--------------------------------

### Print Number of Records and Pairs

Source: https://recordlinkage.readthedocs.io/en/latest/guides/link_two_dataframes.html

Displays the number of records in dfA, dfB, and the total number of generated pairs. This helps in understanding the scale of the indexing process.

```python
print(len(dfA), len(dfB), len(pairs))
```

--------------------------------

### List of Supported Phonetic Algorithms

Source: https://recordlinkage.readthedocs.io/en/latest/ref-preprocessing.html

This variable holds a list of all supported phonetic encoding algorithms available in the toolkit.

```python
preprocessing.phonetic_algorithms = ['soundex', 'nysiis', 'metaphone', 'match_rating']
```

--------------------------------

### Generate Annotation File for Deduplication

Source: https://recordlinkage.readthedocs.io/en/latest/annotation.html

This snippet demonstrates how to generate an annotation file specifically for deduplication tasks. It involves loading a single dataset, indexing potential duplicates, and then saving the annotation file.

```python
import recordlinkage as rl
from recordlinkage.index import Block
from recordlinkage.datasets import load_febrl1

df_a = load_febrl1()

blocker = Block("surname", "surname")
pairs = blocker.index(df_a)

rl.write_annotation_file(
    "annotation_demo_dedup.json",
    pairs[0:50],
    df_a,
    dataset_a_name="Febrl1 A"
)

```

--------------------------------

### recordlinkage.datasets.load_febrl1

Source: https://recordlinkage.readthedocs.io/en/latest/ref-datasets.html

Loads the first FEBRL dataset, which contains 1000 records (500 original and 500 duplicates). This function can optionally return the true links between records.

```APIDOC
## load_febrl1

### Description
Loads the first FEBRL dataset, which contains 1000 records (500 original and 500 duplicates). This function can optionally return the true links between records.

### Method
load_febrl1

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **return_links** (_bool_) - Optional - When True, the function returns also the true links.

### Returns
_pandas.DataFrame_ – A `pandas.DataFrame` with Febrl dataset1.csv. When return_links is True, the function returns also the true links. The true links are all links in the lower triangular part of the matrix.

### Request Example
None

### Response
#### Success Response (200)
None

#### Response Example
None
```

--------------------------------

### One-to-One Linking (Greedy Method)

Source: https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html

Applies one-to-one linking to a set of record pairs. This method ensures that each record from one dataset is matched to at most one record in the other dataset. It is an experimental feature.

```python
one_to_one = OneToOneLinking()
links = one_to_one.compute(links)
```

--------------------------------

### Load and Inspect Krebs Register Dataset

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Loads the Krebs register dataset, separating comparison vectors (krebs_X) from true links (krebs_true_links). Displays the first few rows of the comparison vectors to show the data structure.

```python
krebs_X, krebs_true_links = load_krebsregister(missing_values=0)
krebs_X
```

--------------------------------

### Define a Custom Indexing Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Create a custom indexing algorithm by subclassing `BaseIndexAlgorithm` and overwriting the `_link_index` and optionally `_dedup_index` methods. The `_link_index` method is for linking two datasets, while `_dedup_index` is for finding duplicates within a single dataset.

```python
class CustomIndex(BaseIndexAlgorithm):

    def _link_index(self, df_a, df_b):

        # Custom index for linking.

        return ...

    def _dedup_index(self, df_a):

        # Custom index for duplicate detection, optional.

        return ...
```

--------------------------------

### Initialize Logistic Regression Classifier

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Initializes the LogisticRegressionClassifier from the recordlinkage library.

```python
logreg = rl.LogisticRegressionClassifier()
```

--------------------------------

### Define Custom Address Comparison Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Implements a custom algorithm to compare addresses, handling cases where address components might be swapped between records. This is useful for robust address matching when order might not be consistent.

```python
import recordlinkage as rl
from recordlinkage.base import BaseCompareFeature

class CompareAddress(BaseCompareFeature):

    def _compute_vectorized(self, s1_1, s1_2, s2_1, s2_2):
        """Compare addresses.

        Compare addresses. Compare address_1 of file A with
        address_1 and address_2 of file B. The same for address_2
        of dataset 1.

        """

        return ((s1_1 == s2_1) | (s1_2 == s2_2) | (s1_1 == s2_2) | (s1_2 == s2_1)).astype(float)

comparer = rl.Compare()

# naive
comparer.add(CompareAddress('address_1', 'address_1', label='sim_address_1'))
comparer.add(CompareAddress('address_2', 'address_2', label='sim_address_2'))

```

--------------------------------

### Prepare Golden Pairs for Training

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Selects a subset of record pairs and identifies the true matching pairs from the known links for training supervised classifiers.

```python
golden_pairs = krebs_X[0:5000]
# 2093 matching pairs
golden_matches_index = golden_pairs.index.intersection(krebs_true_links)
golden_matches_index
```

--------------------------------

### BaseIndexAlgorithm Class

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

The BaseIndexAlgorithm class serves as the foundation for creating custom indexing algorithms. It provides abstract methods that can be overridden to define specific logic for linking datasets or detecting duplicates within a single dataset.

```APIDOC
## Class: recordlinkage.base.BaseIndexAlgorithm

### Description
Base class for all index algorithms. This is an abstract class for indexing algorithms.

### Parameters
* **verify_integrity** (bool) - Verify the integrity of the input dataframe(s). The index is checked for duplicate values.
* **suffixes** (tuple) - If the names of the resulting MultiIndex are identical, the suffixes are used to distinguish the names.

### Methods
#### _link_index(df_a, df_b)
Build an index for linking two datasets.

Parameters:
* **df_a** (tuple of pandas.Series) - The data of the left DataFrame to build the index with.
* **df_b** (tuple of pandas.Series) - The data of the right DataFrame to build the index with.

Returns:
pandas.MultiIndex - A pandas.MultiIndex with record pairs. Each record pair contains the index values of two records.

#### _dedup_index(df_a)
Build an index for duplicate detection in a dataset.
This method can be used to implement an algorithm for duplicate detection. This method is optional if method `_link_index()` is implemented.

Parameters:
* **df_a** (tuple of pandas.Series) - The data of the DataFrame to build the index with.

Returns:
pandas.MultiIndex - A pandas.MultiIndex with record pairs. Each record pair contains the index values of two records. The records are sampled from the lower triangular part of the matrix.

#### index(x, x_link=None)
Make an index of record pairs.
Use a custom function to make record pairs of one or two dataframes. Each function should return a pandas.MultiIndex with record pairs.

Parameters:
* **x** (pandas.DataFrame) - A pandas DataFrame. When x_link is None, the algorithm makes record pairs within the DataFrame. When x_link is not empty, the algorithm makes pairs between x and x_link.
* **x_link** (pandas.DataFrame, optional) - A second DataFrame to link with the DataFrame x.

Returns:
pandas.MultiIndex - A pandas.MultiIndex with record pairs. Each record pair contains the index labels of two records.
```

--------------------------------

### String Comparison

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Compares attributes of record pairs using string similarity algorithms. This is a shortcut for using the String comparison algorithm.

```python
from recordlinkage.compare import String

indexer = recordlinkage.Compare()
indexer.add(String())
```

--------------------------------

### Define Custom Zip Code Comparison Algorithm

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Creates a custom comparison algorithm for zip codes. It returns 1.0 for identical zip codes, 0.5 if the first two digits match but the rest differ, and 0.0 otherwise. This is useful for approximate matching of zip codes.

```python
import recordlinkage as rl
from recordlinkage.base import BaseCompareFeature

class CompareZipCodes(BaseCompareFeature):

    def _compute_vectorized(self, s1, s2):
        """Compare zipcodes.

        If the zipcodes in both records are identical, the similarity
        is 1. If the first two values agree and the last two don't, then
        the similarity is 0.5. Otherwise, the similarity is 0.
        """

        # check if the zipcode are identical (return 1 or 0)
        sim = (s1 == s2).astype(float)

        # check the first 2 numbers of the distinct comparisons
        sim[(sim == 0) & (s1.str[0:2] == s2.str[0:2])] = 0.5

        return sim

comparer = rl.Compare()
comparer.extact('given_name', 'given_name', 'y_name')
comparer.string('surname', 'surname', 'y_surname')
comparer.add(CompareZipCodes('postcode', 'postcode', label='y_postcode'))
comparer.compute(pairs, dfA, dfB)

```

--------------------------------

### Date Comparison

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Compares attributes of record pairs using date similarity algorithms. This is a shortcut for using the Date comparison algorithm.

```python
from recordlinkage.compare import Date

indexer = recordlinkage.Compare()
indexer.add(Date())
```

--------------------------------

### recordlinkage.index.Full

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Generates a 'full' index containing all possible combinations of record pairs. This is equivalent to a Cartesian product for linking two DataFrames or the upper triangular matrix for deduplicating a single DataFrame. Use with caution on large datasets due to quadratic scaling.

```APIDOC
## Class: recordlinkage.index.Full

### Description
Class to generate a ‘full’ index, which includes all possible combinations of record pairs. For linking, it generates the Cartesian product of two DataFrames. For deduplication, it generates pairs from the upper triangular matrix of a single DataFrame.

### Parameters
* **kwargs**: Additional keyword arguments to pass to `recordlinkage.base.BaseIndexAlgorithm`.

### Note
This indexation method can be slow for large DataFrame’s as the number of comparisons scales quadratically. It may also not be suitable for all classifiers due to the large number of distinct pairs generated.

## Method: index

### Description
Make an index of record pairs using the full indexer.

### Parameters
* **x** (_pandas.DataFrame_) – The primary DataFrame.
* **x_link** (_pandas.DataFrame_, optional) – A second DataFrame to link with `x`. If None, pairs are generated within `x`.

### Returns
* _pandas.MultiIndex_ – A pandas.MultiIndex with record pairs, containing the index labels of two records.
```

--------------------------------

### Import Libraries and Load Krebs Register Data

Source: https://recordlinkage.readthedocs.io/en/latest/guides/classifiers.html

Imports necessary libraries (pandas, recordlinkage) and loads the Krebs register dataset, which contains comparison vectors for record pairs. This is a prerequisite for using classification algorithms.

```python
import pandas
import recordlinkage as rl
from recordlinkage.datasets import load_krebsregister

```

--------------------------------

### Random indexer with replacement

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Generate random record pairs with replacement using the Random indexer. This is useful for training unsupervised learning models for record linkage.

```python
>>> indexer = recordlinkage.Random(n=1000, replace=True, random_state=42)
>>> indexer.index(census_data_1980, census_data_1990)

```

--------------------------------

### string

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Compares attributes of pairs using the string algorithm. This is a shortcut for `recordlinkage.compare.String`.

```APIDOC
## string

### Description
Compare attributes of pairs with string algorithm.
Shortcut of `recordlinkage.compare.String`:
```
from recordlinkage.compare import String

indexer = recordlinkage.Compare()
indexer.add(String())
```

### Method
string

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Response
#### Success Response (200)
None specified in source.

#### Response Example
None provided in source.
```

--------------------------------

### Load and Inspect Febrl Dataset

Source: https://recordlinkage.readthedocs.io/en/latest/guides/data_deduplication.html

Loads the Febrl dataset into a pandas DataFrame named dfA. This dataset contains fictitious records for testing deduplication. The output shows the first few rows and columns of the DataFrame.

```python
dfA = load_febrl1()
dfA
```

--------------------------------

### recordlinkage.index.SortedNeighbourhood

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Generates candidate record pairs using the SortedNeighbourhood algorithm. It identifies pairs that agree on a sorting key and also considers pairs within a defined 'neighbourhood' (window). This is useful for handling minor spelling mistakes.

```APIDOC
## Class: recordlinkage.index.SortedNeighbourhood

### Description
Make candidate record pairs with the SortedNeighbourhood algorithm. This algorithm returns record pairs that agree on the sorting key, and also records pairs in their neighbourhood. A large window size results in more record pairs. A window size of 1 returns the blocking index.

### Parameters
* **left_on** (_label_, optional) – The column name of the sorting key of the first/left DataFrame.
* **right_on** (_label_, optional) – The column name of the sorting key of the second/right DataFrame.
* **window** (_int_, optional) – The width of the window. Defaults to 3.
* **sorting_key_values** (_array_, optional) – A list of sorting key values.
* **block_on** (_label_) – Additional columns to apply standard blocking on.
* **block_left_on** (_label_) – Additional columns in the left DataFrame to apply standard blocking on.
* **block_right_on** (_label_) – Additional columns in the right DataFrame to apply standard blocking on.
```

--------------------------------

### Generating Full Index

Source: https://recordlinkage.readthedocs.io/en/latest/ref-index.html

Illustrates the creation of a full index, which generates all possible record pairs between two DataFrames or within a single DataFrame. This method can be slow for large datasets.

```python
indexer = recordlinkage.Full()
indexer.index(df_a, df_b)
```

--------------------------------

### exact

Source: https://recordlinkage.readthedocs.io/en/latest/ref-compare.html

Compares attributes of pairs exactly. This is a shortcut for `recordlinkage.compare.Exact`.

```APIDOC
## exact

### Description
Compare attributes of pairs exactly.
Shortcut of `recordlinkage.compare.Exact`:
```
from recordlinkage.compare import Exact

indexer = recordlinkage.Compare()
indexer.add(Exact())
```

### Method
exact

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Response
#### Success Response (200)
None specified in source.

#### Response Example
None provided in source.
```