### Predicate Block Example: Grouping Records by Address Prefix

Source: https://github.com/dedupeio/dedupe/blob/main/docs/how-it-works/Making-smart-comparisons.md

This example demonstrates how a predicate function, specifically extracting the first three characters of an address, can be used to create blocks of records. Records with identical address prefixes are grouped together, significantly reducing the number of comparisons needed. The output is a dictionary where keys are the features (address prefixes) and values are tuples of record IDs belonging to that block.

```python
{
 '160' : (1,2) # tuple of record_ids
 '123' : (3,4)
 }
```

--------------------------------

### Install Dedupe via pip

Source: https://context7.com/dedupeio/dedupe/llms.txt

Command to install the dedupe library and its dependencies into your Python environment.

```bash
pip install dedupe
```

--------------------------------

### dedupe.write_training

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Writes labeled examples to a JSON file.

```APIDOC
## POST /dedupe/write_training

### Description
Write a JSON file that contains labeled examples for the deduplication model.

### Method
POST

### Endpoint
/dedupe/write_training

### Parameters
#### Request Body
- **labeled_pairs** (TrainingData) - Required - A dictionary with two keys, 'match' and 'distinct', containing lists of record pairs.
- **file_obj** (TextIO) - Required - File object to write training data to.

### Request Example
{
  "labeled_pairs": {
    "match": [{"name": "Georgie Porgie"}, {"name": "George Porgie"}],
    "distinct": [{"name": "Georgie Porgie"}, {"name": "Georgette Porgette"}]
  }
}

### Response
#### Success Response (200)
- **status** (string) - Confirmation that the training data was written successfully.
```

--------------------------------

### Write Training Data to File - Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The `write_training` method writes labeled examples to a JSON file. It requires a file object as a parameter where the training data will be written. This is useful for saving and later loading training data.

```python
>>> with open('training.json', 'w') as f:
>>>     matcher.write_training(f)
```

--------------------------------

### Write Training Data to File Object

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Writes labeled training examples to a specified file object. This is useful for saving the progress of training a deduplication model.

```python
with open('training.json', 'w') as f:
    matcher.write_training(f)
```

--------------------------------

### GET /uncertain_pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Retrieves pairs of records that the model is most uncertain about, requiring human labeling.

```APIDOC
## GET /uncertain_pairs

### Description
Returns a list of record pairs that the model is most curious to have labeled to improve accuracy.

### Response Example
```json
[{"name": "Georgie Porgie"}, {"name": "Georgette Porgette"}]
```
```

--------------------------------

### Dedupe Variable Types

Source: https://context7.com/dedupeio/dedupe/llms.txt

This snippet illustrates the various variable types supported by dedupe for field comparisons. It shows examples of String, ShortString, Text, LatLong, Set, Categorical, and Exact variable types.

```python
import dedupe

# String - for fields with potential typos (uses affine gap string distance)
dedupe.variables.String("name")

# ShortString - like String but faster (no index blocking rules)
dedupe.variables.ShortString("zipcode")

# Text - for blocks of text (uses cosine similarity)
dedupe.variables.Text("description", corpus=["sample text 1", "sample text 2"])

# LatLong - for geographic coordinates (uses Haversine formula)
dedupe.variables.LatLong("location")  # field must be (lat, lon) tuples

# Set - for comparing lists/sets of elements
dedupe.variables.Set("tags", corpus=[("python",), ("python", "data")])

# Categorical - for categorical/boolean fields
dedupe.variables.Categorical("business_type", categories=["restaurant", "retail", "service"])

# Exact - for exact match comparison
dedupe.variables.Exact("country_code")
```

--------------------------------

### POST /prepare_training

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes the active learner with dataset and optional existing training data.

```APIDOC
## POST /prepare_training

### Description
Sets up the learner by processing the provided data and preparing samples for active learning.

### Parameters
- **data** (Mapping) - Required - Dictionary of records (record_id -> record_data).
- **training_file** (TextIO) - Optional - File object containing existing training data.
- **sample_size** (int) - Optional - Size of the sample to draw.
- **blocked_proportion** (float) - Optional - Proportion of record pairs sampled from similar records.

### Request Example
```python
matcher.prepare_training(data_d, 150000, 0.5)
```
```

--------------------------------

### Prepare training data

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Shows how to initialize the active learner using either a direct sample size or an existing JSON training file.

```python
matcher.prepare_training(data_d, 150000, .5)

with open('training_file.json') as f:
    matcher.prepare_training(data_d, training_file=f)
```

--------------------------------

### Load StaticGazetteer from Settings File

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes a StaticGazetteer object using pre-saved settings from a file. This allows for efficient loading of trained models for matching. Ensure the settings file is opened in binary read mode.

```python
with open('learned_settings', 'rb') as f:
    matcher = StaticGazetteer(f)
```

--------------------------------

### Initialize StaticRecordLink from Settings

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The StaticRecordLink class allows for record linkage using previously trained settings. It requires a binary file object containing the settings and supports optional configuration for parallel processing and memory usage.

```python
with open('learned_settings', 'rb') as f:
    matcher = StaticRecordLink(f)
```

--------------------------------

### Production Gazetteer Matching with StaticGazetteer

Source: https://context7.com/dedupeio/dedupe/llms.txt

This snippet demonstrates loading trained gazetteer settings for production use. It initializes `StaticGazetteer` with saved settings, indexes canonical records, searches for matches, and unindexes records.

```python
import dedupe

canonical_data = {
    1: {"name": "Acme Corporation", "address": "100 Business Park"},
}

messy_data = {
    "m1": {"name": "Acme Corp", "address": "100 Business Pk"},
}

if __name__ == "__main__":
    with open("gazetteer_settings", "rb") as f:
        matcher = dedupe.StaticGazetteer(f)

    # Index canonical records
    matcher.index(canonical_data)

    # Search for best matches
    matches = matcher.search(messy_data, threshold=0.5, n_matches=1)

    # Remove records from index
    matcher.unindex(canonical_data)
```

--------------------------------

### Prepare training data for RecordLink

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The prepare_training method initializes the active learner with two datasets. It accepts an optional training file and parameters to control sample size and blocking proportions.

```pycon
>>> matcher.prepare_training(data_1, data_2, 150000)

# Or using a file object
>>> with open('training_file.json') as f:
>>>     matcher.prepare_training(data_1, data_2, training_file=f)
```

--------------------------------

### WesternName Variable Parsing in Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/Variable-definition.md

The WesternName variable type is for American names, corporations, and households, using the 'probablepeople' package for component-wise parsing. Installation is available via pip.

```python
import namevariable

namevariable.WesternName("field")
```

--------------------------------

### USAddress Variable Parsing in Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/Variable-definition.md

The USAddress variable type is for United States addresses, utilizing the 'usaddress' package to parse address components for comparison. It requires installation via pip.

```python
import addressvariable

addressvariable.USAddress("address")
```

--------------------------------

### Initialize Dedupe instance

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Demonstrates how to define record fields and instantiate the Dedupe object to begin the deduplication process.

```python
variables = [
    dedupe.variables.String("Site name"),
    dedupe.variables.String("Address"),
    dedupe.variables.String("Zip", has_missing=True),
    dedupe.variables.String("Phone", has_missing=True),
]
deduper = dedupe.Dedupe(variables)
```

--------------------------------

### Perform Production Deduplication with StaticDedupe

Source: https://context7.com/dedupeio/dedupe/llms.txt

Shows how to load previously saved model settings to perform deduplication on new data without requiring further training.

```python
import dedupe

data = {
    1: {"Site name": "Bob's Pizza", "Address": "123 Main St", "Zip": "10001", "Phone": "555-1234"},
    2: {"Site name": "Bobs Pizza", "Address": "123 Main Street", "Zip": "10001", "Phone": "555-1234"},
    3: {"Site name": "Joe's Diner", "Address": "456 Oak Ave", "Zip": "10002", "Phone": "555-5678"},
}

if __name__ == "__main__":
    with open("learned_settings", "rb") as f:
        matcher = dedupe.StaticDedupe(f, num_cores=4)
    duplicates = matcher.partition(data, threshold=0.5)
    for cluster_ids, scores in duplicates:
        print(f"Cluster: {cluster_ids}")
        print(f"Confidence scores: {scores}")
```

--------------------------------

### Initialize Gazetteer Matcher - Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes a Gazetteer object for active learning gazetteer matching. This is useful for matching a messy dataset against a clean, canonical dataset, such as matching addresses against a known list. The constructor requires a list of Variable objects defining the fields to be used for training the matching model.

```python
# initialize from a defined set of fields
variables = [
    dedupe.variables.String("Site name"),
    dedupe.variables.String("Address"),
    dedupe.variables.String("Zip", has_missing=True),
    dedupe.variables.String("Phone", has_missing=True),
]
matcher = dedupe.Gazetteer(variables)
```

--------------------------------

### Production Record Linkage with StaticRecordLink

Source: https://context7.com/dedupeio/dedupe/llms.txt

This snippet shows how to load pre-trained record linkage settings for production use. It initializes the `StaticRecordLink` class with saved settings and then performs record matching on new datasets.

```python
import dedupe

data_1 = {
    "a1": {"name": "John Smith", "address": "123 Main St", "city": "Chicago"},
}

data_2 = {
    "b1": {"name": "J. Smith", "address": "123 Main Street", "city": "Chicago"},
}

if __name__ == "__main__":
    with open("link_settings", "rb") as f:
        matcher = dedupe.StaticRecordLink(f)

    links = matcher.join(data_1, data_2, threshold=0.5, constraint="one-to-one")
    for (id1, id2), score in links:
        print(f"Match: {id1} <-> {id2} (confidence: {score})")
```

--------------------------------

### Prepare Training Data for Dedupe

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes the active learner with datasets and optional existing training files. It samples record pairs based on a specified size and blocked proportion to optimize the training process.

```python
matcher.prepare_training(data_1, data_2, 150000)

with open('training_file.json') as f:
    matcher.prepare_training(data_1, data_2, training_file=f)
```

--------------------------------

### Load Static Dedupe Settings from File

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes a StaticDedupe object using a saved settings file. This allows for deduplication using pre-trained models without retraining.

```python
with open('learned_settings', 'rb') as f:
    matcher = StaticDedupe(f)
```

--------------------------------

### Initialize RecordLink for Cross-Dataset Matching

Source: https://context7.com/dedupeio/dedupe/llms.txt

Sets up the variable definitions required to link records across two distinct datasets.

```python
import dedupe

variables = [
    dedupe.variables.String("name"),
    dedupe.variables.String("address"),
    dedupe.variables.String("city"),
]
```

--------------------------------

### Canonical Matching with Gazetteer

Source: https://context7.com/dedupeio/dedupe/llms.txt

This snippet demonstrates using the `Gazetteer` class for matching messy data against a clean canonical dataset. It covers training, saving settings, indexing canonical data, and searching for matches.

```python
import dedupe

variables = [
    dedupe.variables.String("name"),
    dedupe.variables.String("address"),
]

# Canonical (clean) dataset
canonical_data = {
    1: {"name": "Acme Corporation", "address": "100 Business Park"},
    2: {"name": "Global Industries", "address": "200 Commerce Dr"},
}

# Messy data to match
messy_data = {
    "m1": {"name": "Acme Corp", "address": "100 Business Pk"},
    "m2": {"name": "ACME CORPORATION", "address": "100 Business Park"},
    "m3": {"name": "Global Ind.", "address": "200 Commerce Drive"},
}

if __name__ == "__main__":
    gazetteer = dedupe.Gazetteer(variables)

    # Prepare training
    gazetteer.prepare_training(messy_data, canonical_data, sample_size=1500)

    dedupe.console_label(gazetteer)
    gazetteer.train(recall=1.0)

    with open("gazetteer_settings", "wb") as f:
        gazetteer.write_settings(f)

    # Index the canonical data
    gazetteer.index(canonical_data)

    # Search for matches (returns top n_matches per record)
    matches = gazetteer.search(messy_data, threshold=0.5, n_matches=2)
    # Output: [
    #   ((('m1', 1), 0.92), (('m1', 2), 0.3)),
    #   ((('m2', 1), 0.98),),
    #   ((('m3', 2), 0.95),)
    # ]

    gazetteer.cleanup_training()
```

--------------------------------

### Interactive Labeling and Training Data Management

Source: https://context7.com/dedupeio/dedupe/llms.txt

Demonstrates how to initiate interactive console labeling, generate training data from pre-labeled datasets, and persist training settings to disk. This workflow is essential for training models that can be reused for production matching.

```python
import dedupe

variables = [dedupe.variables.String("name")]
data = {1: {"name": "John"}, 2: {"name": "Jon"}}

if __name__ == "__main__":
    deduper = dedupe.Dedupe(variables)
    deduper.prepare_training(data)
    dedupe.console_label(deduper)

    # Generate training data
    pre_labeled_data = {1: {"name": "John Smith", "cluster_id": "A"}, 2: {"name": "J. Smith", "cluster_id": "A"}}
    training = dedupe.training_data_dedupe(pre_labeled_data, common_key="cluster_id")

    # Read/write training
    with open("training.json", "w") as f:
        dedupe.write_training(training, f)
```

--------------------------------

### Persist Training and Settings

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Methods to export the current training data or the learned model settings to files. This allows for saving progress and reusing trained models in future sessions.

```python
with open('training.json', 'w') as f:
    matcher.write_training(f)

with open('learned_settings', 'wb') as f:
    matcher.write_settings(f)
```

--------------------------------

### Initialize RecordLink for active learning

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The RecordLink class is used for linking two datasets. It requires a list of variable definitions and supports configuration for parallel processing and memory usage.

```python
# initialize from a defined set of fields
variables = [
    dedupe.variables.String("Site name"),
    dedupe.variables.String("Address"),
    dedupe.variables.String("Zip", has_missing=True),
    dedupe.variables.String("Phone", has_missing=True),
]
deduper = dedupe.RecordLink(variables)
```

--------------------------------

### Perform Deduplication with Dedupe Class

Source: https://context7.com/dedupeio/dedupe/llms.txt

Demonstrates how to define variables, prepare training data, perform interactive labeling, and partition a single dataset to find duplicates.

```python
import dedupe

variables = [
    dedupe.variables.String("Site name"),
    dedupe.variables.String("Address"),
    dedupe.variables.String("Zip", has_missing=True),
    dedupe.variables.String("Phone", has_missing=True),
]

data = {
    1: {"Site name": "Bob's Pizza", "Address": "123 Main St", "Zip": "10001", "Phone": "555-1234"},
    2: {"Site name": "Bobs Pizza", "Address": "123 Main Street", "Zip": "10001", "Phone": "555-1234"},
    3: {"Site name": "Joe's Diner", "Address": "456 Oak Ave", "Zip": "10002", "Phone": "555-5678"},
    4: {"Site name": "Joes Diner", "Address": "456 Oak Avenue", "Zip": "10002", "Phone": None},
}

if __name__ == "__main__":
    deduper = dedupe.Dedupe(variables, num_cores=4)
    deduper.prepare_training(data, sample_size=1500, blocked_proportion=0.9)
    dedupe.console_label(deduper)
    deduper.train(recall=1.0, index_predicates=True)
    with open("training.json", "w") as f:
        deduper.write_training(f)
    with open("learned_settings", "wb") as f:
        deduper.write_settings(f)
    duplicates = deduper.partition(data, threshold=0.5)
    deduper.cleanup_training()
```

--------------------------------

### Convenience Functions for Training Data

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

This section covers convenience functions for generating training data, including console-based labeling and functions for creating training data from deduplicated or linked datasets.

```APIDOC
## POST /dedupe/console_label

### Description
Trains a matcher instance (Dedupe, RecordLink, or Gazetteer) from the command line.

### Method
POST

### Endpoint
/dedupe/console_label

### Parameters
#### Path Parameters
None

#### Query Parameters
* **deduper** (object) - Required - The deduper instance to train.

### Request Example
```json
{
  "deduper": { ... deduper object ... }
}
```

### Response
#### Success Response (200)
(The function modifies the deduper instance in place; no specific return value described)

#### Response Example
(No example provided)

## POST /dedupe/training_data_dedupe

### Description
Constructs training data for consumption by the `mark_pairs` method from an already deduplicated dataset.

### Method
POST

### Endpoint
/dedupe/training_data_dedupe

### Parameters
#### Path Parameters
None

#### Query Parameters
* **data** (DataInt or DataStr) - Required - Dictionary of records where the keys are record_ids and the values are dictionaries with the keys being field names.
* **common_key** (str) - Required - The name of the record field that uniquely identifies a match.
* **training_size** (int) - Optional - The rough limit of the number of training examples, defaults to 50000.

### Request Example
```json
{
  "data": {
    "record1": {"name": "John Doe", "city": "New York"},
    "record2": {"name": "Jane Smith", "city": "Los Angeles"}
  },
  "common_key": "id",
  "training_size": 10000
}
```

### Response
#### Success Response (200)
- **TrainingData** (object) - The generated training data.

#### Response Example
(No example provided)

## POST /dedupe/training_data_link

### Description
Constructs training data for consumption by the `mark_pairs` method from already linked datasets.

### Method
POST

### Endpoint
/dedupe/training_data_link

### Parameters
#### Path Parameters
None

#### Query Parameters
* **data_1** (DataInt or DataStr) - Required - Dictionary of records from the first dataset.
* **data_2** (DataInt or DataStr) - Required - Dictionary of records from the second dataset.
* **common_key** (str) - Required - The name of the record field that uniquely identifies a match.
* **training_size** (int) - Optional - The rough limit of the number of training examples, defaults to 50000.

### Request Example
```json
{
  "data_1": {
    "recordA1": {"name": "Alice", "email": "alice@example.com"}
  },
  "data_2": {
    "recordB1": {"name": "Alice", "email": "alice@example.com"}
  },
  "common_key": "id",
  "training_size": 10000
}
```

### Response
#### Success Response (200)
- **TrainingData** (object) - The generated training data.

#### Response Example
(No example provided)
```

--------------------------------

### Perform Record Linkage and Scoring with StaticRecordLink

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Demonstrates how to generate record pairs, calculate similarity scores, and filter matches using the many_to_one method in the StaticRecordLink class.

```python
>>> pairs = matcher.pairs(data)
>>> scores = matcher.scores(pairs, threshold=0.5)
>>> links = matcher.many_to_one(scores)
>>> print(list(links))
[
    ((1, 2), 0.790),
    ((4, 5), 0.720),
    ((7, 2), 0.623),
    ((10, 11), 0.899)
 ]
```

--------------------------------

### Dedupe Active Learning Training

Source: https://context7.com/dedupeio/dedupe/llms.txt

Demonstrates the active learning training process for the dedupe library. Covers preparing training data, loading existing data, labeling pairs, training the model, and saving learned settings. Requires the 'dedupe' library.

```python
import dedupe

variables = [dedupe.variables.String("name")]
data = {1: {"name": "John"}, 2: {"name": "Jon"}, 3: {"name": "Jane"}}

if __name__ == "__main__":
    deduper = dedupe.Dedupe(variables)

    # Prepare training - initializes active learner
    deduper.prepare_training(
        data,
        sample_size=1500,          # Size of sample to draw
        blocked_proportion=0.9     # Proportion from similar records vs random
    )

    # Or load existing training data
    with open("training.json") as f:
        deduper.prepare_training(data, training_file=f)

    # Get uncertain pairs for labeling
    pair = deduper.uncertain_pairs()
    # Returns: [({'name': 'John'}, {'name': 'Jon'})]

    # Mark pairs as match or distinct
    labeled_pairs = {
        "match": [({'name': "John"}, {'name': "Jon"})],
        "distinct": [({'name': "John"}, {'name': "Jane"})],
    }
    deduper.dedupe.mark_pairs(labeled_pairs)

    # Train the model
    deduper.train(
        recall=1.0,              # Proportion of true dupes to cover (0.0-1.0)
        index_predicates=True    # Use index-based blocking (more accurate but slower)
    )

    # Save training data
    with open("training.json", "w") as f:
        deduper.write_training(f)

    # Save learned settings
    with open("settings", "wb") as f:
        deduper.write_settings(f)

    # Free memory after training
    deduper.cleanup_training()
```

--------------------------------

### Write Settings to File - Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The `write_settings` method writes the data model and predicates to a file object. This allows saving the learned configuration of the deduplication model for later use. The file object should be opened in binary write mode.

```python
>>> with open('learned_settings', 'wb') as f:
>>>     matcher.write_settings(f)
```

--------------------------------

### Write Training Data using Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Demonstrates how to write labeled training data to a JSON file using the `dedupe.write_training` function. This function takes a dictionary of matched and distinct record pairs and a file object to write the data.

```python
examples = {
    "match": [
         ({'name' : 'Georgie Porgie'}, {'name' : 'George Porgie'})
    ],
    "distinct": [
        ({'name' : 'Georgie Porgie'}, {'name' : 'Georgette Porgette'})
    ],
}
with open('training.json', 'w') as f:
    dedupe.write_training(examples, f)
```

--------------------------------

### Train and Link Records with Dedupe

Source: https://context7.com/dedupeio/dedupe/llms.txt

This snippet demonstrates the core workflow of dedupe for record linkage. It involves preparing training data, interactively labeling pairs, training a model, saving settings, and then performing linkage with different constraints (one-to-one, many-to-one).

```python
import dedupe

data_1 = {
    "a1": {"name": "John Smith", "address": "123 Main St", "city": "Chicago"},
    "a2": {"name": "Jane Doe", "address": "456 Oak Ave", "city": "New York"},
}

data_2 = {
    "b1": {"name": "J. Smith", "address": "123 Main Street", "city": "Chicago"},
    "b2": {"name": "Janet Doe", "address": "456 Oak Avenue", "city": "New York"},
}

if __name__ == "__main__":
    linker = dedupe.RecordLink(variables)

    # Prepare training with both datasets
    linker.prepare_training(data_1, data_2, sample_size=1500)

    # Interactive labeling
    dedupe.console_label(linker)

    # Train the model
    linker.train(recall=1.0)

    # Save settings
    with open("link_settings", "wb") as f:
        linker.write_settings(f)

    # Find linked pairs with different constraints
    # one-to-one: each record matches at most one record in other dataset
    links = linker.join(data_1, data_2, threshold=0.5, constraint="one-to-one")
    # Output: [(('a1', 'b1'), 0.92), (('a2', 'b2'), 0.85)]

    # many-to-one: multiple records in data_1 can match same record in data_2
    links = linker.join(data_1, data_2, threshold=0.5, constraint="many-to-one")

    linker.cleanup_training()
```

--------------------------------

### Train Matcher from Console (Dedupe)

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Trains a matcher instance (Dedupe, RecordLink, or Gazetteer) directly from the command line interface. This function requires a prepared Dedupe instance and training data.

```python
dedupe.console_label(deduper)
```

--------------------------------

### Gazetteer Matching Lower-Level API

Source: https://context7.com/dedupeio/dedupe/llms.txt

Demonstrates block-based processing for matching messy data against a canonical index. Includes methods for dynamic index updates and retrieving top N matches.

```python
import dedupe

with open("gazetteer_settings", "rb") as f:
    matcher = dedupe.StaticGazetteer(f)

matcher.index(canonical)
score_blocks = matcher.score(matcher.blocks(messy))
matches = matcher.many_to_n(score_blocks, threshold=0.5, n_matches=3)

# Dynamic updates
matcher.index(new_canonical)
matcher.unindex(old_canonical)
```

--------------------------------

### Method: pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Generates candidate pairs of records from two datasets that share common fingerprints.

```APIDOC
## [METHOD] pairs(data_1, data_2)

### Description
Yields pairs of records from two datasets that share common fingerprints. This is the initial step in the record linkage process.

### Parameters
#### Path Parameters
- **data_1** (Mapping) - Required - Dictionary of records from the first dataset.
- **data_2** (Mapping) - Required - Dictionary of records from the second dataset.

### Response
#### Success Response (200)
- **Iterator** (Tuple) - Yields tuples of (record_id, record_data) pairs.

### Response Example
[
    ((1, {"name": "Pat"}), (2, {"name": "Pat"})),
    ((1, {"name": "Pat"}), (3, {"name": "Sam"}))
]
```

--------------------------------

### Write Dedupe Settings to File Object

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Saves the learned data model and predicates to a binary file object. This allows the settings to be loaded later for static deduplication.

```python
with open('learned_settings', 'wb') as f:
    matcher.write_settings(f)
```

--------------------------------

### Class Initialization: Dedupe

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Initializes the Dedupe object with variable definitions and configuration for parallel processing.

```APIDOC
## Constructor: dedupe.Dedupe

### Description
Initializes the active learning deduplication object.

### Parameters
- **variable_definition** (Collection[Variable]) - Required - List of Variable objects describing the fields for the model.
- **num_cores** (int | None) - Optional - Number of CPUs for parallel processing. Defaults to all available.
- **in_memory** (bool) - Optional - If True, uses sqlite3 ':memory:' for pair generation.

### Request Example
```python
variables = [
    dedupe.variables.String("Site name"),
    dedupe.variables.String("Address")
]
deduper = dedupe.Dedupe(variables)
```
```

--------------------------------

### Retrieve and label uncertain pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Retrieves pairs that the model is most uncertain about and demonstrates how to mark them as matches or distinct to update the model.

```python
# Retrieve uncertain pairs
pair = matcher.uncertain_pairs()
print(pair)

# Mark pairs for training
labeled_examples = {
    "match": [],
    "distinct": [
        (
            {"name": "Georgie Porgie"},
            {"name": "Georgette Porgette"},
        )
    ],
}
matcher.mark_pairs(labeled_examples)
```

--------------------------------

### Record Blocking and Predicate Generation

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

This section details the `__call__` method used for generating predicates for records, which is a core part of the record blocking process. It also describes related indexing methods.

```APIDOC
## POST /dedupe/__call__

### Description
Generates the predicates for records. Yields tuples of (predicate, record_id).

### Method
POST

### Endpoint
/dedupe/__call__

### Parameters
#### Path Parameters
None

#### Query Parameters
* **records** (sequence of tuples) - Required - A sequence of tuples of (record_id, record_dict).
* **target** (boolean) - Optional - Indicates whether the data should be treated as the target data. This effects the behavior of search predicates.

### Request Example
```json
{
  "records": [
    [1, {"name": "bob"}],
    [2, {"name": "suzanne"}]
  ],
  "target": false
}
```

### Response
#### Success Response (200)
- **predicate** (string) - The generated predicate.
- **record_id** (integer) - The ID of the record.

#### Response Example
```json
[
  ["foo:1", 1],
  ["bar:1", 100]
]
```

## POST /dedupe/index

### Description
Adds docs to the indices used by fingerprinters. Some fingerprinter methods depend upon having an index of values that a field may have in the data. This method adds those values to the index.

### Method
POST

### Endpoint
/dedupe/index

### Parameters
#### Path Parameters
None

#### Query Parameters
* **docs** (Union[Iterable[str], Iterable[Iterable[str]]]) - Required - An iterator of values from your data to index.
* **field** (str) - Required - Fieldname or key associated with the values you are indexing.

### Request Example
```json
{
  "docs": ["value1", "value2"],
  "field": "name"
}
```

### Response
#### Success Response (200)
(No specific fields described, indicates successful indexing)

#### Response Example
(No example provided)

## POST /dedupe/unindex

### Description
Removes docs from indices used by fingerprinters.

### Method
POST

### Endpoint
/dedupe/unindex

### Parameters
#### Path Parameters
None

#### Query Parameters
* **docs** (Union[Iterable[str], Iterable[Iterable[str]]]) - Required - An iterator of values from your data to remove.
* **field** (str) - Required - Fieldname or key associated with the values you are unindexing.

### Request Example
```json
{
  "docs": ["value1", "value2"],
  "field": "name"
}
```

### Response
#### Success Response (200)
(No specific fields described, indicates successful unindexing)

#### Response Example
(No example provided)

## POST /dedupe/reset_indices

### Description
Resets the fingerprinter indices to free up memory. If you need to block again, the data will need to be re-indexed.

### Method
POST

### Endpoint
/dedupe/reset_indices

### Parameters
None

### Request Example
(No request body)

### Response
#### Success Response (200)
(No specific fields described, indicates successful reset)

#### Response Example
(No example provided)
```

--------------------------------

### Construct Training Data for Deduplication (Dedupe)

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Constructs training data for the `mark_pairs` method from an already deduplicated dataset. It takes a dictionary of records, a common key for identifying matches, and an optional training size. Records without a common key are assumed to be distinct.

```python
dedupe.training_data_dedupe(data, common_key, training_size=50000)
```

--------------------------------

### Index Records with StaticGazetteer

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Adds records to the index for matching. If a record with the same ID already exists, it will be replaced. This method supports both integer and string data types for record IDs and field names.

```python
matcher.index(data)
```

--------------------------------

### POST /mark_pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Updates the matching model with user-labeled record pairs.

```APIDOC
## POST /mark_pairs

### Description
Adds labeled pairs to the training data and updates the matching model.

### Parameters
- **labeled_pairs** (TrainingData) - Required - Dictionary with keys 'match' and 'distinct' containing lists of record pairs.

### Request Example
```json
{
  "match": [],
  "distinct": [
    [{"name": "Georgie Porgie"}, {"name": "Georgette Porgette"}]
  ]
}
```
```

--------------------------------

### Comparing Fields with Multiple Variables

Source: https://github.com/dedupeio/dedupe/blob/main/docs/Variable-definition.md

Illustrates how to define multiple variable types for the same field to apply different comparison logic simultaneously.

```python
[
    dedupe.variables.String("name"),
    dedupe.variables.Text("name")
]
```

--------------------------------

### dedupe.StaticGazetteer

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Provides static gazetteer matching functionalities, mirroring the methods of the Gazetteer class.

```APIDOC
## Class: dedupe.StaticGazetteer

### Description
Provides static gazetteer matching functionalities, mirroring the methods of the Gazetteer class.

### Methods

- **blocks(data)**: Same as [`dedupe.Gazetteer.blocks()`](#dedupe.Gazetteer.blocks)
- **score(blocks)**: Same as [`dedupe.Gazetteer.score()`](#dedupe.Gazetteer.score)
- **many_to_n(score_blocks, threshold=0.0, n_matches=1)**: Same as [`dedupe.Gazetteer.many_to_n()`](#dedupe.Gazetteer.many_to_n)

### Properties

- **fingerprinter**: Instance of [`dedupe.blocking.Fingerprinter`](#dedupe.blocking.Fingerprinter) class.
```

--------------------------------

### Add Labeled Pairs for Training - Python

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The `mark_pairs` method adds user-labeled pairs of records to the training data and updates the matching model. This is useful for building user interfaces for training or adding data from existing sources. It takes a `TrainingData` object as input, which is a dictionary with 'match' and 'distinct' keys, each containing lists of record pairs.

```python
>>> labeled_examples = {
>>>     "match": [],
>>>     "distinct": [
>>>         (
>>>             {"name": "Georgie Porgie"},
>>>             {"name": "Georgette Porgette"},
>>>         )
>>>     ],
>>> }
>>> matcher.mark_pairs(labeled_examples)
```

--------------------------------

### Canonicalization of Duplicate Clusters

Source: https://context7.com/dedupeio/dedupe/llms.txt

Shows how to derive a single canonical representation from a cluster of duplicate records. This is useful for cleaning data after identifying matches.

```python
cluster_records = [
    {"name": "John Smith", "phone": "555-1234"},
    {"name": "J. Smith", "phone": "555-1234"},
    {"name": "John Smith", "phone": None}
]
canonical = dedupe.canonicalize(cluster_records)
```

--------------------------------

### Method: one_to_one

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Decides which pairs refer to the same entity using an injective matching strategy.

```APIDOC
## [METHOD] one_to_one(scores, threshold=0.0)

### Description
Decides which pairs refer to the same entity, ensuring each record in data_1 matches at most one record in data_2 and vice-versa.

### Parameters
#### Request Body
- **scores** (numpy.ndarray) - Required - Structured array containing pairs and similarity scores.
- **threshold** (float) - Optional - Probability threshold (0-1) to consider a match.

### Response
#### Success Response (200)
- **Iterator** (Tuple) - Yields pairs of record IDs and their confidence scores.

### Response Example
[
    ((1, 2), 0.790),
    ((4, 5), 0.720)
]
```

--------------------------------

### Partition Data into Clusters

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Identifies records referring to the same entity and returns clusters of record IDs with corresponding confidence scores. This method is suitable for small to moderately sized datasets.

```python
duplicates = matcher.partition(data, threshold=0.5)
print(duplicates)
```

--------------------------------

### Retrieve Uncertain Record Pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Fetches a list of record pairs that the model is most uncertain about, which is essential for building active learning user interfaces. These pairs are presented to the user for manual labeling.

```python
pair = matcher.uncertain_pairs()
print(pair)
```

--------------------------------

### Calculate Record Similarity via String Distance

Source: https://github.com/dedupeio/dedupe/blob/main/docs/how-it-works/Matching-records.md

Demonstrates calculating similarity by treating an entire record as a single string versus comparing individual fields. Field-level comparison allows for the application of weights to prioritize specific data points.

```python
# Treat record as a single string
record_distance = string_distance('bob roberts 1600 pennsylvania ave. 555-0123',
                                  'Robert Roberts 1600 Pensylvannia Avenue')

# Compare field by field
record_distance = (string_distance('bob', 'Robert')
                   + string_distance('roberts', 'Roberts')
                   + string_distance('1600 pennsylvania ave.', '1600 Pensylvannia Avenue')
                   + string_distance('555-0123', ''))

# Compare field by field with numeric weights
record_distance = (0.5 * string_distance('bob', 'Robert')
                   + 2.0 * string_distance('roberts', 'Roberts')
                   + 2.0 * string_distance('1600 pennsylvania ave.', '1600 Pensylvannia Avenue')
                   + 0.5 * string_distance('555-0123', ''))
```

--------------------------------

### Customize RTD Theme Markup (JavaScript)

Source: https://github.com/dedupeio/dedupe/blob/main/docs/_templates/layout.html

This JavaScript snippet customizes the Read the Docs theme by changing the hardcoded markup in the navigation bar. It targets specific HTML elements to update the text and remove CSS classes. This is useful for branding the documentation site.

```javascript
$(function() { /* Tweak some of the hardcoded markup in the RTD theme */ $('.wy-nav-top > a, .icon.icon-home').html('dedupe library docs'); $('.icon.icon-home').removeClass('icon, icon-home'); });
```

--------------------------------

### Defining Variables and Interaction Fields

Source: https://github.com/dedupeio/dedupe/blob/main/docs/Variable-definition.md

Shows the configuration of various field types including String, Custom, and Interaction variables. It highlights the use of 'has_missing=True' to enable response augmented data processing.

```python
[
    dedupe.variables.String("name", name="name"),
    dedupe.variables.String("address"),
    dedupe.variables.String("city", name="city"),
    dedupe.variables.Custom("zip", comparator=same_or_not_comparator),
    dedupe.variables.String("cuisine", has_missing=True),
    dedupe.vairables.Interaction("name", "city")
]
```

--------------------------------

### Record Linkage Lower-Level API

Source: https://context7.com/dedupeio/dedupe/llms.txt

Provides fine-grained control over matching constraints between two datasets. Supports one-to-one and many-to-one matching strategies using pre-trained settings.

```python
import dedupe

with open("link_settings", "rb") as f:
    matcher = dedupe.StaticRecordLink(f)

pairs = matcher.pairs(data_1, data_2)
scores = matcher.score(pairs)

# One-to-one matching
links = matcher.one_to_one(scores, threshold=0.5)
```

--------------------------------

### Resolve entity matches using one-to-one or many-to-one mapping

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

These methods process similarity scores to determine final matches. one_to_one enforces an injective mapping suitable for distinct sources, while many_to_one allows multiple records to map to a single reference record, ideal for lookup tables.

```python
pairs = matcher.pairs(data)
scores = matcher.scores(pairs, threshold=0.5)
links = matcher.one_to_one(scores)
list(links)
```

--------------------------------

### Perform Record Linkage with join

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The join method identifies pairs of records across two datasets that refer to the same entity. It returns a list of tuples containing record ID pairs and their associated confidence scores.

```python
links = matcher.join(data_1, data_2, threshold=0.5)
list(links)
```

--------------------------------

### Search for Duplicate Records with StaticGazetteer

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Identifies potential duplicate records within a dataset based on a learned model and specified confidence threshold. It can return results as a list or a generator, and allows control over the number of matches per record.

```python
matches = gazetteer.search(data, threshold=0.5, n_matches=2)
print(matches)
```

--------------------------------

### Construct Training Data for Record Linkage (Dedupe)

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Constructs training data for the `mark_pairs` method from already linked datasets. It requires two dictionaries of records, a common key for identifying matches, and an optional training size. This function is designed for record linkage tasks where pairs of records are linked across datasets.

```python
dedupe.training_data_link(data_1, data_2, common_key, training_size=50000)
```

--------------------------------

### Retrieve uncertain pairs for labeling

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The uncertain_pairs method returns pairs of records that the model is most uncertain about, facilitating manual labeling in a user interface.

```pycon
>>> pair = matcher.uncertain_pairs()
>>> print(pair)
[({'name' : 'Georgie Porgie'}, {'name' : 'Georgette Porgette'})]
```

--------------------------------

### Method: many_to_one

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Decides which pairs refer to the same entity using a surjective matching strategy.

```APIDOC
## [METHOD] many_to_one(scores, threshold=0.0)

### Description
Matches multiple records from data_1 to a single record in data_2. Ideal for matching messy data against a golden record lookup table.

### Parameters
#### Request Body
- **scores** (numpy.ndarray) - Required - Structured array containing pairs and similarity scores.
- **threshold** (float) - Optional - Probability threshold (0-1) to consider a match.

### Response
#### Success Response (200)
- **Iterator** (Tuple) - Yields pairs of record IDs and their confidence scores.
```

--------------------------------

### StaticDedupe Class Methods

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

The StaticDedupe class provides methods (fingerprinter, pairs, score, cluster) that mirror the functionality of the Dedupe class. It is designed for scenarios where blocking and pairing logic is static.

```python
# StaticDedupe inherits methods from Dedupe.
# Example usage would be similar to Dedupe class methods.
```

--------------------------------

### dedupe.StaticRecordLink

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Provides methods for record linkage, including finding pairs, scoring them, and linking records one-to-one or many-to-one.

```APIDOC
## Class: dedupe.StaticRecordLink

### Description
Provides methods for record linkage, including finding pairs, scoring them, and linking records one-to-one or many-to-one.

### Methods

- **pairs(data_1, data_2)**: Same as [`dedupe.RecordLink.pairs()`](#dedupe.RecordLink.pairs)
- **score(pairs)**: Same as [`dedupe.RecordLink.score()`](#dedupe.RecordLink.score)
- **one_to_one(scores, threshold=0.0)**: Same as [`dedupe.RecordLink.one_to_one()`](#dedupe.RecordLink.one_to_one)
- **many_to_one(scores, threshold=0.0)**: Same as [`dedupe.RecordLink.many_to_one()`](#dedupe.RecordLink.many_to_one)

### Properties

- **fingerprinter**: Instance of [`dedupe.blocking.Fingerprinter`](#dedupe.blocking.Fingerprinter) class.
```

--------------------------------

### Mark Labeled Record Pairs

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Updates the matching model by incorporating user-labeled record pairs. This method is used to incrementally build a training set by classifying pairs as either matches or distinct.

```python
labeled_examples = {
    "match": [],
    "distinct": [
        (
            {"name": "Georgie Porgie"},
            {"name": "Georgette Porgette"},
        )
    ],
}
matcher.mark_pairs(labeled_examples)
```

--------------------------------

### dedupe.read_training

Source: https://github.com/dedupeio/dedupe/blob/main/docs/API-documentation.md

Reads training data from a previously built training data file object.

```APIDOC
## GET /dedupe/read_training

### Description
Read training data from a previously built training data file object.

### Method
GET

### Endpoint
/dedupe/read_training

### Parameters
#### Query Parameters
- **training_file** (TextIO) - Required - File object containing the training data.

### Response
#### Success Response (200)
- **training_data** (dict) - A dictionary with two keys, 'match' and 'distinct', containing the labeled examples.
```