### Load and Print Example Datasets

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Loads two example datasets, 'linkexample1' and 'linkexample2', and prints them to the console. These datasets are used for demonstrating record linkage procedures.

```R
data("linkexample1", "linkexample2")
print(linkexample1)
print(linkexample2)
```

--------------------------------

### Initialize and Load Data in reclin2

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.html

This snippet demonstrates how to load the reclin2 library into the R environment and import the provided example datasets for record linkage.

```R
library(reclin2)
data("linkexample1", "linkexample2")
print(linkexample1)
```

--------------------------------

### String Comparison Functions in R

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Demonstrates various string comparison functions in reclin2 for calculating similarity scores between 0 and 1. Includes exact match, Jaro-Winkler, Longest Common Subsequence, and Jaccard similarity. Also shows a custom comparison function example.

```r
library(reclin2)

# cmp_identical - exact match (returns TRUE/FALSE or 1/0)
cmp <- cmp_identical()
x <- cmp(c("john", "mary", "susan"), c("john", "mary", "sue"))
print(x)
#> [1]  TRUE  TRUE FALSE

# cmp_jarowinkler - Jaro-Winkler string similarity (0-1 scale)
cmp <- cmp_jarowinkler(threshold = 0.95)
x <- cmp(c("john", "mary", "susan"), c("johan", "mary", "susanna"))
print(x)
#> [1] 0.9333333 1.0000000 0.8492063

# Apply threshold for binary result
cmp(x)
#> [1] FALSE  TRUE FALSE

# cmp_lcs - Longest Common Subsequence similarity
cmp <- cmp_lcs(threshold = 0.8)
x <- cmp(c("rotterdam", "amsterdam"), c("rottrdam", "amstdam"))
print(x)
#> [1] 0.9411765 0.8750000

# cmp_jaccard - Jaccard similarity (character bigrams)
cmp <- cmp_jaccard(threshold = 0.7)
x <- cmp(c("smith", "johnson"), c("smyth", "jonson"))
print(x)

# Custom comparison function example
na_as_class <- function(x, y) {
  factor(
    ifelse(is.na(x) | is.na(y), 2L, (y == x) * 1L),
    levels = 0:2, labels = c("eq", "uneq", "mis"))
}
```

--------------------------------

### Compare pairs without inplace modification

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.html

Compares generated pairs using specified variables and the Jaro-Winkler similarity metric. Unlike the previous example, this version does not modify the 'pairs' object in place, and the 'inplace' argument is ignored for cluster pairs.

```r
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"),
  default_comparator = cmp_jarowinkler(0.9))
```

--------------------------------

### Initialize reclin2 and Load Data

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/introduction.md

Loads the reclin2 library and sample datasets to prepare for record linkage operations.

```R
library(reclin2)
data("linkexample1", "linkexample2")
print(linkexample1)
print(linkexample2)
```

--------------------------------

### Load reclin2 Package

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Loads the reclin2 library into the R session. This is a prerequisite for using any of the package's functionalities.

```R
library(reclin2)
```

--------------------------------

### Create and Use a Cluster for Pair Generation (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Initializes a parallel cluster with 2 nodes and then generates pairs of records from two datasets ('linkexample1', 'linkexample2') based on the 'postcode' variable using blocking. This leverages parallel processing for efficiency.

```R
library(parallel)
cl <- makeCluster(2)
pairs <- cluster_pair_blocking(cl, linkexample1, linkexample2, "postcode")
print(pairs)
```

--------------------------------

### Initialize Cluster and Generate Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Sets up a parallel cluster and generates record pairs using blocking on a specific variable to reduce computational load.

```R
library(reclin2)
library(parallel)
data("linkexample1", "linkexample2")
cl <- makeCluster(2)
pairs <- cluster_pair_blocking(cl, linkexample1, linkexample2, "postcode")
print(pairs)
```

--------------------------------

### Load parallel library and create cluster

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.html

Initializes a parallel cluster with a specified number of nodes. This is a prerequisite for using cluster-based functions in reclin2.

```r
library(parallel)
cl <- makeCluster(2)
```

--------------------------------

### Load and inspect town name data

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/deduplication.md

Initializes the reclin2 library and loads the town_names dataset to inspect the structure of observed vs official names.

```R
library(reclin2)
data(town_names)
head(town_names)
```

--------------------------------

### Compare Records in Pairs

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/introduction.md

Compares generated pairs based on specific linkage keys. Demonstrates both standard comparison and in-place modification for memory efficiency.

```R
# Standard comparison
pairs <- compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"))
print(pairs)

# In-place comparison for memory efficiency
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), inplace = TRUE)
print(pairs)
```

--------------------------------

### Prepare Training Data and Estimate Linkage Model

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.html

This snippet demonstrates how to augment a pair dataset with known match statuses and train a logistic regression model to predict record linkage probabilities. It uses the compare_vars function to merge truth data and glm to perform the classification.

```R
# Prepare known IDs for training
linkexample2$known_id <- linkexample2$id
linkexample2$known_id[c(2,5)] <- NA
setDT(linkexample2)

# Add true match status to pairs
compare_vars(pairs, "y", on_x = "id", on_y = "known_id", y = linkexample2, inplace = TRUE)
compare_vars(pairs, "y_true", on_x = "id", on_y = "id", inplace = TRUE)

# Estimate the model
m <- glm(y ~ lastname + firstname + address + sex, data = pairs, family = binomial())

# Predict probabilities
pairs[, prob := predict(m, type = "response", newdata = pairs)]
```

--------------------------------

### Generate and Compare Record Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.html

This snippet shows how to generate all possible pairs between two datasets using blocking and then compare these pairs based on specified fields. It utilizes the reclin2 package for efficient pair generation and comparison, with Jaro-Winkler similarity for string comparisons.

```R
library(reclin2)
data("linkexample1", "linkexample2")
print(linkexample1)
print(linkexample2)
pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler()))
print(pairs)
```

--------------------------------

### Stratified Sampling of Cluster Pairs using cluster_call (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Demonstrates how to perform a random stratified sample of pairs from a `cluster_pairs` object using the `cluster_call` function. The function is applied to each cluster node to mark sampled pairs, and `cluster_collect` is used to gather the results.

```R
compare_vars(pairs, "id")

cluster_call(pairs, function(pairs, ...) {
  sel1 <- sample(which(pairs$id), 2)
  sel2 <- sample(which(!pairs$id), 2)
  pairs[, sample := FALSE]
  pairs[c(sel1, sel2), sample := TRUE]
  NULL
})

sample <- cluster_collect(pairs, "sample")
```

--------------------------------

### Load Data and Libraries for Deduplication

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/deduplication.md

Loads the reclin2 and data.table libraries and displays the first few rows of the town_names dataset. This is a prerequisite for performing deduplication.

```R
library(reclin2)
library(data.table)
data(town_names)
head(town_names)
```

--------------------------------

### Prepare Training Data for Record Linkage

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/record_linkage_using_machine_learning.md

Prepares the dataset by assigning known identifiers and comparing variables to determine ground truth match status for model training.

```R
linkexample2$known_id <- linkexample2$id
linkexample2$known_id[c(2,5)] <- NA
setDT(linkexample2)
compare_vars(pairs, "y", on_x = "id", on_y = "known_id", y = linkexample2, inplace = TRUE)
compare_vars(pairs, "y_true", on_x = "id", on_y = "id", inplace = TRUE)
print(pairs)
```

--------------------------------

### Evaluate Record Linkage Performance in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Compares selected pairs against a truth dataset using compare_vars and generates a contingency table to calculate recall and sensitivity metrics.

```R
pairs <- compare_vars(pairs, "truth", on_x = "id", on_y = "id")
print(pairs)
table(pairs$truth, pairs$threshold)
```

--------------------------------

### Access Cluster Internals in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Demonstrates how to inspect the underlying data stored within the cluster nodes using clusterCall.

```R
clusterCall(pairs$cluster, function(name) {
  pairs <- reclin2:::reclin_env[[name]]$pairs
  head(pairs, 1)
}, name = pairs$name)
```

--------------------------------

### Perform One-to-One Record Linkage in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Uses greedy or n-to-m optimization algorithms to enforce one-to-one linkage constraints, ensuring each record is selected at most once.

```R
pairs <- select_greedy(pairs, "weights", variable = "greedy", threshold = 0)
table(pairs$truth, pairs$greedy)

pairs <- select_n_to_m(pairs, "weights", variable = "ntom", threshold = 0)
table(pairs$truth, pairs$ntom)
```

--------------------------------

### Generate Pairs using Blocking

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Generates pairs of records from two datasets ('linkexample1', 'linkexample2') based on matching 'postcode' values. This method, called blocking, reduces the number of pairs to compare by only considering records with the same blocking key.

```R
pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
print(pairs)
```

--------------------------------

### Compare Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Compares record pairs across multiple variables using a Jaro-Winkler comparator. Demonstrates both in-place modification and creating new pair objects.

```R
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"),
  default_comparator = cmp_jarowinkler(0.9))
pairs2 <- compare_pairs(pairs, on = 
  c("lastname", "firstname", "address", "sex"), new_name = "pairs2")
```

--------------------------------

### Probabilistic Record Linkage with EM Algorithm (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Applies the Expectation-Maximization (EM) algorithm to probabilistically link records based on the compared pairs. It estimates matching probabilities and then predicts the linkage outcome, adding a 'weights' column to the pairs object.

```R
m <- problink_em(~ lastname + firstname + address + sex, data = pairs)
print(m)
pairs <- predict(m, pairs = pairs, add = TRUE)
print(pairs)
```

--------------------------------

### Analyze Similarity Thresholds and Deduplicate Records in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/deduplication.html

This snippet demonstrates how to evaluate the impact of similarity thresholds on error rates and perform record deduplication. It uses data.table syntax to aggregate error metrics and iteratively tests thresholds to optimize the grouping of records.

```R
# Compare variables and calculate error rates per threshold
compare_vars(pairs, "true", on_x = "official_name", inplace = TRUE)
pairs$threshold <- trunc(pairs$name/0.05)*0.05
thresholds <- pairs[, .(ftrue = mean(true)), by = threshold]

# Select threshold and deduplicate
select_threshold(pairs, "select", "name", threshold = 0.95, inplace = TRUE)
res <- deduplicate_equivalence(pairs, "group", "select")

# Validate quality by checking for multiple official names in one group
qual <- res[, .(errors = length(unique(official_name))-1, n = .N), by = group]
qual$ferrors <- qual$errors/qual$n

# Iterative threshold testing
thresholds <- seq(0.5, 1, by = 0.02)
sizes <- numeric(length(thresholds))
nerrors <- numeric(length(thresholds))
for (i in seq_along(thresholds)) {
  threshold <- thresholds[i]
  select_threshold(pairs, "select", "name", threshold = threshold, inplace = TRUE)
  res <- deduplicate_equivalence(pairs, "group", "select")
  sizes[i] <- length(unique(res$group))
  qual <- res[, .(errors = length(unique(official_name))-1, n = .N), by = group]
  nerrors[i] <- sum(qual$errors)
}
```

--------------------------------

### Deduplicate Equivalence Groups in R

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Demonstrates how to group records based on equivalence, count unique groups, and assign a representative name to each group using data.table syntax.

```R
result <- deduplicate_equivalence(pairs, "group", "select")
length(unique(result$group))
result[, assigned_name := {
  t <- table(name)
  names(sort(t, decreasing = TRUE))[1]
}, by = group]
```

--------------------------------

### Prepare Data with Known IDs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.md

This snippet demonstrates how to add a 'known_id' column to a dataframe and set specific IDs to NA, simulating known identifiers for records. It also converts the dataframe to a data.table for efficient manipulation.

```R
linkexample2$known_id <- linkexample2$id
linkexample2$known_id[c(2,5)] <- NA
setDT(linkexample2)
```

--------------------------------

### Collect Pairs from Cluster (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Collects pairs that have been processed on the cluster nodes locally. It first selects pairs with a threshold of 0 (effectively all pairs after the previous thresholding) and then uses 'cluster_collect' to bring them to the local environment. The 'clear = TRUE' option can be used to remove pairs from the cluster after collection.

```R
pairs <- select_threshold(pairs, "threshold", score = "weights", threshold = 0)
local_pairs <- cluster_collect(pairs, "threshold")
print(local_pairs)
```

--------------------------------

### Standard Probabilistic Linkage Workflow in R

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

A step-by-step implementation of a standard record linkage pipeline using blocking, EM estimation, and one-to-one selection.

```R
pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
pairs <- compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), default_comparator = cmp_jarowinkler(0.9))
model <- problink_em(~ lastname + firstname + address + sex, data = pairs)
pairs <- predict(model, pairs = pairs, add = TRUE)
pairs <- select_n_to_m(pairs, "selected", "weights", threshold = 0)
linked <- link(pairs, selection = "selected")
```

--------------------------------

### Apply EM Algorithm and Select Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Uses the Expectation-Maximization algorithm to estimate weights for record linkage and filters pairs based on a score threshold.

```R
m <- problink_em(~ lastname + firstname + address + sex, data = pairs)
pairs <- predict(m, pairs = pairs, add = TRUE)
pairs <- select_threshold(pairs, "threshold", score = "weights", threshold = 8)
```

--------------------------------

### Machine Learning Linkage Approach in R

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Demonstrates training a supervised classification model (GLM) on known matches to predict and select links between two datasets.

```R
pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), inplace = TRUE)
compare_vars(pairs, "y", on_x = "id", on_y = "known_id", y = linkexample2, inplace = TRUE)
model <- glm(y ~ lastname + firstname + address + sex, data = pairs, family = binomial())
pairs[, prob := predict(model, type = "response", newdata = pairs)]
pairs[, selected := prob > 0.5]
linked <- link(pairs, selection = "selected", all_y = TRUE)
```

--------------------------------

### Compare Pairs and Create New Set (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Compares pairs of records using Jaro-Winkler similarity and creates a new set of pairs named 'pairs2' on the cluster nodes, leaving the original 'pairs' object unmodified. This is useful for exploring different comparison strategies without altering the initial results.

```R
pairs2 <- compare_pairs(pairs, on = 
  c("lastname", "firstname", "address", "sex"), new_name = "pairs2")
print(pairs2)
print(pairs)
```

--------------------------------

### Perform Optimal N-to-M Selection with reclin2

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Uses linear programming to maximize total scores while enforcing one-to-one matching constraints. This method is slower than greedy selection but ensures a globally optimal result.

```R
library(reclin2)
data("linkexample1", "linkexample2")

pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
pairs <- compare_pairs(pairs, c("lastname", "firstname", "address", "sex"))
model <- problink_em(~ lastname + firstname + address + sex, data = pairs)
pairs <- predict(model, pairs = pairs, add = TRUE)

# Optimal one-to-one selection
pairs <- select_n_to_m(pairs, variable = "ntom", score = "weights", threshold = 0)
print(pairs[ntom == TRUE])
```

--------------------------------

### Create Final Linked Dataset in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Uses the link function to merge records based on a previously defined selection variable. Supports optional arguments to control join types.

```R
linked_data_set <- link(pairs, selection = "ntom")
print(linked_data_set)
```

--------------------------------

### Apply Similarity Scores for Fuzzy Matching

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/introduction.md

Uses the Jaro-Winkler similarity score to compare string fields, allowing for robust matching even when data contains typing errors.

```R
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"),
  default_comparator = cmp_jarowinkler(0.9), inplace = TRUE)
print(pairs)
```

--------------------------------

### Generate New Pairs with compare_pairs

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.html

This function generates a new set of pairs for a cluster without modifying the original pairs. It takes the existing pairs object and a new name for the generated pairs as input. The output is a new cluster object with the specified name.

```r
pairs2 <- compare_pairs(pairs, on = 
  c("lastname", "firstname", "address", "sex"), new_name = "pairs2")
print(pairs2)
```

--------------------------------

### Visualize Deduplication Metrics

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/deduplication.md

Generates a 2x2 grid of plots to visualize the relationship between similarity thresholds, group counts, and error rates.

```R
opar = par(mfrow = c(2,2))
plot(thresholds, sizes)
plot(thresholds, nerrors)
plot(sizes, nerrors)
par(opar)
```

--------------------------------

### Final Linkage and Table Generation (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Performs final linkage operations on the locally collected pairs. It compares variables, selects pairs based on a 'ntom' variable, creates a contingency table of truth vs. ntom, and generates the final linked dataset.

```R
local_pairs <- compare_vars(local_pairs, "truth", on_x = "id", on_y = "id")
local_pairs <- select_n_to_m(local_pairs, "weights", variable = "ntom", threshold = 0)
table(local_pairs$truth, local_pairs$ntom)
linked_data_set <- link(local_pairs, selection = "ntom")
print(linked_data_set)
```

--------------------------------

### Execute distributed operations with cluster_call

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.html

Demonstrates how to use cluster_call to perform operations on cluster nodes, such as stratified sampling. The function accepts a pairs object and a custom function, returning the results from all nodes.

```R
compare_vars(pairs, "id")
cluster_call(pairs, function(pairs, ...) {
  sel1 <- sample(which(pairs$id), 2)
  sel2 <- sample(which(!pairs$id), 2)
  pairs[, sample := FALSE]
  pairs[c(sel1, sel2), sample := TRUE]
  NULL
})
sample <- cluster_collect(pairs, "sample")
```

--------------------------------

### Generate All Pairs Between Datasets (R)

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Generates all possible record combinations between two datasets. This method is simple but computationally intensive, making it suitable only for small datasets. It can also be used for deduplication within a single dataset.

```r
library(reclin2)
data("linkexample1", "linkexample2")

# Generate all possible pairs between two datasets
pairs <- pair(linkexample1, linkexample2)
print(pairs)

# For deduplication (comparing dataset to itself)
pairs_dedup <- pair(linkexample1, deduplication = TRUE)
print(pairs_dedup)
```

--------------------------------

### Apply Custom Comparator for Missing Sex Data in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.html

This snippet demonstrates applying a custom comparator function (`na_as_class`) to handle missing 'sex' data during record linkage in R. It first removes the original 'sex' column to avoid conflicts and then re-compares pairs using the new function, treating missing values as a separate category.

```R
pairs[, sex := NULL]
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler(), sex = na_as_class))
print(pairs)
```

--------------------------------

### Perform Greedy Selection with reclin2

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Selects pairs using a greedy algorithm that enforces one-to-one matching. It is computationally efficient but may not guarantee a globally optimal solution.

```R
library(reclin2)
data("linkexample1", "linkexample2")

pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
pairs <- compare_pairs(pairs, c("lastname", "firstname", "address", "sex"))
model <- problink_em(~ lastname + firstname + address + sex, data = pairs)
pairs <- predict(model, pairs = pairs, add = TRUE)

# Greedy selection with one-to-one matching
pairs <- select_greedy(pairs, variable = "greedy", score = "weights", threshold = 0)
print(pairs[greedy == TRUE])

# Allow n-to-m matching
pairs <- select_greedy(pairs, "greedy_n2m", "weights", threshold = 0, n = 2, m = 1)

# Include ties
pairs <- select_greedy(pairs, "greedy_ties", "weights", threshold = 0, include_ties = TRUE)
```

--------------------------------

### Greedy Selection for One-to-One Linkage

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.html

The `select_greedy` function implements a greedy approach to select one-to-one links between records. It prioritizes pairs with higher scores and ensures that each record is selected at most once.

```r
pairs <- select_greedy(pairs, "weights", variable = "greedy", threshold = 0)
table(pairs$truth, pairs$greedy)
```

--------------------------------

### Generate and Compare Record Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.md

This snippet demonstrates the initial steps of record linkage using reclin2. It generates potential record pairs using blocking and then compares these pairs based on specified fields using Jaro-Winkler similarity. Missing values in the 'sex' field are handled by defining a custom comparison function.

```R
library(reclin2)

data("linkexample1", "linkexample2")
print(linkexample1)
print(linkexample2)


pairs <- pair_blocking(linkexample1, linkexample2, "postcode")

compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler()))
print(pairs)

na_as_class <- function(x, y) {
  factor(
    ifelse(is.na(x) | is.na(y), 2L, (y == x)*1L),
    levels = 0:2, labels = c("eq", "uneq", "mis"))
}

pairs[, sex := NULL]

compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler(), sex = na_as_class))
print(pairs)
```

--------------------------------

### Compare Pairs on Multiple Fields

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Compares generated pairs based on exact matches in 'lastname', 'firstname', 'address', and 'sex' fields. The results are stored back into the 'pairs' object.

```R
pairs <- compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex", "sex"))
print(pairs)
```

--------------------------------

### Compare Pairs with Jaro-Winkler Similarity (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/using_a_cluster_for_record_linkage.md

Compares pairs of records based on specified variables ('lastname', 'firstname', 'address', 'sex') using the Jaro-Winkler similarity measure with a threshold of 0.9. The 'inplace = TRUE' argument modifies the existing 'pairs' object directly.

```R
compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
default_comparator = cmp_jarowinkler(0.9), inplace = TRUE)
print(pairs)
```

--------------------------------

### Create Linked Dataset in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/record_linkage_using_machine_learning.md

This snippet uses the 'link' function to create the final linked dataset based on the 'select' column, which indicates the predicted matches. The 'all_y = TRUE' argument ensures all records from the 'y' dataset are considered.

```R
linked_data_set <- link(pairs, selection = "select", all_y = TRUE)
print(linked_data_set)
```

--------------------------------

### Link Datasets with reclin2

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Creates a final linked dataset by joining records from two sources based on previously selected pairs. Supports various join types including inner, left, right, and full outer joins.

```R
library(reclin2)
data("linkexample1", "linkexample2")

pairs <- pair_blocking(linkexample1, linkexample2, "postcode")
pairs <- compare_pairs(pairs, c("lastname", "firstname", "address", "sex"))
model <- problink_em(~ lastname + firstname + address + sex, data = pairs)
pairs <- predict(model, pairs = pairs, add = TRUE)
pairs <- select_n_to_m(pairs, "ntom", "weights", threshold = 0)

# Create linked dataset (inner join)
linked <- link(pairs, selection = "ntom")

# Full outer join
linked_all <- link(pairs, selection = "ntom", all = TRUE)
```

--------------------------------

### Calculate Simple Weighted Score for Record Pairs (R)

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/introduction.md

Calculates a simple score for record pairs based on the sum of similarity scores for specified variables. Allows for custom weights for agreement, non-agreement, and missing values.

```R
pairs <- score_simple(pairs, "score", 
  on = c("lastname", "firstname", "address", "sex"))
```

```R
pairs <- score_simple(pairs, "score", 
  on = c("lastname", "firstname", "address", "sex"), 
  w1 = c(lastname = 2, firstname = 2, address = 1, sex = 0.5),
  w0 = -1, wna = 0)
```

--------------------------------

### N-to-M Selection for Optimized Linkage

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.html

The `select_n_to_m` function performs an n-to-m linkage selection, optimizing the total score of selected records under the constraint that each record can be selected only once. This method can lead to better linkage results than greedy selection.

```r
pairs <- select_n_to_m(pairs, "weights", variable = "ntom", threshold = 0)
table(pairs$truth, pairs$ntom)
```

--------------------------------

### Collect Cluster Pairs Locally in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/using_a_cluster_for_record_linkage.md

Transfers selected pairs from cluster nodes to the local environment for final processing and linkage.

```R
local_pairs <- cluster_collect(pairs, "threshold")
local_pairs <- compare_vars(local_pairs, "truth", on_x = "id", on_y = "id")
local_pairs <- select_n_to_m(local_pairs, "weights", variable = "ntom", threshold = 0)
linked_data_set <- link(local_pairs, selection = "ntom")
```

--------------------------------

### Evaluate threshold performance

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/deduplication.md

Compares pairs against ground truth (official names) to calculate error rates at different similarity thresholds, helping to determine an optimal cutoff.

```R
compare_vars(pairs, "true", on_x = "official_name",
  inplace = TRUE)
pairs$threshold <- trunc(pairs$name/0.05)*0.05
thresholds <- pairs[, .(ftrue = mean(true)), by = threshold]
print(thresholds[order(ftrue)])
```

--------------------------------

### Generate Pairs with Minimum Similarity - R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.html

Generates pairs of records that meet a minimum similarity score on a given set of variables. This method can be computationally intensive as it may require comparing all records. It's useful for identifying potential matches when exact matches are unlikely.

```R
min_sim_pairs <- pairs_minsim(linkexample1, linkexample2, on = c("lastname", "firstname"))
```

--------------------------------

### Generate and Compare Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/vignettes/record_linkage_using_machine_learning.md

Generates record pairs using blocking and compares them based on specified fields using Jaro-Winkler similarity. Handles missing values in the 'sex' field by defining a custom comparison function.

```R
library(reclin2)
library(data.table)

data("linkexample1", "linkexample2")
print(linkexample1)
print(linkexample2)


pairs <- pair_blocking(linkexample1, linkexample2, "postcode")

compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler()))
print(pairs)

na_as_class <- function(x, y) {
  factor(
    ifelse(is.na(x) | is.na(y), 2L, (y == x)*1L),
    levels = 0:2, labels = c("eq", "uneq", "mis"))
}

pairs[, sex := NULL]

compare_pairs(pairs, on = c("lastname", "firstname", "address", "sex"), 
  inplace = TRUE, comparators = list(lastname = cmp_jarowinkler(), 
  firstname = cmp_jarowinkler(), address = cmp_jarowinkler(), sex = na_as_class))
print(pairs)
```

--------------------------------

### Calculate Simple Similarity Scores for Record Pairs in R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.md

Calculates a simple sum of similarity scores for specified variables to score record pairs. This provides a basic measure of record similarity.

```R
pairs <- score_simple(pairs, "score", 
  on = c("lastname", "firstname", "address", "sex"))
```

--------------------------------

### Generate Pairs Using Blocking (R)

Source: https://context7.com/djvanderlaan/reclin2/llms.txt

Generates pairs by requiring exact matches on specified blocking variables, significantly reducing the number of comparisons while retaining most true matches. Supports single or multiple blocking variables and deduplication.

```r
library(reclin2)
data("linkexample1", "linkexample2")

# Generate pairs where postcode must match exactly
pairs <- pair_blocking(linkexample1, linkexample2, on = "postcode")
print(pairs)

# Multiple blocking variables (AND condition - must match on all)
pairs_multi <- pair_blocking(linkexample1, linkexample2, on = c("postcode", "sex"))

# Deduplication mode for finding duplicates within a single dataset
data("town_names")
pairs_dedup <- pair_blocking(town_names, on = "official_name", deduplication = TRUE)
```

--------------------------------

### Generate All Possible Pairs - R

Source: https://github.com/djvanderlaan/reclin2/blob/master/inst/doc/introduction.html

Generates all possible pairs of records between two datasets. This function is used when no blocking strategy is applied, which can be computationally intensive for large datasets. It ensures every record in the first dataset is compared with every record in the second.

```R
all_pairs <- pair(linkexample1, linkexample2)
```