### Quick Start Example

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

A basic example demonstrating how to use the `lt.merge` function for semantic record linkage.

```APIDOC
## Quick Start

This example demonstrates a basic semantic record linkage using `lt.merge`.

```python
import os
import pandas as pd
import linktransformer as lt

# Sample DataFrames
left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})

# Perform semantic merge
out = lt.merge(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
)

# Print the relevant output columns
print(out[["CompanyName_x", "CompanyName_y", "score"]])
```
```

--------------------------------

### Install LinkTransformer

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Install the LinkTransformer package using pip.

```bash
pip install linktransformer
```

--------------------------------

### Installation

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Instructions for installing the LinkTransformer package using pip.

```APIDOC
## Installation

Install the LinkTransformer package using pip:

```bash
pip install linktransformer
```
```

--------------------------------

### Quick Start: Merge DataFrames

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Perform a semantic merge between two pandas DataFrames using a pre-trained sentence transformer model. This is useful for 1:1 or 1:m record linkage.

```python
import os
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})

out = lt.merge(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
)
print(out[["CompanyName_x", "CompanyName_y", "score"]])
```

--------------------------------

### Use Gemini embeddings for matching

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Performs dataset merging using Google Gemini embedding models.

```python
import os
import pandas as pd
import linktransformer as lt

df1 = pd.DataFrame({"Name": ["Apple Inc", "Microsoft"]})
df2 = pd.DataFrame({"Name": ["Apple Computer", "MSFT"]})

# Use Gemini embeddings
result = lt.merge(
    df1=df1,
    df2=df2,
    on="Name",
    model="gemini-embedding-001",
    gemini_key=os.getenv("GEMINI_API_KEY")
)
```

--------------------------------

### lt.all_pair_combos_evaluate

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Computes similarity scores for all possible pairs between left and right columns, creating an n x m similarity matrix.

```APIDOC
## lt.all_pair_combos_evaluate

### Description
Computes similarity scores for all possible pairs between left and right columns, creating an n x m similarity matrix.

### Parameters
#### Request Body
- **df** (DataFrame) - Required - The dataframe containing the items to compare.
- **left_on** (str) - Required - Column name for the first set of items.
- **right_on** (str) - Required - Column name for the second set of items.
- **model** (str) - Required - The sentence-transformer model name to use for scoring.

### Response
#### Success Response (200)
- **all_pairs** (DataFrame) - A dataframe with all n*m combinations and their similarity scores.
```

--------------------------------

### Use OpenAI embeddings for matching and retrieval

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Performs dataset merging or k-NN retrieval using OpenAI embedding models.

```python
import os
import pandas as pd
import linktransformer as lt

df1 = pd.DataFrame({"Name": ["Apple Inc", "Microsoft"]})
df2 = pd.DataFrame({"Name": ["Apple Computer", "MSFT"]})

# Use OpenAI embeddings for matching
result = lt.merge(
    df1=df1,
    df2=df2,
    on="Name",
    model="text-embedding-3-small",  # OpenAI embedding model
    openai_key=os.getenv("OPENAI_API_KEY")
)

# Or use OpenAI for k-NN retrieval
candidates = lt.merge_knn(
    df1=df1,
    df2=df2,
    on="Name",
    model="text-embedding-ada-002",
    k=5,
    openai_key=os.getenv("OPENAI_API_KEY")
)
```

--------------------------------

### Train a classification model

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Trains a text classification model using Hugging Face transformers.

```python
import pandas as pd
import linktransformer as lt

# Training data with labels
train_data = pd.DataFrame({
    "text": [
        "Great product, highly recommend",
        "Waste of money, don't buy",
        "Average quality, met expectations",
        # ... more training examples
    ],
    "sentiment": ["positive", "negative", "neutral"]
})
```

--------------------------------

### Transform rows using LLM prompts

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Standardizes or cleans data in specific columns using LLM-based transformations.

```python
import os
import pandas as pd
import linktransformer as lt

# Standardize company names for better matching
messy_data = pd.DataFrame({
    "CompanyName": ["APPLE INC.", "microsft corp", "Amzon.com Inc"],
    "Address": ["1 Infinite Loop, Cupertino", "Redmond WA", "Seattle"]
})

# Transform/normalize company names
cleaned = lt.transform_rows(
    df=messy_data,
    on="CompanyName",
    model="gpt-4o-mini",
    openai_key=os.getenv("OPENAI_API_KEY"),
    openai_prompt=(
        "Standardize the company name: fix spelling, proper capitalization, "
        "remove Inc/Corp suffixes. Return JSON array of corrected strings."
    ),
    batch_size=50,
    output_column="StandardizedName"
)

print(cleaned[["CompanyName", "StandardizedName"]])
```

--------------------------------

### Train a classification model with LinkTransformer

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Trains a classifier using a transformer model. Requires a pandas DataFrame and specified column mappings.

```python
best_model_path, best_metric, label_map = lt.train_clf_model(
    data=train_data,
    model="distilroberta-base",
    on=["text"],
    label_col_name="sentiment",
    model_save_dir="./trained_models",
    data_dir="./data_splits",
    epochs=5,
    batch_size=16,
    lr=2e-5,
    weighted_loss=True,  # Handle class imbalance
    wandb_log=False
)

print(f"Best model saved at: {best_model_path}")
print(f"Best validation metric: {best_metric}")
print(f"Label mapping: {label_map}")
# Use the trained model for inference:
# lt.classify_rows(df, on="text", model=best_model_path, num_labels=3)
```

--------------------------------

### Core APIs - Evaluate matched pairs

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Includes functions for evaluating the similarity of matched pairs and performing dense pairwise scoring.

```APIDOC
## Core APIs - Evaluate Matched Pairs

### `lt.evaluate_pairs(...)`

#### Description
Calculates similarity scores over known pairs of records to evaluate linkage performance.

### `lt.all_pair_combos_evaluate(...)`

#### Description
Performs dense pairwise scoring across all possible combinations of records for evaluation purposes.
```

--------------------------------

### Core APIs - Cluster and deduplicate

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Provides functions for clustering semantically similar rows and deduplicating rows based on similarity.

```APIDOC
## Core APIs - Cluster and Deduplicate

### `lt.cluster_rows(...)`

#### Description
Clusters semantically similar rows in a dataframe.

### `lt.dedup_rows(...)`

#### Description
Clusters rows based on semantic similarity and keeps representative rows, effectively deduplicating the data.

### Parameters for `dedup_rows`
- `left_df` (pd.DataFrame): The input dataframe.
- `on` (str): The column to use for clustering.
- `model` (str): The model to use for calculating similarity (e.g., SBERT model).
- `cluster_type` (str): The type of clustering algorithm (e.g., 'agglomerative').
- `cluster_params` (dict): Parameters for the clustering algorithm (e.g., `{"threshold": 0.7}`).

### Request Example for `dedup_rows`
```python
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation", "Tech Corp"], "Country": ["USA", "USA"]})

deduped = lt.dedup_rows(
    left_df,
    on="CompanyName",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="agglomerative",
    cluster_params={"threshold": 0.7},
)
```

### Response
#### Success Response (200)
Returns a dataframe with duplicate rows removed based on the specified clustering criteria.
```

--------------------------------

### Top-K Candidate Retrieval with lt.merge_knn

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Employ lt.merge_knn to retrieve the k nearest neighbors from the right dataframe for each row in the left dataframe. This is useful for candidate-based matching workflows. An optional similarity threshold can filter low-scoring matches.

```python
import pandas as pd
import linktransformer as lt

# Product catalog matching
products_df = pd.DataFrame({
    "ProductName": ["iPhone 15 Pro", "MacBook Air M2", "iPad Pro 12.9"],
    "Category": ["Phone", "Laptop", "Tablet"]
})

catalog_df = pd.DataFrame({
    "ItemName": ["iPhone 15 Pro Max", "iPhone 15", "MacBook Pro M2",
                 "MacBook Air", "iPad Air", "iPad Pro"],
    "Type": ["Phone", "Phone", "Laptop", "Laptop", "Tablet", "Tablet"]
})

# Get top 3 matches for each product
matches = lt.merge_knn(
    df1=products_df,
    df2=catalog_df,
    left_on="ProductName",
    right_on="ItemName",
    k=3,  # Number of candidates per row
    model="sentence-transformers/all-MiniLM-L6-v2",
    drop_sim_threshold=0.5  # Optional: filter low-similarity matches
)

# Each left row now has k corresponding right matches
print(matches[["ProductName", "ItemName", "score"]])
```

--------------------------------

### End-to-End Linkage with LLM Adjudication using lt.merge_k_judge

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Utilize lt.merge_k_judge for an end-to-end linkage workflow combining embedding retrieval with LLM-based match verification. This provides match decisions and confidence scores. Ensure to set the LLM provider and API key.

```python
import os
import pandas as pd
import linktransformer as lt

# Company name matching with LLM verification
company_a = pd.DataFrame({
    "Name": ["General Electric", "JP Morgan Chase", "Procter & Gamble"],
    "Industry": ["Industrial", "Banking", "Consumer Goods"]
})

company_b = pd.DataFrame({
    "Name": ["GE Corporation", "JPMorgan", "P&G Company"],
    "Industry": ["Manufacturing", "Financial Services", "FMCG"]
})

# End-to-end linkage with LLM judge
judged_matches = lt.merge_k_judge(
    df1=company_a,
    df2=company_b,
    on=["Name", "Industry"],
    k=3,  # Retrieve top 3 candidates
    knn_sbert_model="sentence-transformers/all-MiniLM-L6-v2",  # For retrieval
    judge_llm_model="gpt-4o-mini",  # For verification
    llm_provider="openai",  # or "gemini"
    openai_key=os.getenv("OPENAI_API_KEY"),
    confidence_threshold=0.7  # Optional: filter by confidence
)

# Output includes retrieval score and LLM judgment
print(judged_matches[["Name_x", "Name_y", "score", "llm_is_match", "llm_confidence"]])
```

--------------------------------

### End-to-End Linkage with merge_k_judge

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Utilize the `merge_k_judge` API for an end-to-end record linkage workflow that includes retrieval and LLM adjudication. This API returns match decisions and confidence scores.

```python
judged = lt.merge_k_judge(
    df1=left_df,
    df2=right_df,
    on=["CompanyName", "Country"],
    k=5,
    knn_sbert_model="sentence-transformers/all-MiniLM-L6-v2",
    judge_llm_model="gpt-4o-mini",
    llm_provider="openai",
    openai_key=os.getenv("OPENAI_API_KEY"),
)

# key output columns:
# - score (retrieval similarity)
# - is_match (bool)
# - confidence (float in [0, 1] when available)
```

--------------------------------

### Core APIs - Link two dataframes

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Provides functions for linking two dataframes using various methods including semantic 1:1/1:m/m:1 linkage, top-k candidate retrieval, fuzzy merging within blocks, and mapping fine rows to coarser labels.

```APIDOC
## Core APIs - Linking DataFrames

### `lt.merge(...)`

#### Description
Performs semantic 1:1, 1:m, or m:1 linkage between two dataframes.

### `lt.merge_knn(...)`

#### Description
Retrieves the top-k semantically similar candidate pairs between two dataframes.

### Request Example for `merge_knn`
```python
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})

matches = lt.merge_knn(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
    k=3,
)
```

### `lt.merge_blocking(...)`

#### Description
Performs a fuzzy merge within blocks defined by exact matches on specified columns.

### `lt.aggregate_rows(...)`

#### Description
Maps fine-grained rows to coarser labels based on semantic similarity.
```

--------------------------------

### Core APIs - Transform rows with LLM prompts

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Utilizes `lt.transform_rows` to normalize, rewrite, or standardize values within specified columns of a dataframe using LLM prompts.

```APIDOC
## Core APIs - Transform Rows

### `lt.transform_rows(...)`

#### Description
Normalizes, rewrites, or standardizes values in one or more columns of a dataframe using LLM prompts. Useful for tasks like fixing OCR errors or standardizing names.

### Parameters
- `left_df` (pd.DataFrame): The input dataframe.
- `on` (list of str): Columns to use for transformation.
- `model` (str): The LLM model to use (e.g., 'gpt-4o-mini').
- `openai_key` (str, optional): OpenAI API key.
- `openai_prompt` (str): The prompt to guide the LLM transformation.

### Request Example
```python
import os
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})

cleaned = lt.transform_rows(
    left_df,
    on=["CompanyName", "Country"],
    model="gpt-4o-mini",
    openai_key=os.getenv("OPENAI_API_KEY"),
    openai_prompt=(
        "Standardize organization names and country strings for record linkage. "
        "Return a JSON list in the same order."
    ),
)
# The output will add columns like: transformed_CompanyName-Country
```

### Response
#### Success Response (200)
Returns the input dataframe with new columns containing the transformed values.
```

--------------------------------

### End-to-end linkage Workflow: merge_k_judge

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

The `merge_k_judge` function provides an end-to-end record linkage solution that includes retrieval of top-k candidates using embeddings and adjudication of candidate pairs using an LLM, returning match decisions and confidence scores.

```APIDOC
## `merge_k_judge` - End-to-End Record Linkage

### Description
Performs end-to-end record linkage by retrieving top-k candidates using embeddings and then adjudicating each candidate pair with an LLM to determine match decisions and confidence scores.

### Method
`lt.merge_k_judge`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
import os
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})

judged = lt.merge_k_judge(
    df1=left_df,
    df2=right_df,
    on=["CompanyName", "Country"],
    k=5,
    knn_sbert_model="sentence-transformers/all-MiniLM-L6-v2",
    judge_llm_model="gpt-4o-mini",
    llm_provider="openai",
    openai_key=os.getenv("OPENAI_API_KEY")
)

# key output columns:
# - score (retrieval similarity)
# - is_match (bool)
# - confidence (float in [0, 1] when available)
```

### Response
#### Success Response (200)
Returns a DataFrame containing linkage results, including similarity scores, match decisions, and confidence scores.

#### Response Example
(DataFrame structure with columns like 'score', 'is_match', 'confidence')
```

--------------------------------

### Map fine-grained to coarse labels with aggregation

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Employ `aggregate_rows` to map detailed records to broader categories using semantic matching. This function is useful for standardizing product names or other granular data to a reference set of coarser labels.

```python
import pandas as pd
import linktransformer as lt

# Map detailed product names to standard categories
detailed_products = pd.DataFrame({
    "ProductName": ["iPhone 15 Pro Max 256GB", "Galaxy S24 Ultra", "Pixel 8 Pro"],
    "Price": [1199, 1299, 999]
})

standard_categories = pd.DataFrame({
    "Category": ["Apple iPhone", "Samsung Galaxy", "Google Pixel"],
    "Brand": ["Apple", "Samsung", "Google"]
})

# Aggregate to standard categories
aggregated = lt.aggregate_rows(
    df=detailed_products,
    ref_df=standard_categories,
    left_on="ProductName",
    right_on="Category",
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(aggregated[["ProductName", "Category", "Brand", "score"]])
```

--------------------------------

### Perform range-based similarity merge

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Utilize `merge_range` to find all matches exceeding a specified similarity threshold, rather than a fixed number of top matches. This is ideal when the number of relevant matches per row is variable, such as finding all similar product descriptions.

```python
import pandas as pd
import linktransformer as lt

# Find all similar product descriptions above threshold
source_products = pd.DataFrame({
    "Description": ["Wireless Bluetooth Headphones", "USB-C Charging Cable"]
})

target_products = pd.DataFrame({
    "Description": ["Bluetooth Wireless Earbuds", "Bluetooth Headset Pro",
                    "USB Type-C Cable", "Lightning Cable", "Wireless Charger"]
})

# Get all matches with similarity >= 0.6
range_matches = lt.merge_range(
    df1=source_products,
    df2=target_products,
    left_on="Description",
    right_on="Description",
    sim_threshold=0.6,  # Minimum similarity threshold
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(range_matches[["Description_x", "Description_y", "score"]])
# Returns variable number of matches per source row
# ...

```

--------------------------------

### Train custom linkage models

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Trains a semantic similarity model using either cluster-based data or paired labeled data.

```python
import pandas as pd
import linktransformer as lt

# Training data: paired examples with or without labels
# Option 1: Positive pairs only (cluster-based)
cluster_data = pd.DataFrame({
    "text": ["Apple Inc", "Apple Computer", "Apple Corp",
             "Microsoft", "MSFT", "Microsoft Corporation"],
    "cluster_id": [1, 1, 1, 2, 2, 2]
})

# Option 2: Paired data with labels
paired_data = pd.DataFrame({
    "left_text": ["Apple Inc", "Microsoft", "Random Corp"],
    "right_text": ["Apple Computer", "Google", "Another Company"],
    "label": [1, 0, 0]  # 1=match, 0=no match
})

# Train using cluster data
best_model = lt.train_model(
    data=cluster_data,
    model_path="sentence-transformers/all-MiniLM-L6-v2",
    clus_id_col_name="cluster_id",
    clus_text_col_names=["text"],
    training_args={
        "num_epochs": 10,
        "train_batch_size": 32,
        "learning_rate": 2e-5,
        "model_save_dir": "./models",
        "model_save_name": "my_linkage_model",
        "loss_type": "supcon"  # or "onlinecontrastive"
    },
    log_wandb=False
)

# Train using paired labeled data
best_model = lt.train_model(
    data=paired_data,
    model_path="sentence-transformers/all-MiniLM-L6-v2",
    left_col_names=["left_text"],
    right_col_names=["right_text"],
    label_col_name="label",
    training_args={
        "num_epochs": 5,
        "model_save_dir": "./models",
        "model_save_name": "paired_linkage_model"
    }
)

print(f"Trained model saved at: {best_model}")
# Use with: lt.merge(df1, df2, model=best_model)
```

--------------------------------

### lt.merge_blocking - Merge with Blocking Variables

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Performs semantic merge within blocks defined by exact matches on blocking variables, significantly improving efficiency for large datasets.

```APIDOC
## lt.merge_blocking - Merge with Blocking Variables

### Description
Performs semantic merge within blocks defined by exact matches on blocking variables, significantly improving efficiency for large datasets.

### Method
`lt.merge_blocking`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **df1** (pd.DataFrame) - The first dataframe.
- **df2** (pd.DataFrame) - The second dataframe.
- **left_on** (str) - The column(s) in df1 to use for matching.
- **right_on** (str) - The column(s) in df2 to use for matching.
- **blocking_vars** (list[str]) - List of columns to use for blocking (exact match).
- **model** (str) - The name of the pre-trained sentence transformer model to use.

### Request Example
```python
import pandas as pd
import linktransformer as lt

transactions_a = pd.DataFrame({
    "Description": ["Wire transfer to ABC Corp", "Payment for services", "Loan disbursement"],
    "Country": ["USA", "USA", "UK"],
    "Amount": [10000, 5000, 25000]
})

transactions_b = pd.DataFrame({
    "Description": ["ABC Corporation wire", "Service payment received", "Loan to customer"],
    "Country": ["USA", "USA", "UK"],
    "Amount": [10000, 5000, 25000]
})

blocked_matches = lt.merge_blocking(
    df1=transactions_a,
    df2=transactions_b,
    left_on="Description",
    right_on="Description",
    blocking_vars=["Country"],
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(blocked_matches[["Description_x", "Description_y", "Country_x", "score"]])
```

### Response
#### Success Response (200)
- **matched_dataframe** (pd.DataFrame) - A dataframe containing the merged results with a 'score' column indicating similarity.

#### Response Example
```json
{
  "Description_x": ["Wire transfer to ABC Corp", "Payment for services"],
  "Description_y": ["ABC Corporation wire", "Service payment received"],
  "Country_x": ["USA", "USA"],
  "score": [0.95, 0.92]
}
```
```

--------------------------------

### Perform blocking merge with exact matches on blocking variables

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Use `merge_blocking` for efficient semantic merging on large datasets by first grouping rows based on exact matches in specified blocking variables. This is useful for transactional data where matching should be confined within specific categories like country.

```python
import pandas as pd
import linktransformer as lt

# Transaction matching within country blocks
transactions_a = pd.DataFrame({
    "Description": ["Wire transfer to ABC Corp", "Payment for services", "Loan disbursement"],
    "Country": ["USA", "USA", "UK"],
    "Amount": [10000, 5000, 25000]
})

transactions_b = pd.DataFrame({
    "Description": ["ABC Corporation wire", "Service payment received", "Loan to customer"],
    "Country": ["USA", "USA", "UK"],
    "Amount": [10000, 5000, 25000]
})

# Match only within same country (blocking)
blocked_matches = lt.merge_blocking(
    df1=transactions_a,
    df2=transactions_b,
    left_on="Description",
    right_on="Description",
    blocking_vars=["Country"],  # Only match within same country
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(blocked_matches[["Description_x", "Description_y", "Country_x", "score"]])
```

--------------------------------

### Compute dense pairwise similarity scores

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Generates an n x m similarity matrix by computing scores for all possible pairs between two columns.

```python
import pandas as pd
import linktransformer as lt

# Score all possible pairs
items = pd.DataFrame({
    "ItemA": ["Red Apple", "Green Apple", "Orange"],
    "ItemB": ["Apple Fruit", "Citrus Orange", "Banana"]
})

# Get all pairwise similarities
all_pairs = lt.all_pair_combos_evaluate(
    df=items,
    left_on="ItemA",
    right_on="ItemB",
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(all_pairs)
```

--------------------------------

### Core APIs - Train linkage models

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Provides functionality to train custom linkage models using paired or clustered data.

```APIDOC
## Core APIs - Train Linkage Models

### `lt.train_model(...)`

#### Description
Trains a linkage model using provided paired or clustered data.

### Parameters
- `data` (pd.DataFrame): Training data, which can be paired or clustered.
- Other model-specific training parameters.
```

--------------------------------

### Evaluate similarity scores for existing pairs

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Calculates similarity scores for pre-matched pairs in a dataframe to validate existing linkages.

```python
import pandas as pd
import linktransformer as lt

# Evaluate existing matched pairs
matched_pairs = pd.DataFrame({
    "SourceName": ["Apple Inc", "Microsoft Corp", "Random Company"],
    "TargetName": ["Apple Computer", "MSFT", "Unrelated Entity"],
    "ManualMatch": [1, 1, 0]
})

# Calculate similarity scores for pairs
scored_pairs = lt.evaluate_pairs(
    df=matched_pairs,
    left_on="SourceName",
    right_on="TargetName",
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(scored_pairs[["SourceName", "TargetName", "ManualMatch", "score"]])
```

--------------------------------

### Core APIs - Classification

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Offers functions for classifying rows using pre-trained models or training custom classification models.

```APIDOC
## Core APIs - Classification

### `lt.classify_rows(...)`

#### Description
Classifies rows using either Hugging Face or OpenAI chat models.

### `lt.train_clf_model(...)`

#### Description
Trains a custom row classifier model.
```

--------------------------------

### lt.cluster_rows - Semantic Clustering

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Clusters semantically similar rows in a dataframe using various clustering algorithms (Agglomerative, HDBScan, SLINK/DBSCAN).

```APIDOC
## lt.cluster_rows - Semantic Clustering

### Description
Clusters semantically similar rows in a dataframe using various clustering algorithms (Agglomerative, HDBScan, SLINK/DBSCAN).

### Method
`lt.cluster_rows`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **df** (pd.DataFrame) - The dataframe to cluster.
- **on** (str) - The column to use for semantic similarity calculation.
- **model** (str) - The name of the pre-trained sentence transformer model to use.
- **cluster_type** (str) - The type of clustering algorithm to use. Options: "agglomerative", "HDBScan", "SLINK".
- **cluster_params** (dict) - Dictionary of parameters specific to the chosen clustering algorithm.

### Request Example
```python
import pandas as pd
import linktransformer as lt

companies = pd.DataFrame({
    "Name": ["Apple Inc", "Apple Computer", "Apple Corp",
             "Microsoft", "MSFT", "Microsoft Corp",
             "Amazon", "Amazon.com", "Amazon Inc"],
    "Revenue": [100, 100, 100, 200, 200, 200, 300, 300, 300]
})

clustered = lt.cluster_rows(
    df=companies,
    on="Name",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="agglomerative",
    cluster_params={
        "threshold": 0.5,
        "clustering linkage": "ward",
        "metric": "euclidean"
    }
)

print(clustered[["Name", "cluster"]])
```

### Response
#### Success Response (200)
- **clustered_dataframe** (pd.DataFrame) - The original dataframe with an added 'cluster' column indicating the cluster assignment for each row.

#### Response Example
```json
{
  "Name": ["Apple Inc", "Apple Computer", "Apple Corp", "Microsoft", "MSFT", "Microsoft Corp", "Amazon", "Amazon.com", "Amazon Inc"],
  "Revenue": [100, 100, 100, 200, 200, 200, 300, 300, 300],
  "cluster": [0, 0, 0, 1, 1, 1, 2, 2, 2]
}
```
```

--------------------------------

### Retrieve Top-K Candidates with merge_knn

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Retrieve the top-k most similar candidate pairs between two dataframes using semantic embeddings. This is a core component for candidate retrieval in linkage tasks.

```python
matches = lt.merge_knn(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
    k=3,
)
```

--------------------------------

### Classify text rows

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Classifies dataframe text using either OpenAI chat models or local transformer models.

```python
import os
import pandas as pd
import linktransformer as lt

# Classify customer feedback
feedback = pd.DataFrame({
    "Text": [
        "This product is amazing, best purchase ever!",
        "Terrible quality, broke after one day",
        "It's okay, nothing special but works fine"
    ],
    "ProductId": [101, 102, 103]
})

# Classification with OpenAI
classified = lt.classify_rows(
    df=feedback,
    on="Text",
    model="gpt-4o-mini",
    openai_key=os.getenv("OPENAI_API_KEY"),
    openai_topic="positive customer sentiment",
    num_labels=2,
    label_map={"Yes": 1, "No": 0}
)

print(classified[["Text", "clf_preds_Text"]])

# Or use a trained local model
classified_local = lt.classify_rows(
    df=feedback,
    on="Text",
    model="path/to/trained/classifier",
    num_labels=3,
    batch_size=64,
    progress_bar=True
)
```

--------------------------------

### Deduplicate rows using name similarity

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Performs row deduplication based on string similarity using a specified sentence-transformer model and clustering algorithm.

```python
deduped = lt.dedup_rows(
    df=contacts,
    on="Name",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="SLINK",
    cluster_params={
        "threshold": 0.3,  # Distance threshold
        "min cluster size": 2,
        "metric": "cosine"
    }
)

print(f"Original: {len(contacts)} rows, Deduplicated: {len(deduped)} rows")
print(deduped[["Name", "Email"]])
```

--------------------------------

### lt.aggregate_rows - Map Fine-Grained to Coarse Labels

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Aggregates rows from a detailed dataframe to a reference dataframe with coarser categories using semantic matching.

```APIDOC
## lt.aggregate_rows - Map Fine-Grained to Coarse Labels

### Description
Aggregates rows from a detailed dataframe to a reference dataframe with coarser categories using semantic matching.

### Method
`lt.aggregate_rows`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **df** (pd.DataFrame) - The detailed dataframe to aggregate.
- **ref_df** (pd.DataFrame) - The reference dataframe with coarser categories.
- **left_on** (str) - The column in `df` to use for matching.
- **right_on** (str) - The column in `ref_df` to use for matching.
- **model** (str) - The name of the pre-trained sentence transformer model to use.

### Request Example
```python
import pandas as pd
import linktransformer as lt

detailed_products = pd.DataFrame({
    "ProductName": ["iPhone 15 Pro Max 256GB", "Galaxy S24 Ultra", "Pixel 8 Pro"],
    "Price": [1199, 1299, 999]
})

standard_categories = pd.DataFrame({
    "Category": ["Apple iPhone", "Samsung Galaxy", "Google Pixel"],
    "Brand": ["Apple", "Samsung", "Google"]
})

aggregated = lt.aggregate_rows(
    df=detailed_products,
    ref_df=standard_categories,
    left_on="ProductName",
    right_on="Category",
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(aggregated[["ProductName", "Category", "Brand", "score"]])
```

### Response
#### Success Response (200)
- **aggregated_dataframe** (pd.DataFrame) - A dataframe containing the aggregated results, mapping detailed rows to reference categories with a 'score' column.

#### Response Example
```json
{
  "ProductName": ["iPhone 15 Pro Max 256GB", "Galaxy S24 Ultra", "Pixel 8 Pro"],
  "Category": ["Apple iPhone", "Samsung Galaxy", "Google Pixel"],
  "Brand": ["Apple", "Samsung", "Google"],
  "score": [0.90, 0.92, 0.88]
}
```
```

--------------------------------

### lt.merge_knn - Top-K Candidate Retrieval

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Retrieves the k nearest neighbors from the right dataframe for each row in the left dataframe.

```APIDOC
## lt.merge_knn

### Description
Retrieves the k nearest neighbors from the right dataframe for each row in the left dataframe, enabling candidate-based matching workflows.

### Parameters
#### Request Body
- **df1** (pd.DataFrame) - Required - The left dataframe.
- **df2** (pd.DataFrame) - Required - The right dataframe.
- **left_on** (str) - Required - Column in df1 to match.
- **right_on** (str) - Required - Column in df2 to match.
- **k** (int) - Required - Number of candidates to retrieve per row.
- **model** (str) - Optional - Embedding model to use.
- **drop_sim_threshold** (float) - Optional - Similarity threshold to filter matches.

### Response
#### Success Response (200)
- **matches** (pd.DataFrame) - Dataframe containing k matches per row with similarity scores.
```

--------------------------------

### Transform Rows with LLM Prompts

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Use `transform_rows` to normalize, rewrite, or standardize values in specified columns using LLMs. This is useful for cleaning and preparing data for linkage, such as fixing OCR errors or standardizing names.

```python
cleaned = lt.transform_rows(
    left_df,
    on=["CompanyName", "Country"],
    model="gpt-4o-mini",
    openai_key=os.getenv("OPENAI_API_KEY"),
    openai_prompt=(
        "Standardize organization names and country strings for record linkage. "
        "Return a JSON list in the same order."
    ),
)
# adds: transformed_CompanyName-Country
```

--------------------------------

### Cluster and Deduplicate Rows

Source: https://github.com/dell-research-harvard/linktransformer/blob/main/README.md

Deduplicate rows in a DataFrame based on semantic similarity using clustering. The `dedup_rows` function clusters semantically similar rows and keeps representative ones.

```python
deduped = lt.dedup_rows(
    left_df,
    on="CompanyName",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="agglomerative",
    cluster_params={"threshold": 0.7},
)
```

--------------------------------

### Semantic 1:1 Dataframe Merge with lt.merge

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Use lt.merge to find the best semantic match for each row in the left dataframe from the right dataframe. Specify columns to match on and the embedding model. Ensure dataframes are prepared with relevant columns.

```python
import pandas as pd
import linktransformer as lt

# Prepare dataframes with company data
left_df = pd.DataFrame({
    "CompanyName": ["Apple Inc.", "Microsoft Corporation", "Amazon.com"],
    "Country": ["USA", "USA", "USA"]
})

right_df = pd.DataFrame({
    "CompanyName": ["Apple Computer", "MSFT Corp", "Amazon Web Services"],
    "Country": ["United States", "United States", "United States"]
})

# Semantic merge on multiple columns
result = lt.merge(
    df1=left_df,
    df2=right_df,
    on=["CompanyName", "Country"],  # Columns to match on
    model="sentence-transformers/all-MiniLM-L6-v2",
    suffixes=("_left", "_right"),
    batch_size=128
)

# Result contains matched rows with similarity scores
print(result[["CompanyName_left", "CompanyName_right", "score"]])
```

--------------------------------

### Perform semantic clustering on dataframe rows

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Use `cluster_rows` to group semantically similar rows within a single dataframe. It supports various clustering algorithms and allows customization of clustering parameters for fine-grained control over the grouping process.

```python
import pandas as pd
import linktransformer as lt

# Cluster similar company names
companies = pd.DataFrame({
    "Name": ["Apple Inc", "Apple Computer", "Apple Corp",
             "Microsoft", "MSFT", "Microsoft Corp",
             "Amazon", "Amazon.com", "Amazon Inc"],
    "Revenue": [100, 100, 100, 200, 200, 200, 300, 300, 300]
})

# Cluster using agglomerative clustering
clustered = lt.cluster_rows(
    df=companies,
    on="Name",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="agglomerative",  # Options: "agglomerative", "HDBScan", "SLINK"
    cluster_params={
        "threshold": 0.5,
        "clustering linkage": "ward",
        "metric": "euclidean"
    }
)

print(clustered[["Name", "cluster"]])
# Output shows cluster assignments for each row
#              Name  cluster
# 0       Apple Inc        0
# 1  Apple Computer        0
# 2      Apple Corp        0
# 3       Microsoft        1
# 4            MSFT        1
# ...

```

--------------------------------

### lt.merge_range - Range-Based Similarity Merge

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Returns all matches above a similarity threshold rather than just top-k, useful when the number of matches per row varies.

```APIDOC
## lt.merge_range - Range-Based Similarity Merge

### Description
Returns all matches above a similarity threshold rather than just top-k, useful when the number of matches per row varies.

### Method
`lt.merge_range`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **df1** (pd.DataFrame) - The first dataframe.
- **df2** (pd.DataFrame) - The second dataframe.
- **left_on** (str) - The column(s) in df1 to use for matching.
- **right_on** (str) - The column(s) in df2 to use for matching.
- **sim_threshold** (float) - The minimum similarity threshold for a match.
- **model** (str) - The name of the pre-trained sentence transformer model to use.

### Request Example
```python
import pandas as pd
import linktransformer as lt

source_products = pd.DataFrame({
    "Description": ["Wireless Bluetooth Headphones", "USB-C Charging Cable"]
})

target_products = pd.DataFrame({
    "Description": ["Bluetooth Wireless Earbuds", "Bluetooth Headset Pro",
                    "USB Type-C Cable", "Lightning Cable", "Wireless Charger"]
})

range_matches = lt.merge_range(
    df1=source_products,
    df2=target_products,
    left_on="Description",
    right_on="Description",
    sim_threshold=0.6,
    model="sentence-transformers/all-MiniLM-L6-v2"
)

print(range_matches[["Description_x", "Description_y", "score"]])
```

### Response
#### Success Response (200)
- **matched_dataframe** (pd.DataFrame) - A dataframe containing all matches above the similarity threshold, with a 'score' column.

#### Response Example
```json
{
  "Description_x": ["Wireless Bluetooth Headphones", "Wireless Bluetooth Headphones", "USB-C Charging Cable"],
  "Description_y": ["Bluetooth Wireless Earbuds", "Bluetooth Headset Pro", "USB Type-C Cable"],
  "score": [0.85, 0.70, 0.90]
}
```
```

--------------------------------

### lt.evaluate_pairs

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Evaluates similarity scores for pre-matched pairs in a dataframe, useful for validating existing linkages.

```APIDOC
## lt.evaluate_pairs

### Description
Evaluates similarity scores for pre-matched pairs in a dataframe, useful for validating existing linkages.

### Parameters
#### Request Body
- **df** (DataFrame) - Required - The dataframe containing the pairs to evaluate.
- **left_on** (str) - Required - Column name in the dataframe for the left side of the pair.
- **right_on** (str) - Required - Column name in the dataframe for the right side of the pair.
- **model** (str) - Required - The sentence-transformer model name to use for scoring.

### Response
#### Success Response (200)
- **scored_pairs** (DataFrame) - A dataframe containing the original pairs with an added 'score' column.
```

--------------------------------

### lt.merge - Semantic 1:1 Dataframe Merge

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Merges two dataframes using language model embeddings to find the best semantic match for each row.

```APIDOC
## lt.merge

### Description
Merges two dataframes using language model embeddings, finding the best match for each row in the left dataframe from the right dataframe based on semantic similarity.

### Parameters
#### Request Body
- **df1** (pd.DataFrame) - Required - The left dataframe to merge.
- **df2** (pd.DataFrame) - Required - The right dataframe to merge.
- **on** (list) - Required - Columns to match on.
- **model** (str) - Optional - The embedding model to use (e.g., 'sentence-transformers/all-MiniLM-L6-v2').
- **suffixes** (tuple) - Optional - Suffixes to apply to overlapping columns.
- **batch_size** (int) - Optional - Batch size for embedding generation.

### Response
#### Success Response (200)
- **result** (pd.DataFrame) - A merged dataframe containing matched rows and similarity scores.
```

--------------------------------

### lt.classify_rows

Source: https://context7.com/dell-research-harvard/linktransformer/llms.txt

Classifies text in dataframe rows using either a trained transformer classifier or OpenAI chat models.

```APIDOC
## lt.classify_rows

### Description
Classifies text in dataframe rows using either a trained transformer classifier or OpenAI chat models.

### Parameters
#### Request Body
- **df** (DataFrame) - Required - The dataframe containing text to classify.
- **on** (str) - Required - The column containing text to classify.
- **model** (str) - Required - The model name or path to a trained classifier.
- **openai_key** (str) - Optional - API key for OpenAI if using LLM models.
- **openai_topic** (str) - Optional - Topic context for LLM classification.
- **num_labels** (int) - Required - Number of classification labels.
- **label_map** (dict) - Optional - Mapping of labels to numeric values.

### Response
#### Success Response (200)
- **classified** (DataFrame) - The original dataframe with added prediction columns.
```