### Clone ClinLinker-KB Repository

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/README.md

Clone the repository to your local machine to begin setup. Ensure you have Git installed.

```bash
git clone https://github.com/ICB-UMA/ClinLinker-KB.git
```

--------------------------------

### Install Project Requirements

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/README.md

Navigate to the cloned repository directory and install all necessary Python packages using pip.

```bash
cd ClinLinker-KB
pip install -r requirements
```

--------------------------------

### Reranking and Accuracy Calculation Start (Dataset 3)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs the completion of reranking and the start of top-k accuracy calculation for a dataset of size 1730.

```text
2025-04-07 10:23:50,049 - INFO - Reranking complete. (crossEncoder.py:205)
2025-04-07 10:23:50,049 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 10:23:50,049 - INFO - DataFrame size: 1730, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
```

--------------------------------

### Initialize and Fit FAISS Encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

This snippet initializes and fits a FAISS encoder, logging its progress. Ensure the FAISS library is installed and configured.

```python
baseline_preds = test_df.copy()
baseline_preds["candidates"] = candidates
baseline_preds["codes"] = codes
models.append("SapBERT-XLM-R-large")
results.append(baseline_preds)
del faiss_encoder
```

--------------------------------

### Reranking and Accuracy Calculation Start (Dataset 4)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs the completion of reranking and the start of top-k accuracy calculation for a dataset of size 878.

```text
2025-04-07 10:25:19,640 - INFO - Reranking complete. (crossEncoder.py:205)
2025-04-07 10:25:19,640 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 10:25:19,641 - INFO - DataFrame size: 878, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
```

--------------------------------

### Top-k Accuracy Calculation Start (FAISS)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs the start of top-k accuracy calculation after FAISS index fitting, specifying DataFrame size and top-k values.

```text
2025-04-07 10:28:24,170 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 10:28:24,171 - INFO - DataFrame size: 3618, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
```

--------------------------------

### Reranking and Accuracy Calculation Start (Dataset 2)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs the completion of reranking and the initiation of top-k accuracy calculation for a dataset of size 3512.

```text
2025-04-07 10:21:07,153 - INFO - Reranking complete. (crossEncoder.py:205)
2025-04-07 10:21:07,153 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 10:21:07,153 - INFO - DataFrame size: 3512, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
```

--------------------------------

### Reranking Completion and Accuracy Calculation Start

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Indicates the completion of the reranking process and the start of Top-k accuracy calculations. This log output is helpful for understanding the workflow and timing of evaluation steps.

```python
2025-04-07 11:23:29,952 - INFO - Reranking complete. (crossEncoder.py:205)
2025-04-07 11:23:29,952 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 11:23:29,953 - INFO - DataFrame size: 2848, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)

```

--------------------------------

### Reranking and Accuracy Calculation Start

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs indicate the completion of reranking and the start of top-k accuracy calculation, including DataFrame size and top-k values.

```text
2025-04-07 10:16:48,005 - INFO - Reranking complete. (crossEncoder.py:205)
2025-04-07 10:16:48,005 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 10:16:48,005 - INFO - DataFrame size: 3618, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
```

--------------------------------

### Initialize Grandparents Pairs Generation

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Initializes the dictionary for generating positive pairs related to grandparents. This is the starting point for creating triplets involving two-level hierarchical relationships.

```python
%%time
grandparents_dict = {}
```

--------------------------------

### Get Shape of Combined Concept DataFrame

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Displays the dimensions of the DataFrame after combining the original Spanish concepts with the newly integrated trade names. This confirms the total number of entries in the enriched dataset.

```python
conso_es_td_df.shape
```

--------------------------------

### Get Shape of TradeNames DataFrame

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Displays the dimensions (number of rows and columns) of the DataFrame containing trade name information. This provides a quick overview of the dataset size after filtering and deduplication.

```python
rel_ht_df.shape
```

--------------------------------

### FAISS Index Initialization and Fitting

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Logs indicate successful initialization of the vocabulary and fitting of the FAISS index.

```text
2025-04-07 10:25:19,970 - INFO - Vocabulary initialized successfully. (faissEncoder.py:76)
2025-04-07 10:27:44,725 - INFO - FAISS index fitted successfully. (faissEncoder.py:154)
```

--------------------------------

### Define constants and semantic tag mapping

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

Sets up configuration constants for data paths, model types, and evaluation parameters. Includes a dictionary to map domain-specific semantic tags to standardized English terms.

```python
CORPUS = "SympTEMIST"
DATA_PATH = "/scratch/data/"

models = []
results = []
F_TYPE = "FlatIP"
MAX_LENGTH = 256
TOP_N = 3
TOP_K_VALUES = [1,5,25,50,100,200]
mapped_semtag = {
    "estructura corporal": "body structure",
    "sustancia": "substance",
    "organismo": "organism",
    "anomalía morfológica": "morphologic abnormality",
    "hallazgo": "finding"
}
```

--------------------------------

### Initialize and Use SapBERT FAISS Encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Initializes the FaissEncoder with the 'cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large' model, fits the index, and retrieves candidate entities. This snippet is used for the SapBERT baseline model evaluation.

```python
faiss_encoder = faiss_enc.FaissEncoder("cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large", F_TYPE, MAX_LENGTH, link_gaz_df)
faiss_encoder.fit_faiss()
candidates, codes, _ = faiss_encoder.get_candidates(test_df["term"].tolist(), k=200)
baseline_preds = test_df.copy()
baseline_preds["candidates"] = candidates
baseline_preds["codes"] = codes
models.append("SapBERT-XLM-R-large")
results.append(baseline_preds)
del faiss_encoder
```

--------------------------------

### Initialize and Use ClinLinker-KB-P FAISS Encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Initializes the FaissEncoder with the 'ICB-UMA/ClinLinker-KB-P' model, fits the index, and retrieves candidate entities. This snippet is used for the primary ClinLinker-KB-P model evaluation.

```python
faiss_encoder = faiss_enc.FaissEncoder("ICB-UMA/ClinLinker-KB-P", F_TYPE, MAX_LENGTH, link_gaz_df)
faiss_encoder.fit_faiss()
candidates, codes, _ = faiss_encoder.get_candidates(test_df["term"].tolist(), k=200)
clinlinker_parent_preds = test_df.copy()
clinlinker_parent_preds["candidates"] = candidates
clinlinker_parent_preds["codes"] = codes
models.append("ClinLinker-KB-P")
results.append(clinlinker_parent_preds)
del faiss_encoder
```

--------------------------------

### Define Corpus and Model Parameters

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Sets up constants for the corpus name, data path, model list, and FAISS encoder parameters. These variables configure the data loading and model initialization.

```python
CORPUS = "DisTEMIST"
DATA_PATH = "/scratch/data/"
models = []
results = []
F_TYPE = "FlatIP"
MAX_LENGTH = 256
TOP_N = 2
TOP_K_VALUES = [1,5,25,50,100,200]
```

--------------------------------

### Initialize and Fit FAISS Encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Initializes a FAISS encoder with specified parameters and fits it to the linked gazetteer data. It then retrieves candidate terms and codes for the test set.

```python
faiss_encoder = faiss_enc.FaissEncoder("ICB-UMA/ClinLinker-KB-GP", F_TYPE, MAX_LENGTH, link_gaz_df)
faiss_encoder.fit_faiss()
candidates, codes, _ = faiss_encoder.get_candidates(test_df["term"].tolist(), k=200)
clinlinker_granparent_preds = test_df.copy()
clinlinker_granparent_preds["candidates"] = candidates
clinlinker_granparent_preds["codes"] = codes
models.append("ClinLinker-KB-GP")
results.append(clinlinker_granparent_preds)
del faiss_encoder
```

--------------------------------

### Initialize and Use ClinLinker FAISS Encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Initializes the FaissEncoder with the 'ICB-UMA/ClinLinker' model, fits the index, and retrieves candidate entities. This snippet is used for the ClinLinker model evaluation.

```python
faiss_encoder = faiss_enc.FaissEncoder("ICB-UMA/ClinLinker", F_TYPE, MAX_LENGTH, link_gaz_df)
faiss_encoder.fit_faiss()
candidates, codes, _ = faiss_encoder.get_candidates(test_df["term"].tolist(), k=200)
clinlinker_preds = test_df.copy()
clinlinker_preds["candidates"] = candidates
clinlinker_preds["codes"] = codes
models.append("ClinLinker")
results.append(clinlinker_preds)
del faiss_encoder
```

--------------------------------

### Load MRHIER.RRF File

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Loads the MRHIER.RRF file and prints the number of lines. Ensure the UMLS_PATH environment variable is set correctly.

```python
with open(os.path.join(UMLS_PATH, "MRHIER.RRF"), "r") as f:
    lines = f.readlines()
print (len(lines))
```

--------------------------------

### Initialize and fit SapBERT FAISS encoder

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

Initializes a `FaissEncoder` with the 'cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large' model, fits it to the linked gazetteer data, and retrieves candidate medical codes for the test set terms.

```python
faiss_encoder = faiss_enc.FaissEncoder("cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large", F_TYPE, MAX_LENGTH, link_gaz_df)
faiss_encoder.fit_faiss()
candidates, codes, _ = faiss_encoder.get_candidates(test_df["term"].tolist(), k=200)
```

--------------------------------

### Logging output from faissEncoder and metrics modules

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

This log output shows the progress and results of FAISS index fitting and Top-k accuracy calculations, including vocabulary initialization and specific accuracy scores for different k values.

```text
2025-04-07 08:29:47,037 - INFO - Vocabulary initialized successfully. (faissEncoder.py:76)
2025-04-07 08:30:59,571 - INFO - FAISS index fitted successfully. (faissEncoder.py:154)
2025-04-07 08:31:21,732 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 08:31:21,732 - INFO - DataFrame size: 2598, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
2025-04-07 08:31:21,900 - INFO - Top-1 accuracy: 0.5855 (metrics.py:67)
2025-04-07 08:31:21,901 - INFO - Top-5 accuracy: 0.7244 (metrics.py:67)
2025-04-07 08:31:21,901 - INFO - Top-25 accuracy: 0.8114 (metrics.py:67)
2025-04-07 08:31:21,901 - INFO - Top-50 accuracy: 0.8418 (metrics.py:67)
2025-04-07 08:31:21,902 - INFO - Top-100 accuracy: 0.8622 (metrics.py:67)
2025-04-07 08:31:21,902 - INFO - Top-200 accuracy: 0.8737 (metrics.py:67)
2025-04-07 08:31:21,903 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
2025-04-07 08:31:21,903 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-07 08:31:21,903 - INFO - DataFrame size: 2507, Top-k values: [1, 5, 25, 50, 100, 200] (metrics.py:40)
2025-04-07 08:31:22,064 - INFO - Top-1 accuracy: 0.6055 (metrics.py:67)

```

--------------------------------

### Import Libraries and Define Constants

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Data_exploration.ipynb

Imports essential Python libraries and defines constants for corpora, data paths, and output paths. Ensure the 'utils' module is accessible in the system path.

```python
import os
import sys
import pandas as pd

sys.path.append(os.path.join(os.getcwd(), '../src'))
from utils import load_corpus_data
```

```python
CORPORA = ["DisTEMIST", "MedProcNER", "SympTEMIST"]
DATA_PATH = "/scratch/data/"
OUTPUT_PATH = "../data/"
```

--------------------------------

### Load and preprocess corpus data

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

Loads training, testing, and gazetteer data using the `load_corpus_data` utility. It then maps semantic tags, concatenates training and gazetteer data for linking, and filters out entries with 'NO_CODE' or composite codes.

```python
test_df, train_df, gaz_df = load_corpus_data(DATA_PATH, CORPUS)
gaz_df['semantic_tag'] = gaz_df['semantic_tag'].replace(mapped_semtag)
link_gaz_df = pd.concat([train_df[['code', 'term']], gaz_df[['code', 'term']]], ignore_index=True) 
no_code_count = test_df['code'].str.contains('NO_CODE').sum()
composite_count = test_df['code'].str.contains(r'\+').sum()
print(f"Number of NO_CODE: {no_code_count}")
print(f"Number of composite(+): {composite_count}")
test_df = test_df[~test_df['code'].str.contains('NO_CODE|\+', regex=True)]
```

--------------------------------

### Configure UMLS Paths and Limits

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Sets the UMLS version and base path, and defines maximum limits for parent and grandparent concepts to be considered during data processing.

```python
UMLS_VERSION = "2023AA"
UMLS_PATH = f"/scratch/data/UMLS/{UMLS_VERSION}/META/"
MAX_NOPARENTS = 15
MAX_PARENTS = 15
MAX_GRAND_PARENTS = 45
```

--------------------------------

### FAISS Encoder Logging

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

This snippet shows log messages indicating the successful initialization and fitting of a FAISS index. These logs are useful for monitoring the FAISS encoder's operational status.

```log
2025-04-26 14:30:41,890 - INFO - Vocabulary initialized successfully. (faissEncoder.py:76)
2025-04-26 14:35:36,317 - INFO - FAISS index fitted successfully. (faissEncoder.py:154)
```

--------------------------------

### Import Libraries and Modules

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Imports necessary Python libraries and custom modules for data manipulation, machine learning, and NLP tasks. Ensure these modules are available in the Python path.

```python
import sys
import os
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from itertools import combinations
import textwrap

sys.path.append(os.path.join(os.getcwd(), '../src'))
from utils import load_corpus_data
from metrics import calculate_topk_accuracy
import faissEncoder as faiss_enc
```

--------------------------------

### Top-k Accuracy Results (Dataset 3)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Presents the top-k accuracy metrics for a dataset containing 1730 items.

```text
2025-04-07 10:23:50,154 - INFO - Top-1 accuracy: 0.4173 (metrics.py:67)
2025-04-07 10:23:50,155 - INFO - Top-5 accuracy: 0.6665 (metrics.py:67)
2025-04-07 10:23:50,155 - INFO - Top-25 accuracy: 0.8098 (metrics.py:67)
2025-04-07 10:23:50,155 - INFO - Top-50 accuracy: 0.8439 (metrics.py:67)
2025-04-07 10:23:50,156 - INFO - Top-100 accuracy: 0.8642 (metrics.py:67)
2025-04-07 10:23:50,156 - INFO - Top-200 accuracy: 0.8711 (metrics.py:67)
2025-04-07 10:23:50,156 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
```

--------------------------------

### Top-k Accuracy Results (Dataset 4)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Presents the top-k accuracy metrics for the smallest dataset evaluated, containing 878 items.

```text
2025-04-07 10:25:19,695 - INFO - Top-1 accuracy: 0.3633 (metrics.py:67)
2025-04-07 10:25:19,695 - INFO - Top-5 accuracy: 0.5991 (metrics.py:67)
2025-04-07 10:25:19,696 - INFO - Top-25 accuracy: 0.7620 (metrics.py:67)
2025-04-07 10:25:19,696 - INFO - Top-50 accuracy: 0.7973 (metrics.py:67)
2025-04-07 10:25:19,696 - INFO - Top-100 accuracy: 0.8121 (metrics.py:67)
2025-04-07 10:25:19,697 - INFO - Top-200 accuracy: 0.8166 (metrics.py:67)
2025-04-07 10:25:19,697 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
```

--------------------------------

### Prepare and Integrate TradeNames into Concept Data

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Extracts trade name CUIs, filters the English concept DataFrame, and then constructs a new DataFrame ('tradenames_es') with the structure of the Spanish concept DataFrame. It maps trade names and their attributes, then concatenates this with the original Spanish concept data.

```python
tradenames_CUIs = rel_ht_df.AUI1.to_list()
conso_tradenames = conso_en_df[conso_en_df.AUI.isin(tradenames_CUIs)]


# 3. Generamos dataframe con la estructura de conso_es
# Crea el nuevo dataframe transformado
tradenames_es = pd.DataFrame()
tradenames_es['CUI'] = rel_ht_df['CUI2']
tradenames_es['STR'] = ''
tradenames_es['LAT'] = 'ENG'
tradenames_es['TS'] = 'S'
tradenames_es['STT'] = ''
tradenames_es['ISPREF'] = ''
tradenames_es['AUI'] = rel_ht_df['AUI1']
# 4. Completamos los campos que quedan con conso_tradenames. 
# Filtrar filas con valor único de CUI y STR
unique_tradenames_rows = conso_tradenames.groupby(['AUI', 'STR']).filter(lambda x: len(x) == 1)
# Generar el diccionario con clave CUI y valor una lista de STR
trade_aui_str = unique_tradenames_rows.groupby('AUI')['STR'].apply(list).to_dict()
trade_aui_stt = unique_tradenames_rows.groupby('AUI')['STT'].apply(list).to_dict()
trade_aui_ispref = unique_tradenames_rows.groupby('AUI')['ISPREF'].apply(list).to_dict()
# Mapeamos el STR con el CUI
tradenames_es["STR"] = tradenames_es.AUI.map(trade_aui_str)
tradenames_es["STT"] = tradenames_es.AUI.map(trade_aui_stt)
tradenames_es["ISPREF"] = tradenames_es.AUI.map(trade_aui_ispref)
tradenames_es["STR"] = tradenames_es.STR.explode()
tradenames_es["STT"] = tradenames_es.STT.explode()
tradenames_es["ISPREF"] = tradenames_es.ISPREF.explode()
# Ahora combinamos conso_es con los fármacos
conso_es_td_df = pd.concat([conso_es_df,tradenames_es]).reset_index(drop=True).copy()
```

--------------------------------

### Train Cross-Encoder Model

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/README.md

Execute the cross-encoder training script. This script is located within the 'scripts' directory.

```bash
cd scripts/
python cross_encoder_training.py
```

--------------------------------

### Import Libraries for UMLS Processing

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Imports necessary Python libraries including pandas, tqdm, and custom utilities for handling UMLS data.

```python
import os, sys
import pandas as pd
from tqdm.auto import tqdm
import gc
import random
import itertools

sys.path.append(os.path.join(os.getcwd(), '../src'))
from utils import extract_column_names_from_ctl_file
```

--------------------------------

### Structure Concepts into DataFrame

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Organizes the cleaned UMLS concept data into a pandas DataFrame, extracts column names from a CTL file, removes duplicate entries, and displays the first few rows.

```python
colnames = extract_column_names_from_ctl_file(os.path.join(UMLS_PATH, "MRCONSO.ctl"))
conso_es_en_df = pd.DataFrame(cleaned, columns=colnames)[["CUI","STR","LAT","TS","STT","ISPREF","AUI"]]
conso_es_en_df.drop_duplicates(inplace=True)
conso_es_en_df.head()
```

--------------------------------

### Import necessary libraries

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

Imports essential Python libraries for data manipulation, natural language processing, and machine learning. Includes custom modules for data loading and metric calculation.

```python
import sys
import os
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

sys.path.append(os.path.join(os.getcwd(), '../src'))
from utils import load_corpus_data
from metrics import calculate_topk_accuracy
import faissEncoder as faiss_enc
```

--------------------------------

### Load and Preprocess Corpus Data

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Loads corpus data using the 'load_corpus_data' function and preprocesses the test set. It concatenates training and gazetteer data for linking, counts 'NO_CODE' and composite entries, and filters them out.

```python
test_df, train_df, gaz_df = load_corpus_data(DATA_PATH, CORPUS)
link_gaz_df = pd.concat([train_df[['code', 'term']], gaz_df[['code', 'term']]], ignore_index=True)
no_code_count = test_df['code'].str.contains('NO_CODE').sum()
composite_count = test_df['code'].str.contains(r'\+').sum()
print(f"Number of NO_CODE: {no_code_count}")
print(f"Number of composite(+): {composite_count}")
test_df = test_df[~test_df['code'].str.contains('NO_CODE|\+', regex=True)]
```

--------------------------------

### Define corpus evaluation constants and model mapping

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Sets up constants for corpora, a mapping for model names, FAISS index type, maximum sequence length, data paths, and top-k values for accuracy calculation.

```python
CORPORA = ["DisTEMIST", "MedProcNER", "SympTEMIST"]
model_map = {
    "SapBERT-UMLS-2020AB-all-lang-from-XLMR-large" : "SapBERT-XLM-R-large",
    "SapBERT-UMLS-2020AB-all-lang-from-XLMR" : "SapBERT-XLM-R-base",
    "sapbert_15_noparents_1epoch" : "ClinLinker",
    "sapbert_15_parents_1epoch" : "ClinLinker-KB-P",
    "sapbert_15_grandparents_1epoch" : "ClinLinker-KB-GP",
    "distemist-biencoder" : "DisTEMIST-bi-encoder",
    "medprocner-biencoder" : "MedProcNER-bi-encoder",
    "symptemist-biencoder" : "SympTEMIST-bi-encoder",
    "roberta-base-biomedical-clinical-es": "RoBERTa-base-biomedical"
}

F_TYPE = "FlatIP"
MAX_LENGTH = 256

DATA_PATH = "/scratch/data/"
TOP_K_VALUES = [1, 5, 25, 50, 100, 200]


```

--------------------------------

### Tqdm progress bar warning

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

This is a warning message from the tqdm library indicating that IProgress is not found. It suggests updating jupyter and ipywidgets for better integration with notebooks.

```text
/home/fernandogd/Documents/Investigacion/Transformers/Repositories/ClinLinker-KB/venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm


```

--------------------------------

### Display head of gazetteer DataFrame

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

Shows the first few rows of the gazetteer DataFrame to provide a glimpse of the data structure, including columns for code, language, term, semantic tag, and main term.

```python
gaz_df.head()
```

--------------------------------

### Import necessary libraries for corpus evaluation

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Imports essential libraries such as pandas, os, sys, and custom modules for encoding, cross-encoding, and evaluation metrics. Ensure these modules are available in your Python path.

```python
import os, sys
import pandas as pd
import json

sys.path.append(os.path.join(os.getcwd(), '../src'))
import faissEncoder as faiss_enc
from crossEncoder import CrossEncoderReranker
from metrics import calculate_topk_accuracy
from utils import load_corpus_data, evaluate_model

```

--------------------------------

### Create CUI to STR Dictionary

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Creates a dictionary mapping CUIs (Concept Unique Identifiers) to their corresponding strings (STR). This is useful for quick lookups of concept names.

```python
CUI_STR = dict(zip(grouped_conso_es_td_df.CUI.to_list(), grouped_conso_es_td_df.STR.to_list()))
```

--------------------------------

### Load and Filter UMLS Concepts

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Reads the MRCONSO.RRF file from UMLS, filters concepts to include only English (ENG) and Spanish (SPA) languages, and prints the total number of lines before and after filtering.

```python
with open(os.path.join(UMLS_PATH, "MRCONSO.RRF"), "r") as f:
    lines = f.readlines()
print (len(lines))

cleaned = []
for l in tqdm(lines):
    lst = l.rstrip("\n").split("|")
    cui, lang, synonym = lst[0], lst[1], lst[14]
    if lang not in ["ENG", "SPA"]: continue 
    cleaned.append(lst[0:-1])
print (len(cleaned))
```

--------------------------------

### Display Head of Consolidated DataFrame

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Displays the first few rows of the grouped_conso_es_td_df DataFrame after consolidation, showing the aggregated data.

```python
grouped_conso_es_td_df.head()
```

--------------------------------

### Select and Display Final Columns

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Selects and displays the 'CUI', 'STR', 'PCUI', and 'GPCUI' columns from the grouped_conso_es_td_df DataFrame. The head(25) shows the first 25 rows of the final processed data.

```python
grouped_conso_es_td_df = grouped_conso_es_td_df[["CUI","STR","PCUI","GPCUI"]]
grouped_conso_es_td_df.head(25)
```

--------------------------------

### Plot Comparison Heatmap

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Generates and saves a heatmap visualization of the model comparison matrix. Labels are wrapped for better readability on the plot.

```python
def plot_heatmap(
    comparisons_df: 'pd.DataFrame',
    corpus: str
) -> None:
    """
    This function generates and saves a heatmap based on the given DataFrame.
    
    Parameters:
    comparisons_df (pd.DataFrame): DataFrame with the data for the heatmap.
    corpus (str): Name of the corpus for naming the saved file.
    """
    plt.figure(figsize=(10, 8))

    ax = sns.heatmap(
        comparisons_df,
        annot=True,
        fmt='d',
        cmap='coolwarm',
        cbar=False,
        annot_kws={'size': 16}
    )

    ax.set_title(
        'Cleaned gold standard comparison',
        fontsize=20,
        fontweight='bold'
    )

    ax.set_xlabel('')
    ax.set_ylabel('')

    # Envolver etiquetas largas en X
    wrapped_x = ['\n'.join(textwrap.wrap(label, 10)) for label in comparisons_df.columns]
    ax.set_xticklabels(wrapped_x, rotation=0, ha='center', fontsize=14)

    # Envolver etiquetas largas en Y
    wrapped_y = ['\n'.join(textwrap.wrap(label, 10)) for label in comparisons_df.index]
    ax.set_yticklabels(wrapped_y, rotation=0, fontsize=14)

    plt.tight_layout()
    plt.savefig(
        f'../figures/{corpus}_cleaned_gold_standard_comparison.pdf',
        bbox_inches='tight'
    )
    plt.show()
```

--------------------------------

### Reranking Progress Indicator (Dataset 4)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Shows the progress of reranking for the smallest dataset (878 items), indicating completion and iteration speed.

```text
Reranking candidates: 100%|██████████| 878/878 [01:29<00:00, 9.81it/s]
```

--------------------------------

### Write Positive Pairs to Training File

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

This snippet writes the collected positive pairs to a text file. The filename includes the UMLS version and the maximum number of grandparent pairs used.

```python
with open(f'../data/triplets/training_file_umls{UMLS_VERSION.lower()}_es_uncased_no_dup_pairwise_pair_th{MAX_GRAND_PARENTS}_grandparents.txt', 'w') as f:
    for line in pos_pairs_gpcui:
        f.write("%s\n" % line)
```

--------------------------------

### Top-k Accuracy Results (Dataset 2)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Details the top-k accuracy metrics calculated for a dataset of 3512 items.

```text
2025-04-07 10:21:07,352 - INFO - Top-1 accuracy: 0.6777 (metrics.py:67)
2025-04-07 10:21:07,352 - INFO - Top-5 accuracy: 0.8274 (metrics.py:67)
2025-04-07 10:21:07,353 - INFO - Top-25 accuracy: 0.9006 (metrics.py:67)
2025-04-07 10:21:07,353 - INFO - Top-50 accuracy: 0.9188 (metrics.py:67)
2025-04-07 10:21:07,354 - INFO - Top-100 accuracy: 0.9288 (metrics.py:67)
2025-04-07 10:21:07,354 - INFO - Top-200 accuracy: 0.9325 (metrics.py:67)
2025-04-07 10:21:07,354 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
```

--------------------------------

### Reranking Progress Indicator (Dataset 3)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Shows reranking progress for a dataset of 1730 items, indicating completion and iteration speed.

```text
Reranking candidates: 100%|██████████| 1730/1730 [02:42<00:00, 10.63it/s]
```

--------------------------------

### Reranking Progress Indicator (Dataset 2)

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Shows the progress of reranking for a different dataset size, indicating completion and iteration speed.

```text
Reranking candidates: 100%|██████████| 3512/3512 [04:18<00:00, 13.56it/s]
```

--------------------------------

### Top-k Accuracy Calculation Logging

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

These log messages detail the process and results of calculating top-k accuracy for different models and tags. They indicate the DataFrame size, top-k values used, and the final accuracy percentage.

```log
2025-04-26 14:36:36,302 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-26 14:36:36,302 - INFO - DataFrame size: 2068, Top-k values: [25] (metrics.py:40)
2025-04-26 14:36:36,412 - INFO - Top-25 accuracy: 0.9483 (metrics.py:67)
2025-04-26 14:36:36,412 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
2025-04-26 14:36:36,414 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-26 14:36:36,414 - INFO - DataFrame size: 568, Top-k values: [25] (metrics.py:40)
2025-04-26 14:36:36,448 - INFO - Top-25 accuracy: 0.8310 (metrics.py:67)
2025-04-26 14:36:36,448 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
2025-04-26 14:36:36,449 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-26 14:36:36,449 - INFO - DataFrame size: 68, Top-k values: [25] (metrics.py:40)
2025-04-26 14:36:36,453 - INFO - Top-25 accuracy: 0.8971 (metrics.py:67)
2025-04-26 14:36:36,454 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
2025-04-26 14:36:36,455 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-26 14:36:36,455 - INFO - DataFrame size: 37, Top-k values: [25] (metrics.py:40)
2025-04-26 14:36:36,458 - INFO - Top-25 accuracy: 0.4865 (metrics.py:67)
2025-04-26 14:36:36,458 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
2025-04-26 14:36:36,460 - INFO - Calculating Top-k accuracy... (metrics.py:39)
2025-04-26 14:36:36,460 - INFO - DataFrame size: 2068, Top-k values: [25] (metrics.py:40)
2025-04-26 14:36:36,569 - INFO - Top-25 accuracy: 0.9497 (metrics.py:67)
2025-04-26 14:36:36,569 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
```

--------------------------------

### Display DataFrame Shapes

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Prints the dimensions (number of rows and columns) of the English and Spanish concept DataFrames.

```python
conso_en_df.shape, conso_es_df.shape
```

--------------------------------

### Reranking Candidates Progress

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Displays the progress of the reranking process. This snippet is useful for monitoring long-running reranking tasks.

```python
Reranking candidates: 100%|██████████| 2848/2848 [03:36<00:00, 13.14it/s]

```

--------------------------------

### Compare Model Predictions

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Comparative_performance_EA.ipynb

Compares the prediction results from different models and displays the head of the resulting comparison DataFrame. This is used to quantify the overlap and differences between model predictions.

```python
comparisons_df = compare_predictions(results, models)
comparisons_df.head()
```

--------------------------------

### Create Dictionaries for Hierarchical Relations

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/triplets_definition.ipynb

Iterates through the grouped_hier_df to create two dictionaries: cui_pcui_dict mapping CUIs to their PCUI lists, and cui_ab_cui_dict mapping CUIs to their AB_CUI lists. These dictionaries store the extracted hierarchical relations.

```python
cui_pcui_dict = {}
cui_ab_cui_dict = {}

for index, row in grouped_hier_df.iterrows():
    cui = index
    pcui_list = row['PCUI']
    ab_cui_list = row['AB_CUI']

    cui_pcui_dict[cui] = pcui_list

    cui_ab_cui_dict[cui] = ab_cui_list
```

--------------------------------

### Semantic Tagging and Model Evaluation

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Semantic_tag_EA.ipynb

This code snippet performs semantic tagging on test data, calculates top tags, and evaluates model accuracy across different tags. It prepares data for visualization.

```python
code_sem_tag_dict = gaz_df.set_index('code')['semantic_tag'].to_dict()
test_df['sem_tag'] = test_df['code'].apply(lambda code: code_sem_tag_dict.get(code, "CODE_NOT_IN_DICT"))
test_sem_tag_counts = test_df['sem_tag'].value_counts().to_dict()
top_tags = [k for k, v in sorted(test_sem_tag_counts.items(), key=lambda item: item[1], reverse=True)[:TOP_N]]

for df in results:
    df['tag'] = df['code'].map(code_sem_tag_dict).fillna('others')
    df['tag'] = df['tag'].apply(lambda x: x if x in top_tags else 'others')

results_dict = {}
topk_values = [25]
for i, model in enumerate(models):
    df = results[i]
    for tag in top_tags + ['others']:  
        filtered_df = df[df['tag'] == tag]
        accuracy_dict = calculate_topk_accuracy(filtered_df, topk_values)
        accuracy = accuracy_dict[25]
        if model not in results_dict:
            results_dict[model] = {}
        results_dict[model][tag] = accuracy

data = []
for model, tags in results_dict.items():
    for tag, accuracy in tags.items():
        data.append([model, tag, accuracy])

results_df = pd.DataFrame(data, columns=['Model', 'Tag', 'Accuracy'])
```

--------------------------------

### Evaluate models across multiple corpora and save results

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Iterates through defined corpora, loads data, prepares training and ground truth dataframes, and evaluates a list of models using the `evaluate_model` function. Results are saved to TSV files.

```python
for corpus in CORPORA:
    gs_results, clean_results, um_results, uc_results = dict(), dict(), dict(), dict()
    um_df = pd.read_csv(f"../data/{corpus}/df_um.tsv", sep="\t", dtype={"code":str})
    uc_df = pd.read_csv(f"../data/{corpus}/df_uc.tsv", sep="\t", dtype={"code":str})
    gs_df, train_df, gaz_df = load_corpus_data(DATA_PATH, corpus)
    train_gaz_df = pd.concat([train_df[["term", "code"]], gaz_df[["term","code"]]], ignore_index=True)
    clean_df = gs_df[
        gs_df['code'].notna() & 
        (gs_df['code'] != "NO_CODE") & 
        (~gs_df['code'].str.contains("\+", na=False))
    ]

    model_list = [
        "/scratch/models/NEL/spanish_sapbert_models/sapbert_15_grandparents_1epoch",
        "/scratch/models/NEL/spanish_sapbert_models/sapbert_15_parents_1epoch",
        "/scratch/models/NEL/spanish_sapbert_models/sapbert_15_noparents_1epoch",
        f"/scratch/models/NEL/corpus-specific_bi-encoders/{corpus.lower()}-biencoder",
        "cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR-large",
        "cambridgeltl/SapBERT-UMLS-2020AB-all-lang-from-XLMR",
        "PlanTL-GOB-ES/roberta-base-biomedical-clinical-es"
    ]
    for model in model_list:
        gs_results, clean_results, um_results, uc_results=evaluate_model(
            model_name=model,
            f_type=F_TYPE,
            max_length=MAX_LENGTH,
            train_gaz_df=train_gaz_df,
            gs_df=gs_df,
            clean_df=clean_df,
            um_df=um_df,
            uc_df=uc_df,
            gs_results=gs_results,
            clean_results=clean_results,
            um_results=um_results,
            uc_results=uc_results,
            model_map=model_map,
            top_k_values=TOP_K_VALUES,
            corpus=corpus
        )
    pd.DataFrame.from_dict(gs_results, orient='index').reset_index().rename(columns={'index': 'name'}).to_csv(f"../output/{corpus}/gs_results.tsv", sep='\t', index=False)
    pd.DataFrame.from_dict(clean_results, orient='index').reset_index().rename(columns={'index': 'name'}).to_csv(f"../output/{corpus}/clean_results.tsv", sep='\t', index=False)
    pd.DataFrame.from_dict(um_results, orient='index').reset_index().rename(columns={'index': 'name'}).to_csv(f"../output/{corpus}/um_results.tsv", sep='\t', index=False)
    pd.DataFrame.from_dict(uc_results, orient='index').reset_index().rename(columns={'index': 'name'}).to_csv(f"../output/{corpus}/uc_results.tsv", sep='\t', index=False)



```

--------------------------------

### Corpus Data Loading and Filtering

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Data_exploration.ipynb

Iterates through specified corpora, loads test, train, and gazetteer data, filters for relevant entries (excluding composite codes and 'NO_CODE'), and saves processed data. This snippet is crucial for preparing datasets for NER model training and evaluation.

```python
for corpus in CORPORA:
    test_df, train_df, gaz_df = load_corpus_data(DATA_PATH, corpus)
    train_gaz_df = pd.concat([train_df[["term", "code"]], gaz_df[["term","code"]]], ignore_index=True)
    clean_df = test_df[
        test_df['code'].notna() & 
        (test_df['code'] != "NO_CODE") & 
        (~test_df['code'].str.contains("\+", na=False))
    ]
    train_gaz_df.drop_duplicates(inplace=True)
    aux_path = os.path.join(OUTPUT_PATH, corpus)
    os.makedirs(aux_path, exist_ok=True)
    
    df_um = test_df[~test_df['term'].isin(train_df['term']) & ~test_df['term'].isin(gaz_df['term'])]
    df_uc = test_df[~test_df['code'].isin(train_df['code'])]
    df_um_filtered = df_um[~df_um['code'].str.contains(r'\+ |NO_CODE', na=False)]
    df_uc_filtered = df_uc[~df_uc['code'].str.contains(r'\+|NO_CODE', na=False)]
    
    df_um_filtered.to_csv(os.path.join(aux_path, "df_um.tsv"), sep="\t", index=False)
    df_uc_filtered.to_csv(os.path.join(aux_path, "df_uc.tsv"), sep="\t", index=False)

    print(f"CORPUS: {corpus}")
    print(f"Train + Gaz: {train_gaz_df.shape[0]}")
    print(f"Gold standard:  {test_df.shape[0]}")
    print(f"Cleaned: {clean_df.shape[0]}")
    print(f"Unseen mentions (filtered): {df_um_filtered.shape[0]}")
    print(f"Unseen codes (filtered): {df_uc_filtered.shape[0]}")
    print("="*50)
```

--------------------------------

### Top-k Accuracy Metrics

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Reports the Top-k accuracy for a given dataset. This snippet is crucial for evaluating the performance of a retrieval model at different recall levels.

```python
2025-04-07 11:23:30,117 - INFO - Top-1 accuracy: 0.6254 (metrics.py:67)
2025-04-07 11:23:30,117 - INFO - Top-5 accuracy: 0.7753 (metrics.py:67)

```

--------------------------------

### Top-k Accuracy Results

Source: https://github.com/icb-uma/clinlinker-kb/blob/master/notebooks/Corpus_evaluation.ipynb

Presents the calculated top-k accuracy values for different k, ranging from 1 to 200.

```text
2025-04-07 10:16:48,207 - INFO - Top-1 accuracy: 0.6622 (metrics.py:67)
2025-04-07 10:16:48,207 - INFO - Top-5 accuracy: 0.8076 (metrics.py:67)
2025-04-07 10:16:48,208 - INFO - Top-25 accuracy: 0.8789 (metrics.py:67)
2025-04-07 10:16:48,208 - INFO - Top-50 accuracy: 0.8969 (metrics.py:67)
2025-04-07 10:16:48,208 - INFO - Top-100 accuracy: 0.9069 (metrics.py:67)
2025-04-07 10:16:48,209 - INFO - Top-200 accuracy: 0.9104 (metrics.py:67)
2025-04-07 10:16:48,209 - INFO - Top-k accuracy calculation complete. (metrics.py:69)
```