### Install pytorch-pretrained-bert

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Install the necessary library for using ClinicalBERT.

```bash
pip install pytorch-pretrained-bert
```

--------------------------------

### Continue Pretraining (512 Sequence Length)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Continues the BERT pretraining process using TFRecords with a sequence length of 512, starting from a checkpoint generated at 128 sequence length. This step trains for another 100,000 steps.

```bash
python run_pretraining.py \
  --input_file=tf_examples_512.tfrecord \
  --output_dir=./pretraining_output_512 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json \
  --init_checkpoint=./pretraining_output_128/model.ckpt-100000 \
  --train_batch_size=16 \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5
```

--------------------------------

### Get Unactionable Admits for Test

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

This comment indicates a section for retrieving unactionable admissions, likely for testing purposes. No code is provided.

```python
## get unactionable admits for Test
```

--------------------------------

### Get Actionable IDs for 2 Days Stay

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters the 'df_adm1' DataFrame to get 'HADM_ID' for records where 'STAY_DAYS' is greater than or equal to 2. This identifies actionable patient IDs for the 2-day category.

```python
actionable_ID_2days = df_adm1[df_adm1['STAY_DAYS'] >= 2].HADM_ID
```

--------------------------------

### Create Training ID Folds

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Generates the training ID sets for each of the 4 folds. It combines the initial 'not_readmit_ID_use' and 'readmit_ID' lists with the test and validation IDs for each fold, then removes duplicates to get the final training IDs.

```python
train_id_folds={}
for i in range(4):
    all_index = pd.concat([not_readmit_ID_use, readmit_ID, pd.Series(test_id_folds[i]), pd.Series(val_id_folds[i])])
    train_id_folds[i] = all_index.drop_duplicates(keep=False).values
```

--------------------------------

### Get Actionable IDs for 3 Days Stay

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters the 'df_adm1' DataFrame to get 'HADM_ID' for records where 'STAY_DAYS' is greater than or equal to 3. This identifies actionable patient IDs for the 3-day category.

```python
actionable_ID_3days = df_adm1[df_adm1['STAY_DAYS'] >= 3].HADM_ID
```

--------------------------------

### Get Length of Except Fold 1 Readmitted IDs

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates and prints the number of IDs in 'except_fold1_f', which represents readmitted patients excluding fold 1.

```python
len(except_fold1_f)
```

--------------------------------

### Download Model Checkpoints

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Reference links for downloading XLNet and BERT implementations and initial checkpoints.

```bash
# STEP 6: Download the Implementations and Initial Checkpoint with the SentencePiece Model and Vocab 
# Github Repo:
# XLNet: git clone https://github.com/zihangdai/xlnet.git
# BERT: git clone https://github.com/google-research/bert.git
# Model: 
# XLNet-Base: https://storage.googleapis.com/xlnet/released_models/cased_L-12_H-768_A-12.zip
# BERT-Base-Uncased: https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
```

--------------------------------

### Initial BERT Pre-training (128 Sequence Length)

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Use this command to perform initial pre-training for BERT with a maximum sequence length of 128. Ensure the input file, output directory, and configuration files are correctly specified.

```bash
!python run_pretraining.py \
  --input_file=PRETRAIN_DATA_PATH/tf_examples_128.tfrecord \
  --output_dir=PRETRAINED_MODEL_PATH/pretraining_output \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=INITIAL_DATA_PATH/bert_config.json \
  --init_checkpoint=INITIAL_DATA_PATH/bert_model.ckpt \
  --train_batch_size=64 \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5
```

--------------------------------

### Run Pretraining (128 Sequence Length)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Initiates the BERT pretraining process using the generated TFRecords with a sequence length of 128 for 100,000 steps. Requires BERT configuration and an initial checkpoint.

```bash
python run_pretraining.py \
  --input_file=tf_examples_128.tfrecord \
  --output_dir=./pretraining_output_128 \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=./uncased_L-12_H-768_A-12/bert_config.json \
  --init_checkpoint=./uncased_L-12_H-768_A-12/bert_model.ckpt \
  --train_batch_size=64 \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5
```

--------------------------------

### Get Readmission Prediction

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Perform inference using a trained model to calculate the probability of readmission.

```python
with torch.no_grad():
    logits = model(input_ids, segment_ids, input_mask)
    prob = torch.sigmoid(logits).item()

print(f"Readmission probability: {prob:.4f}")
```

--------------------------------

### Initialize ClinicalBERT Environment

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Imports necessary libraries and initializes the BERT tokenizer for the ClinicalBERT model.

```python
import torch
from torch import nn
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
sns.set_style('white')
sns.set_context('talk', font_scale = 1)
import os

import math
from pytorch_pretrained_bert import BertTokenizer
os.chdir('../')
from modeling_readmission import BertForSequenceClassification
import modeling_readmission

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```

--------------------------------

### Load Model Configuration and Weights

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Loads the model configuration from a JSON file and initializes the sequence classification model with pre-trained weights.

```python
bert_config = modeling_readmission.BertConfig.from_json_file('../model2/bert_config.json')
model = BertForSequenceClassification(bert_config)
```

```python
dicts=model.load_state_dict(torch.load('../model2/pytorch_model.bin',map_location='cpu'))
```

--------------------------------

### Get Length of Test IDs in Fold 2

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Retrieves and prints the number of test IDs for the second fold (index 2) from the 'not_re' dictionary, which contains non-readmitted IDs for each fold.

```python
len(not_re[2])
```

--------------------------------

### Download and Cache Model Files

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Provides utility functions for downloading and caching pretrained model files from URLs. Demonstrates using `cached_path` with default and custom cache directories, and setting the cache location via an environment variable.

```python
from file_utils import cached_path, url_to_filename, get_from_cache

# Download and cache a model from URL
model_url = "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz"
local_path = cached_path(model_url)
print(f"Model cached at: {local_path}")

# Custom cache directory
custom_cache = "/path/to/model/cache"
local_path = cached_path(model_url, cache_dir=custom_cache)

# The cache stores metadata for version tracking
# Files are stored with hashed filenames based on URL and ETag
filename = url_to_filename(model_url, etag="abc123")
print(f"Cache filename: {filename}")

# Environment variable for cache location
import os
os.environ['PYTORCH_PRETRAINED_BERT_CACHE'] = '/custom/cache/path'
```

--------------------------------

### Further BERT Pre-training (512 Sequence Length)

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

This command continues BERT pre-training for another 100,000 steps, increasing the maximum sequence length to 512. The init_checkpoint should point to the previously pre-trained model.

```bash
!python run_pretraining.py \
  --input_file=PRETRAIN_DATA_PATH/tf_examples_512.tfrecord \
  --output_dir=PRETRAINED_MODEL_PATH/pretraining_output \
  --do_train=True \
  --do_eval=True \
  --bert_config_file=INITIAL_DATA_PATH/bert_config.json \
  --init_checkpoint=PRETRAINED_MODEL_PATH/pretraining_output_128/model.ckpt-100000 \
  --train_batch_size=16 \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --num_train_steps=100000 \
  --num_warmup_steps=10 \
  --learning_rate=2e-5
```

--------------------------------

### Get Length of Except Fold 1 IDs

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates and prints the number of unique IDs in the 'except_fold1' DataFrame. This indicates the total number of IDs available after excluding fold 1.

```python
len(except_fold1)
```

--------------------------------

### Clinical XLNet Pre-training

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Command for pre-training Clinical XLNet for 200,000 steps. This includes specific parameters for sequence length, memory length, and attention mechanisms.

```bash
!python train_gpu.py \
    --record_info_dir=PRETRAIN_DATA_PATH/xlnet_tfrecord/tfrecords/ \
    --model_dir=PRETRAINED_MODEL_PATH/xlnet_model/ \
    --init_checkpoint=INITIAL_DATA_PATH/xlnet_cased_L-12_H-768_A-12/xlnet_model.ckpt \
    --train_batch_size=8 \
    --seq_len=512 \
    --reuse_len=256 \
    --mem_len=384 \
    --perm_size=256 \
    --n_layer=12 \
    --d_model=768 \
    --d_embed=768 \
    --n_head=12 \
    --d_head=64 \
    --d_inner=3072 \
    --untie_r=True \
    --mask_alpha=6 \
    --mask_beta=1 \
    --num_predict=85 \
    --num_hosts=1 \
    --num_core_per_host=2 \
    --train_steps=200000 \
    --iterations=500 \
    --save_steps=5000 
```

--------------------------------

### Import Dependencies

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Required libraries for data manipulation, regex processing, and progress tracking.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import spacy
import re
from tqdm import tqdm
import string
```

--------------------------------

### File System Structure for Models

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Expected directory structure for storing pretrained and fine-tuned ClinicalBERT model weights and configurations.

```text
-model
	-discharge_readmission
		-bert_config.json
		-pytorch_model.bin
	-early_readmission
		-bert_config.json
		-pytorch_model.bin
	-pretraining
		-bert_config.json
		-pytorch_model.bin
		-vocab.txt
```

--------------------------------

### Create Training DataFrame with Labels

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Combines training IDs with their corresponding readmission labels (1 for readmitted, 0 for not readmitted) and creates a DataFrame.

```python
id_train = pd.concat([id_train_t, id_train_f])
train_id_label = pd.DataFrame(data = list(zip(id_train, [1]*len(id_train_t)+[0]*len(id_train_f))), columns = ['id','label'])
```

--------------------------------

### Import Data Libraries

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Standard imports for data manipulation and visualization.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
```

--------------------------------

### File System Structure for Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Expected directory structure for input data files, including train, validation, and test sets for different time windows.

```text
-data
  -discharge
    -train.csv
    -val.csv
    -test.csv
  -3days
    -train.csv
    -val.csv
    -test.csv
  -2days
    -test.csv
```

--------------------------------

### Export Pretraining Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Write the segmented sentences to a text file for pretraining.

```python
# STEP 5: Create Pretraining File
file=open('PRETRAIN_DATA_PATH/clinical_sentences_pretrain.txt','w')
pretrain_sent = pretrain_sent.values
for i in tqdm(range(len(pretrain_sent))):
    if len(pretrain_sent[i]) > 0:
        # remove the one token note
        note = pretrain_sent[i]
        for sent in note:
            file.write(sent+'\n')
        file.write('\n')
```

--------------------------------

### Initialize and Configure KFold

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Initializes a KFold object with 4 splits, shuffling enabled, and a fixed random state for reproducibility. This object will be used for cross-validation splitting.

```python
from sklearn.model_selection import KFold
kf = KFold(n_splits = 4, shuffle = True, random_state = 1)
kf.get_n_splits(except_fold1_f)
```

--------------------------------

### Load Less Than 3 Days Notes

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Loads a CSV file containing notes for patients with less than 3 days of stay. This is the initial step for creating the training dataset.

```python
want_early = pd.read_csv('less_3_days_notes.csv')
#early notes fold 1
```

--------------------------------

### Calculate Time Intervals

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculate days until next admission and total stay duration in days.

```python
df_adm['DAYS_NEXT_ADMIT']=  (df_adm.NEXT_ADMITTIME - df_adm.DISCHTIME).dt.total_seconds()/(24*60*60)
```

```python
df_adm1['STAY_DAYS']=  (df_adm1.DISCHTIME - df_adm1.ADMITTIME).dt.total_seconds()/(24*60*60)
```

--------------------------------

### Inspect Model Configuration

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Accesses the configuration object and specific attributes like the number of attention heads.

```python
bert_config
```

```python
bert_config.num_attention_heads
```

--------------------------------

### Process and Inspect Early Discharge Data (2 Days)

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Loads early discharge notes for stays less than 2 days, filters them based on the previously identified actionable IDs, and displays the label distribution for the test set.

```python
early_test_2days = want_2days[want_2days.ID.isin(test_actionable_id_2days)]
early_test_2days.Label.value_counts()
```

--------------------------------

### Value Counts for Shuffled Training Snippets

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Shows the label distribution after shuffling and combining the training snippets.

```python
discharge_train_snippets.Label.value_counts()
```

--------------------------------

### Select and Label Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Subset the DataFrame and create a binary label for readmission within 30 days.

```python
df_adm1 = df_adm[['SUBJECT_ID','HADM_ID','ADMITTIME','DISCHTIME','DAYS_NEXT_ADMIT','NEXT_ADMITTIME','ADMISSION_TYPE','DEATHTIME']]
```

```python
df_adm1['OUTPUT_LABEL'] = (df_adm1.DAYS_NEXT_ADMIT < 30).astype('int')
```

--------------------------------

### Initialize BertForSequenceClassification Model

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Load a pretrained ClinicalBERT model with a classification head for binary readmission prediction. This involves loading configuration, model weights, and initializing a tokenizer.

```python
import torch
from pytorch_pretrained_bert import BertTokenizer
from modeling_readmission import BertForSequenceClassification, BertConfig

# Load configuration and model
config = BertConfig.from_json_file('./model/discharge_readmission/bert_config.json')
model = BertForSequenceClassification.from_pretrained(
    './model/discharge_readmission',
    num_labels=1
)
model.eval()

# Tokenize clinical text
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = "patient admitted with acute respiratory failure requiring intubation"
tokens = tokenizer.tokenize(text)
input_ids = tokenizer.convert_tokens_to_ids(['[CLS]'] + tokens + ['[SEP]'])

# Prepare tensors
input_ids = torch.tensor([input_ids])
segment_ids = torch.zeros_like(input_ids)
input_mask = torch.ones_like(input_ids)

```

--------------------------------

### Load Early Discharge Datasets

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Loads datasets related to early discharge prediction, specifically for notes less than 3 days and less than 2 days.

```python
want_early = pd.read_csv('less_3_days_notes.csv')
```

```python
want_2days = pd.read_csv('less_2_days_notes.csv')
actionable_ID_3days = df_adm1[df_adm1['STAY_DAYS'] >= 3].HADM_ID
actionable_ID_2days = df_adm1[df_adm1['STAY_DAYS'] >= 2].HADM_ID
```

--------------------------------

### Preprocess MIMIC-III Data

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Calculate readmission labels by sorting admissions and filtering out elective readmissions, newborns, and deaths.

```python
import pandas as pd
import re

# Load MIMIC-III data
df_adm = pd.read_csv('ADMISSIONS.csv')
df_notes = pd.read_csv('NOTEEVENTS.csv')

# Parse timestamps
df_adm.ADMITTIME = pd.to_datetime(df_adm.ADMITTIME)
df_adm.DISCHTIME = pd.to_datetime(df_adm.DISCHTIME)
df_adm.DEATHTIME = pd.to_datetime(df_adm.DEATHTIME)

# Calculate readmission labels
df_adm = df_adm.sort_values(['SUBJECT_ID', 'ADMITTIME'])
df_adm['NEXT_ADMITTIME'] = df_adm.groupby('SUBJECT_ID').ADMITTIME.shift(-1)
df_adm['NEXT_ADMISSION_TYPE'] = df_adm.groupby('SUBJECT_ID').ADMISSION_TYPE.shift(-1)

# Filter out elective readmissions (only count unplanned)
rows = df_adm.NEXT_ADMISSION_TYPE == 'ELECTIVE'
df_adm.loc[rows, 'NEXT_ADMITTIME'] = pd.NaT
df_adm.loc[rows, 'NEXT_ADMISSION_TYPE'] = None

# Backfill to handle gaps from filtered electives
df_adm[['NEXT_ADMITTIME', 'NEXT_ADMISSION_TYPE']] = df_adm.groupby('SUBJECT_ID')[
    ['NEXT_ADMITTIME', 'NEXT_ADMISSION_TYPE']
].fillna(method='bfill')

# Calculate days to next admission and create label
df_adm['DAYS_NEXT_ADMIT'] = (df_adm.NEXT_ADMITTIME - df_adm.DISCHTIME).dt.total_seconds() / (24*60*60)
df_adm['OUTPUT_LABEL'] = (df_adm.DAYS_NEXT_ADMIT < 30).astype('int')

# Filter out newborns and deaths
df_adm = df_adm[df_adm['ADMISSION_TYPE'] != 'NEWBORN']
df_adm = df_adm[df_adm.DEATHTIME.isnull()]
```

--------------------------------

### Create Test DataFrame with Labels

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Combines test IDs with their corresponding readmission labels and creates a DataFrame.

```python
id_test = pd.concat([id_test_t, id_test_f])
test_id_label = pd.DataFrame(data = list(zip(id_test, [1]*len(id_test_t)+[0]*len(id_test_f))), columns = ['id','label'])
```

--------------------------------

### Load Admissions Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Read the admissions CSV file into a pandas DataFrame.

```python
df_adm = pd.read_csv('/Users/KexinHuang/Downloads/ADMISSIONS.csv')
```

--------------------------------

### Create Training Set for Readmitted IDs

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Creates the training set for readmitted patients by dropping the IDs selected for validation/testing.

```python
id_train_t = readmit_ID.drop(id_val_test_t.index)
```

--------------------------------

### Execute Attention Score Extraction

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Runs the attention score extraction for a sample text input.

```python
text=' he has experienced acute on chronic diastolic heart failure in the setting of volume overload due to his sepsis.'
x,tokens=get_attention_scores(model,0,text)
```

--------------------------------

### Load Note Datasets

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Load the raw clinical notes from a CSV file.

```python
# STEP 1: load Note datasets
df_notes = pd.read_csv('NOTE_DATA_PATH/NOTEEVENTS.csv')
```

--------------------------------

### Prepare Pretraining Corpus with spaCy

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Loads clinical notes, filters them, preprocesses text by removing special characters and numbers, and converts text into sentences using spaCy's sentencizer. Excludes test set admissions to prevent data leakage. Writes sentences to a file, with blank lines separating documents.

```python
import pandas as pd
from spacy.lang.en import English
import re
import string
from tqdm import tqdm

# Load and filter notes
df_notes = pd.read_csv('NOTEEVENTS.csv')
df_notes = df_notes[df_notes.CATEGORY.isin(['Physician ', 'Nursing', 'Nursing/Others'])]

# IMPORTANT: Exclude test set admissions to prevent data leakage
df_test_ids = pd.read_csv('./data/discharge/test.csv').ID.unique()
df_notes = df_notes[~df_notes.HADM_ID.isin(df_test_ids)]

# Preprocess text
def preprocess_for_pretraining(text):
    text = re.sub('\[(.*?)\]', '', text)
    text = re.sub('[0-9]+\.', '', text)
    text = re.sub('dr\.', 'doctor', text)
    text = re.sub('m\.d\.', 'md', text)
    text = re.sub('admission date:', '', text)
    text = re.sub('discharge date:', '', text)
    text = re.sub('--|__|==', '', text)
    text = text.translate(str.maketrans("", "", string.digits))
    text = " ".join(text.split())
    return text.lower()

df_notes['TEXT'] = df_notes['TEXT'].apply(preprocess_for_pretraining)

# Convert to sentences using spaCy
nlp = English()
nlp.add_pipe(nlp.create_pipe('sentencizer'))

def to_sentences(text):
    doc = nlp(text)
    sentences = []
    for sent in doc.sents:
        s = str(sent).strip()
        if len(s) >= 20:
            sentences.append(s)
        elif sentences:
            sentences[-1] = sentences[-1] + ' ' + s
    return sentences

# Write pretraining file (one sentence per line, blank line between documents)
with open('clinical_sentences_pretrain.txt', 'w') as f:
    for text in tqdm(df_notes['TEXT']):
        sentences = to_sentences(text)
        for sent in sentences:
            f.write(sent + '\n')
        f.write('\n')
```

--------------------------------

### Sample and Filter Early Discharge Training Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Samples a subset of the early discharge data (less than 3 days) and checks its length. This is used to create a balanced training set.

```python
ear_train = want_early[want_early.ID.isin(df.sample(n=400, random_state=1))]
len(ear_train)
```

--------------------------------

### Count Training Samples

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Reports the total number of samples in the training set.

```python
len(train_id_label)
```

--------------------------------

### Create Training Set for Non-Readmitted IDs

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Creates the training set for non-readmitted patients by dropping the IDs selected for validation/testing.

```python
id_train_f = not_readmit_ID_use.drop(id_val_test_f.index)
```

--------------------------------

### Count Labels in Early Test Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates and displays the value counts for the 'Label' column in the 'early_test' DataFrame. This shows the class distribution in the test set for 3-day stays.

```python
early_test.Label.value_counts()
```

--------------------------------

### Count Output Labels After Dropping Newborns

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates the value counts for the 'OUTPUT_LABEL' column after filtering out newborn admissions.

```python
df_adm1.OUTPUT_LABEL.value_counts()
```

--------------------------------

### Load Less Than 2 Days Notes

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Loads a CSV file containing notes for patients with less than 2 days of stay. This is used for preparing the 2-day test set.

```python
want_2days = pd.read_csv('less_2_days_notes.csv')
```

--------------------------------

### Generate TFRecords for Pretraining (128 Sequence Length)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Generates TFRecords from the preprocessed text file for BERT pretraining with a maximum sequence length of 128. Requires vocabulary file and specifies various pretraining parameters.

```bash
python create_pretraining_data.py \
  --input_file=clinical_sentences_pretrain.txt \
  --output_file=tf_examples_128.tfrecord \
  --vocab_file=./uncased_L-12_H-768_A-12/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=3
```

--------------------------------

### Save Training IDs to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the training set IDs and labels to a CSV file named 'good_train_id_label.csv'.

```python
train_id_label.to_csv('./good_datasets/good_train_id_label.csv')
```

--------------------------------

### Save 2-Day Test Data to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the 'early_test_2days' DataFrame to a CSV file named 'test_snippets.csv' in the './good_datasets/2days/' directory.

```python
early_test_2days.to_csv('./good_datasets/2days/test_snippets.csv')
```

--------------------------------

### Load Gensim Word2Vec Model

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Load pretrained Word2Vec models for clinical notes using Gensim. This allows for direct use of word embeddings.

```python
import gensim
word2vec = gensim.models.KeyedVectors.load('word2vec.model')
weights = (m[m.wv.vocab])
```

--------------------------------

### Shift Admission Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Create columns for the next admission time and type per subject.

```python
df_adm['NEXT_ADMITTIME'] = df_adm.groupby('SUBJECT_ID').ADMITTIME.shift(-1)
```

```python
df_adm['NEXT_ADMISSION_TYPE'] = df_adm.groupby('SUBJECT_ID').ADMISSION_TYPE.shift(-1)
```

--------------------------------

### Evaluate Early Notes Prediction Model

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Run evaluation for early readmission prediction using ClinicalBERT. Ensure the data directory and model path are correctly specified.

```bash
python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_eval \
  --data_dir ./data/3days(2days)/ \
  --bert_model ./model/early_readmission \
  --max_seq_length 512 \
  --output_dir ./result_early
```

--------------------------------

### Save Processed Early Discharge Data (3 Days)

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the processed training, validation, and test sets for the early discharge (less than 3 days) task to CSV files. The file paths are dynamically generated based on the fold number.

```python
file = './good_datasets/fold'+str(i+2)+'/3days/train_snippets.csv'
early_train_snippets.to_csv(file)
```

```python
file = './good_datasets/fold'+str(i+2)+'/3days/val_snippets.csv'
early_val.to_csv(file)
file = './good_datasets/fold'+str(i+2)+'/3days/test_snippets.csv'
early_test.to_csv(file)
```

--------------------------------

### Sample and Filter Training Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Samples a subset of the readmission data and checks its length. This is used to create a balanced training set.

```python
dis_train = want[want.ID.isin(df.sample(n=437, random_state=1))]
len(dis_train)
```

--------------------------------

### Count Labels in Early Training Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates and displays the value counts for the 'Label' column in the 'early_train' DataFrame. This shows the distribution of classes in the training set.

```python
early_train.Label.value_counts()
```

--------------------------------

### Generate Basic Attention Heatmap with Seaborn

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Creates a basic heatmap visualization of attention scores using Seaborn. This is useful for a quick overview of attention patterns. Ensure Matplotlib and Seaborn are imported.

```python
f, ax = plt.subplots(figsize=(6,8))
sns.heatmap(map1, annot=False, fmt="f", ax=ax, xticklabels = False, yticklabels = False, vmax=0.4, cmap='gray', cbar_kws={'label':'Attention Weight', 'orientation':'horizontal'}, rasterized = True)
```

--------------------------------

### Run Readmission Prediction with Early Notes (3 days)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Perform readmission prediction using early admission notes (first 2-3 days). This command evaluates a model trained on 3-day notes against a 3-day test set.

```python
python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_eval \
  --data_dir ./data/3days/ \
  --bert_model ./model/early_readmission \
  --max_seq_length 512 \
  --output_dir ./result_early
```

--------------------------------

### Aggregate Predictions and Calculate Metrics

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Aggregates chunk-level predictions to admission-level using weighted voting and calculates AUROC and recall at 80% precision.

```python
import numpy as np
import pandas as pd
from sklearn.metrics import roc_curve, auc, precision_recall_curve

def vote_score(df_test, chunk_scores):
    """
    Aggregate chunk predictions to admission level.
    Score = (max + sum/2) / (1 + count/2)
    """
    df_test['pred_score'] = chunk_scores
    df_sort = df_test.sort_values(by=['ID'])

    # Aggregate by admission ID
    grouped = df_sort.groupby(['ID'])['pred_score']
    temp = (grouped.agg('max') + grouped.agg('sum') / 2) / (1 + grouped.agg('len') / 2)

    # Get true labels
    y_true = df_sort.groupby(['ID'])['Label'].agg(np.min).values
    y_score = temp.values

    # Calculate AUROC
    fpr, tpr, thresholds = roc_curve(y_true, y_score)
    auroc = auc(fpr, tpr)

    print(f"AUROC: {auroc:.4f}")
    return fpr, tpr, y_true, y_score

def calculate_rp80(y_true, y_score):
    """Calculate recall at 80% precision."""
    precision, recall, thresholds = precision_recall_curve(y_true, y_score)

    # Find recall where precision >= 0.80
    pr_df = pd.DataFrame({'precision': precision[:-1], 'recall': recall[:-1]})
    high_precision = pr_df[pr_df.precision >= 0.80]

    if len(high_precision) > 0:
        rp80 = high_precision.iloc[0].recall
    else:
        rp80 = 0.0

    print(f"Recall at Precision 80%: {rp80:.4f}")
    return rp80

# Example usage
fpr, tpr, y_true, y_score = vote_score(df_test, logits_history)
rp80 = calculate_rp80(y_true, y_score)
```

--------------------------------

### Generate TFRecords for Pretraining (512 Sequence Length)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Generates TFRecords for BERT pretraining with a maximum sequence length of 512. Similar to the 128 sequence length generation but with adjusted parameters for longer sequences.

```bash
python create_pretraining_data.py \
  --input_file=clinical_sentences_pretrain.txt \
  --output_file=tf_examples_512.tfrecord \
  --vocab_file=./uncased_L-12_H-768_A-12/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=3
```

--------------------------------

### Inspect Model Architecture

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Displays the structure of a specific attention layer within the model.

```python
model.bert.encoder.layer[0].attention.self
```

--------------------------------

### Value Counts for Training Discharge Labels

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Shows the distribution of labels (1.0 and 0.0) in the training set for the discharge task.

```python
discharge_train.Label.value_counts()
```

--------------------------------

### Create Validation DataFrame with Labels

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Combines validation IDs with their corresponding readmission labels and creates a DataFrame.

```python
id_val = pd.concat([id_val_t, id_val_f])
val_id_label = pd.DataFrame(data = list(zip(id_val, [1]*len(id_val_t)+[0]*len(id_val_f))), columns = ['id','label'])
```

--------------------------------

### Save Training Snippets to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the processed 'early_train_snippets' DataFrame to a CSV file named 'train_snippets.csv' in the './good_datasets/3days/' directory.

```python
early_train_snippets.to_csv('./good_datasets/3days/train_snippets.csv')
```

--------------------------------

### Filter Test Data for Actionable 3-Day Stays

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters the 'want_early' DataFrame to include only IDs present in 'test_actionable_id_label'. This selects the test subset for the 3-day stay category.

```python
early_test = want_early[want_early.ID.isin(test_actionable_id_label.id)]
```

--------------------------------

### Run Readmission Prediction with Early Notes (2 days)

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Evaluate readmission prediction using early admission notes (2 days) with a model trained on 3-day data. The model is capable of evaluating on both 2-day and 3-day test sets.

```python
python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode early \
  --do_eval \
  --data_dir ./data/2days/ \
  --bert_model ./model/early_readmission \
  --max_seq_length 512 \
  --output_dir ./result_early_2days
```

--------------------------------

### Filter Test Data for Actionable 2-Day Stays

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters the 'want_2days' DataFrame to include only IDs present in 'test_actionable_id_label_2days'. This selects the test subset for the 2-day stay category.

```python
early_test_2days = want_2days[want_2days.ID.isin(test_actionable_id_label_2days.id)]
```

--------------------------------

### Create Customized Attention Heatmap with Labels

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/attention_visualization.ipynb

Generates a detailed heatmap visualization of attention scores with custom labels for both query and key tokens. This provides a more interpretable view of attention relationships. Ensure Matplotlib and Seaborn are imported.

```python
f=plt.figure(figsize=(10,10))
ax = f.add_subplot(1,1,1)
i=ax.imshow(map1,interpolation='nearest',cmap='gray')

ax.set_yticks(range(len(tokens)))
ax.set_yticklabels(tokens)

ax.set_xticks(range(len(tokens)))
ax.set_xticklabels(tokens,rotation=60)

ax.set_xlabel('key')
ax.set_ylabel('query')

ax.grid(linewidth = 0.8)
```

```python
f, ax = plt.subplots(figsize=(8,6))
sns.heatmap(map1, annot=False, fmt="f", ax=ax, xticklabels = False, yticklabels = False, vmin=0, vmax=0.07, cmap='gray', cbar_kws={'label':'Attention Weight'})

ax.set_xlabel('Key Tokens')
ax.set_ylabel('Query Tokens')
```

--------------------------------

### Sample Data for Filtering

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Samples 500 rows randomly from the 'df' DataFrame with a fixed random state for reproducibility. This sampled data is used for filtering.

```python
not_readmit_ID_more = df.sample(n=500, random_state=1)
```

--------------------------------

### Analyze Death and Label Counts

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Count deceased patients and the distribution of the output label.

```python
len(df_adm1[df_adm1.DEATHTIME.notnull()])
```

```python
df_adm1.OUTPUT_LABEL.value_counts()
```

--------------------------------

### Save Training Snippets to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the processed training snippets dataset for the discharge task to a CSV file.

```python
discharge_train_snippets.to_csv('./good_datasets/discharge/train_snippets.csv')
```

--------------------------------

### Count Labels in 2-Day Test Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates and displays the value counts for the 'Label' column in the 'early_test_2days' DataFrame. This shows the class distribution in the 2-day test set.

```python
early_test_2days.Label.value_counts()
```

--------------------------------

### Check Label Distribution After Concatenation

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Displays the label distribution for the combined and shuffled early discharge training data.

```python
early_train_snippets.Label.value_counts()
```

--------------------------------

### Save Test Data to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the 'early_test' DataFrame to a CSV file named 'test_snippets.csv' in the './good_datasets/3days/' directory.

```python
early_test.to_csv('./good_datasets/3days/test_snippets.csv')
```

--------------------------------

### Shuffle and Reset Index for Training Snippets

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Shuffles the combined training snippets dataset and resets the index for clean processing.

```python
discharge_train_snippets = discharge_train_snippets.sample(frac=1, random_state=1).reset_index(drop=True)
```

--------------------------------

### Count Training Discharge Samples

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Reports the number of samples in the filtered training set for the discharge task.

```python
len(discharge_train)
```

--------------------------------

### Split Readmitted IDs for Testing

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Samples 20% of readmitted patient IDs for the validation/testing set.

```python
id_val_test_t=readmit_ID.sample(frac=0.2,random_state=1)
```

--------------------------------

### Visualize Self-Attention Patterns

Source: https://context7.com/kexinhuang12345/clinicalbert/llms.txt

Extracts attention scores using forward hooks and generates a heatmap to visualize model focus on clinical concepts.

```python
import torch
from torch import nn
import matplotlib.pyplot as plt
import seaborn as sns
import math
from pytorch_pretrained_bert import BertTokenizer
from modeling_readmission import BertForSequenceClassification, BertConfig

def transpose_for_scores(config, x):
    """Reshape tensor for multi-head attention visualization."""
    new_x_shape = x.size()[:-1] + (
        config.num_attention_heads,
        int(config.hidden_size / config.num_attention_heads)
    )
    x = x.view(*new_x_shape)
    return x.permute(0, 2, 1, 3)

def get_attention_scores(model, config, tokenizer, layer_idx, text):
    """Extract attention scores from a specific transformer layer."""
    tokens = tokenizer.tokenize(text)
    input_ids = tokenizer.convert_tokens_to_ids(tokens)
    segment_ids = [0] * len(input_ids)

    t_tensor = torch.tensor([input_ids])
    s_tensor = torch.tensor([segment_ids])

    outputs_query = []
    outputs_key = []

    def hook_query(module, input, output):
        outputs_query.append(output)

    def hook_key(module, input, output):
        outputs_key.append(output)

    # Register hooks to capture query and key tensors
    model.bert.encoder.layer[layer_idx].attention.self.query.register_forward_hook(hook_query)
    model.bert.encoder.layer[layer_idx].attention.self.key.register_forward_hook(hook_key)

    with torch.no_grad():
        model(t_tensor, s_tensor)

    # Calculate attention scores
    query_layer = transpose_for_scores(config, outputs_query[0])
    key_layer = transpose_for_scores(config, outputs_key[0])

    attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
    attention_scores = attention_scores / math.sqrt(config.hidden_size // config.num_attention_heads)
    attention_probs = nn.Softmax(dim=-1)(attention_scores)

    return attention_probs, tokens

# Load model and tokenizer
config = BertConfig.from_json_file('./model/discharge_readmission/bert_config.json')
model = BertForSequenceClassification(config, num_labels=1)
model.load_state_dict(torch.load('./model/discharge_readmission/pytorch_model.bin', map_location='cpu'))
model.eval()
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Get attention for clinical text
text = "he has experienced acute on chronic diastolic heart failure in the setting of volume overload due to his sepsis"
attention_probs, tokens = get_attention_scores(model, config, tokenizer, layer_idx=0, text=text)

# Visualize attention head 1
attention_map = attention_probs[0, 1].detach().numpy()

plt.figure(figsize=(10, 10))
plt.imshow(attention_map, cmap='gray')
plt.xticks(range(len(tokens)), tokens, rotation=60)
plt.yticks(range(len(tokens)), tokens)
plt.xlabel('Key')
plt.ylabel('Query')
plt.title('ClinicalBERT Attention Weights')
plt.colorbar(label='Attention Weight')
plt.tight_layout()
plt.savefig('attention_visualization.png')
```

--------------------------------

### Convert Date Columns

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Convert admission, discharge, and death time columns to datetime objects.

```python
df_adm.ADMITTIME = pd.to_datetime(df_adm.ADMITTIME, format = '%Y-%m-%d %H:%M:%S', errors = 'coerce')
df_adm.DISCHTIME = pd.to_datetime(df_adm.DISCHTIME, format = '%Y-%m-%d %H:%M:%S', errors = 'coerce')
df_adm.DEATHTIME = pd.to_datetime(df_adm.DEATHTIME, format = '%Y-%m-%d %H:%M:%S', errors = 'coerce')
```

--------------------------------

### Save Test IDs to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the test set IDs and labels to a CSV file named 'good_test_id_label.csv'.

```python
test_id_label.to_csv('./good_datasets/good_test_id_label.csv')
```

--------------------------------

### Filter Training Data for Less Than 3 Days

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters the 'want_early' DataFrame to include only IDs present in 'train_id_label'. This selects the training subset for the 3-day stay category.

```python
early_train = want_early[want_early.ID.isin(train_id_label.id)]
```

--------------------------------

### Split Early Discharge Data for 3 Days

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Splits the early discharge data (less than 3 days) into training, validation, and test sets based on folds and actionable IDs. Filters test set for actionable IDs with stay >= 3 days.

```python
i=3
early_train = want_early[want_early.ID.isin(train_id_folds[i])]
early_val = want_early[want_early.ID.isin(val_id_folds[i])]
test_actionable_id_3days = pd.Series(test_id_folds[i])[pd.Series(test_id_folds[i]).isin(actionable_ID_3days)]
early_test = want_early[want_early.ID.isin(test_actionable_id_3days)]

early_train.Label.value_counts()
```

--------------------------------

### Filter Test IDs for Actionable 2-Day Stays

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters 'test_id_label' to include only IDs that are present in 'actionable_ID_2days'. This selects test set IDs corresponding to patients with 2-day or longer stays.

```python
test_actionable_id_label_2days = test_id_label[test_id_label.id.isin(actionable_ID_2days)]
```

--------------------------------

### Sum Output Labels for Fold 0 Test Set

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Calculates the sum of 'OUTPUT_LABEL' from 'df_adm1' for the patient IDs in the test set of the first fold ('test_id_folds[0]'). This shows the label distribution in the first test fold.

```python
sum(df_adm1[df_adm1.HADM_ID.isin(test_id_folds[0])].OUTPUT_LABEL)
```

--------------------------------

### Generate TFRecords

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/pretrain.ipynb

Convert text data into TFRecord format for BERT and XLNet pretraining.

```bash
# STEP 7: Generate Pretraining Tensorflow TF_Records

# For Clinical BERT
# cd to the git repo

# Generate datasets for 128 max seq
!python create_pretraining_data.py \
  --input_file=PRETRAIN_DATA_PATH/clinical_sentences_pretrain.txt \
  --output_file=PRETRAIN_DATA_PATH/tf_examples_128.tfrecord \
  --vocab_file=INITIAL_MODEL_PATH/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=128 \
  --max_predictions_per_seq=20 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=3

# Generate datasets for 512 max seq
!python create_pretraining_data.py \
  --input_file=PRETRAIN_DATA_PATH/clinical_sentences_pretrain.txt \
  --output_file=PRETRAIN_DATA_PATH/tf_examples_512.tfrecord \
  --vocab_file=INITIAL_MODEL_PATH/vocab.txt \
  --do_lower_case=True \
  --max_seq_length=512 \
  --max_predictions_per_seq=76 \
  --masked_lm_prob=0.15 \
  --random_seed=12345 \
  --dupe_factor=3

# For Clinical XLNet

!python data_utils.py \
    --bsz_per_host=6 \
    --num_core_per_host=1 \
    --seq_len=512 \
    --reuse_len=256 \
    --input_glob=/scratch/kh2383/MechVent/data/clinical_sentences_pretrain_xlnet.txt \
    --save_dir=/scratch/kh2383/MechVent/data/xlnet_tfrecord/ \
    --num_passes=5 \
    --bi_data=True \
    --sp_path=/scratch/kh2383/clibert/xlnet_cased_L-12_H-768_A-12/spiece.model \
    --mask_alpha=6 \
    --mask_beta=1 \
    --num_predict=85
```

--------------------------------

### Filter Test IDs for Actionable 3-Day Stays

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Filters 'test_id_label' to include only IDs that are present in 'actionable_ID_3days'. This selects test set IDs corresponding to patients with 3-day or longer stays.

```python
test_actionable_id_label = test_id_label[test_id_label.id.isin(actionable_ID_3days)]
```

--------------------------------

### Save Validation IDs to CSV

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Saves the validation set IDs and labels to a CSV file named 'good_val_id_label.csv'.

```python
val_id_label.to_csv('./good_datasets/good_val_id_label.csv')
```

--------------------------------

### Concatenate and Shuffle Training Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Combines the sampled data with the original training data and shuffles it. The index is reset after shuffling.

```python
discharge_train_snippets = pd.concat([dis_train, discharge_train])
discharge_train_snippets = discharge_train_snippets.sample(frac=1, random_state=1).reset_index(drop=True)
```

--------------------------------

### Perform K-Fold Split for Readmitted IDs

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Iterates through the KFold splits of 'except_fold1_t' (readmitted IDs). For each split, it stores the test indices in the 're' dictionary, indexed by fold number.

```python
kf.get_n_splits(except_fold1_t)
for i, (train_index, test_index) in  enumerate(kf.split(except_fold1_t)):
    #print("TRAIN:", train_index) 
    #print("TEST:", test_index)
    re[i] = except_fold1_t[test_index]
```

--------------------------------

### Backfill Missing Admission Data

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/notebook/Dataset_Split.ipynb

Fill missing next admission values using backfilling per subject.

```python
df_adm = df_adm.sort_values(['SUBJECT_ID','ADMITTIME'])

df_adm[['NEXT_ADMITTIME','NEXT_ADMISSION_TYPE']] = df_adm.groupby(['SUBJECT_ID'])[['NEXT_ADMITTIME','NEXT_ADMISSION_TYPE']].fillna(method = 'bfill')
```

--------------------------------

### Evaluate Discharge Summary Prediction Model

Source: https://github.com/kexinhuang12345/clinicalbert/blob/master/README.md

Run evaluation for discharge summary readmission prediction using ClinicalBERT. Ensure the data directory and model path are correctly specified.

```bash
python ./run_readmission.py \
  --task_name readmission \
  --readmission_mode discharge \
  --do_eval \
  --data_dir ./data/discharge/ \
  --bert_model ./model/discharge_readmission \
  --max_seq_length 512 \
  --output_dir ./result_discharge
```