### Training Initialization and Setup

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Shows the output messages indicating the successful loading of extension modules and the start of the training process. It includes details about the dataset size, batch configuration, and trainable parameters.

```text
Output:
Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
***** Running training *****
  Num examples = 9712
  Num Epochs = 1
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Gradient Accumulation steps = 1
  Total optimization steps = 9712
  Number of trainable parameters = 2510851


```

--------------------------------

### Training Initialization Output

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Shows the start of the training process, including the number of examples, epochs, batch sizes, and total optimization steps. This output is generated after loading necessary extensions.

```text
Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
***** Running training *****
  Num examples = 2691
  Num Epochs = 20
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 8
  Total optimization steps = 6720
  Number of trainable parameters = 3558401
```

--------------------------------

### Training Initialization Output

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

This output indicates the successful loading of custom PyTorch extensions and the start of the training process, including the number of examples and epochs.

```text
Output:
Using /homes/schmirx6/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
***** Running training *****
  Num examples = 1056
  Num Epochs = 10


```

--------------------------------

### Install Protobuf for T5 Tokenizer

Source: https://github.com/agemagician/prottrans/blob/master/README.md

Installs the protobuf library to resolve potential UnboundLocalError with the T5 tokenizer.

```console
pip install protobuf
```

--------------------------------

### Install Necessary Libraries

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/TensorFlow/Advanced/ProtT5-XL-UniRef50.ipynb

Installs the SentencePiece and Transformers libraries required for using the ProtT5 model. This is a prerequisite for loading the model and tokenizer.

```python
!pip install -q SentencePiece transformers
```

--------------------------------

### Install Libraries and Clone Repository

Source: https://github.com/agemagician/prottrans/blob/master/Visualization/ProtAlbert_attention_head_view.ipynb

Installs the transformers library and clones the bertviz repository. These are necessary for loading the model and for visualization.

```python
!pip install -q transformers
!git clone https://github.com/jessevig/bertviz.git
```

--------------------------------

### Evaluation Phase Start

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

Indicates the beginning of the evaluation phase, specifying the number of examples and batch size used for evaluation.

```text
Output:
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1


```

--------------------------------

### Install Transformers and SentencePiece

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Basic/ProtAlbert.ipynb

Installs the necessary libraries for using Hugging Face Transformers and SentencePiece, which is required for some tokenizers.

```bash
!pip install -q transformers sentencePiece
```

--------------------------------

### Evaluation Phase Start

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Indicates the beginning of the evaluation phase, showing the number of examples and batch size used for evaluation. This output may appear multiple times during the training process.

```text
***** Running Evaluation *****
  Num examples = 299
  Batch size = 16
```

--------------------------------

### Start Benchmark Loop

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtElectra.ipynb

Initiates the benchmarking process by iterating through different sequence lengths and batch sizes. It prints the device being used and starts timing the operations within a `torch.no_grad()` context to disable gradient calculations.

```python
device_name = torch.cuda.get_device_name(device.index) if device.type == 'cuda' else 'CPU'

with torch.no_grad():
    print((' Benchmarking using ' + device_name + ' ').center(80, '*'))
    print(' Start '.center(80, '*'))
    for sequence_length in range(min_sequence_length,max_sequence_length+1,inc_sequence_length):
        for batch_size in range(min_batch_size,max_batch_size+1,inc_batch_size):
            start = time.time()
            for i in range(iterations):

```

--------------------------------

### Install Transformers and Dependencies

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/TensorFlow/Advanced/ProtT5-XL-BFD.ipynb

Installs the Hugging Face Transformers library from GitHub and SentencePiece. Ensure you have the correct commit hash for compatibility.

```bash
!pip install -q SentencePiece git+https://github.com/huggingface/transformers.git@40ecaf0c2b1c0b3894e9abf619f32472c5a3b3ca
```

--------------------------------

### Start Training

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Initiates the fine-tuning process using the trainer.train() method. This will train the model based on the provided arguments and datasets.

```python
trainer.train()
```

--------------------------------

### Inference Output Example

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/Onnx/ProtBert-BFD.ipynb

Example output showing the inference time in milliseconds for ProtBert-BFD for different sequence lengths and batch sizes.

```text
ProtBert-BFD model took 36.103 ms for sequence length between 0.500 and 0.500000 of total sequences 399
ProtBert-BFD model took 34.940 ms for sequence length between 0.500 and 0.500000 of total sequences 1025
ProtBert-BFD model took 31.531 ms for sequence length between 0.500 and 0.500000 of total sequences 99
```

--------------------------------

### Repeated Evaluation Phase Start

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

Indicates the beginning of the evaluation phase, specifying the number of examples and batch size used for evaluation. This log appears multiple times during training.

```text
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1
***** Running Evaluation *****
  Num examples = 118
  Batch size = 1


```

--------------------------------

### Install ProtTrans Dependencies

Source: https://github.com/agemagician/prottrans/blob/master/README.md

Installs the necessary Python libraries for using ProtTrans models with PyTorch and Hugging Face Transformers.

```console
pip install torch
pip install transformers
pip install sentencepiece
```

--------------------------------

### Initialize Tokenizer

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/Onnx/ProtBert-BFD.ipynb

Loads the tokenizer for the ProtBert-BFD model. Ensure the 'transformers' library is installed.

```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("Rostlab/prot_bert_bfd", do_lower_case=False)
```

--------------------------------

### Install Hugging Face Transformers

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Advanced/ProtAlbert.ipynb

Installs the Hugging Face Transformers library, which is necessary for loading and using pretrained models like ProtAlbert.

```python
!pip install -q transformers
```

--------------------------------

### Define Example Sequence

Source: https://github.com/agemagician/prottrans/blob/master/Generate/ProtXLNet.ipynb

Defines a sample protein sequence string for demonstration purposes.

```python
sequences_Example = "A E T C Z A O"
```

--------------------------------

### Inference Output Example

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/Onnx/ProtBert-BFD.ipynb

Sample output showing the inference time in milliseconds for the ProtBert-BFD model, broken down by sequence length and batch size.

```text
ProtBert-BFD model took 35.942 ms for sequence length between 0.500 and 0.500000 of total sequences 399
ProtBert-BFD model took 39.580 ms for sequence length between 0.500 and 0.500000 of total sequences 1025

```

--------------------------------

### Training Log Output

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Example log output during the training process, showing global step, learning rate, momentum, and performance metrics.

```text
[2023-07-11 16:23:42,930] [INFO] [logging.py:75:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]
[2023-07-11 16:23:42,932] [INFO] [timer.py:198:stop] epoch=0/micro_step=48000/global_step=6000, RunningAvgSamplesPerSec=4.1635646382033755, CurrSamplesPerSec=4.158338530053567, MemAllocated=4.52GB, MaxMemAllocated=9.11GB
```

--------------------------------

### Evaluation Phase Output

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Indicates the start of the evaluation phase and reports the number of examples and batch size used for evaluation. This output appears multiple times during the training process, typically after certain milestones.

```text
Output:
***** Running Evaluation *****
  Num examples = 1080
  Batch size = 1
***** Running Evaluation *****
  Num examples = 1080
  Batch size = 1
***** Running Evaluation *****
  Num examples = 1080
  Batch size = 1


```

--------------------------------

### Import Core Dependencies

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Imports essential libraries for model training, data manipulation, and utility functions. Ensure these are installed in your environment.

```python
#import dependencies
import os.path
os.chdir("set a path here")


import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss
from torch.utils.data import DataLoader

import re
import numpy as np
import pandas as pd
import copy

import transformers, datasets
from transformers.modeling_outputs import TokenClassifierOutput
from transformers.models.t5.modeling_t5 import T5Config, T5PreTrainedModel, T5Stack
from transformers.utils.model_parallel_utils import assert_device_map, get_device_map
from transformers import T5EncoderModel, T5Tokenizer
from transformers import TrainingArguments, Trainer, set_seed
from transformers import DataCollatorForTokenClassification

from evaluate import load
from datasets import Dataset

from tqdm import tqdm
import random

from scipy import stats
from sklearn.metrics import accuracy_score

import matplotlib.pyplot as plt
```

--------------------------------

### Execute PT5 Model Training

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Example of how to call the `train_per_residue` function with specific parameters for fine-tuning. This demonstrates setting the training and validation data, number of labels, batch sizes, epochs, random seed, and GPU selection.

```python
tokenizer, model, history = train_per_residue(my_train, my_valid, num_labels=3, batch=1, accum=1, epochs=1, seed=42, gpu=2)
```

--------------------------------

### Utility Functions for Training Setup

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Provides functions to set random seeds for reproducibility across different libraries (PyTorch, NumPy, random) and to create a Hugging Face Dataset from protein sequences and labels, including tokenization.

```python
# Set random seeds for reproducibility of your trainings run
def set_seeds(s):
    torch.manual_seed(s)
    np.random.seed(s)
    random.seed(s)
    set_seed(s)

# Dataset creation
def create_dataset(tokenizer,seqs,labels):
    tokenized = tokenizer(seqs, max_length=1024, padding=False, truncation=True)
    dataset = Dataset.from_dict(tokenized)
    dataset = dataset.add_column("labels", labels)

    return dataset
```

--------------------------------

### Install Required Libraries

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/Onnx/ProtBert-BFD.ipynb

Installs the necessary Python packages for using transformers, biopython, and ONNX runtime with GPU support. Ensure you have a compatible CUDA version for onnxruntime-gpu.

```python
!pip install -q transformers
!pip install -q biopython
!pip install -q onnxruntime-gpu==1.4.0
!pip install -q onnxruntime-tools==1.4.2
```

--------------------------------

### Prepare Protein Sequences for Embedding

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Basic/ProtXLNet.ipynb

Prepares protein sequences by adding a start and end token, and then tokenizes them using the ProtXLNet tokenizer. Sequences should be in FASTA format.

```python
sequences = ["MAAHKLTENELQRM"] # Example sequence

# Add start and end tokens
sequences_tk = ["<start> " + seq + " <end>" for seq in sequences]

# Tokenize sequences
inputs = tokenizer(sequences_tk, return_tensors="pt", padding=True, truncation=True)
```

--------------------------------

### Model Reloading and Prediction Setup

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

This code reloads a previously trained model, sets the computation device (GPU or CPU), and prepares the test dataset and dataloader for making predictions. It includes necessary imports for PyTorch and Hugging Face's DataCollatorForTokenRegression.

```python
#Use reloaded model
model = model_reload
del model_reload

# Set the device to use
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# Create Dataset
test_set=create_dataset(tokenizer,list(test['sequence']),list(test['label']))
# Make compatible with torch DataLoader
test_set = test_set.with_format("torch", device=device)

# For token classification we need a data collator here to pad correctly
data_collator = DataCollatorForTokenRegression(tokenizer)

# Create a dataloader for the test dataset
test_dataloader = DataLoader(test_set, batch_size=16, shuffle = False, collate_fn = data_collator)

# Put the model in evaluation mode
model.eval()

# Make predictions on the test dataset
predictions = []
# We need to collect the batch["labels"] as well, this allows us to filter out all positions with a -100 afterwards
padded_labels = []

with torch.no_grad():
    for batch in tqdm(test_dataloader):
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        # Padded labels from the data collator
        padded_labels += batch['labels'].tolist()
        # Add batch results(logits) to predictions, we take the argmax here to get the predicted class
        predictions += model(input_ids, attention_mask=attention_mask).logits.squeeze().tolist()
```

--------------------------------

### Optimizer and Utility Loading Confirmation

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Confirms the loading of utility operations and the creation of the AdamW optimizer with specified parameters. This output is typically seen after the initial setup and before the main training loop begins.

```text
Output:
Time to load utils op: 0.002664327621459961 seconds
Adam Optimizer #0 is created with AVX2 arithmetic capability.
Config: alpha=0.000300, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1


```

--------------------------------

### Load ProtXLNet Model and Tokenizer

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Basic/ProtXLNet.ipynb

Loads the pre-trained ProtXLNet model and its corresponding tokenizer from Hugging Face Transformers. Ensure you have 'torch' and 'transformers' installed.

```python
from transformers import ProtT5ForConditionalGeneration, T5Tokenizer

model = ProtT5ForConditionalGeneration.from_pretrained("Rostlab/prot_t5_xl_uniref50.3M_prot_m")
tokenizer = T5Tokenizer.from_pretrained("Rostlab/prot_t5_xl_uniref50.3M_prot_m", do_lower_case=False)
```

--------------------------------

### Perform Prediction

Source: https://github.com/agemagician/prottrans/blob/master/Prediction/ProtBert_BFD_Predict_MS.ipynb

Runs the preprocessed example sequences through the loaded prediction pipeline to classify them as 'Membrane' or 'Soluble'.

```python
pipeline(sequences_Example)
```

--------------------------------

### Import Libraries

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtBert.ipynb

Imports essential Python libraries for model loading, tensor operations, timing, file handling, and progress display. Ensure these are installed before running.

```python
import torch
from transformers import BertModel
import time
from datetime import timedelta
import os
import requests
from tqdm.auto import tqdm
```

--------------------------------

### Download and Load Secondary Structure Dataset

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Downloads the secondary structure dataset from a GitHub repository and loads it into a pandas DataFrame. This example uses Biopython for parsing FASTA files.

```python
# For this example we import the secondary_structure dataset from https://github.com/J-SNACKKB/FLIP
# For details, see publication here: https://openreview.net/forum?id=p2dMLEwL8tF
import requests
import zipfile
from io import BytesIO
from Bio import SeqIO
import tempfile

# Download the zip file from GitHub
url = 'https://github.com/J-SNACKKB/FLIP/raw/main/splits/secondary_structure/splits.zip'

response = requests.get(url)
zip_file = zipfile.ZipFile(BytesIO(response.content))

# Extract the fasta file to a temporary directory
# Sequence File
with tempfile.TemporaryDirectory() as temp_dir:
    zip_file.extract('splits/sequences.fasta', temp_dir)

    # Load the fasta files
    fasta_file = open(temp_dir + '/splits/sequences.fasta')
    
    # Load FASTA file using Biopython
    sequences = []
    for record in SeqIO.parse(fasta_file, "fasta"):
        sequences.append([record.name, str(record.seq)])

    # Create dataframe
    df = pd.DataFrame(sequences, columns=["name", "sequence"])
```

--------------------------------

### Initiate PT5 Model Training

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

This snippet shows how to call the `train_per_residue` function with specific hyperparameters to start the fine-tuning process. Adjust parameters like batch size, accumulation steps, epochs, and seed according to your dataset and computational resources.

```python
tokenizer, model, history = train_per_residue(train, valid, num_labels=1, batch=1, accum=1, epochs=10, seed=42, gpu=4)
```

--------------------------------

### Prepare Sequences

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Basic/ProtXLNet.ipynb

Defines example protein sequences and then uses regular expressions to replace rare amino acids (U, Z, O, B) with the unknown token '<unk>'.

```python
sequences_Example = ["A E T C Z A O","S K T Z P"]
sequences_Example = [re.sub(r"[UZOBX]", "<unk>", sequence) for sequence in sequences_Example]
```

--------------------------------

### Prepare Protein Sequences for Embedding

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Advanced/ProtT5-XL-UniRef50.ipynb

Prepares a list of protein sequences by adding a start and end token, and then tokenizing them. This step is crucial for the model to correctly process the input sequences. Ensure sequences are in the correct format.

```python
sequences = [
    "AEMRQ" # Example sequence
]

# Add start and end tokens, and tokenize
sequences_tk = ["<start>"] + [seq.replace("", " ").split("") for seq in sequences] + ["<end>"]
sequences_str = [" ".join(seq) for seq in sequences_tk]

inputs = tokenizer(sequences_str, return_tensors="pt", padding=True, truncation=True)

# Move inputs to GPU if available
if torch.cuda.is_available():
    inputs = inputs.to('cuda')

```

--------------------------------

### PyTorch Batch Collation Utility

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

This utility function collates a list of examples into a PyTorch tensor batch, applying padding if necessary based on the tokenizer's configuration and a specified `pad_to_multiple_of` value. It raises an error if the tokenizer lacks a pad token.

```python
def _torch_collate_batch(examples, tokenizer, pad_to_multiple_of: Optional[int] = None):
    """Collate `examples` into a batch, using the information in `tokenizer` for padding if necessary."""
    import torch

    # Tensorize if necessary.
    if isinstance(examples[0], (list, tuple, np.ndarray)):
        examples = [torch.tensor(e, dtype=torch.long) for e in examples]

    length_of_first = examples[0].size(0)

    # Check if padding is necessary.

    are_tensors_same_length = all(x.size(0) == length_of_first for x in examples)
    if are_tensors_same_length and (pad_to_multiple_of is None or length_of_first % pad_to_multiple_of == 0):
        return torch.stack(examples, dim=0)

    # If yes, check if we have a `pad_token`.
    if tokenizer._pad_token is None:
        raise ValueError(
            "You are attempting to pad samples but the tokenizer you are using"
            f" ({tokenizer.__class__.__name__}) does not have a pad token."
        )

    # Creating the full tensor and filling it with our data.
    max_length = max(x.size(0) for x in examples)
    if pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
        max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
    result = examples[0].new_full([len(examples), max_length], tokenizer.pad_token_id)
    for i, example in enumerate(examples):
        if tokenizer.padding_side == "right":
            result[i, : example.shape[0]] = example
        else:
            result[i, -example.shape[0] :] = example
```

--------------------------------

### Utils Op Load Time and Optimizer Initialization

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Reports the time taken to load the utils operation and details the initialization of the ZeRO stage 2 optimizer, including memory usage.

```log
ninja: no work to do.
Time to load utils op: 0.3791632652282715 seconds
Rank: 0 partition count [1] and sizes[(3558402, False)] 
[2023-07-11 13:05:25,513] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states
[2023-07-11 13:05:25,514] [INFO] [utils.py:826:see_memory_usage] MA 4.52 GB         Max_MA 4.52 GB         CA 4.54 GB         Max_CA 5 GB 
[2023-07-11 13:05:25,515] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 38.29 GB, percent = 20.5%
[2023-07-11 13:05:25,671] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states
[2023-07-11 13:05:25,672] [INFO] [utils.py:826:see_memory_usage] MA 4.52 GB         Max_MA 4.52 GB         CA 4.54 GB         Max_CA 5 GB 
[2023-07-11 13:05:25,673] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 38.32 GB, percent = 20.5%
[2023-07-11 13:05:25,674] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized
[2023-07-11 13:05:25,822] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer

```

--------------------------------

### Model Parallelization Setup for T5 Token Classifier

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Configures the model for parallel processing across multiple devices. It assigns layers to specific devices based on the encoder's block count.

```python
def parallelize(self, device_map=None):
    self.device_map = (
        get_device_map(len(self.encoder.block), range(torch.cuda.device_count()))
        if device_map is None
        else device_map
    )
    assert_device_map(self.device_map, len(self.encoder.block))
    self.encoder.parallelize(self.device_map)
    self.classifier = self.classifier.to(self.encoder.first_device)
    self.model_parallel = True
```

--------------------------------

### Define Example Sequences

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Basic/ProtBert-BFD.ipynb

Defines a list of example protein sequences for feature extraction.

```python
sequences_Example = ["A E T C Z A O","S K T Z P"]
```

--------------------------------

### Input Embeddings Management for T5 Token Classifier

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Provides methods to get and set the input embeddings for the T5 token classification model. Ensures consistency between shared embeddings and the encoder.

```python
def get_input_embeddings(self):
    return self.shared

def set_input_embeddings(self, new_embeddings):
    self.shared = new_embeddings
    self.encoder.set_input_embeddings(new_embeddings)
```

--------------------------------

### Download Model Files if Not Present

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtXLNet.ipynb

Downloads the ProtXLNet model weights, configuration, and tokenizer files if they do not already exist in the specified local path. This avoids re-downloading.

```python
if not os.path.exists(modelFilePath):
    download_file(modelUrl, modelFilePath)

if not os.path.exists(configFilePath):
    download_file(configUrl, configFilePath)

if not os.path.exists(tokenizerFilePath):
    download_file(tokenizerUrl, tokenizerFilePath)
```

--------------------------------

### Define PT5 Model Training Function

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Defines the main training function for fine-tuning a PT5 model per residue class. It handles data preprocessing, dataset creation, trainer argument setup, metric computation, and model training. It supports configurations for batch size, learning rate, epochs, deepspeed, and mixed precision.

```python
def train_per_residue(
        train_df,         #training data
        valid_df,         #validation data      
        num_labels= 3,    #number of classes
    
        # effective training batch size is batch * accum
        # we recommend an effective batch size of 8 
        batch= 4,         #for training
        accum= 2,         #gradient accumulation
    
        val_batch = 16,   #batch size for evaluation
        epochs= 10,       #training epochs
        lr= 3e-4,         #recommended learning rate
        seed= 42,         #random seed
        deepspeed= True,  #if gpu is large enough disable deepspeed for training speedup
        mixed= False,     #enable mixed precision training
        gpu= 1 ):         #gpu selection (1 for first gpu)

    # Set gpu device
    os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu-1)
    
    # Set all random seeds
    set_seeds(seed)
    
    # load model
    model, tokenizer = PT5_classification_model(num_labels=num_labels)

    # Preprocess inputs
    # Replace uncommon AAs with "X"
    train_df["sequence"]=train_df["sequence"].str.replace('|'.join(["O","B","U","Z"]и),"X",regex=True)
    valid_df["sequence"]=valid_df["sequence"].str.replace('|'.join(["O","B","U","Z"]и),"X",regex=True)
    # Add spaces between each amino acid for PT5 to correctly use them
    train_df['sequence']=train_df.apply(lambda row : " ".join(row["sequence"]), axis = 1)
    valid_df['sequence']=valid_df.apply(lambda row : " ".join(row["sequence"]), axis = 1)


    # Create Datasets
    train_set=create_dataset(tokenizer,list(train_df['sequence']),list(train_df['label']))
    valid_set=create_dataset(tokenizer,list(valid_df['sequence']),list(valid_df['label']))

    # Huggingface Trainer arguments
    args = TrainingArguments(
        "./",
        evaluation_strategy = "steps",
        eval_steps = 500,
        logging_strategy = "epoch",
        save_strategy = "no",
        learning_rate=lr,
        per_device_train_batch_size=batch,
        #per_device_eval_batch_size=val_batch,
        per_device_eval_batch_size=batch,
        gradient_accumulation_steps=accum,
        num_train_epochs=epochs,
        seed = seed,
        deepspeed= ds_config if deepspeed else None,
        fp16 = mixed,
    ) 

    # Metric definition for validation data
    def compute_metrics(eval_pred):

        metric = load("accuracy")
        predictions, labels = eval_pred
        
        labels = labels.reshape((-1,))
        
        predictions = np.argmax(predictions, axis=2)
        predictions = predictions.reshape((-1,))
        
        predictions = predictions[labels!=-100]
        labels = labels[labels!=-100]
        
        return metric.compute(predictions=predictions, references=labels)

    # For token classification we need a data collator here to pad correctly
    data_collator = DataCollatorForTokenClassification(tokenizer) 

    # Trainer          
    trainer = Trainer(
        model,
        args,
        train_dataset=train_set,
        eval_dataset=valid_set,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics
    )    
    
    # Train model
    trainer.train()

    return tokenizer, model, trainer.state.log_history
```

--------------------------------

### PT5 Model Training Function

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

This function orchestrates the fine-tuning of a PT5 model for per-residue prediction. It handles data preprocessing, model loading, dataset creation, Hugging Face Trainer setup, and the training execution. Ensure all necessary libraries and the PT5 model architecture are available.

```python
import os
import numpy as np
import pandas as pd
from transformers import TrainingArguments, Trainer, DataCollatorForTokenRegression
from datasets import load_dataset
from evaluate import load

# Assuming PT5_classification_model, create_dataset, and set_seeds are defined elsewhere
# from your_module import PT5_classification_model, create_dataset, set_seeds

# Placeholder for PT5_classification_model
def PT5_classification_model(num_labels):
    # This is a placeholder. Replace with actual model loading.
    class MockTokenizer: pass
    class MockModel: pass
    return MockModel(), MockTokenizer()

# Placeholder for create_dataset
def create_dataset(tokenizer, sequences, labels):
    # This is a placeholder. Replace with actual dataset creation.
    class MockDataset: pass
    return MockDataset()

# Placeholder for set_seeds
def set_seeds(seed):
    # This is a placeholder. Replace with actual seed setting.
    pass

# Placeholder for ds_config
ds_config = {}

def train_per_residue(
        train_df,         #training data
        valid_df,         #validation data      
        num_labels= 1,    #number of classes
    
        # effective training batch size is batch * accum
        # we recommend an effective batch size of 8 
        batch= 4,         #for training
        accum= 2,         #gradient accumulation
    
        val_batch = 16,   #batch size for evaluation
        epochs= 10,       #training epochs
        lr= 3e-4,         #recommended learning rate
        seed= 42,         #random seed
        deepspeed= True,  #if gpu is large enough disable deepspeed for training speedup
        mixed= False,     #enable mixed precision training  
        gpu= 1 ):         #gpu selection (1 for first gpu)

    # Set gpu device
    os.environ["CUDA_VISIBLE_DEVICES"]=str(gpu-1)
    
    # Set all random seeds
    set_seeds(seed)
    
    # load model
    model, tokenizer = PT5_classification_model(num_labels=num_labels)

    # Preprocess inputs
    # Replace uncommon AAs with "X"
    train_df["sequence"]=train_df["sequence"].str.replace('|'.join(["O","B","U","Z"]і),"X",regex=True)
    valid_df["sequence"]=valid_df["sequence"].str.replace('|'.join(["O","B","U","Z"]і),"X",regex=True)
    # Add spaces between each amino acid for PT5 to correctly use them
    train_df['sequence']=train_df.apply(lambda row : " ".join(row["sequence"]), axis = 1)
    valid_df['sequence']=valid_df.apply(lambda row : " ".join(row["sequence"]), axis = 1)


    # Create Datasets
    train_set=create_dataset(tokenizer,list(train_df['sequence']),list(train_df['label']))
    valid_set=create_dataset(tokenizer,list(valid_df['sequence']),list(valid_df['label']))

    # Huggingface Trainer arguments
    args = TrainingArguments(
        "./",
        evaluation_strategy = "steps",
        eval_steps = 528,
        logging_strategy = "epoch",
        save_strategy = "no",
        learning_rate=lr,
        per_device_train_batch_size=batch,
        #per_device_eval_batch_size=val_batch,
        per_device_eval_batch_size=batch,
        gradient_accumulation_steps=accum,
        num_train_epochs=epochs,
        seed = seed,
        deepspeed= ds_config if deepspeed else None,
        fp16 = mixed,
    ) 

    # Metric definition for validation data
    def compute_metrics(eval_pred):

        metric = load("spearmanr")
        predictions, labels = eval_pred
        predictions=predictions.flatten()
        labels=labels.flatten()

        valid_labels=labels[np.where((labels != -100 ) & (labels < 900 ))]
        valid_predictions=predictions[np.where((labels != -100 ) & (labels < 900 ))]

        return metric.compute(predictions=valid_predictions, references=valid_labels)

    # For token classification we need a data collator here to pad correctly
    data_collator = DataCollatorForTokenRegression(tokenizer) 

    # Trainer          
    trainer = Trainer(
        model,
        args,
        train_dataset=train_set,
        eval_dataset=valid_set,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics
    )    
    
    # Train model
    trainer.train()

    return tokenizer, model, trainer.state.log_history

```

--------------------------------

### Construct File Paths for Model Components

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtXLNet.ipynb

Creates the full file paths for the model weights, configuration, and tokenizer files based on the download folder. This prepares for checking existence and downloading.

```python
modelFolderPath = downloadFolderPath

modelFilePath = os.path.join(modelFolderPath, 'pytorch_model.bin')

configFilePath = os.path.join(modelFolderPath, 'config.json')

tokenizerFilePath = os.path.join(modelFolderPath, 'spm_model.model')
```

--------------------------------

### Prepare Model and Data for Inference

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_class.ipynb

Loads the model, sets the device (GPU or CPU), creates a PyTorch Dataset from the test sequences and labels, and configures a DataLoader with a Data Collator for token classification.

```python
#Use reloaded model
model = model_reload
del model_reload

# Set the device to use
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# Create Dataset
test_set=create_dataset(tokenizer,list(my_test['sequence']),list(my_test['label']))
# Make compatible with torch DataLoader
test_set = test_set.with_format("torch", device=device)

# For token classification we need a data collator here to pad correctly
data_collator = DataCollatorForTokenClassification(tokenizer)

# Create a dataloader for the test dataset
test_dataloader = DataLoader(test_set, batch_size=16, shuffle = False, collate_fn = data_collator)
```

--------------------------------

### Memory Usage Before and After Optimizer Initialization

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

Logs showing memory usage before and after initializing the ZeRO optimizer, including CPU virtual memory.

```log
ninja: no work to do.
Time to load utils op: 0.5580213069915771 seconds
Rank: 0 partition count [1] and sizes[(2508802, False)] 
[2023-10-20 16:11:47,892] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states
[2023-10-20 16:11:47,893] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB 
[2023-10-20 16:11:47,894] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 51.81 GB, percent = 27.8%
[2023-10-20 16:11:48,079] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states
[2023-10-20 16:11:48,080] [INFO] [utils.py:826:see_memory_usage] MA 4.51 GB         Max_MA 4.51 GB         CA 4.52 GB         Max_CA 5 GB 
[2023-10-20 16:11:48,081] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 51.84 GB, percent = 27.8%
[2023-10-20 16:11:48,082] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized
[2023-10-20 16:11:48,235] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer
```

--------------------------------

### Define Training Arguments

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Sets up the training arguments for the fine-tuning process using the Seq2SeqTrainingArguments class. This includes output directory, learning rate, and number of epochs.

```python
from transformers import Seq2SeqTrainingArguments

args = Seq2SeqTrainingArguments(
    output_dir="./pt5_lora_finetuned",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    warmup_steps=100,
    num_train_epochs=2,
    learning_rate=2e-4,
    logging_steps=10,
    save_strategy="epoch",
    evaluation_strategy="epoch",
    fp16=True,
    push_to_hub=False,
)
```

--------------------------------

### Define Example Protein Sequences

Source: https://github.com/agemagician/prottrans/blob/master/Prediction/ProtBert_BFD_Predict_MS.ipynb

Defines a list of example protein sequences for testing the prediction pipeline. These sequences are strings of amino acid codes.

```python
sequences_Example = ["M A K S K N H T A H N Q T R K A H R N G I K K P K T Y K Y P S L K G V D P K F R R N H K H A L H G T A K A L A A A K K",
                     "M G L P V S W A P P A L W V L G C C A L L L S L W A L C T A C R R P E D A V A P R K R A R R Q R A R L Q G S A T A A E A S L L R R T H L C S L S K S D T R L H E L H R G P R S S R A L R P A S M D L L R P H W L E V S R D I T G P Q A A P S A F P H Q E L P R A L P A A A A T A G C A G L E A T Y S N V G L A A L P G V S L A A S P V V A E Y A R V Q K R K G T H R S P Q E P Q Q G K T E V T P A A Q V D V L Y S R V C K P K R R D P G P T T D P L D P K G Q G A I L A L A G D L A Y Q T L P L R A L D V D S G P L E N V Y E S I R E L G D P A G R S S T C G A G T P P A S S C P S L G R G W R P L P A S L P"]
```

--------------------------------

### DeepSpeed Optimizer Initialization

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

Logs detailing the initialization of DeepSpeed's ZeRO stage 2 optimizer with CPU offload enabled.

```log
ninja: no work to do.
Time to load cpu_adam op: 3.048384428024292 seconds
[2023-10-20 16:11:47,034] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2023-10-20 16:11:47,071] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2023-10-20 16:11:47,071] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2023-10-20 16:11:47,072] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer
[2023-10-20 16:11:47,073] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 200000000
[2023-10-20 16:11:47,073] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 200000000
[2023-10-20 16:11:47,074] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True
[2023-10-20 16:11:47,074] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False
```

--------------------------------

### Download ProtBert Model Files

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtBert.ipynb

Downloads the ProtBert model weights, configuration, and vocabulary files if they do not already exist locally. This step ensures all necessary files are present before loading the model.

```python
if not os.path.exists(modelFilePath):
    download_file(modelUrl, modelFilePath)

if not os.path.exists(configFilePath):
    download_file(configUrl, configFilePath)

if not os.path.exists(vocabFilePath):
    download_file(vocabUrl, vocabFilePath)
```

--------------------------------

### Training Batch Size and Optimization Steps

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_residue_reg.ipynb

Details the effective batch size per device, total batch size considering distributed training and accumulation, and the total number of optimization steps required for training.

```text
Output:
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Gradient Accumulation steps = 1
  Total optimization steps = 10560
  Number of trainable parameters = 2508801


```

--------------------------------

### Define and Preprocess Example Sequences

Source: https://github.com/agemagician/prottrans/blob/master/Embedding/PyTorch/Advanced/ProtAlbert.ipynb

Defines a list of example protein sequences and then uses regular expressions to replace rarely occurring amino acids (U, Z, O, B) with 'X'. This standardization helps the model handle variations in sequence data.

```python
sequences_Example = ["A E T C Z A O","S K T Z P"]
sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]
```

--------------------------------

### Initialize PT5 Model and Load Finetuned Weights

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Initializes a new PT5 model and loads finetuned weights from a specified file. This is useful for resuming training or using a pre-finetuned model.

```python
model, tokenizer = PT5_classification_model(num_labels=num_labels, half_precision=mixed)

# Load the non-frozen parameters from the saved file
non_frozen_params = torch.load(filepath)

# Assign the non-frozen parameters to the corresponding parameters of the model
for param_name, param in model.named_parameters():
    if param_name in non_frozen_params:
        param.data = non_frozen_params[param_name].data

return tokenizer, model
```

--------------------------------

### Evaluation and Training Completion Log

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Log messages indicating the start of an evaluation phase and the completion of the training process.

```text
***** Running Evaluation *****
  Num examples = 299
  Batch size = 16
***** Running Evaluation *****
  Num examples = 299
  Batch size = 16
***** Running Evaluation *****
  Num examples = 299
  Batch size = 16


Training completed. Do not forget to share your model on huggingface.co/models =)
```

--------------------------------

### Load Model and Tokenizer

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Loads the pre-trained PT5 model and its corresponding tokenizer. Ensure you have the 'transformers' and 'torch' libraries installed.

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = "./pt5_model"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
```

--------------------------------

### Load ProtXLNet Model

Source: https://github.com/agemagician/prottrans/blob/master/Benchmark/ProtXLNet.ipynb

Loads the ProtXLNet model from the specified local folder using Hugging Face's `from_pretrained` method. This prepares the model for inference.

```python
model = XLNetModel.from_pretrained(modelFolderPath)
```

--------------------------------

### Prepare Model and Dataset for Prediction

Source: https://github.com/agemagician/prottrans/blob/master/Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

Loads the reloaded model, sets the appropriate device (GPU or CPU), creates a PyTorch Dataset from the test data, and formats it for the specified device. Requires a tokenizer and DataLoader.

```python
#Use reloaded model
model = model_reload
del model_reload

# Set the device to use
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# create Dataset
test_set=create_dataset(tokenizer,list(my_test['sequence']),list(my_test['label']))
# make compatible with torch DataLoader
test_set = test_set.with_format("torch", device=device)

# Create a dataloader for the test dataset
test_dataloader = DataLoader(test_set, batch_size=16, shuffle=False)

# Put the model in evaluation mode
model.eval()

# Make predictions on the test dataset
predictions = []
with torch.no_grad():
    for batch in tqdm(test_dataloader):
        input_ids = batch['input_ids'].to(device)
        attention_mask = batch['attention_mask'].to(device)
        #add batch results(logits) to predictions
        predictions += model(input_ids, attention_mask=attention_mask).logits.tolist()
```