### Manual Installation of MethylBERT Source: https://github.com/compepigen/methylbert/blob/main/README.md Sets up the MethylBERT environment manually by cloning the repository and installing the package using pip. ```bash conda create -n methylbert -c conda-forge python=3.11 cudatoolkit==11.8 pip freetype-py conda activate methylbert git clone https://github.com/hanyangii/methylbert.git cd methylbert pip3 install . ``` -------------------------------- ### Initialize and Train MethylBERT Model Source: https://context7.com/compepigen/methylbert/llms.txt Sets up the MethylBertFinetuneTrainer with specified hyperparameters and data loaders, then loads a pre-trained model and starts the training process. ```python import torch from torch.utils.data import DataLoader from methylbert.data.vocab import MethylVocab from methylbert.data.dataset import MethylBertFinetuneDataset from methylbert.trainer import MethylBertFinetuneTrainer tokenizer = MethylVocab(k=3) train_dataset = MethylBertFinetuneDataset("data/train_seq.csv", tokenizer, seq_len=150) test_dataset = MethylBertFinetuneDataset("data/test_seq.csv", tokenizer, seq_len=150) train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=8) test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False, num_workers=8) trainer = MethylBertFinetuneTrainer( vocab_size=len(tokenizer), save_path="model/bert.model/", train_dataloader=train_loader, test_dataloader=test_loader, lr=1e-4, beta=(0.9, 0.98), weight_decay=0.01, with_cuda=torch.cuda.is_available(), log_freq=100, eval_freq=10, gradient_accumulation_steps=4, max_grad_norm=1.0, warmup_step=100, decrease_steps=200, loss="focal_bce" ) trainer.load("hanyangii/methylbert_hg19_12l") # Or create model without pre-training # trainer.create_model(config_file="hanyangii/methylbert_hg19_6l") trainer.train(steps=600) trainer.save("model/final_model/") ``` -------------------------------- ### BAM/SAM Read with XM Tag Example Source: https://github.com/compepigen/methylbert/blob/main/tutorials/01_Data_Preparation.md Illustrates a BAM/SAM file line showing read-level methylation calls using the XM tag. Ensure your BAM/SAM files are Bismark-aligned and contain XM tags for methylation status. ```text SRR5390326.sra.2060072_2060072_length=150 16 chr1 3000485 42 118M * 0 0 AATTTCAACTCTAAATTTAATTATTTCCTACTATCTACTCATCTTAAATAAATTTACTTCCTTTTATTCTAAAACTTCTAAATTTACTATCAAACTACTAATATATACTCTAATTTCC JA-FFJJJFJJJJJJJJJJJJJJFJJJJJJJJJFJJJFJJFJJJJJJJJJJJJJJJJFJJJJJJJJJJJJJJJFJJFJFJFJJJFJJJJJJJFJJAJJ methylbert MethylBERT v2.0.1 One option must be given from ['preprocess_finetune', 'finetune', 'deconvolute'] ``` -------------------------------- ### Finetune Model Configuration (JSON) Source: https://context7.com/compepigen/methylbert/llms.txt JSON configuration for the model finetuning process. Includes dataset paths, model architecture parameters, training hyperparameters, and optimization settings. ```json // finetune_config.json { "train_dataset": "data/train_seq.csv", "test_dataset": "data/test_seq.csv", "output_path": "model/", "pretrain": null, "n_encoder": 12, "n_mers": 3, "seq_len": 150, "batch_size": 256, "gradient_accumulation_steps": 4, "steps": 600, "num_workers": 8, "with_cuda": true, "log_freq": 100, "eval_freq": 10, "lr": 0.0001, "adam_weight_decay": 0.01, "adam_beta1": 0.9, "adam_beta2": 0.98, "warm_up": 100, "decrease_steps": 200, "loss": "focal_bce", "seed": 950410 } ``` -------------------------------- ### Create MethylBERT Model Trainer Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Initializes the MethylBertFinetuneTrainer with DataLoader objects and hyperparameters. The trainer is configured for CPU usage by default, and logging frequency is set. Evaluation frequency can be activated by uncommenting the eval_freq parameter. ```python from methylbert.trainer import MethylBertFinetuneTrainer import os trainer = MethylBertFinetuneTrainer(len(tokenizer), save_path=output_path, train_dataloader=train_data_loader, test_dataloader=test_data_loader, lr=1e-4, with_cuda=False, log_freq=1, #eval_freq=10, #activate this only when you want to evaluate the model with test_data_loader warmup_step=3) ``` -------------------------------- ### Train MethylBERT Model Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Initiates the training process for the MethylBERT model for a specified number of steps. Monitor the loss and learning rate for training progress. Note the potential UserWarning regarding CPU autocast. ```python trainer.train(steps=10) ``` -------------------------------- ### Load and Inspect Training Dataset Source: https://context7.com/compepigen/methylbert/llms.txt Loads a training dataset using MethylBertFinetuneDataset and prints its statistics. Ensure 'data/train_seq.csv' and the tokenizer are available. ```python train_dataset = MethylBertFinetuneDataset( f_path="data/train_seq.csv", vocab=tokenizer, seq_len=150, n_cores=10, n_seqs=None # Load all sequences, or specify number to subset ) print(f"Number of sequences: {len(train_dataset)}") print(f"Number of DMRs: {train_dataset.num_dmrs()}") print(f"Class distribution: {train_dataset.ctype_label_count}") sample = train_dataset[0] print(f"DNA sequence shape: {sample['dna_seq'].shape}") # torch.Size([151]) print(f"Methylation pattern shape: {sample['methyl_seq'].shape}") # torch.Size([151]) print(f"DMR label: {sample['dmr_label']}") print(f"Cell type label: {sample['ctype_label']}") # 0=normal, 1=tumour train_loader = DataLoader( train_dataset, batch_size=64, num_workers=8, shuffle=True, pin_memory=True ) ``` -------------------------------- ### Preprocessing Finetune Configuration (JSON) Source: https://context7.com/compepigen/methylbert/llms.txt JSON configuration for the `finetune_data_generate` command. Specifies input files, reference genome, k-mer size, and processing parameters. ```json // preprocess_finetune_config.json { "sc_dataset": null, "input_file": "data/bulk.bam", "f_dmr": "data/dmrs.csv", "output_path": "processed/", "f_ref": "data/genome.fa", "n_mers": 3, "methylcaller": "bismark", "split_ratio": 0.8, "n_dmrs": -1, "n_cores": 50, "seed": 950410, "ignore_sex_chromo": true } ``` -------------------------------- ### Create MethylBERT Model from Config Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Creates a new MethylBERT model instance using a specified configuration file. Ensure the config file path is correct. ```python trainer.create_model(config_file="../test/data/pretrained_model/config.json") ``` -------------------------------- ### Load Fine-tuned MethylBERT Model Source: https://github.com/compepigen/methylbert/blob/main/tutorials/05_tumour_deconvolution.ipynb Initializes the MethylBertFinetuneTrainer and loads a pre-trained MethylBERT model from a specified directory. The `load` function automatically finds necessary model files. ```python from methylbert.trainer import MethylBertFinetuneTrainer restore_dir = "tmp/fine_tune/" trainer = MethylBertFinetuneTrainer(len(tokenizer), train_dataloader=data_loader, test_dataloader=data_loader, ) trainer.load(restore_dir) # Load the fine-tuned MethylBERT model ``` -------------------------------- ### Command Line Interface - Fine-tuning Model Source: https://context7.com/compepigen/methylbert/llms.txt The `finetune` command trains a MethylBERT model on preprocessed methylation data using a pre-trained BERT backbone with the specified number of encoder layers. ```APIDOC ## finetune ### Description Trains a MethylBERT model on preprocessed methylation data using a pre-trained BERT backbone with the specified number of encoder layers. ### Method CLI command ### Endpoint methylbert finetune ### Parameters #### Path Parameters - **config** (string) - Required - Path to a JSON configuration file for fine-tuning. #### Query Parameters - **-c, --csv_train** (string) - Required - Path to the training CSV file. - **-t, --csv_test** (string) - Required - Path to the test CSV file. - **-o, --output** (string) - Required - Output directory for the trained model. - **-l, --layers** (int) - Optional - Number of encoder layers in the BERT backbone (e.g., 2, 4, 6, 8, 12). Default is 12. - **-s, --seq_len** (int) - Optional - Maximum sequence length for training (default: 150). - **-b, --batch_size** (int) - Optional - Batch size for training (default: 256). - **--gradient_accumulation_steps** (int) - Optional - Number of steps for gradient accumulation (default: 4). - **-e, --epochs** (int) - Optional - Number of training epochs (default: 600). - **-w, --workers** (int) - Optional - Number of data loader workers (default: 8). - **--log_freq** (int) - Optional - Frequency of logging training information (default: 1). - **--eval_freq** (int) - Optional - Frequency of evaluation during training (default: 1). - **--warm_up** (int) - Optional - Number of warm-up steps for learning rate scheduler (default: 1). - **--lr** (float) - Optional - Learning rate (default: 1e-4). - **--decrease_steps** (int) - Optional - Number of steps after which to decrease learning rate (default: 200). - **--with_cuda** (boolean) - Optional - Flag to enable CUDA (GPU) training. - **--loss** (string) - Optional - Loss function to use (e.g., `focal_bce`). ### Request Example ```bash # Fine-tune with 12 encoder blocks (largest model) methylbert finetune \ -c data/train_seq.csv \ -t data/test_seq.csv \ -o model/ \ -l 12 \ -s 150 \ -b 256 \ --gradient_accumulation_steps 4 \ -e 600 \ -w 8 \ --log_freq 1 \ --eval_freq 1 \ --warm_up 1 \ --lr 1e-4 \ --decrease_steps 200 \ --with_cuda # Fine-tune with focal loss for imbalanced datasets methylbert finetune \ -c data/train_seq.csv \ -t data/test_seq.csv \ -o model/ \ -l 6 \ --loss focal_bce \ -e 600 \ --with_cuda # Using a JSON configuration file methylbert finetune config/finetune_config.json ``` ### Response This command does not produce a direct response in terms of data output. It saves the trained model checkpoints in the specified output directory. ``` -------------------------------- ### Display BERT Base Uncased Configuration Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Displays the configuration file for the baseline BERT model. This is useful for understanding the model's architecture and parameters before creating a new MethylBERT model. ```bash cat ../test/data/config.json ``` -------------------------------- ### List files in tmp directory Source: https://github.com/compepigen/methylbert/blob/main/tutorials/02_Preprocessing_training_data.ipynb Use this command to list the contents of the temporary directory where preprocessed data is stored. ```bash ls tmp/ ``` -------------------------------- ### Initialize MethylVocab for DNA Sequence Tokenization Source: https://context7.com/compepigen/methylbert/llms.txt Creates a k-mer vocabulary for tokenizing DNA sequences, including special tokens for padding, unknown, and sequence boundaries. Supports converting between DNA k-mers and token indices. ```python from methylbert.data.vocab import MethylVocab # Create a 3-mer vocabulary (default) tokenizer = MethylVocab(k=3) print(f"Vocabulary size: {len(tokenizer)}") # 69 tokens (64 k-mers + 5 special) # Convert DNA sequence to token indices dna_kmers = ["ATG", "TGC", "GCA", "CAT"] token_indices = tokenizer.to_seq(dna_kmers) print(f"Token indices: {token_indices}") # [9, 56, 33, 17] # Convert token indices back to k-mers decoded = tokenizer.from_seq(token_indices, join=True) print(f"Decoded sequence: {decoded}") # "ATG TGC GCA CAT" # Access special token indices print(f"PAD index: {tokenizer.pad_index}") # 0 print(f"UNK index: {tokenizer.unk_index}") # 1 print(f"EOS index: {tokenizer.eos_index}") # 2 print(f"SOS index: {tokenizer.sos_index}") # 3 print(f"MASK index: {tokenizer.mask_index}") # 4 ``` -------------------------------- ### Load Pre-trained Model from Local Directory Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Loads a pre-trained MethylBERT model from a local directory containing 'config.json' and 'pytorch_model.bin' files. This method is useful if you have downloaded or saved models locally. ```bash ls ../test/data/pretrained_model/ ``` ```python trainer.load("../test/data/pretrained_model/") ``` -------------------------------- ### MethylBertFinetuneDataset Usage Source: https://context7.com/compepigen/methylbert/llms.txt Demonstrates loading and inspecting a training dataset using MethylBertFinetuneDataset. ```APIDOC ## Python API - MethylBertFinetuneDataset ### Description This section shows how to load and inspect the training dataset for MethylBERT. ### Usage Example ```python from methylbert.data.vocab import MethylVocab from methylbert.data.dataset import MethylBertFinetuneDataset # Load training dataset tokenizer = MethylVocab(k=3) train_dataset = MethylBertFinetuneDataset( f_path="data/train_seq.csv", vocab=tokenizer, seq_len=150, n_cores=10, n_seqs=None # Load all sequences, or specify number to subset ) print(f"Number of sequences: {len(train_dataset)}") print(f"Number of DMRs: {train_dataset.num_dmrs()}") print(f"Class distribution: {train_dataset.ctype_label_count}") # Access a single sample sample = train_dataset[0] print(f"DNA sequence shape: {sample['dna_seq'].shape}") # torch.Size([151]) print(f"Methylation pattern shape: {sample['methyl_seq'].shape}") # torch.Size([151]) print(f"DMR label: {sample['dmr_label']}") print(f"Cell type label: {sample['ctype_label']}") # 0=normal, 1=tumour # Create DataLoader for training from torch.utils.data import DataLoader train_loader = DataLoader( train_dataset, batch_size=64, num_workers=8, shuffle=True, pin_memory=True ) ``` ``` -------------------------------- ### Load and display preprocessed data with pandas Source: https://github.com/compepigen/methylbert/blob/main/tutorials/02_Preprocessing_training_data.ipynb Load the 'test_seq.csv' file using pandas, specifying tab as the separator. Displays the first 5 rows of the loaded data. ```python import pandas as pd pd.read_csv("tmp/test_seq.csv", sep='\t').head() ``` -------------------------------- ### MethylBERT Command Line Tool Source: https://github.com/compepigen/methylbert/blob/main/README.md MethylBERT provides a command-line interface for various tasks. Use -h or --help for detailed arguments for each function. ```APIDOC ## MethylBERT Command Line Tool ### Description MethylBERT supports a command line tool for various functions including data preprocessing, fine-tuning, and deconvolution. ### Usage ``` > methylbert MethylBERT v2.0.1 One option must be given from ['preprocess_finetune', 'finetune', 'deconvolute'] ``` ### Available Commands - `preprocess_finetune`: Preprocesses data for fine-tuning MethylBERT. - `finetune`: Fine-tunes the MethylBERT model. - `deconvolute`: Performs tumor deconvolution using MethylBERT. ### Command Help Use `-h` or `--help` to get detailed arguments for each function. For example: ``` > methylbert preprocess_finetune --help ``` ``` -------------------------------- ### Generate Finetuning Data from Single-Cell BAM Files Source: https://context7.com/compepigen/methylbert/llms.txt Processes multiple single-cell BAM files with cell type labels to generate datasets for fine-tuning. Specify a file listing BAM paths and their corresponding cell types. ```python # sc_samples.txt format: # /path/to/cell1.bam Tumour # /path/to/cell2.bam Normal df_reads = finetune_data_generate( f_dmr="data/dmrs.csv", output_dir="processed_data/", f_ref="genome.fa", sc_dataset="sc_samples.txt", train_valid_test_ratio=[0.7, 0.15, 0.15], # Train/valid/test split n_cores=50, seed=950410, ignore_sex_chromo=True, methyl_caller="bismark" ) ``` -------------------------------- ### Display BAM list file content Source: https://github.com/compepigen/methylbert/blob/main/tutorials/02_Preprocessing_training_data.ipynb Use `cat` to display the content of the BAM list file, which specifies tumour and normal samples. ```bash cat ../test/data/bam_list.txt ``` -------------------------------- ### Command Line Interface - Preprocessing Data Source: https://context7.com/compepigen/methylbert/llms.txt The `preprocess_finetune` command extracts DNA methylation sequences from BAM files overlapping with specified DMRs and prepares training/test datasets for model fine-tuning. ```APIDOC ## preprocess_finetune ### Description Extracts DNA methylation sequences from BAM files overlapping with specified DMRs and prepares training/test datasets for model fine-tuning. ### Method CLI command ### Endpoint methylbert preprocess_finetune ### Parameters #### Path Parameters - **config** (string) - Required - Path to a JSON configuration file for preprocessing. #### Query Parameters - **-f, --file** (string) - Required - Path to a single BAM file. - **-s, --samples** (string) - Required - Path to a text file containing paths to multiple single-cell BAM files. - **-d, --dmrs** (string) - Required - Path to a CSV file containing Differentially Methylated Regions (DMRs). - **-r, --ref** (string) - Required - Path to the reference genome FASTA file. - **-p, --padding** (float) - Optional - Padding value for methylation sequences (default: 0.8). - **-c, --chunksize** (int) - Optional - Chunk size for processing BAM files (default: 50). - **-o, --output** (string) - Required - Output directory for processed data. - **--ignore_sex_chromo** (boolean) - Optional - Flag to ignore sex chromosomes. ### Request Example ```bash # Basic preprocessing with a single BAM file methylbert preprocess_finetune \ -f bulk.bam \ -d dmrs.csv \ -r genome.fa \ -p 0.8 \ -c 50 \ -o data/ # Preprocessing with multiple single-cell BAM files methylbert preprocess_finetune \ -s sc_samples.txt \ -d dmrs.csv \ -r genome.fa \ -p 0.8 \ -c 50 \ -o data/ \ --ignore_sex_chromo # Using a JSON configuration file methylbert preprocess_finetune config/preprocess_finetune_config.json ``` ### Response This command does not produce a direct response in terms of data output. It generates processed data files in the specified output directory. ``` -------------------------------- ### Generate Finetune Data with MethylBERT Source: https://github.com/compepigen/methylbert/blob/main/tutorials/03_Preprocessing_bulk_data.ipynb Use `finetune_data_generate` to preprocess bulk BAM and DMR files for MethylBERT fine-tuning. Specify input files, reference genome, output directory, n-mers, and number of cores. ```python from methylbert.data import finetune_data_generate as fdg f_bam = "../test/data/bulk.bam" f_dmr = "../test/data/dmrs.csv" f_ref = "../../../genome/hg19.fa" out_dir = "tmp/" fdg.finetune_data_generate( input_file = f_bam, f_dmr = f_dmr, f_ref = f_ref, output_dir=out_dir, n_mers=3, # 3-mer DNA sequences n_cores=20 ) ``` -------------------------------- ### MethylBERT Preprocess Finetune Command Source: https://github.com/compepigen/methylbert/blob/main/README.md This command preprocesses data for fine-tuning MethylBERT. It takes BAM files, DMR information, and reference genome as input. ```APIDOC ## preprocess_finetune ### Description Preprocesses data for fine-tuning MethylBERT. This involves preparing input files in the required format. ### Method Command Line Tool ### Endpoint `methylbert preprocess_finetune` ### Parameters #### Command Line Arguments - **-f, --input_file** (string) - Required - Path to the input .bam file. - **-d, --f_dmr** (string) - Required - Path to the .bed or .csv file containing DMRs information. - **-r, --f_ref** (string) - Required - Path to the reference genome .fasta file. - **-o, --output_path** (string) - Required - Directory where all generated results will be saved. - **-s, --sc_dataset** (string) - Optional - A file listing all single-cell bam files. The first and second columns must indicate file names and cell types if cell types are given. Otherwise, each line must have one file path. - **-nm, --n_mers** (integer) - Optional - K for K-mer sequences (default: 3). - **-m, --methylcaller** (string) - Optional - Used methylation caller. It must be either 'bismark' or 'dorado' (default: bismark). - **-p, --split_ratio** (float) - Optional - The ratio between train and test dataset (default: 0.8). - **-nd, --n_dmrs** (integer) - Optional - Number of DMRs to take from the dmr file. If the value is not given, all DMRs will be used. - **-c, --n_cores** (integer) - Optional - Number of cores for multiprocessing (default: 1). - **--seed** (integer) - Optional - Random seed number (default: 950410). - **--ignore_sex_chromo** (boolean) - Optional - Whether DMRs at sex chromosomes (chrX and chrY) will be ignored (default: True). ### Request Example ```bash methylbert preprocess_finetune -f bulk.bam -d dmrs.csv -r genome.fa -p 0.8 -c 50 -o data/ ``` ### Response This command does not produce a direct API response. It generates output files in the specified output directory. ``` -------------------------------- ### MethylBERT Fine-tuning Command Source: https://github.com/compepigen/methylbert/blob/main/README.md Use this command to fine-tune a MethylBERT model. Specify training and testing datasets, output path, and various training hyperparameters. ```bash methylbert finetune -c data/train_seq.csv -t data/test_seq.csv -o model/ -l 12 -s 150 -b 256 --gradient_accumulation_steps 4 -e 600 -w 8 --log_freq 1 --eval_freq 1 --warm_up 1 --lr 1e-4 --decrease_steps 200 ``` -------------------------------- ### Set Random Seed and Configuration Source: https://github.com/compepigen/methylbert/blob/main/tutorials/05_tumour_deconvolution.ipynb Sets the random seed for reproducibility and defines configuration parameters like sequence length, n-mers, batch size, and output path. ```python from methylbert.utils import set_seed set_seed(42) seq_len=150 n_mers=3 batch_size=5 num_workers=20 output_path="tmp/deconvolution/" ``` -------------------------------- ### Run MethylBERT Deconvolution Source: https://github.com/compepigen/methylbert/blob/main/tutorials/05_tumour_deconvolution.ipynb Initiates the deconvolution process with specified training data, trainer, tokenizer, data loader, and output path. Ensure the training data is provided as a pandas DataFrame. ```python import pandas as pd from methylbert.deconvolute import deconvolute deconvolute(trainer = trainer, tokenizer = tokenizer, data_loader = data_loader, output_path = output_path, df_train = pd.read_csv("tmp/train_seq.csv", sep="\t")) ``` -------------------------------- ### Python API - MethylVocab Source: https://context7.com/compepigen/methylbert/llms.txt The `MethylVocab` class creates a k-mer vocabulary for tokenizing DNA sequences with special tokens for padding, unknown, end-of-sequence, start-of-sequence, and masking. ```APIDOC ## MethylVocab ### Description Creates a k-mer vocabulary for tokenizing DNA sequences with special tokens for padding, unknown, end-of-sequence, start-of-sequence, and masking. ### Method Python Class ### Endpoint from methylbert.data.vocab import MethylVocab ### Parameters #### Initialization Parameters - **k** (int) - Optional - The size of the k-mer (default: 3). ### Methods - **to_seq(kmers: list[str]) -> list[int]**: Converts a list of k-mers to their corresponding token indices. - **from_seq(indices: list[int], join: bool = False) -> str or list[str]**: Converts a list of token indices back to k-mers. If `join` is True, returns a single string; otherwise, returns a list of k-mers. ### Properties - **pad_index** (int): Index for the padding token. - **unk_index** (int): Index for the unknown token. - **eos_index** (int): Index for the end-of-sequence token. - **sos_index** (int): Index for the start-of-sequence token. - **mask_index** (int): Index for the mask token. ### Request Example ```python from methylbert.data.vocab import MethylVocab # Create a 3-mer vocabulary (default) tokenizer = MethylVocab(k=3) print(f"Vocabulary size: {len(tokenizer)}") # Convert DNA sequence to token indices dna_kmers = ["ATG", "TGC", "GCA", "CAT"] token_indices = tokenizer.to_seq(dna_kmers) print(f"Token indices: {token_indices}") # Convert token indices back to k-mers decoded = tokenizer.from_seq(token_indices, join=True) print(f"Decoded sequence: {decoded}") # Access special token indices print(f"PAD index: {tokenizer.pad_index}") print(f"UNK index: {tokenizer.unk_index}") print(f"EOS index: {tokenizer.eos_index}") print(f"SOS index: {tokenizer.sos_index}") print(f"MASK index: {tokenizer.mask_index}") ``` ### Response - **Vocabulary size** (int) - The total number of tokens in the vocabulary. - **Token indices** (list[int]) - A list of integer indices corresponding to the input k-mers. - **Decoded sequence** (str) - The DNA sequence reconstructed from token indices. - **PAD index** (int) - The index for the padding token. - **UNK index** (int) - The index for the unknown token. - **EOS index** (int) - The index for the end-of-sequence token. - **SOS index** (int) - The index for the start-of-sequence token. - **MASK index** (int) - The index for the mask token. ``` -------------------------------- ### Generate MethylBERT fine-tuning data Source: https://github.com/compepigen/methylbert/blob/main/tutorials/02_Preprocessing_training_data.ipynb The `finetune_data_generate` function preprocesses BAM files and DMRs to create training and validation datasets. Specify the BAM list file, DMR CSV file, reference genome FASTA file, and output directory. Optional parameters include `split_ratio`, `n_mers`, and `n_cores`. ```python from methylbert.data import finetune_data_generate as fdg f_bam_file_list = "../test/data/bam_list.txt" f_dmr = "../test/data/dmrs.csv" f_ref = "../../../genome/hg19.fa" out_dir = "tmp/" fdg.finetune_data_generate( sc_dataset = f_bam_file_list, f_dmr = f_dmr, f_ref = f_ref, output_dir=out_dir, split_ratio = 0.8, # Split ratio to make training and validation data n_mers=3, # 3-mer DNA sequences n_cores=20 ) ``` -------------------------------- ### Load Training and Evaluation Data Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Loads preprocessed data into PyTorch DataLoader objects for training and testing. A tokenizer is created using MethylVocab, and datasets are instantiated with specified sequence lengths. DataLoader parameters like batch size and number of workers can be adjusted. ```python from torch.utils.data import DataLoader from methylbert.data.vocab import MethylVocab from methylbert.data.dataset import MethylBertFinetuneDataset # Creat a look-up table tokenizer = MethylVocab(n_mers) # Load the data files int a data set object train_dataset = MethylBertFinetuneDataset("tmp/train_seq.csv", tokenizer, seq_len=seq_len) test_dataset = MethylBertFinetuneDataset("tmp/test_seq.csv", tokenizer,seq_len=seq_len) # Load the data into a data loader train_data_loader = DataLoader(train_dataset, batch_size= batch_size, num_workers= num_workers, pin_memory=False, shuffle=True) test_data_loader = DataLoader(test_dataset, batch_size= batch_size, num_workers= num_workers, pin_memory=True, shuffle=False) ``` -------------------------------- ### Read and Inspect Preprocessed Data Source: https://github.com/compepigen/methylbert/blob/main/tutorials/03_Preprocessing_bulk_data.ipynb Read the generated `data.csv` file using pandas to inspect the preprocessed bulk data. The `head()` method displays the first few rows of the DataFrame. ```python import pandas as pd pd.read_csv("tmp/data.csv", sep="\t").head() ``` -------------------------------- ### Load Pre-trained Model from Hugging Face Source: https://github.com/compepigen/methylbert/blob/main/tutorials/04_Fine-tuning_MethylBERT_model.ipynb Loads a pre-trained MethylBERT model from Hugging Face using a specified model identifier. The model can have varying numbers of encoder blocks (e.g., 4l for 4 layers). Some weights may be newly initialized and require training on a downstream task. ```python trainer.load("hanyangii/methylbert_hg19_4l") ``` -------------------------------- ### Fine-tune MethylBERT Model Source: https://context7.com/compepigen/methylbert/llms.txt Trains a MethylBERT model using preprocessed data. Supports specifying the number of encoder layers, loss function (including focal loss), and various training parameters. Can be configured via JSON. ```bash methylbert finetune \ -c data/train_seq.csv \ -t data/test_seq.csv \ -o model/ \ -l 12 \ -s 150 \ -b 256 \ --gradient_accumulation_steps 4 \ -e 600 \ -w 8 \ --log_freq 1 \ --eval_freq 1 \ --warm_up 1 \ --lr 1e-4 \ --decrease_steps 200 \ --with_cuda ``` ```bash methylbert finetune \ -c data/train_seq.csv \ -t data/test_seq.csv \ -o model/ \ -l 6 \ --loss focal_bce \ -e 600 \ --with_cuda ``` ```bash methylbert finetune config/finetune_config.json ```