### Install Packages from Requirements File Source: https://context7.com/magics-lab/dnabert_2/llms.txt Alternatively, install all required packages by running pip install with a requirements.txt file. ```bash pip install -r requirements.txt ``` -------------------------------- ### Setup Environment for DNABERT-2 Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Create and activate a Conda virtual environment for Python 3.8. Optionally, install Triton for flash attention. Finally, install the required packages from the requirements.txt file. ```bash conda create -n dna python=3.8 conda activate dna # (optional if you would like to use flash attention) # install triton from source git clone https://github.com/openai/triton.git; cd triton/python; pip install cmake; # build-time dependency pip install -e . # install required packages python3 -m pip install -r requirements.txt ``` -------------------------------- ### Install Triton for Flash Attention Source: https://context7.com/magics-lab/dnabert_2/llms.txt Optional steps to clone the Triton repository, navigate to its Python directory, install build-time dependencies, and then install Triton. This is for Flash Attention support. ```bash git clone https://github.com/openai/triton.git cd triton/python pip install cmake # Build-time dependency pip install -e . cd ../.. ``` -------------------------------- ### Distributed Training Setup (DistributedDataParallel) Source: https://context7.com/magics-lab/dnabert_2/llms.txt Environment variable setup for scaling DNABERT-2 training across multiple GPUs using PyTorch's DistributedDataParallel. This configuration is for multi-GPU efficiency. ```bash # Set number of GPUs export num_gpu=4 export DATA_PATH=./sample_data export MAX_LENGTH=100 export LR=3e-5 ``` -------------------------------- ### Install Required Python Packages Source: https://context7.com/magics-lab/dnabert_2/llms.txt Install essential Python packages for DNABERT-2 using pip. Ensure you have the correct versions as specified. ```bash pip install einops==0.6.1 pip install transformers==4.29.2 pip install peft==0.3.0 pip install omegaconf==2.3.0 pip install torch==1.13.1 pip install evaluate==0.4.0 pip install accelerate==0.20.3 pip install scikit-learn==1.2.2 ``` -------------------------------- ### Fine-tune DNABERT-2 with DistributedDataParallel Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md This command fine-tunes DNABERT-2 using DistributedDataParallel for more efficient multi-GPU training. Set num_gpu according to your hardware setup. ```python # Training use DistributedDataParallel (more efficient) export num_gpu=4 # please change the value based on your setup torchrun --nproc_per_node=${num_gpu} train.py \ --model_name_or_path zhihan1996/DNABERT-2-117M \ --data_path ${DATA_PATH} \ --kmer -1 \ --run_name DNABERT2_${DATA_PATH} \ --model_max_length ${MAX_LENGTH} \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 16 \ --gradient_accumulation_steps 1 \ --learning_rate ${LR} \ --num_train_epochs 5 \ --fp16 \ --save_steps 200 \ --output_dir output/dnabert2 \ --evaluation_strategy steps \ --eval_steps 200 \ --warmup_steps 50 \ --logging_steps 100 \ --overwrite_output_dir True \ --log_level info \ --find_unused_parameters False ``` -------------------------------- ### Fine-tune Nucleotide Transformer with LoRA Source: https://context7.com/magics-lab/dnabert_2/llms.txt Example of fine-tuning a Nucleotide Transformer model using Low-Rank Adaptation (LoRA). This method is efficient for large models and requires specifying LoRA parameters. ```python python train.py \ --model_name_or_path InstaDeepAI/nucleotide-transformer-500m-1000g \ --data_path ./sample_data \ --kmer -1 \ --run_name NT_lora_custom \ --model_max_length 100 \ --use_lora \ --lora_r 8 \ --lora_alpha 16 \ --lora_dropout 0.05 \ --lora_target_modules 'query,value,key,dense' \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 16 \ --gradient_accumulation_steps 1 \ --learning_rate 1e-4 \ --num_train_epochs 3 \ --fp16 \ --save_steps 200 \ --output_dir output/nt_lora \ --evaluation_strategy steps \ --eval_steps 200 \ --warmup_steps 50 \ --logging_steps 100 \ --overwrite_output_dir True \ --log_level info \ --find_unused_parameters False ``` -------------------------------- ### Calculate DNA Sequence Embedding Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Tokenize a DNA sequence, obtain its input IDs, and pass them through the DNABERT-2 model to get hidden states. Calculate embeddings using mean or max pooling. ```python dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC" inputs = tokenizer(dna, return_tensors = 'pt')["input_ids"] hidden_states = model(inputs)[0] # [1, sequence_length, 768] # embedding with mean pooling embedding_mean = torch.mean(hidden_states[0], dim=0) print(embedding_mean.shape) # expect to be 768 # embedding with max pooling embedding_max = torch.max(hidden_states[0], dim=0)[0] print(embedding_max.shape) # expect to be 768 ``` -------------------------------- ### Data Folder Structure Source: https://context7.com/magics-lab/dnabert_2/llms.txt Organize your training, validation, and test data into separate CSV files within a designated data folder. The expected files are `train.csv`, `dev.csv`, and `test.csv`. ```text # File structure required: data_folder/ ├── train.csv # Training data ├── dev.csv # Validation data (for model selection) └── test.csv # Test data (final evaluation) ``` -------------------------------- ### Fine-tune DNABERT-2 with train.py (DataParallel) Source: https://context7.com/magics-lab/dnabert_2/llms.txt Command-line interface usage for fine-tuning DNABERT-2 on custom datasets using DataParallel training. Requires CSV files with 'sequence' and 'label' columns. Supports LoRA fine-tuning. ```bash # Set environment variables export DATA_PATH=./sample_data export MAX_LENGTH=100 # 0.25 * sequence length (e.g., 250 for 1000bp sequences) export LR=3e-5 # Fine-tune DNABERT-2 on custom dataset with DataParallel python train.py \ --model_name_or_path zhihan1996/DNABERT-2-117M \ --data_path ${DATA_PATH} \ --kmer -1 \ --run_name DNABERT2_custom \ --model_max_length ${MAX_LENGTH} \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 16 \ --gradient_accumulation_steps 1 \ --learning_rate ${LR} \ --num_train_epochs 5 \ --fp16 \ --save_steps 200 \ --output_dir output/dnabert2 \ --evaluation_strategy steps \ --eval_steps 200 \ --warmup_steps 50 \ --logging_steps 100 \ --overwrite_output_dir True \ --log_level info \ --find_unused_parameters False ``` ```text # Example data format (train.csv): # sequence,label # AAAAAGCCTGTGAAGCACAGAGAGCAGCCAGCCAGAGCTGATGCTCAATGGCAGAAACTGCTTAGTCACGCTGAAAGGGAGCCAAGGCAATAGCAGAGTGG,1 # ACCTGCTAACAATTAAGGCCTCCAGGTCTACCCTGCAGCTGGGCCTGAGGAGGTCCTCTTGAAAGGAGTGGGTAACAGCGCACTATTGAGGGCCTGTGAAG,0 ``` -------------------------------- ### Evaluate DNABERT-2 on GUE Benchmark Source: https://context7.com/magics-lab/dnabert_2/llms.txt Execute this script to perform a comprehensive evaluation of DNABERT-2 on all 28 GUE benchmark tasks. Ensure the GUE dataset is downloaded and the DATA_PATH environment variable is set correctly. ```bash # Download GUE dataset first from: # https://drive.google.com/file/d/1uOrwlf07qGQuruXqGXWMpPn8avBoW7T-/view?usp=sharing export DATA_PATH=/path/to/GUE cd finetune # Evaluate DNABERT-2 on all GUE tasks sh scripts/run_dnabert2.sh ${DATA_PATH} ``` -------------------------------- ### Training Arguments Reference Source: https://context7.com/magics-lab/dnabert_2/llms.txt Reference for key training arguments used in `train.py`. This includes arguments for model selection, LoRA configuration, data handling, and general training parameters. ```python # ModelArguments # --model_name_or_path: Pre-trained model (default: "facebook/opt-125m") # --use_lora: Enable LoRA fine-tuning (default: False) # --lora_r: LoRA hidden dimension (default: 8) # --lora_alpha: LoRA alpha parameter (default: 32) # --lora_dropout: LoRA dropout rate (default: 0.05) # --lora_target_modules: Comma-separated modules for LoRA (default: "query,value") # DataArguments # --data_path: Path to folder containing train.csv, dev.csv, test.csv # --kmer: K-mer size for DNABERT-1 (-1 for DNABERT-2 BPE tokenization) # TrainingArguments (extends transformers.TrainingArguments) # --model_max_length: Maximum sequence length (default: 512) # --per_device_train_batch_size: Batch size per GPU for training (default: 1) # --per_device_eval_batch_size: Batch size per GPU for evaluation (default: 1) # --gradient_accumulation_steps: Gradient accumulation steps (default: 1) # --learning_rate: Learning rate (default: 1e-4) # --num_train_epochs: Number of training epochs (default: 1) # --fp16: Enable mixed precision training (default: False) # --save_steps: Save checkpoint every N steps (default: 100) # --eval_steps: Evaluate every N steps (default: 100) # --warmup_steps: Warmup steps for scheduler (default: 50) # --output_dir: Output directory (default: "output") # --save_model: Save final model (default: False) # --eval_and_save_results: Evaluate on test set and save results (default: True) # --log_level: Set the logging level (default: info) # --find_unused_parameters: Find unused parameters (default: False) ``` -------------------------------- ### Evaluate DNABERT-1 with K-mer Tokenization Source: https://context7.com/magics-lab/dnabert_2/llms.txt Run evaluations for DNABERT-1 using different k-mer sizes (3, 4, 5, or 6). This allows comparison of k-mer based tokenization performance against other models. ```bash # Evaluate DNABERT-1 with k-mer tokenization (3, 4, 5, or 6-mer) sh scripts/run_dnabert1.sh ${DATA_PATH} 3 # 3-mer sh scripts/run_dnabert1.sh ${DATA_PATH} 6 # 6-mer ``` -------------------------------- ### Fine-tune DNABERT-2 with DistributedDataParallel Source: https://context7.com/magics-lab/dnabert_2/llms.txt Use this command to fine-tune DNABERT-2 using DistributedDataParallel for efficient training across multiple GPUs. Ensure you have set the necessary environment variables for model path, data path, and learning rate. ```bash torchrun --nproc_per_node=${num_gpu} train.py \ --model_name_or_path zhihan1996/DNABERT-2-117M \ --data_path ${DATA_PATH} \ --kmer -1 \ --run_name DNABERT2_distributed \ --model_max_length ${MAX_LENGTH} \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 16 \ --gradient_accumulation_steps 1 \ --learning_rate ${LR} \ --num_train_epochs 5 \ --fp16 \ --save_steps 200 \ --output_dir output/dnabert2 \ --evaluation_strategy steps \ --eval_steps 200 \ --warmup_steps 50 \ --logging_steps 100 \ --overwrite_output_dir True \ --log_level info \ --find_unused_parameters False ``` -------------------------------- ### Evaluate DNABERT-2 on GUE Dataset Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Run this script to evaluate the DNABERT-2 model on the GUE dataset. Ensure the DATA_PATH environment variable is set correctly. ```bash export DATA_PATH=/path/to/GUE #(e.g., /home/user) cd finetune # Evaluate DNABERT-2 on GUE sh scripts/run_dnabert2.sh DATA_PATH ``` -------------------------------- ### Create and Activate Conda Environment Source: https://context7.com/magics-lab/dnabert_2/llms.txt Use these commands to create a new conda environment named 'dna' with Python 3.8 and activate it. This isolates project dependencies. ```bash conda create -n dna python=3.8 conda activate dna ``` -------------------------------- ### Evaluate Nucleotide Transformers Source: https://context7.com/magics-lab/dnabert_2/llms.txt Evaluate Nucleotide Transformer models with different pre-training configurations. Choose the appropriate configuration index (0-3) based on your needs. ```bash # Evaluate Nucleotide Transformers # 0: 500m-1000g, 1: 500m-human-ref, 2: 2.5b-1000g, 3: 2.5b-multi-species sh scripts/run_nt.sh ${DATA_PATH} 0 # NT-500M-1000g sh scripts/run_nt.sh ${DATA_PATH} 3 # NT-2.5B-multi-species ``` -------------------------------- ### Evaluate DNABERT (3-mer) on GUE Dataset Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Use this script to evaluate a DNABERT model with a 3-mer k-mer size on the GUE dataset. Adjust the second argument for different k-mer sizes (4, 5, or 6). ```bash # Evaluate DNABERT (e.g., DNABERT with 3-mer) on GUE # 3 for 3-mer, 4 for 4-mer, 5 for 5-mer, 6 for 6-mer sh scripts/run_dnabert1.sh DATA_PATH 3 ``` -------------------------------- ### Load DNABERT-2 Model (Transformers > v4.28) Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Load the DNABERT-2 model from Hugging Face for transformers versions greater than 4.28. This involves loading the configuration first, then the model, ensuring trust_remote_code is True. ```python from transformers.models.bert.configuration_bert import BertConfig config = BertConfig.from_pretrained("zhihan1996/DNABERT-2-117M") model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True, config=config) ``` -------------------------------- ### Load DNABERT-2 Model (Transformers v4.28) Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Load the DNABERT-2 model and tokenizer from Hugging Face using the transformers library version 4.28. Ensure trust_remote_code is set to True. ```python import torch from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True) model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True) ``` -------------------------------- ### Evaluate Nucleotide Transformers on GUE Dataset Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md Evaluate Nucleotide Transformers on the GUE dataset. The second argument specifies the model variant (0-3). ```bash # Evaluate Nucleotide Transformers on GUE # 0 for 500m-1000g, 1 for 500m-human-ref, 2 for 2.5b-1000g, 3 for 2.5b-multi-species sh scripts/run_nt.sh DATA_PATH 0 ``` -------------------------------- ### Load DNABERT-2 Model (Transformers v4.28) Source: https://context7.com/magics-lab/dnabert_2/llms.txt Load the pre-trained DNABERT-2 model and tokenizer using Hugging Face Transformers version 4.28 for inference and embedding generation. Supports mean and max pooling for embeddings. ```python import torch from transformers import AutoTokenizer, AutoModel # Load tokenizer and model tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True) model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True) # Example DNA sequence dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC" # Tokenize and get embeddings inputs = tokenizer(dna, return_tensors='pt')["input_ids"] hidden_states = model(inputs)[0] # [1, sequence_length, 768] # Embedding with mean pooling embedding_mean = torch.mean(hidden_states[0], dim=0) print(embedding_mean.shape) # torch.Size([768]) # Embedding with max pooling embedding_max = torch.max(hidden_states[0], dim=0)[0] print(embedding_max.shape) # torch.Size([768]) ``` -------------------------------- ### Load DNABERT-2 Model (Transformers > v4.28) Source: https://context7.com/magics-lab/dnabert_2/llms.txt Load the DNABERT-2 model with an explicit BertConfig for compatibility with Hugging Face Transformers versions newer than 4.28. Supports mean and max pooling for embeddings. ```python import torch from transformers import AutoTokenizer, AutoModel from transformers.models.bert.configuration_bert import BertConfig # Load tokenizer with explicit config for newer transformers versions tokenizer = AutoTokenizer.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True) config = BertConfig.from_pretrained("zhihan1996/DNABERT-2-117M") model = AutoModel.from_pretrained("zhihan1996/DNABERT-2-117M", trust_remote_code=True, config=config) # Example DNA sequence dna = "ACGTAGCATCGGATCTATCTATCGACACTTGGTTATCGATCTACGAGCATCTCGTTAGC" # Tokenize and get embeddings inputs = tokenizer(dna, return_tensors='pt')["input_ids"] hidden_states = model(inputs)[0] # [1, sequence_length, 768] # Embedding with mean pooling embedding_mean = torch.mean(hidden_states[0], dim=0) print(embedding_mean.shape) # torch.Size([768]) # Embedding with max pooling embedding_max = torch.max(hidden_states[0], dim=0)[0] print(embedding_max.shape) # torch.Size([768]) ``` -------------------------------- ### Data Format Specification - Classification Source: https://context7.com/magics-lab/dnabert_2/llms.txt Prepare datasets in CSV format for classification tasks. For single sequence classification, use 'sequence' and 'label' columns. For sequence-pair classification, use 'sequence1', 'sequence2', and 'label'. ```csv sequence,label AAAAAGCCTGTGAAGCACAGAGAGCAGCCAGCCAGAGCTGATGCTCAATGGCAGAAACTGCTTAGTCACGCTGAAAGGGAGCCAAGGCAATAGCAGAGTGG,1 ACCTGCTAACAATTAAGGCCTCCAGGTCTACCCTGCAGCTGGGCCTGAGGAGGTCCTCTTGAAAGGAGTGGGTAACAGCGCACTATTGAGGGCCTGTGAAG,0 ACTGACCATGTGCATCCTCACTGATACCAGTCTTGCCACAGTGTGCCTTGGAAACTCTTTCACAGGCAGTTATGGTCCCTACAGATAGGGGGCAGAGTATG,0 ``` ```python # For sequence-pair classification tasks, use 3 columns: # sequence1,sequence2,label # ACGT...,TGCA...,1 ``` -------------------------------- ### Fine-tune DNABERT-2 with DataParallel Source: https://github.com/magics-lab/dnabert_2/blob/main/README.md This command fine-tunes DNABERT-2 on a custom dataset using DataParallel. Adjust MAX_LENGTH based on your sequence length and ensure DATA_PATH is set. ```python cd finetune export DATA_PATH=$path/to/data/folder # e.g., ./sample_data export MAX_LENGTH=100 # Please set the number as 0.25 * your sequence length. # e.g., set it as 250 if your DNA sequences have 1000 nucleotide bases # This is because the tokenized will reduce the sequence length by about 5 times export LR=3e-5 # Training use DataParallel python train.py \ --model_name_or_path zhihan1996/DNABERT-2-117M \ --data_path ${DATA_PATH} \ --kmer -1 \ --run_name DNABERT2_${DATA_PATH} \ --model_max_length ${MAX_LENGTH} \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 16 \ --gradient_accumulation_steps 1 \ --learning_rate ${LR} \ --num_train_epochs 5 \ --fp16 \ --save_steps 200 \ --output_dir output/dnabert2 \ --evaluation_strategy steps \ --eval_steps 200 \ --warmup_steps 50 \ --logging_steps 100 \ --overwrite_output_dir True \ --log_level info \ --find_unused_parameters False ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.