### Setup Caduceus Environment

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Commands to create the conda environment, prepare directories, and verify the installation.

```bash
# Create conda environment from provided yml file
conda env create -f caduceus_env.yml

# Activate the environment
conda activate caduceus_env

# Create directories for outputs and checkpoints
mkdir outputs
mkdir watch_folder

# Verify installation
python -c "from caduceus import Caduceus, CaduceusConfig; print('Caduceus installed successfully')"
```

--------------------------------

### Launch Pretraining Run

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Command to start the pretraining process using the train module with specific experiment and model configurations.

```bash
python -m train \
  experiment=hg38/hg38 \
  callbacks.model_checkpoint_every_n_steps.every_n_train_steps=500 \
  dataset.max_length=1024 \
  dataset.batch_size=1024 \
  dataset.mlm=true \
  dataset.mlm_probability=0.15 \
  dataset.rc_aug=false \
  model=caduceus \
  model.config.d_model=128 \
  model.config.n_layer=4 \
  model.config.bidirectional=true \
  model.config.bidirectional_strategy=add \
  model.config.bidirectional_weight_tie=true \
  model.config.rcps=true \
  optimizer.lr="8e-3" \
  train.global_batch_size=1024 \
  trainer.max_steps=10000 \
  +trainer.val_check_interval=10000 \
  wandb=null
```

--------------------------------

### CaduceusForMaskedLM for Pre-training

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Utilize the masked language modeling head for pre-training. Input DNA tokens should start at ID 7, and non-masked tokens in the labels should be set to -100 to be ignored by the loss function.

```python
import torch
from caduceus import CaduceusForMaskedLM, CaduceusConfig

config = CaduceusConfig(
    d_model=128,
    n_layer=4,
    vocab_size=12,
    bidirectional=True,
    rcps=False,
    rms_norm=True,
    fused_add_norm=False,
)

model = CaduceusForMaskedLM(config)
model.train()

# Sample input with masked tokens (mask_token_id=3)
input_ids = torch.randint(7, 12, (4, 128))  # DNA tokens start at id 7
labels = input_ids.clone()

# Mask 15% of tokens
mask = torch.rand(input_ids.shape) < 0.15
input_ids[mask] = 3  # Replace with [MASK] token
labels[~mask] = -100  # Ignore non-masked tokens in loss

# Forward pass with labels computes loss automatically
outputs = model(
    input_ids=input_ids,
    labels=labels,
    return_dict=True,
)

print(f"Loss: {outputs.loss.item():.4f}")
print(f"Logits shape: {outputs.logits.shape}")
# Output: torch.Size([4, 128, 12]) - (batch, seq_len, vocab_size)
```

--------------------------------

### Filter dataset by distance to nearest TSS

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Selects data entries that fall within a specified range of distances to the nearest Transcription Start Site (TSS). This function filters PyTorch tensors within a dictionary.

```python
def dataset_tss_filter(data: dict, min_distance: int, max_distance: int) -> dict:
    """Filter the data to items that fall within TSS bucket"""
    distance_mask = ((data["distance_to_nearest_tss"] >= min_distance) 
                     & (data["distance_to_nearest_tss"] <= max_distance))
    new_data = dict()
    for data_key in data.keys():
        new_data[data_key] = data[data_key][distance_mask]

    return new_data
```

--------------------------------

### Create Output and Watch Directories

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Create the 'outputs' and 'watch_folder' directories, which are required for storing saved models and slurm logs, respectively.

```bash
mkdir outputs
mkdir watch_folder
```

--------------------------------

### Instantiate Caduceus Model from Scratch

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Instantiate a Caduceus model from scratch using a configuration, allowing for training on custom data. You can override default configurations as needed.

```python
from transformers import AutoConfig, AutoModelForMaskedLM

# Add any config overrides here, see the `config.json` file on the hub for details.
config_overrides = {}
# See the `Caduceus` collection page on the hub for list of available models.
config = AutoConfig.from_pretrained(
 "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16",
 **config_overrides,
)
model = AutoModelForMaskedLM.from_config(config)
```

--------------------------------

### Instantiate Caduceus Model from Scratch

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Create a new Caduceus model from scratch using a configuration from HuggingFace, with optional parameter overrides. This is useful for custom training scenarios where a pre-trained model is not suitable.

```python
from transformers import AutoConfig, AutoModelForMaskedLM

# Add any config overrides here, see the `config.json` file on the hub for details.
config_overrides = {
    "d_model": 128,      # Model dimension
    "n_layer": 8,        # Number of layers
}

# See the `Caduceus` collection page on the hub for list of available models.
config = AutoConfig.from_pretrained(
    "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16",
    **config_overrides,
)
model = AutoModelForMaskedLM.from_config(config)

print(f"Model parameters: {sum(p.numel() for p in model.parameters()):,}")
```

--------------------------------

### Fine-tune Caduceus on GenomicBenchmarks

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Launches a fine-tuning run for the dummy_mouse_enhancers_ensembl task. Requires paths to the model configuration and pre-trained checkpoint.

```bash
python -m train \
    experiment=hg38/genomic_benchmark \
    callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \
    dataset.dataset_name="dummy_mouse_enhancers_ensembl" \
    dataset.train_val_split_seed=1 \
    dataset.batch_size=256 \
    dataset.rc_aug=false \
    +dataset.conjoin_train=false \
    +dataset.conjoin_test=false \
    loader.num_workers=2 \
    model=caduceus \
    model._name_=dna_embedding_caduceus \
    +model.config_path="<path to model_config.json>" \
    +model.conjoin_test=false \
    +decoder.conjoin_train=true \
    +decoder.conjoin_test=false \
    optimizer.lr="1e-3" \
    trainer.max_epochs=10 \
    train.pretrained_model_path="<path to .ckpt file>" \
    wandb=null
```

--------------------------------

### Activate Conda Environment

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Activate the newly created conda environment to begin using the Caduceus project.

```bash
conda activate caduceus_env
```

--------------------------------

### Pre-training on Human Reference Genome

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Launch pre-training on the HG38 human reference genome using the Hydra configuration system. Ensure the necessary data (HG38 FASTA and BED files) is downloaded first.

```bash
# Download the Human Reference Genome data first
mkdir -p data/hg38/
curl https://storage.googleapis.com/basenji_barnyard2/hg38.ml.fa.gz > data/hg38/hg38.ml.fa.gz
gunzip data/hg38/hg38.ml.fa.gz
curl https://storage.googleapis.com/basenji_barnyard2/sequences_human.bed > data/hg38/human-sequences.bed

# Launch pre-training run
python -m train \
  experiment=hg38/hg38 \
  callbacks.model_checkpoint_every_n_steps.every_n_train_steps=500 \
  dataset.max_length=1024 \
  dataset.batch_size=1024 \
  dataset.mlm=true \
  dataset.mlm_probability=0.15 \
  dataset.rc_aug=false \
  model=caduceus \
  model.config.d_model=128 \
  model.config.n_layer=4 \
  model.config.bidirectional=true \
  model.config.bidirectional_strategy=add \
  model.config.bidirectional_weight_tie=true \
  model.config.rcps=true \
  optimizer.lr="8e-3" \
  train.global_batch_size=1024 \
  trainer.max_steps=10000 \
  +trainer.val_check_interval=10000 \
  wandb=null
```

--------------------------------

### Submit Pretraining Batch Job via Slurm

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Commands to navigate to the script directory and submit a pretraining job to a Slurm cluster.

```bash
cd slurm_scripts
sbatch run_pretrain_caduceus.sh
```

--------------------------------

### Create Conda Environment for Caduceus

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Create a conda environment using the provided `caduceus_env.yml` file to set up the necessary dependencies for the project.

```bash
conda env create -f caduceus_env.yml
```

--------------------------------

### Download Human Reference Genome Data

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Commands to create the data directory and download the required FASTA and BED files for the hg38 genome.

```bash
mkdir -p data/hg38/
curl https://storage.googleapis.com/basenji_barnyard2/hg38.ml.fa.gz > data/hg38/hg38.ml.fa.gz
gunzip data/hg38/hg38.ml.fa.gz  # unzip the fasta file
curl https://storage.googleapis.com/basenji_barnyard2/sequences_human.bed > data/hg38/human-sequences.bed
```

--------------------------------

### Fine-tune Caduceus on Nucleotide Transformer Tasks

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Launches a fine-tuning run for Nucleotide Transformer tasks using environment variables for task-specific configuration.

```bash
python -m train \
    experiment=hg38/nucleotide_transformer \
    callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \
    dataset.dataset_name="${task}" \
    dataset.train_val_split_seed=${seed} \
    dataset.batch_size=${batch_size} \
    dataset.rc_aug="${rc_aug}" \
    +dataset.conjoin_test="${CONJOIN_TEST}" \
    loader.num_workers=2 \
    model._name_=dna_embedding_caduceus \
    +model.config_path="<path to model_config.json>" \
    +model.conjoin_test=false \
    +decoder.conjoin_train=true \
    +decoder.conjoin_test=false \
    optimizer.lr="1e-3" \
    trainer.max_epochs=10 \
    train.pretrained_model_path="<path to .ckpt file>" \
    trainer.max_epochs=20 \
    wandb=null
```

--------------------------------

### Define constants and output paths

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Sets up constants for distance thresholds to TSS, tissue usage flags, SVM regularization parameters, and the output directory for downstream analysis.

```python
DIST_TO_TSS = [[0, 30_000], [30_000, 100_000], [100_000, np.infty]]
USE_TISSUE = [True]  # used as another for loop for fitting SVM, whether to use tissue embed or not
Cs = [1, 5, 10]  # for loop in fitting SVM, inverse of L2 penalty (sklearn hyperparam)
PATH_TO_OUTPUTS = "./outputs/downstream/vep_embeddings"
```

--------------------------------

### Initialize DNAEmbeddingModelCaduceus Configuration

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Sets up the configuration object for the DNA embedding model wrapper used in downstream classification tasks.

```python
import torch
from caduceus.configuration_caduceus import CaduceusConfig
from src.models.sequence.dna_embedding import DNAEmbeddingModelCaduceus

# Create config
config = CaduceusConfig(
    d_model=128,
    n_layer=4,
    vocab_size=12,
    bidirectional=True,
    rcps=True,  # Enable RC equivariance
    rms_norm=True,
    fused_add_norm=False,
)
```

--------------------------------

### Initialize BiMambaWrapper for Bi-directional Processing

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Wraps the Mamba SSM module to enable bi-directional sequence processing with configurable weight tying and combination strategies.

```python
import torch
from caduceus.modeling_caduceus import BiMambaWrapper

# Create bi-directional Mamba wrapper
bi_mamba = BiMambaWrapper(
    d_model=128,
    bidirectional=True,
    bidirectional_strategy="add",      # "add" or "ew_multiply"
    bidirectional_weight_tie=True,     # Tie weights between forward/reverse
    layer_idx=0,
    # Additional Mamba SSM kwargs
    d_state=16,
    d_conv=4,
    expand=2,
)

# Forward pass with bidirectional processing
hidden_states = torch.randn(4, 64, 128)  # (batch, seq_len, d_model)
output = bi_mamba(hidden_states)

print(f"Input shape: {hidden_states.shape}")
print(f"Output shape: {output.shape}")
# Both have shape: torch.Size([4, 64, 128])
```

--------------------------------

### Fine-tune Caduceus on Genomic Benchmarks

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Configures training for specific genomic tasks using the train module with custom dataset and model parameters.

```bash
python -m train \
    experiment=hg38/genomic_benchmark \
    callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \
    dataset.dataset_name="dummy_mouse_enhancers_ensembl" \
    dataset.train_val_split_seed=1 \
    dataset.batch_size=256 \
    dataset.rc_aug=false \
    +dataset.conjoin_train=false \
    +dataset.conjoin_test=false \
    loader.num_workers=2 \
    model=caduceus \
    model._name_=dna_embedding_caduceus \
    +model.config_path="/path/to/pretrained/model_config.json" \
    +model.conjoin_test=false \
    +decoder.conjoin_train=true \
    +decoder.conjoin_test=false \
    optimizer.lr="1e-3" \
    trainer.max_epochs=10 \
    train.pretrained_model_path="/path/to/pretrained/model.ckpt" \
    wandb=null
```

```bash
# Fine-tune on Nucleotide Transformer downstream task
python -m train \
    experiment=hg38/nucleotide_transformer \
    callbacks.model_checkpoint_every_n_steps.every_n_train_steps=5000 \
    dataset.dataset_name="enhancers" \
    dataset.train_val_split_seed=42 \
    dataset.batch_size=128 \
    dataset.rc_aug=false \
    +dataset.conjoin_test=false \
    loader.num_workers=2 \
    model._name_=dna_embedding_caduceus \
    +model.config_path="/path/to/pretrained/model_config.json" \
    +model.conjoin_test=false \
    +decoder.conjoin_train=true \
    +decoder.conjoin_test=false \
    optimizer.lr="1e-3" \
    trainer.max_epochs=20 \
    train.pretrained_model_path="/path/to/pretrained/model.ckpt" \
    wandb=null
```

--------------------------------

### Initialize Caduceus Tokenizer for DNA

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Initialize the CaduceusTokenizer with a specified maximum sequence length and define the DNA characters and their complement mappings. This tokenizer is essential for processing DNA sequences into token IDs for the model.

```python
from caduceus.tokenization_caduceus import CaduceusTokenizer

# Initialize tokenizer with max sequence length
tokenizer = CaduceusTokenizer(
    model_max_length=1024,
    characters=("A", "C", "G", "T", "N"),  # DNA bases
    complement_map={"A": "T", "C": "G", "G": "C", "T": "A", "N": "N"},
)

# Tokenize a DNA sequence
dna_seq = "ATCGATCGNNACGT"
tokens = tokenizer(dna_seq, return_tensors="pt")
print(f"Input IDs: {tokens['input_ids']}")
print(f"Vocab size: {tokenizer.vocab_size}")

# Access complement map for RC operations
print(f"Complement map: {tokenizer.complement_map}")
```

--------------------------------

### Configure SequenceDecoder for Classification

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Set up a classification head for pooling hidden states, supporting both standard and conjoined inputs.

```python
import torch
from src.tasks.decoders import SequenceDecoder

# Create decoder for sequence classification
decoder = SequenceDecoder(
    d_model=128,           # Input dimension from backbone
    d_output=5,            # Number of output classes
    l_output=0,            # 0 means squeeze to single output per sequence
    use_lengths=False,     # Use full sequences
    mode="pool",           # Pooling mode: "last", "first", "pool", "sum"
    conjoin_train=True,    # Average fwd/RC during training
    conjoin_test=True,     # Average fwd/RC during eval
)

# Input from backbone (batch, seq_len, d_model, 2) for conjoined
# or (batch, seq_len, d_model) for standard
hidden_states = torch.randn(4, 64, 128)

# During training (no conjoining needed for standard input)
decoder.train()
output_train = decoder(hidden_states)
print(f"Training output shape: {output_train.shape}")
# Output: torch.Size([4, 5])

# For conjoined input
hidden_conjoined = torch.randn(4, 64, 256)  # d_model * 2 for fwd + RC
output_conjoined = decoder(hidden_conjoined)
print(f"Conjoined output shape: {output_conjoined.shape}")
```

--------------------------------

### Model Name Replacements for Display

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

This dictionary provides replacements for model names to improve their display in reports or visualizations. It includes adding details like dataset size and line breaks for better readability.

```python
model_name_replacement = {
    "Caduceus w/o Equiv.": "Caduceus w/o\nEquiv. (7.7M)",
    "Caduceus-Ph": "Caduceus-Ph\n(7.7M)",
    "Caduceus-PS": "Caduceus-PS\n(7.7M)",

```

--------------------------------

### Configure models for testing embeddings

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Defines a dictionary to configure various embedding models for testing, including paths to pre-computed embeddings, reverse complement augmentation settings, and concatenation strategies.

```python
# Embeddings to test
model_dict = {
    "HyenaDNA": dict(
        embed_path="hyena_downstream-seqlen=131k",
        rc_aug=False,
        conjoin_train=False,
        conjoin_test=False,
        key="concat_avg_ws",
    ),
    "Caduceus-Ph": dict(
        embed_path="caduceus-ph_downstream-seqlen=131k",
        rc_aug=False,
        conjoin_train=False,
        conjoin_test=True,
        key="concat_avg_ws",
    ),
    "Caduceus w/o Equiv.": dict(
        embed_path="caduceus-ph_downstream-seqlen=131k",
        rc_aug=False,
        conjoin_train=False,
        conjoin_test=False,
        key="concat_avg_ws",
    ),
    "Caduceus-PS": dict(
        embed_path="caduceus-ps_downstream-seqlen=131k",
        rc_aug=False,
        conjoin_train=True,
        conjoin_test=False,
        key="concat_avg_ws",
    ),
    "Enformer": dict(
        embed_path="enformer-seqlen=196k",
        rc_aug=False,
        conjoin_train=False,
        conjoin_test=False,
        key="concat_avg_ws",
    ),
    "NTv2": dict(
        embed_path="NTv2_downstream-seqlen=12k",
        rc_aug=False,
        conjoin_train=False,
        conjoin_test=False,
        key="concat_avg_ws",
    ),
}
```

--------------------------------

### Fit and Test SVM Models

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

This script iterates through various datasets and hyperparameters to fit and evaluate SVM models. It handles data loading, filtering by TSS buckets, and includes options for using tissue embeddings. Ensure all necessary libraries (torch, fsspec, sklearn, etc.) and predefined variables (PATH_TO_OUTPUTS, DIST_TO_TSS, USE_TISSUE, Cs, model_dict) are correctly set up before execution.

```python
metrics = {
    "model_name": [],
    "bucket_id": [],
    "use_tissue": [],
    "C": [],
    "seed": [],
    "AUROC": [],
}

for model_name, downstream_kwargs in model_dict.items():
    print(f"********** Gathering results for: {model_name} ********** ")
    embed_path = downstream_kwargs["embed_path"]
    rc_aug = downstream_kwargs["rc_aug"]
    conjoin_train = downstream_kwargs["conjoin_train"]
    conjoin_test = downstream_kwargs["conjoin_test"]
    key = downstream_kwargs["key"]
    
    if "NT" in model_name: assert (rc_aug == False) and (conjoin_train == False) and (conjoin_test == False)
    
    base_embeds_path = PATH_TO_OUTPUTS
    embeds_path = osp.join(base_embeds_path, embed_path)
    
    print(f"Embed Path: {embeds_path}")
    with fsspec.open(osp.join(embeds_path, "train_embeds_combined.pt"), "rb") as f:
        train_val_ds_raw = torch.load(f, map_location="cpu")
        train_val_ds_raw = dataset_nan_filter(train_val_ds_raw, data_key=key)
    with fsspec.open(osp.join(embeds_path, "test_embeds_combined.pt"), "rb") as f:
        test_ds_raw = torch.load(f, map_location="cpu")
        test_ds_raw = dataset_nan_filter(test_ds_raw, data_key=key)
    print(f"Total Train size: {len(train_val_ds_raw[key])},", end=" ")
    print(f"Total Test size: {len(test_ds_raw[key])},", end=" ")
    print(f"Shape: {test_ds_raw[key].shape[1:]}")


    for bucket_id, (min_dist, max_dist) in enumerate(DIST_TO_TSS):
        # Filter data to desired TSS bucket
        train_val_ds_filter = dataset_tss_filter(train_val_ds_raw, min_dist, max_dist)
        test_ds_filter = dataset_tss_filter(test_ds_raw, min_dist, max_dist)
        print(f"- TSS bucket: [{min_dist}, {max_dist}],", end=" ")
        print(f"Train size: {len(train_val_ds_filter[key])},", end=" ")
        print(f"Test size: {len(test_ds_filter[key])}")
    
        for use_tissue in USE_TISSUE:
            for C in Cs:
                for seed in range(1, 6):     
                    # Re-seed for SVM fitting
                    random.seed(seed)
                    np.random.seed(seed)
                    torch.manual_seed(seed)
                    torch.cuda.manual_seed_all(seed)

                    svm_clf = make_pipeline(
                        StandardScaler(),
                        SVC(C=C, random_state=seed),
                    )

                    # Setup Train/Test dataset
                    if conjoin_train:
                        X = np.array(train_val_ds_filter[key])
                        X += np.array(train_val_ds_filter[f"rc_{key}"])
                        X /= 2
                    else:
                        X = np.array(train_val_ds_filter[key])
                    X_with_tissue = np.concatenate(
                        [X, np.array(train_val_ds_filter["tissue_embed"])[..., None]],
                        axis=-1
                    )
                    y = train_val_ds_filter["labels"]
                    if conjoin_train or conjoin_test:
                        X_test = np.array(test_ds_filter[key])
                        X_test += np.array(test_ds_filter[f"rc_{key}"])
                        X_test /= 2
                    else:
                        X_test = np.array(test_ds_filter[key])
                    X_test_with_tissue = np.concatenate(
                        [X_test, np.array(test_ds_filter["tissue_embed"])[..., None]],
                        axis=-1
                    )
                    y_test = test_ds_filter["labels"]

                    print(f"\tFitting SVM ({use_tissue=}, {C=}, {seed=})...", end=" ")
                    
                    mask = np.random.choice(len(X), size=5000, replace= 5000 > len(X) )
                    if use_tissue: 
                        X_train = X_with_tissue[mask]
                        X_test = X_test_with_tissue
                    else: 
                        X_train = X[mask]
                    y_train = y[mask]

                    start = time.time()
                    svm_clf.fit(X_train, y_train)
                    svm_y_pred = svm_clf.predict(X_test)
                    svm_aucroc = roc_auc_score(y_test, svm_y_pred)
                    end = time.time()
                    print(f"Completed! ({end - start:0.3f} s) -", end=" ")
                    print(f"AUROC: {svm_aucroc}")
                     
                    metrics["model_name"] += [model_name]
                    metrics["bucket_id"] += [bucket_id]
                    metrics["use_tissue"] += [use_tissue]
                    metrics["C"] += [C]
                    metrics["seed"] += [seed]
                    metrics["AUROC"] += [svm_aucroc]

```

```python
df_metrics = pd.DataFrame.from_dict(metrics)
df_metrics.to_csv(osp.join(PATH_TO_OUTPUTS, "SVM_results.csv"))
```

--------------------------------

### Import necessary libraries for Caduceus project

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Imports essential Python libraries for data science and machine learning tasks, including data manipulation, plotting, and deep learning frameworks.

```python
import random
import time
from os import path as osp

import fsspec
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import torch
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from tqdm.auto import tqdm
```

--------------------------------

### Load Pre-trained Caduceus Model for Masked Language Modeling

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Use this snippet to load a pre-trained Caduceus model and tokenizer from the Hugging Face hub for masked language modeling tasks. Ensure you specify the correct model name.

```python
from transformers import AutoModelForMaskedLM, AutoTokenizer

# See the `Caduceus` collection page on the hub for list of available models.
model_name = "kuleshov-group/caduceus-ph_seqlen-131k_d_model-256_n_layer-16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
```

--------------------------------

### Configure Caduceus Model Parameters

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Define a CaduceusConfig object to specify model parameters, including dimensions, Mamba SSM settings, normalization, bi-directionality, and reverse-complement parameter sharing (RCPS). The `complement_map` is typically set during tokenizer instantiation.

```python
from caduceus.configuration_caduceus import CaduceusConfig

config = CaduceusConfig(
    # Core model dimensions
    d_model=256,              # Model embedding dimension
    n_layer=16,               # Number of Mamba layers
    vocab_size=12,            # Vocabulary size (DNA bases + special tokens)

    # SSM configuration
    ssm_cfg={
        "d_state": 16,        # SSM state dimension
        "d_conv": 4,          # Convolution kernel size
        "expand": 2,          # Expansion factor for inner dimension
        "dt_rank": "auto",    # Rank of dt projection
        "dt_min": 0.001,
        "dt_max": 0.1,
    },

    # Normalization settings
    rms_norm=True,
    fused_add_norm=True,
    residual_in_fp32=False,
    norm_epsilon=1e-5,

    # Caduceus-specific params for bi-directionality
    bidirectional=True,
    bidirectional_strategy="add",      # "add" or "ew_multiply"
    bidirectional_weight_tie=True,     # Tie forward/reverse weights

    # Reverse-complement parameter sharing
    rcps=True,                         # Enable RCPS equivariance
    complement_map=None,               # Set during instantiation from tokenizer
)

print(f"Config: bidirectional={config.bidirectional}, rcps={config.rcps}")
```

--------------------------------

### Configure RCPSEmbedding for Reverse-Complement Equivariance

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Creates an embedding layer that doubles output dimensionality to support both forward and reverse-complement representations.

```python
import torch
from caduceus.modeling_rcps import RCPSEmbedding

# Define complement mapping for DNA tokens
complement_map = {
    0: 0,   # [CLS] -> [CLS]
    1: 1,   # [SEP] -> [SEP]
    2: 2,   # [BOS] -> [BOS]
    3: 3,   # [MASK] -> [MASK]
    4: 4,   # [PAD] -> [PAD]
    5: 5,   # [RESERVED] -> [RESERVED]
    6: 6,   # [UNK] -> [UNK]
    7: 10,  # A -> T
    8: 9,   # C -> G
    9: 8,   # G -> C
    10: 7,  # T -> A
    11: 11, # N -> N
}

# Create RCPS embedding
rcps_embed = RCPSEmbedding(
    vocab_size=12,
    d_model=64,  # Output will be 2 * d_model = 128
    complement_map=complement_map,
)

# Embed sequences
input_ids = torch.randint(7, 12, (2, 32))  # DNA token ids
embeddings = rcps_embed(input_ids)

print(f"Input shape: {input_ids.shape}")
print(f"Embedding shape: {embeddings.shape}")
# Output: torch.Size([2, 32, 128]) - doubled dimension for fwd + RC
```

--------------------------------

### Format and Aggregate SVM Results

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Reads SVM results from a CSV, renames columns, replaces categorical values, aggregates results by averaging over seeds, and selects the best hyperparameters based on AUROC. Displays the pivoted results and selected hyperparameter details.

```python
# Formatting changes to df
df = pd.read_csv(osp.join(PATH_TO_OUTPUTS, "SVM_results.csv"), index_col=0)
df_display = df.rename(columns={"bucket_id": "Distance to TSS"})
df_display = df_display.replace({"Distance to TSS": {0: "0 - 30k", 1: "30 - 100k", 2: "100k+"}})
df_display = df_display.replace({"model_name": model_name_replacement})

# Take average over seeds
df_display_selected = df_display.groupby(
    ["model_name", "Distance to TSS", "use_tissue", "C"]
).agg(AUROC=("AUROC", np.mean)).reset_index()

# Select best hyperparam by model/bucket
best_ids = df_display_selected.groupby(["model_name", "Distance to TSS"])["AUROC"].idxmax()
df_display_selected = df_display_selected.loc[best_ids.reset_index()["AUROC"].values]
display(
    df_display_selected.pivot(
        index="model_name", columns="Distance to TSS", values="AUROC"
    )[["0 - 30k", "30 - 100k", "100k+"]]
)
display(df_display_selected[["model_name", "Distance to TSS", "C", "use_tissue"]])
```

--------------------------------

### Check file existence using fsspec

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

A utility function to check if a file exists, compatible with various file systems supported by fsspec.

```python
def fsspec_exists(filename: str) -> bool:
    """Check if file exists in manner compatible with fsspec."""
    fs, _ = fsspec.core.url_to_fs(filename)
    return fs.exists(filename)
```

--------------------------------

### Extract DNA Embeddings

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Initialize the Caduceus embedding model and extract hidden states from input sequences.

```python
# Create embedding model
embedding_model = DNAEmbeddingModelCaduceus(
    config=config,
    conjoin_train=False,
    conjoin_test=False,
)
embedding_model.eval()

# Extract embeddings
input_ids = torch.randint(7, 12, (4, 128))

with torch.no_grad():
    hidden_states, _ = embedding_model(input_ids)

# For RCPS models, hidden_states includes both strands stacked
print(f"Hidden states shape: {hidden_states.shape}")
# Output: torch.Size([4, 128, 128, 2]) - last dim separates fwd/RC

# Average both strands for final representation
final_embedding = hidden_states.mean(dim=-1)
print(f"Final embedding shape: {final_embedding.shape}")
```

--------------------------------

### Extract Embeddings with VEP Script

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Use this script to extract embeddings from the pre-trained Caduceus model. It supports distributed data parallelism for faster processing. Ensure the `--rcps` flag is set appropriately based on the model's reverse-complement equivariance.

```bash
torchrun \
    --standalone \
    --nnodes=1 \
    --nproc-per-node=8 \
    vep_embeddings.py \
      --num_workers=2 \
      --seq_len=131072  \
      --bp_per_token=1  \
      --embed_dump_batch_size=1 \
      --name="caduceus-ps_downstream-seqlen=131k"  \
      --model_name_or_path="kuleshov-group/caduceus-ps_seqlen-131k_d_model-256_n_layer-16" \
      --rcps
```

--------------------------------

### CaduceusForSequenceClassification for Downstream Tasks

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Fine-tune Caduceus for sequence classification. Ensure `num_labels` in the config matches the number of classes. The `pooling_strategy` can be adjusted based on requirements.

```python
import torch
from caduceus import CaduceusForSequenceClassification, CaduceusConfig

config = CaduceusConfig(
    d_model=128,
    n_layer=4,
    vocab_size=12,
    bidirectional=True,
    rcps=False,
    rms_norm=True,
    fused_add_norm=False,
    num_labels=5,  # Number of classification classes
)

model = CaduceusForSequenceClassification(
    config,
    pooling_strategy="mean",   # Options: "mean", "max", "first", "last"
    conjoin_train=False,       # Average forward/RC predictions during training
    conjoin_eval=False,        # Average forward/RC predictions during eval
)
model.train()

# Sample classification input
input_ids = torch.randint(7, 12, (8, 256))  # batch_size=8, seq_len=256
labels = torch.randint(0, 5, (8,))  # 5-class labels

outputs = model(
    input_ids=input_ids,
    labels=labels,
    return_dict=True,
)

print(f"Classification loss: {outputs.loss.item():.4f}")
print(f"Logits shape: {outputs.logits.shape}")
# Output: torch.Size([8, 5]) - (batch, num_labels)

# Get predictions
predictions = outputs.logits.argmax(dim=-1)
accuracy = (predictions == labels).float().mean()
print(f"Accuracy: {accuracy.item():.2%}")
```

--------------------------------

### Extract Embeddings for Variant Effect Prediction

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Uses torchrun to perform distributed embedding extraction for variant effect prediction tasks.

```bash
# Extract embeddings using torchrun for distributed processing
torchrun \
    --standalone \
    --nnodes=1 \
    --nproc-per-node=8 \
    vep_embeddings.py \
      --num_workers=2 \
      --seq_len=131072 \
      --bp_per_token=1 \
      --embed_dump_batch_size=1 \
      --name="caduceus-ps_downstream-seqlen=131k" \
      --model_name_or_path="kuleshov-group/caduceus-ps_seqlen-131k_d_model-256_n_layer-16" \
      --rcps  # Use --no-rcps for non-RC-equivariant models
```

--------------------------------

### Caduceus Model Forward Pass

Source: https://context7.com/kuleshov-group/caduceus/llms.txt

Run a forward pass through the Caduceus backbone model to obtain hidden state representations. Ensure `fused_add_norm` is disabled for CPU compatibility.

```python
import torch
from caduceus import Caduceus, CaduceusConfig

# Create config with RCPS enabled
config = CaduceusConfig(
    d_model=128,
    n_layer=4,
    vocab_size=12,
    bidirectional=True,
    bidirectional_strategy="add",
    bidirectional_weight_tie=True,
    rcps=False,  # Set to True for RC equivariance
    rms_norm=True,
    fused_add_norm=False,  # Disable for CPU compatibility
)

model = Caduceus(config)
model.eval()

# Create sample input (batch_size=2, seq_len=64)
input_ids = torch.randint(0, 12, (2, 64))

# Forward pass
with torch.no_grad():
    outputs = model(
        input_ids=input_ids,
        output_hidden_states=True,
        return_dict=True,
    )

print(f"Last hidden state shape: {outputs.last_hidden_state.shape}")
# Output: torch.Size([2, 64, 128]) - (batch, seq_len, d_model)

if outputs.hidden_states:
    print(f"Number of hidden states: {len(outputs.hidden_states)}")
```

--------------------------------

### Caduceus Citation

Source: https://github.com/kuleshov-group/caduceus/blob/main/README.md

Please cite the Caduceus paper using this BibTeX entry if you find the work useful.

```bibtex
@article{schiff2024caduceus,
  title={Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling},
  author={Schiff, Yair and Kao, Chia-Hsiang and Gokaslan, Aaron and Dao, Tri and Gu, Albert and Kuleshov, Volodymyr},
  journal={arXiv preprint arXiv:2403.03234},
  year={2024}
}
```

--------------------------------

### Plot SVM Results by Distance to TSS

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Filters results to selected hyperparameters, then generates a seaborn catplot to visualize AUROC scores across different models and distances to the TSS. Includes displaying bar values and saving the plot.

```python
# Filter results to selected hyperparams
df_plot = pd.merge(
    df_display, df_display_selected,
    on=["model_name", "Distance to TSS", "use_tissue", "C"]
).drop(columns=["AUROC_y"]).rename(columns={"AUROC_x": "AUROC"})

# Plot results by distance to TSS
sns.set_style("whitegrid")
g = sns.catplot(
    data=df_plot,
    x="model_name",
    y="AUROC",
    col="Distance to TSS",
    hue="Distance to TSS",
    kind="bar",
    errorbar="sd",
    height=12,
    aspect=1,
    dodge=False,
    order=list(model_name_replacement.values()),
)
g.set_xticklabels(rotation=60, fontsize=30)
g.set(xlabel="")
g.set(ylim=(0.4, 0.7))
g.set_titles(template="Dist. to TSS: {col_name}", fontsize=40)
g.fig.suptitle("Predicting Effects of Variants on Gene Expression", y=1.1, fontsize=40)
g._legend.remove()
# Display bar values
# (See: https://stackoverflow.com/questions/55586912/seaborn-catplot-set-values-over-the-bars)
for ax in tqdm(g.axes.ravel(), leave=False):
    title = ax.title.get_text()
    ax.set_title(title, fontsize=35)
    for c in tqdm(ax.containers, leave=False):
        labels = [f"{v.get_height():0.3f}" for v in c]
        ax.bar_label(c, labels=labels, label_type="center", color="white", weight="bold", fontsize=24)
plt.show()
g.savefig(osp.join(PATH_TO_OUTPUTS, "SVM_results.png"))
```

--------------------------------

### Filter dataset for NaN values in embeddings

Source: https://github.com/kuleshov-group/caduceus/blob/main/vep_svm.ipynb

Removes entries from a dataset dictionary where embeddings or their reverse complement counterparts contain NaN values. This function operates on PyTorch tensors.

```python
def dataset_nan_filter(data: dict, data_key: str) -> dict:
    """Filter any items that have NaN in embedding within TSS bucket"""
    mask_out = torch.logical_or(
        torch.any(data[data_key].isnan(), dim=1),
        torch.any(data[f"rc_{data_key}"].isnan(), dim=1)
    )
    
    new_data = dict()
    for data_key in data.keys():
        new_data[data_key] = data[data_key][~mask_out]

    return new_data
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.