### Example ChromBPNet Marginal Footprints Command Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/marginal_footprints/README.md An example command demonstrating how to run the marginal footprinting script with specific file paths and parameters. ```bash chrombpnet_marginal_footprints -g /path/to/genome_fasta -r /path/to/bed_file -chr chr1 -m /path/to/model.h5 -o /path/to/output_dir/outputprefix -pwm_f /path/to/motif_to_pwm.tsv ``` -------------------------------- ### Motif to PWM TSV File Content Example Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/marginal_footprints/README.md This is an example of the content expected in the motifs_to_pwm_file. It maps motif names to their corresponding PWM strings. ```text tn5_1 GCACAGTACAGAGCTG tn5_2 GTGCACAGTTCTAGAGTGTGCAG tn5_3 CCTCTACACTGTGCAGAA tn5_4 GCACAGTTCTAGACTGTGCAG tn5_5 CTGCACAGTGTAGAGTTGTGC dnase_1 TTTACAAGTCCA dnase_2 TGTACTTACGAA NRF1 GCGCATGCGC AP1 CGATATGACTCATCCC CTCF TTGGCCACTAGGGGGCGCTAT ETS CCGAAAGCGGAAGTGAGAC SP1 AAGGGGGCGGGGCCTAA RUNX CCCTAACCACAGCCC NFKB GCAAGGGAAATTCCCCAGG GATA+TAL GGCTGGGGGGGGCAGATAAGGCC TAL GGCTGGG NFYB CCAGCCAATCAGAGC GABPA GAAACCGGAAGTGGCC BACH1+MAFK AACTGCTGAGTCATCCCG NRF1 CCCCGCGCATGCGCAGTGC HNF4G CCGTTGGACTTTGGACCCTG ``` -------------------------------- ### Build PWM from BigWig Example Usage Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/preprocessing/analysis/README.md An example command demonstrating how to build a PWM. This includes specifying the input bigwig file, genome, output path, chromosome, chromosome sizes file, and the desired PWM width. ```bash python build_pwm_from_bigwig.py -i unstranded.bw -g GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta -o /path/name_of_pwm -c chr20 -cz hg38.chrom.sizes -pw 24 ``` -------------------------------- ### Install ChromBPNet from PyPI Source: https://github.com/kundajelab/chrombpnet/wiki/Installation Install the ChromBPNet package directly from the Python Package Index. This is a straightforward method for users who prefer using pip. ```bash pip install chrombpnet ``` -------------------------------- ### Start Interactive GPU Node Session (Bash) Source: https://github.com/kundajelab/chrombpnet/wiki/FAQ Use this command to start an interactive session on a GPU node. The session will remain active for one day. For scalable runs, consider using `sbatch`. ```bash srun -p gpu -c 4 --gres gpu:1 --time 1-0 --pty bash ``` -------------------------------- ### Install ChromBPNet from Source Source: https://github.com/kundajelab/chrombpnet/wiki/Installation Clone the ChromBPNet repository and install it in editable mode using pip. This method is useful for developers or users who need to modify the source code. ```bash git clone https://github.com/kundajelab/chrombpnet.git ``` ```bash pip install -e chrombpnet ``` -------------------------------- ### Install ChromBPNet Locally with Conda Source: https://context7.com/kundajelab/chrombpnet/llms.txt Install ChromBPNet locally using Conda for environment management and pip for package installation. Includes system dependencies. ```bash # Create isolated environment conda create -n chrombpnet python=3.8 conda activate chrombpnet # Install system / bioinformatics dependencies conda install -y -c conda-forge -c bioconda \ samtools bedtools ucsc-bedgraphtobigwig pybigwig meme # Install ChromBPNet from PyPI pip install chrombpnet # OR install from source (editable) git clone https://github.com/kundajelab/chrombpnet.git pip install -e chrombpnet ``` -------------------------------- ### Example MODISCO Script Execution Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/modisco/README.md An example of how to execute the `modisco.sh` script with specific paths and parameters. Ensure the output directory exists prior to execution. ```bash modisco.sh /path/to/deepshap_scores/ /path/to/store/output/ counts_or_profiles 200000 1000 [meme_db] [meme_logos] [vier_logos] [vier_html] [html_link] ``` -------------------------------- ### Download Genome and Generate GC Buckets Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/make_gc_matched_negatives/get_genomewide_gc_buckets/README.md Example demonstrating how to download a reference genome (hg38), decompress it, and then run the `chrombpnet_genomewide_gc` script with specific input length and stride parameters. ```bash wget https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz gzip -d GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz chrombpnet_genomewide_gc -g GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta -o genomewide_gc_hg38_stride_1000_inputlen_2114 -f 2114 -s 1000 ``` -------------------------------- ### Example Usage: Fragment File to Bigwig Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/preprocessing/README.md Example command to convert an ATAC-seq fragment file to a Bigwig file. Ensure the output directory exists before running. ```bash python reads_to_bigwig.py -ifrag my_sample.frag.tsv.gz \ -g hg38.fa \ -c hg38.chrom.sizes \ -o /output/directory/my_sample\ -d ATAC ``` -------------------------------- ### Run ChromBPNet Pipeline with Singularity (Bash) Source: https://context7.com/kundajelab/chrombpnet/llms.txt Example of running the ChromBPNet pipeline on a High-Performance Computing (HPC) cluster using Singularity. This command specifies input files, data types, genome information, and output directories. ```bash # 1. Request a GPU node interactively srun -p gpu -c 4 --gres gpu:1 --time 1-0 --pty bash # 2. Load CUDA/cuDNN matching TensorFlow 2.8.0 ml cudnn/8.1 ml cuda/11.2.0 # 3. Run ChromBPNet inside Singularity, binding your data directory to /mnt singularity exec \ --nv \ -e \ --no-mount hostfs \ --bind ~/chrombpnet_tutorial/:/mnt \ docker://kundajelab/chrombpnet:latest \ chrombpnet pipeline \ -ibam /mnt/data/downloads/merged.bam \ -d ATAC \ -g /mnt/data/downloads/hg38.fa \ -c /mnt/data/downloads/hg38.chrom.sizes \ -p /mnt/data/peaks_no_blacklist.bed \ -n /mnt/data/negatives.bed \ -fl /mnt/data/splits/fold_0.json \ -b /mnt/bias_model/ENCSR868FGK_bias_fold_0.h5 \ -o /mnt/chrombpnet_model/ ``` -------------------------------- ### Import Libraries for Data Analysis and Visualization Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_dnase_tf_new.ipynb Imports necessary libraries for data manipulation, statistical analysis, and plotting. Ensure these libraries are installed before running. ```python import pickle as pkl import viz_sequence from scipy.stats import spearmanr, pearsonr import matplotlib.pyplot as plt from scipy.spatial.distance import jensenshannon import numpy as np from scipy.stats import entropy import matplotlib.pyplot as plt import scipy.ndimage import random ``` -------------------------------- ### Generate GC-Matched Background Regions Source: https://context7.com/kundajelab/chrombpnet/llms.txt Use `chrombpnet prep nonpeaks` to produce GC-matched non-peak background regions in narrowPeak BED format. Essential for bias model training and marginal footprinting. Example shows key arguments. ```bash chrombpnet prep nonpeaks \ -g ~/data/hg38.fa \ -p ~/data/peaks_no_blacklist.bed \ -c ~/data/hg38.chrom.sizes \ -fl ~/data/splits/fold_0.json \ -br ~/data/blacklist.bed.gz \ -o ~/data/output # Outputs: # ~/data/output_negatives.bed — final GC-matched non-peak regions (10-col narrowPeak) # ~/data/output_auxiliary/ — intermediate files (gc buckets, candidate beds, etc.) ``` -------------------------------- ### Generate Chromosome Train/Valid/Test Folds Source: https://context7.com/kundajelab/chrombpnet/llms.txt Use `chrombpnet prep splits` to create a JSON file assigning chromosomes to train, validation, and test sets. This is required for all training commands. Example keeps primary chromosomes. ```bash # Keep only the primary chromosomes (chr1–chr22, chrX, chrY) head -n 24 ~/data/hg38.chrom.sizes > ~/data/hg38.chrom.subset.sizes # Generate fold_0: test on chr1+chr3+chr6, validate on chr8+chr20 chrombpnet prep splits \ -c ~/data/hg38.chrom.subset.sizes \ -tcr chr1 chr3 chr6 \ -vcr chr8 chr20 \ -op ~/data/splits/fold_0 # Output: ~/data/splits/fold_0.json # { # "train": ["chr2","chr4","chr5","chr7","chr9",...], # "valid": ["chr8","chr20"], # "test": ["chr1","chr3","chr6"] # } ``` -------------------------------- ### ChromBPNet CLI Help Source: https://github.com/kundajelab/chrombpnet/wiki/Tools-overview Display the main help message for the ChromBPNet CLI, listing available subcommands. ```bash chrombpnet -h ``` -------------------------------- ### ChromBPNet Pipeline Usage Help Source: https://github.com/kundajelab/chrombpnet/wiki/ChromBPNet-training Display the help message for the 'chrombpnet pipeline' command to understand its arguments and options for training and quality checks. ```bash chrombpnet pipeline -h ``` -------------------------------- ### Install Non-Python Requirements via Conda Source: https://github.com/kundajelab/chrombpnet/wiki/Installation Install essential bioinformatics tools like samtools, bedtools, and UCSC utilities using conda. This command ensures compatibility and proper function of ChromBPNet. ```bash conda install -y -c conda-forge -c bioconda samtools bedtools ucsc-bedgraphtobigwig pybigwig meme ``` -------------------------------- ### Get Input Keys and Print Count Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_deepshaps_new.ipynb Retrieves the keys for input regions from the loaded data and prints the total number of keys. ```python input_key_list=data_500M["seq"].keys() input_key_list=data_500M["seq"].keys() ``` ```python print(len(input_key_list)) ``` -------------------------------- ### Run ChromBPNet in Docker Source: https://github.com/kundajelab/chrombpnet/wiki/Installation Use this command to launch a Docker container with all ChromBPNet dependencies pre-installed. Ensure Docker and NVIDIA Container Toolkit are set up on your host machine. ```bash docker run -it --rm --memory=100g --gpus device=0 kundajelab/chrombpnet:latest ``` -------------------------------- ### Create Data Download Directory Source: https://github.com/kundajelab/chrombpnet/wiki/Download-test-data Creates a directory to store downloaded data. Use this before downloading any files. ```bash mkdir -p ~/chrombpnet_tutorial/data/downloads ``` -------------------------------- ### Get ChromBPNet Predictions Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/custom_sequences/custom_sequence_pred.ipynb Generates predictions from the loaded ChromBPNet model using the one-hot encoded sequences. It outputs prediction logits, log counts, and final processed predictions. ```python # ---- Step 4: Get prediction ---- pred_logits_wo_bias, pred_logcts_wo_bias = model.predict(one_hot_seqs) def softmax(x, temp=1): norm_x = x - np.mean(x,axis=1, keepdims=True) return np.exp(temp*norm_x)/np.sum(np.exp(temp*norm_x), axis=1, keepdims=True) predictions = softmax(pred_logits_wo_bias) * (np.expand_dims(np.exp(pred_logcts_wo_bias)[:,0],axis=1)) # final predcitions you can use print("Prediction logits shape:", pred_logits_wo_bias.shape) # (N, 1000) print("Prediction logcnt shape:", pred_logcts_wo_bias.shape) # (N, 1) print("Prediction shape:", predictions.shape) # (N, 1000) ``` -------------------------------- ### Train ChromBPNet with Bias Model Source: https://github.com/kundajelab/chrombpnet/blob/master/README.md Use this command to initiate ChromBPNet training with a pre-trained bias model. Ensure all input file paths and parameters are correctly specified. ```bash chrombpnet pipeline \ -ibam /path/to/input.bam \ -ifrag /path/to/input.tsv \ -itag /path/to/input.tagAlign \ -d "ATAC" \ -g /path/to/hg38.fa \ -c /path/to/hg38.chrom.sizes \ -p /path/to/peaks.bed \ -n /path/to/nonpeaks.bed \ -fl /path/to/fold_0.json \ -b /path/to/bias.h5 \ -o path/to/output/dir/ \ ``` -------------------------------- ### Initialize Paths and Reference Genome Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/correlation_plots_region_wise_gm12878.ipynb Sets up directory paths for scores, peaks, and output, and loads the human reference genome (hg38) using pysam for sequence fetching. ```python import math import h5py import pickle import random import pysam import numpy as np import pandas as pd import os import pickle as pkl import viz_sequence from scipy.stats import spearmanr, pearsonr import matplotlib.pyplot as plt from scipy.spatial.distance import jensenshannon import numpy as np from scipy.stats import entropy import matplotlib.pyplot as plt import scipy.ndimage scores_dir = '/oak/stanford/groups/akundaje/projects/chrombpnet/deepshap_from_merged_bed' peak_dir = '/oak/stanford/groups/akundaje/projects/chrombpnet/bigwigs_unplugged_bias/joint_overlap' outdir = '/oak/stanford/groups/akundaje/projects/chrombpnet/modisco_inputs' hg38_fasta = '/oak/stanford/groups/akundaje/refs/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta' ref_genome = pysam.FastaFile(hg38_fasta) ``` -------------------------------- ### Visualize SHAP Scores for a Specific Sequence Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/custom_sequences/custom_sequence_interpret.ipynb Visualizes the SHAP scores for a specific example from the 'counts' head. It calculates observed SHAP scores by element-wise multiplication with one-hot encoded sequences and plots the contributions within a specified genomic region. ```python index=0 # plot contributions for example 0 # Multiple counts_shap_scores with one_hot_seqs to get observed shap scores on counts head observed_counts_scores = counts_shap_scores * one_hot_seqs ymin = counts_shap_scores[index].min() ymax = counts_shap_scores[index].max() viz_sequence.plot_weights(observed_counts_scores[index, 1057-100:1057+100,:],subticks_frequency=5,ylim=[ymin,ymax]) plt.show() ``` -------------------------------- ### ChromBPNet CLI Usage Source: https://github.com/kundajelab/chrombpnet/wiki/Tools-overview Shows the usage structure of the ChromBPNet command-line interface, detailing positional arguments and available subcommands. ```text usage: chrombpnet [-h] {pipeline,train,qc,bias,prep,pred_bw,contribs_bw,modisco_motifs,footprints,snp_score} ... ================================================================================================== Bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants ================================================================================================== positional arguments: {pipeline,train,qc,bias,prep,pred_bw,contribs_bw,modisco_motifs,footprints,snp_score} Must be eithier 'pipeline', 'train', 'qc', 'bias', 'prep', 'pred_bw', 'contribs_bw', 'modisco_motifs' ,'footprints', or 'snp_score'. pipeline End-to-end pipline with train, quality check and test for bias factorized ChromBPNet model train Train bias factorized ChromBPNet model qc Do quality checks and test for bias factorized ChromBPNet model bias Tools to train, quality check and test bias model prep Tools to generate preprocessing data for chrombpnet pred_bw Get model prediction bigwigs (Metrics calculated if observed bigwig provided) contribs_bw Get contribution score bigwigs modisco_motifs Summarize motifs from contribution scores with TFModisco footprints Get marginal footprinting for given model and given motifs snp_score Score SNPs with model ``` -------------------------------- ### Train ChromBPNet Bias Model Source: https://context7.com/kundajelab/chrombpnet/llms.txt Trains a BPNet-style model on non-peak regions to capture enzyme sequence preferences. The `-b` (bias_threshold_factor) hyperparameter is critical: start with 0.5 for ATAC-seq and 0.8 for DNase-seq. Incorrect values can lead to models learning AT-rich bias or absorbing TF motifs. ```bash chrombpnet bias pipeline \ -ibam ~/data/merged.bam \ -d "ATAC" \ -g ~/data/hg38.fa \ -c ~/data/hg38.chrom.sizes \ -p ~/data/peaks_no_blacklist.bed \ -n ~/data/output_negatives.bed \ -fl ~/data/splits/fold_0.json \ -b 0.5 \ -o ~/bias_model/ \ -fp k562 ``` -------------------------------- ### Run ChromBPNet Docker Container Source: https://context7.com/kundajelab/chrombpnet/llms.txt Run the ChromBPNet Docker container with GPU access. Requires NVIDIA Container Toolkit on the host. ```bash # Run with GPU access (requires NVIDIA Container Toolkit on host) docker run -it --rm --memory=100g --gpus device=0 kundajelab/chrombpnet:latest ``` -------------------------------- ### ChromBPNet Pipeline Command Arguments Source: https://github.com/kundajelab/chrombpnet/wiki/ChromBPNet-training Shows the general usage signature for the 'chrombpnet pipeline' command, detailing required and optional arguments for training a ChromBPNet model. ```bash chrombpnet pipeline [-h] -g GENOME -c CHROM_SIZES (-ibam INPUT_BAM_FILE | -ifrag INPUT_FRAGMENT_FILE | -itag INPUT_TAGALIGN_FILE) -o OUTPUT_DIR -d {ATAC,DNASE} -p PEAKS -n NONPEAKS -fl CHR_FOLD_PATH [-oth OUTLIER_THRESHOLD] [--ATAC-ref-path ATAC_REF_PATH] [--DNASE-ref-path DNASE_REF_PATH] [--num-samples NUM_SAMPLES] [-il INPUTLEN] [-ol OUTPUTLEN] [-s SEED] [-e EPOCHS] [-es EARLY_STOP] [-l LEARNING_RATE] [-track [TRACKABLES [TRACKABLES ...]]] [-a ARCHITECTURE_FROM_FILE] [-fp FILE_PREFIX] -b BIAS_MODEL_PATH [-sr NEGATIVE_SAMPLING_RATIO] [-fil FILTERS] [-dil N_DILATION_LAYERS] [-j MAX_JITTER] [-bs BATCH_SIZE] ``` -------------------------------- ### Run ChromBPNet Pipeline with Singularity (Bash) Source: https://github.com/kundajelab/chrombpnet/wiki/FAQ Execute the ChromBPNet pipeline within a Singularity container on a GPU node. Ensure necessary data directories are bound to the container. The `--nv` flag enables GPU usage, `-e` cleans the environment, `--no-mount hostfs` prevents host filesystem conflicts, and `--bind` maps host paths to container paths. ```bash # we will mount all the required files in our current directory (~/chrombpnet_tutorial/), which contains the data and output folders from tutorial, to the /mnt directory on the singularity container and run chrombpnet pipeline by using the command below. singularity exec --nv -e --no-mount hostfs --bind ~/chrombpnet_tutorial/:/mnt docker://kundajelab/chrombpnet:latest chrombpnet pipeline -ibam /mnt/data/downloads/merged.bam -d ATAC -g /mnt/data/downloads/hg38.fa -c /mnt/data/downloads/hg38.chrom.sizes -p /mnt/data/peaks_no_blacklist.bed -n /mnt/data/negatives.bed -f /mnt/data/splits/fold_0.json -b /mnt/bias_model/ENCSR868FGK_bias_fold_0.h5 -o /mnt/chrombpnet_model/ ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/kundajelab/chrombpnet/wiki/Installation Set up a dedicated conda environment for ChromBPNet with Python 3.8. This isolates dependencies and prevents conflicts. ```bash conda create -n chrombpnet python=3.8 ``` ```bash conda activate chrombpnet ``` -------------------------------- ### Define Data Paths and Load DeepSHAP Data Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_deepshaps_new.ipynb Sets up paths to DeepSHAP data files for different subsample sizes and loads the data using pickle. ```python main = "/srv/scratch/anusri/chrombpnet_paper/results/chrombpnet/ATAC/K562/" path_500M = main + "4_4_shifted_ATAC_09.16.2021_subsample_500M/with_gm12878_bias_final_model/unplug/deepshap/" path_100M = main + "4_4_shifted_ATAC_09.16.2021_subsample_100M/with_gm12878_bias_final_model/unplug/deepshap/" path_25M = main + "4_4_shifted_ATAC_09.16.2021_subsample_25M/with_gm12878_bias_final_model/unplug/deepshap/" path_5M = main + "4_4_shifted_ATAC_09.16.2021_subsample_5M/with_gm12878_bias_final_model/unplug/deepshap/" data_500M = pkl.load(open(path_500M+"20K.fold0.deepSHAP", "rb")) data_100M = pkl.load(open(path_100M+"20K.fold0.deepSHAP", "rb")) data_25M = pkl.load(open(path_25M+"20K.fold0.deepSHAP", "rb")) data_5M = pkl.load(open(path_5M+"20K.fold0.deepSHAP", "rb")) ``` -------------------------------- ### Download Pre-trained Bias Model Source: https://github.com/kundajelab/chrombpnet/wiki/ChromBPNet-training Download a pre-trained bias model provided with the ChromBPNet repository. This command creates a directory and downloads a specific bias model file. ```bash mkdir ~/chrombpnet_tutorial/bias_model wget https://storage.googleapis.com/chrombpnet_data/input_files/bias_models/ATAC/ENCSR868FGK_bias_fold_0.h5 -O ~/chrombpnet_tutorial/bias_model/ENCSR868FGK_bias_fold_0.h5 ``` -------------------------------- ### ChromBPNet Command Line Interface Usage Source: https://github.com/kundajelab/chrombpnet/wiki/Tutorial This displays the available subcommands for the ChromBPNet CLI, including pipeline, train, qc, bias, prep, pred_bw, contribs_bw, modisco_motifs, footprints, and snp_score. Each subcommand has a specific function for model training, evaluation, or data processing. ```bash usage: chrombpnet [-h] {pipeline,train,qc,bias,prep,pred_bw,contribs_bw,modisco_motifs,footprints,snp_score} ... ================================================================================================== Bias factorized, base-resolution deep learning models of chromatin accessibility reveal cis-regulatory sequence syntax, transcription factor footprints and regulatory variants ================================================================================================== positional arguments: {pipeline,train,qc,bias,prep,pred_bw,contribs_bw,modisco_motifs,footprints,snp_score} Must be eithier 'pipeline', 'train', 'qc', 'bias', 'prep', 'pred_bw', 'contribs_bw', 'modisco_motifs' ,'footprints', or 'snp_score'. pipeline End-to-end pipline with train, quality check and test for bias factorized ChromBPNet model train Train bias factorized ChromBPNet model qc Do quality checks and test for bias factorized ChromBPNet model bias Tools to train, quality check and test bias model prep Tools to generate preprocessing data for chrombpnet pred_bw Get model prediction bigwigs (Metrics calculated if observed bigwig provided) contribs_bw Get contribution score bigwigs modisco_motifs Summarize motifs from contribution scores with TFModisco footprints Get marginal footprinting for given model and given motifs snp_score Score SNPs with model ``` -------------------------------- ### Load chrombpnet_nobias.h5 Model with bpnet-lite Source: https://github.com/kundajelab/chrombpnet/wiki/Custom-sequence-prediction Load a ChromBPNet accessibility model (nobias) using the bpnet-lite library. This is useful for predictions that do not require bias correction. ```python from bpnetlite import BPNet accessibility_model=/path/to/chrombpnet_nobias.h5 acc = BPNet.from_chrombpnet(accessibility_model) ``` -------------------------------- ### Build PWM from BigWig Usage Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/preprocessing/analysis/README.md Use this command to build a PWM matrix centered at non-zero entries in a bigwig file. Specify input bigwig, genome, output prefix, chromosome, chromosome sizes, and PWM width. ```bash python build_pwm_from_bigwig.py -i [bigwig] -g [genome] -o [output_prefix] -c [chr] -cz [chrom_sizes] -pw [pwm_width] ``` -------------------------------- ### Generate and display footprinting plots Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/footprinting.ipynb Sets up the plotting environment, iterates through the selected motifs, and plots the footprinting tracks for each subsample size using the plot_tracks function. Finally, it displays the generated figure. ```python num=31213 import numpy as np plt.rcParams["figure.figsize"] = (12,5) #fig, axs = plt.subplots(len(index),5,sharex=True, sharey=True) fig, axs = plt.subplots(len(index),5,sharex=True) for i in range(len(index)): idx =index[i] plot_tracks(np.array(data_500M[idx]).reshape(num,1000), axs[i,0], ylims[i]) plot_tracks(np.array(data_100M[idx]).reshape(num,1000), axs[i,1], ylims[i]) plot_tracks(np.array(data_50M[idx]).reshape(num,1000), axs[i,2], ylims[i]) plot_tracks(np.array(data_25M[idx]).reshape(num,1000), axs[i,3], ylims[i]) plot_tracks(np.array(data_5M[idx]).reshape(num,1000), axs[i,4], ylims[i]) fig.subplots_adjust(wspace=0.6, hspace=0.2) plt.show() ``` -------------------------------- ### Load and Inspect TF and Signal Profile Data Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_dnase_tf_new.ipynb Loads TF and signal profile data from pickle files and prints the shape of the TF profile data. Adjust file paths as necessary. ```python tf_prof = "tf_deepshap/SPI1.dnase.strand0.fold0.deepSHAP" sig_prof = "all_deepshap/gm12878.dnase.fold0.deepSHAP" tf_data = pkl.load(open(tf_prof, "rb")) sig_data = pkl.load(open(sig_prof, "rb")) histone="dnase_old" print(np.array(list(tf_data['profile_shap'].values())).shape) input_length_tf=2114 input_length_sig=71000 ``` -------------------------------- ### Initialize DeepExplainer for Profile Head Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/custom_sequences/custom_sequence_interpret.ipynb Initializes a SHAP DeepExplainer for the 'profile' head of the model. This explainer is configured to use a weighted sum of mean-normalized logits and a shuffling strategy for background data, suitable for capturing sequence contributions to motif activity. ```python # Get profile shap scores profile_model_input = model.input profile_input = one_hot_seqs weightedsum_meannormed_logits = shap_utils.get_weightedsum_meannormed_logits(model) profile_model_profile_explainer = shap.explainers.deep.TFDeepExplainer( (profile_model_input, weightedsum_meannormed_logits), shap_utils.shuffle_several_times, combine_mult_and_diffref=shap_utils.combine_mult_and_diffref) ``` -------------------------------- ### Prepare Metrics for Table Generation Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/correlation_plots_region_wise_gm12878.ipynb Organizes profile and count correlation metrics into separate lists, preparing them for display in a tabular format. Defines the header for the table. ```python header=["NormedJSD", "NormedPearsonr", "NormedSpearmanr", "Pearsonr", "Pearsonr_random", "Spearmanr", "Spearmanr_random", "JSD", "JSD_random", "JSD_perturb"] prof_metrics = [normed_prof_shap, normed_prof_shap1, normed_prof_shap2, prof_shap_pearsonr_corr, prof_shap_pearsonr_random, prof_shap_spearmanr_corr, prof_shap_spearmanr_random, prof_shap_jsd_corr, prof_shap_jsd_random1, prof_shap_jsd_random] count_metrics = [normed_counr_shap, normed_counr_shap1, normed_counr_shap2, count_shap_pearsonr_corr, count_shap_pearsonr_random, count_shap_spearmanr_corr, count_shap_spearmanr_random, count_shap_jsd_corr, count_shap_jsd_random1, count_shap_jsd_random] heading = ["Statistic", "Count", "Profile"] rows=[] for idx in range(len(header)): ``` -------------------------------- ### Run chrombpnet_genomewide_gc Script Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/make_gc_matched_negatives/get_genomewide_gc_buckets/README.md Use this command to bin the genome into regions of a specified length and stride for GC content calculation. Ensure the output directory exists. ```bash chrombpnet_genomewide_gc -g [genome] -o [output_prefix] -f [inputlen] -s [stride] ``` -------------------------------- ### Load chrombpnet.h5 Model with bpnet-lite Source: https://github.com/kundajelab/chrombpnet/wiki/Custom-sequence-prediction Load a full ChromBPNet model, including bias and accessibility components, using the bpnet-lite library. This is for comprehensive predictions that account for sequence bias. ```python from bpnetlite import ChromBPNet bias_model=/path/to/bias_model_scaled.h5 accessibility_model=/path/to/chrombpnet_nobias.h5 chrombpnet = ChromBPNet.from_chrombpnet(bias_model, accessibility_model) ``` -------------------------------- ### Load and Prepare Data for Plotting Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/arch_compare_footprinting.ipynb Loads data for 'universal_transfer' and 'pre_worked_model' using the `load_data` function. This prepares the datasets for subsequent visualization and comparison. ```python universal_transfer_data, univ_tn5_transfer=load_data(universal_transfer) #universal_transfer_data_500filt, tn5_transfer_500_filt=load_data(universal_transfer_500filt) pre_worked_model_data, pre_worked_invivo = load_data(pre_worked_model) #universal_retrain_data, tn5_retrain =load_data(universal_retrain) #invivo_data, tn5_invivo =load_data(invivo) #invivo_data2, tn5_invivo2 =load_data(invivo2) ``` -------------------------------- ### Initialize SHAP Explainer for Counts Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/custom_sequences/custom_sequence_interpret.ipynb Initializes a DeepExplainer from the SHAP library to compute SHAP values for the counts output of the ChromBPNet model. It uses a background dataset generated by shuffling input sequences. ```python # Get count shap scores counts_model_input = model.input counts_input = one_hot_seqs profile_model_counts_explainer = shap.explainers.deep.TFDeepExplainer( (counts_model_input, tf.reduce_sum(model.outputs[1], axis=-1)), shap_utils.shuffle_several_times, combine_mult_and_diffref=shap_utils.combine_mult_and_diffref) ``` -------------------------------- ### Train ChromBPNet Model with Bias Factorization Source: https://github.com/kundajelab/chrombpnet/wiki/ChromBPNet-training Train a bias-factorized ChromBPNet model using the downloaded pre-trained bias model and specified input data. This command runs the entire pipeline, including quality checks. ```bash chrombpnet pipeline \ -ibam ~/chrombpnet_tutorial/data/downloads/merged.bam \ -d "ATAC" \ -g ~/chrombpnet_tutorial/data/downloads/hg38.fa \ -c ~/chrombpnet_tutorial/data/downloads/hg38.chrom.sizes \ -p ~/chrombpnet_tutorial/data/peaks_no_blacklist.bed \ -n ~/chrombpnet_tutorial/data/output_negatives.bed \ -fl ~/chrombpnet_tutorial/data/splits/fold_0.json \ -b ~/chrombpnet_tutorial/bias_model/ENCSR868FGK_bias_fold_0.h5 \ -o ~/chrombpnet_tutorial/chrombpnet_model/ ``` -------------------------------- ### Load ChromBPNet Models with bpnet-lite in Python Source: https://context7.com/kundajelab/chrombpnet/llms.txt Loads trained ChromBPNet models into PyTorch using `bpnet-lite` for custom sequence prediction and contribution scoring in Python. ```python from bpnetlite import BPNet, ChromBPNet # Load the TF-only (bias-corrected) model acc = BPNet.from_chrombpnet("/path/to/chrombpnet_nobias.h5") # Load the full bias-factorized model (bias + TF) chrombpnet_model = ChromBPNet.from_chrombpnet( "/path/to/bias_model_scaled.h5", "/path/to/chrombpnet_nobias.h5" ) ``` -------------------------------- ### Initialize ChromBPNet Batch Generator Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/data_generator_examples/batchgen_generator_example.ipynb Initializes the ChromBPNetBatchGenerator for validation data. This generator is used to create batches of sequences and their corresponding counts for model training or evaluation. Ensure that `val_peaks_seqs`, `val_nonpeaks_seqs`, `val_peaks_cts`, `val_nonpeaks_cts`, `negative_sampling`, `negative_sampling_ratio`, `input_len`, `output_len`, and `batch_size` are defined. ```python from generators.batchgen_generator import * batch_size=50 valid_generator = ChromBPNetBatchGenerator(val_peaks_seqs,val_nonpeaks_seqs, val_peaks_cts,val_nonpeaks_cts, negative_sampling, negative_sampling_ratio, input_len, output_len, batch_size) ``` -------------------------------- ### ChromBPNet Bias Pipeline Usage Source: https://github.com/kundajelab/chrombpnet/wiki/Bias-model-training General usage syntax for the chrombpnet bias pipeline command. Refer to the documentation for detailed explanations of each argument. ```bash chrombpnet bias pipeline [-h] -g GENOME -c CHROM_SIZES (-ibam INPUT_BAM_FILE | -ifrag INPUT_FRAGMENT_FILE | -itag INPUT_TAGALIGN_FILE) -o OUTPUT_DIR -d {ATAC,DNASE} -p PEAKS -n NONPEAKS -fl CHR_FOLD_PATH [-oth OUTLIER_THRESHOLD] [--ATAC-ref-path ATAC_REF_PATH] [--DNASE-ref-path DNASE_REF_PATH] [--num-samples NUM_SAMPLES] [-il INPUTLEN] [-ol OUTPUTLEN] [-s SEED] [-e EPOCHS] [-es EARLY_STOP] [-l LEARNING_RATE] [-track [TRACKABLES [TRACKABLES ...]]] [-a ARCHITECTURE_FROM_FILE] [-fp FILE_PREFIX] -b BIAS_THRESHOLD_FACTOR [-fil FILTERS] [-dil N_DILATION_LAYERS] [-j MAX_JITTER] [-bs BATCH_SIZE] ``` -------------------------------- ### Plotting JSD Distribution (Combined) Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_dnase_tf_new.ipynb Compares the Jensen-Shannon Divergence (JSD) for profile, profile shap, and count shap metrics against each other. ```python #print(count_shap_jsd_corr) plt.figure() plt.hist(prof_jsd_corr, bins=100, range=(0,1), alpha=0.5, label="SPI1 TF versus H3K27Ac -- profile") plt.hist(prof_shap_jsd_corr, bins=100, range=(0,1), alpha=0.5, label="SPI1 TF versus H3K27Ac -- profile shap") plt.hist(count_shap_jsd_corr, bins=100, range=(0,1), alpha=0.5, label="SPI1 TF versus H3K27Ac -- count shap") plt.legend() plt.xlabel("JSD") plt.ylabel("Frequency") plt.show() ``` -------------------------------- ### Plotting JSD Distribution (Profile Shap) Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_dnase_tf_new.ipynb Visualizes the Jensen-Shannon Divergence (JSD) for profile shap values, comparing the metric against a random baseline. ```python #print(count_shap_jsd_corr) plt.figure() plt.hist(prof_shap_jsd_corr, bins=100, range=(0,1), alpha=0.5, label="profile shap jsd") plt.hist(prof_shap_jsd_random, bins=100, range=(0,1), alpha=0.5, label="upper bound") plt.legend() plt.xlabel("JSD") plt.ylabel("Frequency") plt.show() ``` -------------------------------- ### Import Libraries and Define Paths Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_jsd.ipynb Imports necessary libraries for data manipulation, scientific computing, and plotting. It also defines base paths for different subsampling experiment results. ```python import pickle as pkl #import viz_sequence from scipy.stats import spearmanr, pearsonr import matplotlib.pyplot as plt from scipy.spatial.distance import jensenshannon import h5py from scipy.special import softmax import scipy import matplotlib import matplotlib.pyplot as plt import numpy as np from statsmodels.distributions.empirical_distribution import ECDF smoothing_flag=True main = "/srv/scratch/anusri/chrombpnet_paper/results/chrombpnet/ATAC/K562/" path_500M = main + "4_4_shifted_ATAC_09.16.2021_subsample_500M/with_gm12878_bias_final_model/" path_100M = main + "4_4_shifted_ATAC_09.16.2021_subsample_100M/with_gm12878_bias_final_model/" path_25M = main + "4_4_shifted_ATAC_09.16.2021_subsample_25M/with_gm12878_bias_final_model/" path_5M = main + "4_4_shifted_ATAC_09.16.2021_subsample_5M/with_gm12878_bias_final_model/" ``` -------------------------------- ### Save Data to Pickle File (Commented Out) Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/correlation_plots_region_wise_gm12878.ipynb This is a commented-out line that shows how to save the 'output_reg' variable to a pickle file. It is not active code but indicates a potential saving mechanism. ```python #pkl.dump(output_reg, open(out_dir+"/"+celltype+".atac.dnase.pkl","wb")) ``` -------------------------------- ### ChromBPNet Footprints Required Arguments Source: https://github.com/kundajelab/chrombpnet/wiki/Marginal-footprinting Defines the essential arguments for the `chrombpnet footprints` command. These include paths to the model, background regions, genome, chromosome fold information, output prefix, and a file mapping motifs to PWMs. ```bash required arguments: -m MODEL_H5, --model-h5 MODEL_H5 Path model .h5 file -r REGIONS, --regions REGIONS 10 column bed file of non-peak regions -g GENOME, --genome GENOME Genome fasta -fl CHR_FOLD_PATH, --chr-fold-path CHR_FOLD_PATH Fold information - dictionary with test,valid and train keys and values with corresponding chromosomes -op OUTPUT_PREFIX, --output-prefix OUTPUT_PREFIX Output prefix for bigwig files -pwm_f MOTIFS_TO_PWM, --motifs-to-pwm MOTIFS_TO_PWM Path to a TSV file containing motifs in first column and motif string to use for footprinting in second column ``` -------------------------------- ### Calculate JSD for Observed vs. Predicted Profiles Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/get_predictions_bias.ipynb Calculates and prints the Jensen-Shannon Divergence (JSD) between observed and predicted sequence profiles for different subsampling sizes (500M, 50M, 25M). Requires pre-computed profile labels and predictions. ```python stats_o500M_vs_p500M, vals_o500M_vs_p500M = jsd_pred(profile_labels_500M, profile_pred_softmax_500M) print("observed 500M vs predicted 500M JSD", stats_o500M_vs_p500M ) stats_o500M_vs_p50M, vals_o500M_vs_p50M = jsd_pred(profile_labels_500M, profile_pred_softmax_50M) print("observed 500M vs predicted 50M JSD", stats_o500M_vs_p50M) stats_o500M_vs_p25M, vals_o500M_vs_p25M = jsd_pred(profile_labels_500M, profile_pred_softmax_25M) print("observed 500M vs predicted 25M JSD", stats_o500M_vs_p25M) ``` -------------------------------- ### Run GC-Matched Negatives Script Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/helpers/make_gc_matched_negatives/README.md Execute the main bash script to generate GC-matched negative regions. Ensure the script has execute permissions first. ```bash chmod +x make_gc_matched_negatives.sh ./make_gc_matched_negatives.sh [foreground_bed] [exclude_bed] [inputlen] [output_dir] [genome] [genomewide_gc] [fold] [chrom_sizes] ``` -------------------------------- ### Merge and Index BAM Files (Bash) Source: https://context7.com/kundajelab/chrombpnet/llms.txt Standard preprocessing step to merge multiple replicate BAM files into a single sorted and indexed BAM file using samtools. This is a prerequisite for downstream analysis. ```bash # 1. Merge and index replicate BAMs samtools merge -f merged_unsorted.bam rep1.bam rep2.bam rep3.bam samtools sort -@4 merged_unsorted.bam -o merged.bam samtools index merged.bam ``` -------------------------------- ### Train ChromBPNet Bias Model Source: https://github.com/kundajelab/chrombpnet/wiki/Bias-model-training Use this command to train a bias model. Ensure you have downloaded and preprocessed the tutorial data. Adjust parameters like '-b' based on assay type (0.5 for ATAC, 0.8 for DNASE). ```bash chrombpnet bias pipeline \ -ibam ~/chrombpnet_tutorial/data/downloads/merged.bam \ -d "ATAC" \ -g ~/chrombpnet_tutorial/data/downloads/hg38.fa \ -c ~/chrombpnet_tutorial/data/downloads/hg38.chrom.sizes \ -p ~/chrombpnet_tutorial/data/peaks_no_blacklist.bed \ -n ~/chrombpnet_tutorial/data/output_negatives.bed \ -fl ~/chrombpnet_tutorial/data/splits/fold_0.json \ -b 0.5 \ -o ~/chrombpnet_tutorial/bias_model/ \ -fp k562 \ ``` -------------------------------- ### Import Libraries Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/figure_notebooks/subsampling/comparing_deepshaps_new.ipynb Imports necessary libraries for data manipulation, statistical analysis, and visualization. ```python import pickle as pkl import viz_sequence from scipy.stats import spearmanr, pearsonr import matplotlib.pyplot as plt from scipy.spatial.distance import jensenshannon import numpy as np from scipy.stats import entropy import matplotlib.pyplot as plt import scipy.ndimage import random random.seed(1) ``` -------------------------------- ### MODISCO Script Usage Source: https://github.com/kundajelab/chrombpnet/blob/master/chrombpnet/evaluation/modisco/README.md This shows the general command-line usage for the `modisco.sh` script, outlining the required arguments. ```bash modisco.sh [scores_prefix] [output_dir] [score_type] [seqlets] [crop] [meme_db] [meme_logos] [vier_logos] [vier_html] [html_link] ```