### Install tensorQTL using pip Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Install tensorQTL using pip for easy setup. ```bash pip3 install tensorqtl ``` -------------------------------- ### Install Pgenlib from source Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Install Pgenlib from its source code if direct pip installation is not suitable. ```bash git clone git@github.com:chrchang/plink-ng.git cd plink-ng/2.0/Python/ python3 setup.py build_ext python3 setup.py install ``` -------------------------------- ### Install tensorQTL from repository with Conda Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Clone the repository and install tensorQTL using a Conda environment file. ```bash $ git clone git@github.com:broadinstitute/tensorqtl.git $ cd tensorqtl # install into a new virtual environment and load $ mamba env create -f install/tensorqtl_env.yml $ conda activate tensorqtl ``` -------------------------------- ### Import Libraries and Setup Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Imports necessary libraries for data manipulation, plotting, and TensorQTL. Sets up the computation device (GPU or CPU) and printing configurations. ```python import pandas as pd import numpy as np import tensorqtl from tensorqtl import pgen, cis, post import qtl.plot import matplotlib.pyplot as plt import matplotlib matplotlib.rcParams['figure.dpi'] = 72 plt.rcParams.update({'font.family': 'Helvetica', 'svg.fonttype':'none', 'pdf.fonttype':42}) import torch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print(f"torch: {torch.__version__} (CUDA {torch.version.cuda}), device: {device}") print(f"pandas {pd.__version__}") ``` -------------------------------- ### Install R and qvalue Package Source: https://github.com/broadinstitute/tensorqtl/blob/master/install/INSTALL.md Installs R and the 'qvalue' package using BiocManager, which is required for computing q-values. ```r if (!require("BiocManager", quietly = TRUE)) install.packages("BiocManager") BiocManager::install("qvalue") ``` -------------------------------- ### Setup and Data Loading for tensorQTL Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Initializes the environment, loads necessary libraries, and prepares genotype, phenotype, and covariate data for tensorQTL analysis. Ensure data paths are correctly set. ```python import pandas as pd import torch import tensorqtl from tensorqtl import pgen, cis, trans, post device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print(f"torch: {torch.__version__} (CUDA {torch.version.cuda}), device: {device}") print(f"pandas: {pd.__version__}") # define paths to data plink_prefix_path = 'data/GEUVADIS.445_samples.GRCh38.20170504.maf01.filtered.nodup.chr18' expression_bed = 'data/GEUVADIS.445_samples.expression.bed.gz' covariates_file = 'data/GEUVADIS.445_samples.covariates.txt' prefix = 'GEUVADIS.445_samples' # load phenotypes and covariates phenotype_df, phenotype_pos_df = tensorqtl.read_phenotype_bed(expression_bed) covariates_df = pd.read_csv(covariates_file, sep='\t', index_col=0).T # PLINK reader for genotypes pgr = pgen.PgenReader(plink_prefix_path) genotype_df = pgr.load_genotypes() variant_df = pgr.variant_df ``` -------------------------------- ### Install latest tensorQTL from Git repository Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Install the latest version of tensorQTL directly from its GitHub repository using pip. ```bash pip install pip@git+https://github.com/broadinstitute/tensorqtl.git ``` -------------------------------- ### Install CUDA Drivers Source: https://github.com/broadinstitute/tensorqtl/blob/master/install/INSTALL.md Installs CUDA drivers on Ubuntu 22.04 LTS. Requires a reboot to take effect. Verify installation with nvidia-smi. ```bash sudo ./install_cuda.sh sudo reboot # verify nvidia-smi ``` -------------------------------- ### Install Pgenlib for PLINK 2 Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Install the Pgenlib Python package, required for using PLINK 2 binary files. ```bash pip install Pgenlib ``` -------------------------------- ### Install rmate (Optional) Source: https://github.com/broadinstitute/tensorqtl/blob/master/install/INSTALL.md Installs the rmate tool, which allows remote editing of files. It requires Ruby and adds the rmate executable to the user's bin directory and configures the bashrc. ```bash sudo apt install -y ruby mkdir ~/bin curl -Lo ~/bin/rmate https://raw.githubusercontent.com/textmate/rmate/master/bin/rmate chmod a+x ~/bin/rmate echo 'export RMATE_PORT=${rmate_port}' >> ~/.bashrc ``` -------------------------------- ### Verify PyTorch and CUDA Installation Source: https://github.com/broadinstitute/tensorqtl/blob/master/install/INSTALL.md Verifies the installed PyTorch version and checks if CUDA is available, printing the CUDA device name if available. Expected output format is shown. ```python import torch; print(torch.__version__); print('CUDA available: {} ({})'.format(torch.cuda.is_available(), torch.cuda.get_device_name(torch.cuda.current_device()))) ``` -------------------------------- ### Run tensorQTL help command Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Display all available command-line options for tensorQTL. ```bash python3 -m tensorqtl --help ``` -------------------------------- ### Create and Activate Conda Environment Source: https://github.com/broadinstitute/tensorqtl/blob/master/install/INSTALL.md Creates a conda environment using the provided 'tensorqtl_env.yml' file and activates it. This environment includes PyTorch and TensorQTL. ```bash mamba env create -f tensorqtl_env.yml conda activate tensorqtl ``` -------------------------------- ### Run cis-QTL mapping with permutations (Shell) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Execute this shell command to perform cis-QTL mapping. The `${prefix}` argument specifies the output file name. ```shell python3 -m tensorqtl ${plink_prefix_path} ${expression_bed} ${prefix} \ --covariates ${covariates_file} \ --mode cis ``` -------------------------------- ### Run trans-QTL mapping (Shell) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Execute this shell command to perform trans-QTL mapping. By default, it generates sparse output for associations with p-value < 1e-5 and filters out cis-associations. ```shell python3 -m tensorqtl ${plink_prefix_path} ${expression_bed} ${prefix} \ --covariates ${covariates_file} \ --mode trans ``` -------------------------------- ### Map cis-QTL interactions (Shell) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Use this shell command to map cis-QTLs with an interaction term. The `--interaction` option specifies the interaction file. `--best_only` disables full summary statistics output. ```shell python3 -m tensorqtl ${plink_prefix_path} ${expression_bed} ${prefix} \ --covariates ${covariates_file} \ --interaction ${interactions_file} \ --best_only \ --mode cis_nominal ``` -------------------------------- ### Run trans-QTL mapping (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md This Python code performs trans-QTL mapping, generating sparse output of associations with p-value < 1e-5. It includes filtering of cis-associations. ```python trans_df = trans.map_trans(genotype_df, phenotype_df, covariates_df, return_sparse=True, pval_threshold=1e-5, maf_threshold=0.05, batch_size=20000) # remove cis-associations trans_df = trans.filter_cis(trans_df, phenotype_pos_df.T.to_dict(), variant_df, window=5000000) ``` -------------------------------- ### Run cis-QTL mapping with permutations (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Use this Python code to perform cis-QTL mapping and generate phenotype-level summary statistics with empirical p-values for genome-wide FDR calculation. ```python cis_df = cis.map_cis(genotype_df, variant_df, phenotype_df, phenotype_pos_df, covariates_df) tensorqtl.calculate_qvalues(cis_df, qvalue_lambda=0.85) ``` -------------------------------- ### Import tensorQTL modules Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Import necessary pandas and tensorQTL modules for running analyses in Python. ```python import pandas as pd import tensorqtl from tensorqtl import genotypeio, cis, trans ``` -------------------------------- ### Run cis-QTL Mapping and Calculate Q-values Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Performs cis-QTL mapping using the loaded genotype, phenotype, and covariate data. It then calculates q-values based on the empirical p-values with a specified FDR. ```python # run permutations and compute q-values cis_df = cis.map_cis(genotype_df, pgr.variant_df, phenotype_df, phenotype_pos_df, covariates_df=covariates_df, seed=3042235018, warn_monomorphic=False) post.calculate_qvalues(cis_df, fdr=0.05, qvalue_lambda=0.85) ``` -------------------------------- ### Load Genotypes Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Loads genotype data from a PLINK prefix file using PgenReader. This is a prerequisite for running the cis-QTL analysis. ```python plink_prefix_path = '/resources/vcfs/GTEx_Analysis_2017-06-05_v8_WholeGenomeSeq_838Indiv_Analysis_Freeze.SHAPEIT2_phased.MAF01' pgr = pgen.PgenReader(plink_prefix_path) genotype_df = pgr.load_genotypes() ``` -------------------------------- ### Generate cis-QTL summary statistics (Shell) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Use this shell command to generate nominal cis-QTL associations. The results are saved to parquet files per chromosome. ```shell python3 -m tensorqtl ${plink_prefix_path} ${expression_bed} ${prefix} \ --covariates ${covariates_file} \ --mode cis_nominal ``` -------------------------------- ### Calculate Q-values for Cis-Associations Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Calculates empirical q-values from a DataFrame of cis-associations. Requires a DataFrame with cis-associations and specifies FDR and q-value lambda parameters. ```python post.calculate_qvalues(cis_df, fdr=0.05, qvalue_lambda=0.85) ``` -------------------------------- ### Read cis-QTL summary statistics from Parquet (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md This Python snippet demonstrates how to read the parquet files containing cis-QTL summary statistics using pandas. ```python df = pd.read_parquet(file_name) ``` -------------------------------- ### Generate cis-QTL summary statistics (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md This Python code generates nominal associations for all variant-phenotype pairs. Results are written to parquet files. ```python cis.map_nominal(genotype_df, variant_df, phenotype_df, phenotype_pos_df, prefix, covariates_df, output_dir='.') ``` -------------------------------- ### Map conditionally independent cis-QTLs (Shell) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Execute this shell command to map conditionally independent cis-QTLs. The `--cis_output` option specifies the input from the permutation step. ```shell python3 -m tensorqtl ${plink_prefix_path} ${expression_bed} ${prefix} \ --covariates ${covariates_file} \ --cis_output ${prefix}.cis_qtl.txt.gz \ --mode cis_independent ``` -------------------------------- ### Plotting q-value comparison with zoom Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb This snippet visualizes the comparison of q-values between FastQTL v8 and cis-QTL results, with a zoomed-in view on a specific threshold. ```python # zoom on FDR threshold ax[3].set_xscale('log') ax[3].set_yscale('log') ax[3].scatter(egenes_v8_df['qval'], cis_df['qval'], **args) ax[3].set_title('q-values (zoom)', fontsize=12) ax[3].set_xlabel('FastQTL (v8)', fontsize=12) b = [0.01, 0.1] ax[3].plot(b, [0.05, 0.05], 'k--') ax[3].plot([0.05, 0.05], b, 'k--') ax[3].set_xlim(b) ax[3].set_ylim(b) ax[3].set_xticks([0.01, 0.05, 0.1]) ax[3].set_yticks([0.01, 0.05, 0.1]); ``` -------------------------------- ### Load subset of samples with PlinkReader Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Load genotypes using PlinkReader, selecting only a specific subset of samples to potentially save memory. ```python pr = genotypeio.PlinkReader(plink_prefix_path, select_samples=phenotype_df.columns) ``` -------------------------------- ### Perform Trans-QTL Mapping Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Initiates trans-QTL mapping using genotype, phenotype, and covariate data. Allows for batch processing, sparse output, and filtering by p-value and minor allele frequency. ```python # run mapping # to limit output size, only associations with p-value <= 1e-5 are returned trans_df = trans.map_trans(genotype_df, phenotype_df, covariates_df, batch_size=10000, return_sparse=True, pval_threshold=1e-5, maf_threshold=0.05) ``` -------------------------------- ### Load genotypes from PLINK reader Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Initialize a PlinkReader and load genotypes and variant information into pandas DataFrames. ```python pr = genotypeio.PlinkReader(plink_prefix_path) # load genotypes and variants into data frames genotype_df = pr.load_genotypes() variant_df = pr.bim.set_index('snp')[['chrom', 'pos']] ``` -------------------------------- ### Run Nominal cis-QTL Mapping Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Performs nominal cis-QTL mapping for a specified chromosome, saving the results to a specified output directory. Requires genotype, phenotype, and covariate data. ```python # nominal mode (limit to one chromosome for illustration purposes) chrom = 'chr9' m = phenotype_pos_df['chr'] == chrom cis.map_nominal(genotype_df, pgr.variant_df, phenotype_df[m], phenotype_pos_df[m], tissue_id, covariates_df=covariates_df, output_dir='/mnt/disks/scratch/tmp/') nominal_df = pd.read_parquet(f'/mnt/disks/scratch/tmp/Brain_Cortex.cis_qtl_pairs.{chrom}.parquet') ``` -------------------------------- ### Map cis-QTL interactions with eigenMT (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md This Python code maps cis-QTLs including an interaction term and computes eigenMT-adjusted p-values. It requires interaction data and can write top associations and full summary statistics. ```python cis.map_nominal(genotype_df, variant_df, phenotype_df, phenotype_pos_df, prefix, covariates_df=covariates_df, interaction_df=interaction_df, maf_threshold_interaction=0.05, run_eigenmt=True, output_dir='.', write_top=True, write_stats=True) ``` -------------------------------- ### Load GTEx v8 Results and Plot Comparisons Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Loads pre-computed GTEx v8 eQTL results and generates plots to compare nominal p-values, empirical p-values, and q-values between the GTEx v8 results and the TensorQTL output. ```python # load V8 results and plot comparisons egenes_v8_df = pd.read_csv(f'/resources/V8_release/GTEx_Analysis_v8_eQTL/{tissue_id}.v8.egenes.txt.gz', sep='\t', index_col=0) args = {'ec':'none', 'alpha':0.5} ax = qtl.plot.get_axgrid(1, 4) ax[0].set_xscale('log') ax[0].set_yscale('log') ax[0].scatter(egenes_v8_df['pval_nominal'], cis_df['pval_nominal'], **args) ax[0].set_title('Nominal p-values', fontsize=12) ax[0].set_xlabel('FastQTL (v8)', fontsize=12) ax[0].set_ylabel('TensorQTL', fontsize=12) ax[1].set_xscale('log') ax[1].set_yscale('log') ax[1].scatter(egenes_v8_df['pval_beta'], cis_df['pval_beta'], **args) ax[1].set_title('Empirical p-values', fontsize=12) ax[1].set_xlabel('FastQTL (v8)', fontsize=12) ax[2].set_xscale('log') ax[2].set_yscale('log') ax[2].scatter(egenes_v8_df['qval'], cis_df['qval'], **args) ax[2].set_title('q-values', fontsize=12) ax[2].set_xlabel('FastQTL (v8)', fontsize=12); ``` -------------------------------- ### Load phenotypes and covariates Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Load phenotype data from a BED file and covariates from a tab-delimited file into pandas DataFrames. ```python phenotype_df, phenotype_pos_df = tensorqtl.read_phenotype_bed(phenotype_bed_file) covariates_df = pd.read_csv(covariates_file, sep='\t', index_col=0).T # samples x covariates ``` -------------------------------- ### Load and Inspect Cis-QTL Nominal Results Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Loads the results of the cis-QTL nominal association mapping from a Parquet file and displays the first few rows. This is useful for reviewing the calculated p-values and effect sizes. ```python # load results pairs_df = pd.read_parquet(f'{prefix}.cis_qtl_pairs.chr18.parquet') pairs_df.head() ``` -------------------------------- ### Map conditionally independent cis-QTLs (Python) Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md This Python code maps conditionally independent cis-QTLs using a stepwise regression procedure. It requires the output from the permutation step. ```python indep_df = cis.map_independent(genotype_df, variant_df, cis_df, phenotype_df, phenotype_pos_df, covariates_df) ``` -------------------------------- ### Load and Subset V8 Summary Statistics Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Loads GTEx v8 summary statistics for a given tissue and subsets them to a specific chromosome. Resets the index after subsetting. ```python # load V8 summary statistics nominal_v8_df = pd.read_parquet( f"gs://gtex-resources/GTEx_Analysis_v8_QTLs/GTEx_Analysis_v8_eQTL_all_associations/{tissue_id}.v8.allpairs.parquet") # subset chr9 nominal_v8_df = nominal_v8_df[nominal_v8_df['variant_id'].str.startswith(chrom)] nominal_v8_df.reset_index(drop=True, inplace=True) ``` -------------------------------- ### Calculate eGene Overlap Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Calculates and prints the number of eGenes and their intersection between FastQTL v8 and TensorQTL results at a 0.05 q-value threshold. ```python # eGenes overlap igenes_v8 = egenes_v8_df[egenes_v8_df['qval'] <= 0.05].index egenes = cis_df[cis_df['qval'] <= 0.05].index ix = egenes[egenes.isin(egenes_v8)] print(f"eGenes, v8 (all biotypes): {len(egenes_v8)}") print(f"eGenes, rep. (all biotypes): {len(egenes)}") print(f" * intersection: {len(ix)} ({len(ix)/len(egenes_v8)*100:.1f}%)") ``` -------------------------------- ### Load Phenotype and Covariate Data Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Loads normalized expression data (phenotypes) and covariate data for a specific tissue. Phenotype data is loaded from a BED file, and covariates from a tab-separated text file. ```python tissue_id = 'Brain_Cortex' covariates_df = pd.read_csv(f"/resources/V8_release/GTEx_Analysis_v8_eQTL_covariates/{tissue_id}.v8.covariates.txt", sep='\t', index_col=0).T phenotype_df, phenotype_pos_df = tensorqtl.read_phenotype_bed( f'/resources/V8_release/GTEx_Analysis_v8_eQTL_expression_matrices/{tissue_id}.v8.normalized_expression.bed.gz') ``` -------------------------------- ### Display Head of Cis-Association DataFrame Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Displays the first few rows of a pandas DataFrame containing cis-association results. Useful for inspecting the structure and content of the data. ```python cis_df.head() ``` -------------------------------- ### Convert VCF to PLINK2 pgen format Source: https://github.com/broadinstitute/tensorqtl/blob/master/README.md Convert VCF files to PLINK2 pgen/pvar/psam format using plink2. Ensure to use --output-chr chrM for compatibility. ```bash plink2 \ --output-chr chrM \ --vcf ${plink_prefix_path}.vcf.gz \ --out ${plink_prefix_path} ``` -------------------------------- ### Plot Nominal p-value Comparison Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Generates a scatter plot comparing nominal p-values from GTEx v8 and TensorQTL results. Both axes are set to a logarithmic scale. ```python # plot p-values ax = qtl.plot.get_axgrid(1, 1)[0] ax.set_xscale('log') ax.set_yscale('log') ax.scatter(nominal_v8_df['pval_nominal'], nominal_df['pval_nominal'], **args) ax.set_title('Nominal p-values', fontsize=12) ax.set_xlabel('FastQTL p-value', fontsize=12) ax.set_ylabel('TensorQTL p-value', fontsize=12); ``` -------------------------------- ### Display Head of Trans-Association DataFrame Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Displays the first few rows of a pandas DataFrame containing trans-association results after filtering. Useful for inspecting the filtered data. ```python trans_df.head() ``` -------------------------------- ### Compare V8 and TensorQTL Nominal p-values Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/GTEx_v8_example.ipynb Asserts the equality of gene IDs, variant IDs, and TSS distances between GTEx v8 and TensorQTL nominal results for a specific chromosome. This is a validation step before plotting. ```python # check that outputs are identical assert nominal_v8_df['gene_id'].equals(nominal_df['phenotype_id']) assert nominal_v8_df['variant_id'].equals(nominal_df['variant_id']) assert nominal_v8_df['tss_distance'].equals(nominal_df['start_distance']) ``` -------------------------------- ### Cis-QTL Empirical P-value Calculation Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Calculates empirical p-values for cis-QTL associations using a specified random seed for reproducibility. This snippet also focuses on a specific chromosome (chr18). ```python # all genes # cis_df = cis.map_cis(genotype_df, variant_df, phenotype_df, phenotype_pos_df, covariates_df=covariates_df) # genes on chr18 cis_df = cis.map_cis(genotype_df, variant_df, phenotype_df.loc[phenotype_pos_df['chr'] == 'chr18'], phenotype_pos_df.loc[phenotype_pos_df['chr'] == 'chr18'], covariates_df=covariates_df, seed=123456) ``` -------------------------------- ### Cis-QTL Nominal Association Mapping Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Performs cis-QTL mapping to calculate nominal p-values for all variant-phenotype pairs within a specified window. This snippet focuses on a specific chromosome (chr18). ```python # map all cis-associations (results for each chromosome are written to file) # all genes # cis.map_nominal(genotype_df, variant_df, phenotype_df, phenotype_pos_df, prefix, covariates_df=covariates_df) # genes on chr18 cis.map_nominal(genotype_df, variant_df, phenotype_df.loc[phenotype_pos_df['chr'] == 'chr18'], phenotype_pos_df.loc[phenotype_pos_df['chr'] == 'chr18'], prefix, covariates_df=covariates_df) ``` -------------------------------- ### Filter Cis-Associations from Trans-QTL Results Source: https://github.com/broadinstitute/tensorqtl/blob/master/example/tensorqtl_examples.ipynb Removes cis-associations from a trans-QTL results DataFrame. Requires the trans-QTL DataFrame, phenotype position data, variant data, and a window size for filtering. ```python # remove cis-associations trans_df = trans.filter_cis(trans_df, phenotype_pos_df, variant_df, window=5000000) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.