### Install scperturb and scanpy Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/get_obs.ipynb Installs the scperturb and scanpy libraries. Use --quiet to suppress installation output. ```python !pip install scanpy scperturb --quiet ``` -------------------------------- ### Install Dependencies Source: https://github.com/sanderlab/scperturb/blob/master/website/datavzrd/datavzrd_notebook.ipynb Install the scanpy library in the current environment. ```python !pip install --quiet scanpy ``` -------------------------------- ### Load Example Dataset Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/getExplanatoryPCs_py.ipynb Loads an example dataset using scanpy. This is a prerequisite for using other functions in the library. ```python adata = sc.read('../exampledataset/exampledataset.h5') ``` -------------------------------- ### Import Libraries and Setup Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/delete_me.ipynb Imports core Python libraries for data manipulation, scientific computing, and visualization. Includes setup for scvelo, matplotlib, and custom utilities. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.auto import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * # scperturb package sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../../figures/') ``` -------------------------------- ### Initialize Environment and Imports Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/scANVI_run_notebook.ipynb Setup necessary libraries and append the project root to the system path for module resolution. ```python import scvi import numpy as np import pandas as pd import scanpy as sc import os from sklearn.metrics import confusion_matrix import seaborn as sns import sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '../../'))) import scanvi import importlib ``` -------------------------------- ### Install and Import scperturb Source: https://github.com/sanderlab/scperturb/blob/master/package/notebooks/e-distance.ipynb Installs the scperturb library and reloads the Python environment to make its functions available. This is typically done after installing the library. ```python # !pip install scperturb --upgrade # from scperturb import * import sys sys.path.append(".."") %reload_ext autoreload %autoreload 2 from src.scperturb import * ``` -------------------------------- ### Import Libraries and Setup Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/getExplanatoryPCs_py.ipynb Imports necessary libraries for data analysis and visualization, and configures display settings for notebooks. ```python from tqdm.notebook import tqdm import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline import scvelo as scv scv.settings.verbosity=1 ``` -------------------------------- ### Install scperturb and Dependencies Source: https://github.com/sanderlab/scperturb/blob/master/package/notebooks/e-distance.ipynb Installs the scperturb library and its dependencies using pip. This is a prerequisite for using the library. ```python !pip install scanpy scikit-misc --quiet ``` -------------------------------- ### Install Required Packages Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/perturbation_statistics.ipynb Installs the scmmd and torch-two-sample packages from GitHub. These are necessary for advanced statistical comparisons, such as MMD calculations. ```bash # !pip install git+https://github.com/calico/scmmd # !pip install git+https://github.com/josipd/torch-two-sample ``` -------------------------------- ### Install Dependencies Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/XuCao2023.ipynb Install the required Python packages for data analysis. ```python !pip install chembl_webresource_client scanpy scperturb --quiet ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/LotfollahiTheis2023.ipynb Installs all necessary Python packages for the scPerturb project using pip. Ensure you are in a suitable environment before running. ```python !pip install mygene statannotations scrublet scanpy scvelo decoupler matplotlib_venn goatools gseapy scperturb biomart PyComplexHeatmap statsmodels omnipath git+https://github.com/saezlab/pypath.git --quiet ``` -------------------------------- ### Install Project Dependencies Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/SunshineHein2023.ipynb Installs all necessary Python packages for the scperturb project using pip. This command should be run in a compatible environment. ```python !pip install mygene statannotations scrublet scanpy scvelo decoupler goatools gseapy scperturb chembl_webresource_client biomart PyComplexHeatmap statsmodels omnipath git+https://github.com/saezlab/pypath.git --quiet ``` -------------------------------- ### Import Libraries and Setup Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Imports necessary Python libraries for data analysis and sets up the environment for Jupyter notebooks. Includes custom utility functions. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') from utils import * ``` -------------------------------- ### Partial KNN Classifier Setup Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Classifier_loss_similarity.ipynb A truncated snippet showing the initialization of a KNN classifier pipeline. ```python for selected_class in pd.unique(adata.obs.perturbation)[:10]: if selected_class == 'control': continue mask = (adata.obs.perturbation == selected_class) | (adata.obs.perturbation == 'control') # Define and split training and test data from sklearn.model_selection import train_test_split X = adata.obsm['X_pca'][mask] # data y = list(adata.obs.perturbation[mask]) # labels X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y) # stratify=labels makes sure same label ratio in splits # Feature scaling from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(X_train) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test) # select and train classifier (here KNN classifier) from sklearn.neighbors import KNeighborsClassifier ``` -------------------------------- ### Setup Figure and Load Icons for Plotting Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/data_qc.ipynb Initializes a Matplotlib figure with multiple subplots and loads various icon images for use in annotations. This setup is for creating complex visualizations with custom icons. ```python from matplotlib.offsetbox import TextArea, DrawingArea, OffsetImage, AnnotationBbox import matplotlib.image as mpimg n = len(B.index) m = len(B.columns) # setup figure # with sns.axes_style("whitegrid"): fig, axs = pl.subplots(3, 1, figsize=[17, 10], sharex=True, dpi=300, gridspec_kw={'height_ratios': [1.5, 1, 1]}) # fig, ax = pl.subplots(figsize=[17,5], dpi=300) text_options = {'fontsize': 6, 'ha': 'center', 'va': 'center'} zoom_factor=0.6 mouse = OffsetImage(mpimg.imread('icons/mouse.png'), zoom=0.06*zoom_factor) stem = OffsetImage(mpimg.imread('icons/stem.png'), zoom=0.04*zoom_factor) human = OffsetImage(mpimg.imread('icons/person-standing.png'), zoom=0.08*zoom_factor) primary = OffsetImage(mpimg.imread('icons/scalpel.png'), zoom=0.06*zoom_factor) cell_line = OffsetImage(mpimg.imread('icons/petri-dish.png'), zoom=0.05*zoom_factor) CRISPR = OffsetImage(mpimg.imread('icons/gene.png'), zoom=0.05*zoom_factor) drug = OffsetImage(mpimg.imread('icons/pills.png'), zoom=0.05*zoom_factor) True_ = OffsetImage(mpimg.imread('icons/check.png'), zoom=0.04*zoom_factor) False_ = OffsetImage(mpimg.imread('icons/close.png'), zoom=0.035*zoom_factor) ``` -------------------------------- ### Import Libraries and Setup Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/theory.ipynb Imports necessary libraries and sets up the Python environment for the scperturb project, including custom paths and autoreload functionality. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.auto import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * # scperturb package sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../figures/') ``` -------------------------------- ### Import Libraries and Setup Environment Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/LotfollahiTheis2023.ipynb Imports essential Python libraries for single-cell data analysis and sets up the plotting environment. Includes custom utility functions and scperturb package. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../') from utils import * # scperturb package sys.path.insert(1, '../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../figures/') ``` -------------------------------- ### Split String Example Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/scANVI.ipynb Demonstrates splitting a string by an underscore to extract the first part. This is useful for parsing dataset identifiers. ```python stringthing = "a_b" ``` ```python stringthing.split("_")[0] ``` -------------------------------- ### Run datavzrd Configuration Source: https://github.com/sanderlab/scperturb/blob/master/website/datavzrd/README.md Use this command to run datavzrd with a specified configuration file and output directory. Ensure datavzrd version 2.27.0 or later is installed. ```bash datavzrd config.yaml --output test/ ``` -------------------------------- ### jQuery Animation Timer Start Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/CuiHacohen2023.nb.html The `at` function is used to get the current timestamp, typically for starting animation timers. ```javascript function at(){return C.setTimeout(function(){Ze=void 0}),Ze=Date.now()} ``` -------------------------------- ### Initialize Environment and Load Data Paths Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/present_2_selected_datasets.ipynb Sets up the data directory path and imports necessary utility functions. ```python # sys.path.insert(1, '/extra/stefan/utils/scrnaseq_utils/') from scrnaseq_util_functions import * data_path = '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' ``` ```python data_path = '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' ``` -------------------------------- ### Generate Guide IDs from Counts Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/data_processing_JS.ipynb Determines the most likely guide ID for each cell based on the maximum count in HTO and GDO data. 'control' is assigned if GDO starts with 'NT'. ```python htoguides = list(hto.idxmax()) gdoguides = ["control" if x.startswith("NT") else x for x in gdo.idxmax()] ``` -------------------------------- ### Initialize Environment and Paths Source: https://github.com/sanderlab/scperturb/blob/master/notebooks/Fig2.ipynb Sets up necessary imports, Jupyter notebook configurations, and file system paths for data processing. ```python import subprocess import os import sys import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns from statsmodels.stats.multitest import multipletests import scvelo as scv scv.settings.verbosity=1 # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../') from utils import * # path with scPerturb data (replace accordingly) data_path = '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' # temp path SDIR = '/fast/scratch/users/peidlis_c/perturbation_resource_paper/' # output from snakemake (tables) table_path = '/fast/work/users/peidlis_c/projects/perturbation_resource_paper/single_cell_perturbation_data/code/notebooks/data_analysis/analysis_screens/tables/' # path for figures website_path = '../website/' # path for supplemental figures and tables supp_path = '../supplement/' ``` -------------------------------- ### Get Chunk Boundaries Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/JoungZhang2023.ipynb Displays the start and end row indices for a specific chunk (determined by variable 'i' and 'chunk_size'). This is useful for verifying which part of the dataset is being processed or has been processed. ```python i*chunk_size, (i+1)*chunk_size ``` -------------------------------- ### Initialize Utility Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/data_qc_2.ipynb Sets up the Python path and imports utility functions for single-cell RNA sequencing analysis. ```python sys.path.insert(1, utils_path) from scrnaseq_util_functions import * ``` -------------------------------- ### Create Conda Environment for Reproducibility Source: https://github.com/sanderlab/scperturb/blob/master/README.md Create a new conda environment with all necessary packages to reproduce the figures and tables from the scPerturb paper. This requires having conda installed. ```bash conda env create -f sc_env.yaml ``` -------------------------------- ### Initialize Environment and Paths Source: https://github.com/sanderlab/scperturb/blob/master/notebooks/Fig3-updated.ipynb Sets up the Jupyter environment, imports necessary libraries, and loads configuration paths from a YAML file. ```python import os import matplotlib.pyplot as pl import pandas as pd import numpy as np import seaborn as sns # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # paths import yaml config = yaml.safe_load(open('../config.yaml', "r")) data_path = config['DOWNDIR'] SDIR = config['DIR'] # output from snakemake (tables) table_path = '/fast/work/users/peidlis_c/projects/perturbation_resource_paper/single_cell_perturbation_data/code/notebooks/data_analysis/analysis_screens/tables/' # path for figures figure_path = '../figures/' # path for supplemental figures and tables supp_path = '../supplement/' dayfirst = True ``` -------------------------------- ### Import Dependencies and Configure Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/singlecell_vs_pseudobulk.ipynb Initializes necessary libraries and Jupyter notebook settings for scperturb analysis. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import scvelo as scv scv.settings.verbosity=1 # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') from utils import * ``` -------------------------------- ### Example Data Processing Workflow Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/LaraAstiasoHuntly2023.ipynb Demonstrates the typical workflow of merging and harmonizing data for a 'leukemia' experiment. This involves calling `merge_data` followed by `harmonize_data`. ```python adata = merge_data('leukemia') bdata = harmonize_data(adata, 'leukemia') ``` -------------------------------- ### jQuery AJAX Request Methods (GET, POST) Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/CuiHacohen2023.nb.html Convenience methods for making GET and POST requests. Use `get` for retrieving data and `post` for sending data. They simplify common AJAX calls. ```javascript S.each(["get","post"],function(e,i){S[i]=function(e,t,n,r){return m(t)&&(r=r||n,n=t,t=void 0),S.ajax(S.extend({url:e,type:i,dataType:r,data:t,success:n},S.isPlainObject(e)&&e))}}) ``` -------------------------------- ### Initialize Environment and Imports Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/bio_interpretation_cluster_perturbations.ipynb Sets up the Python environment with necessary libraries for single-cell analysis and plotting. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * # scperturb package sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../figures/') ``` -------------------------------- ### Initialize Environment and Imports Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/DEG_profile.ipynb Sets up the Python environment with necessary libraries for single-cell analysis and visualization. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') from utils import * ``` -------------------------------- ### Initialize scperturb Analysis Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/Feature_Selection.ipynb Sets up the Python environment with necessary libraries, custom paths, and Jupyter display settings for scperturb analysis. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.auto import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * # scperturb package sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../../figures/') ``` -------------------------------- ### Map Targets to Guides Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Creates a dictionary mapping each unique target gene to a list of its corresponding guides. ```python # dict of all targets : [guides] A = {} for target in unique_targets: A[target] = [x for x in sim.columns if target in x] ``` -------------------------------- ### Initialize scperturb environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/Edist_results.ipynb Imports core libraries, configures scvelo verbosity, adjusts Jupyter notebook width, and adds the local utils directory to the system path. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) # Custom functions sys.path.insert(1, '../..') from utils import * ``` -------------------------------- ### Initialize McFarland H5 Files List Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/quality_control_JS.ipynb Finds all H5 files in the McFarlandTshemiak2020 directory and processes the first one for QC. ```python mcfarlandh5 = glob.glob('/n/data1/hms/cellbio/sander/judy/resource_paper/McFarlandTshemiak2020/*.h5') qc_h5(mcfarlandh5[0], writeadata=True) ``` -------------------------------- ### Get and Set CSS Properties with jQuery Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/CuiHacohen2023.nb.html Use the .css() method to get or set CSS properties for elements. When setting, you can pass a single property-value pair or an object of multiple properties. When getting, it returns the computed value of the first element in the set. ```javascript S.extend({cssHooks:{opacity:{get:function(e,t){if(t){var n=We(e,"opacity");return""===n?"1":n}}}},cssNumber:{animationIterationCount:!0,columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,gridArea:!0,gridColumn:!0,gridColumnEnd:!0,gridColumnStart:!0,gridRow:!0,gridRowEnd:!0,gridRowStart:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{},style:function(e,t,n,r){if(e&&3!==e.nodeType&&8!==e.nodeType&&e.style){var i,o,a,s=X(t),u=Xe.test(t),l=e.style;if(u||(t=ze(s)),a=S.cssHooks[t]||S.cssHooks[s],void 0===n)return a&&"get"in a&&void 0!==(i=a.get(e,!1,r))?i:l[t] ("string"===(o=typeof n)&&(i=te.exec(n))&&i[1]&&(n=se(e,t,i),o="number"),null!=n&&n==n&&("number"!==o||u||(n+=i&&i[3]||(S.cssNumber[s] ?"" :"px")),y.clearCloneStyle||""!==n||0!==t.indexOf("background")||(l[t]="inherit"),a&&"set"in a&&void 0==(n=a.set(e,n,r))||(u?l.setProperty(t,n):l[t]=n))}},css:function(e,t,n,r){var i,o,a,s=X(t);return Xe.test(t)||(t=ze(s)),(a=S.cssHooks[t]||S.cssHooks[s])&&"get"in a&&(i=a.get(e,!0,n)),void 0===i&&(i=We(e,t,r)),"normal"===i&&t in Ge&&(i=Ge[t]),""===n||n?(o=parseFloat(i),!0===n||isFinite(o)?o||0:i):i}}),S.each(["height","width"],function(e,u){S.cssHooks[u]={get:function(e,t,n){if(t)return!Ue.test(S.css(e,"display"))||e.getClientRects().length&&e.getBoundingClientRect().width?Je(e,u,n):Me(e,Ve,function(){return Je(e,u,n)})},set:function(e,t,n){var r,i=Re(e),o=!y.scrollboxSize()&&"absolute"===i.position,a=(o||n)&&"border-box"===S.css(e,"boxSizing",!1,i),s=n?Qe(e,u,n,a,i):0;return a&&o&&(s-=Math.ceil(e["offset"+u[0].toUpperCase()+u.slice(1)]-parseFloat(i[u])-Qe(e,u,"border",!1,i)-.5)),s&&(r=te.exec(t))&&"px"!==(r[3]||"px")&&(e.style[u]=t,t=S.css(e,u)),Ye(0,t,s)}}}),S.cssHooks.marginLeft=Fe(y.reliableMarginLeft,function(e,t){if(t)return(parseFloat(We(e,"marginLeft"))||e.getBoundingClientRect().left-Me(e,{marginLeft:0},function(){return e.getBoundingClientRect().left}))+"px"}),S.each({margin:"",padding:"",border:"Width"},function(i,o){S.cssHooks[i+o]={expand:function(e){for(var t=0,n={},r="string"==typeof e?e.split(" "):[e];t<4;t++)n[i+ne[t]+o]=r[t]||r[t-2]||r[0];return n}},"margin"!==i&&(S.cssHooks[i+o].set=Ye)}),S.fn.extend({css:function(e,t){return $(this,function(e,t,n){var r,i,o={},a=0;if(Array.isArray(t)){for(r=Re(e),i=t.length;a.container { width:95% !important; }")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') from utils import * # paths at_home = False if '/fast/work/users/' in os.getcwd() else True data_path = '/extra/stefan/data/perturbation_resource_paper/' if at_home else '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' signatures_path = '/home/peidli/utils/scrnaseq_signature_collection/' if at_home else '/fast/work/users/peidlis_c/utils/scrnaseq_signature_collection/' utils_path = '/extra/stefan/utils/scrnaseq_utils/' if at_home else '/fast/work/users/peidlis_c/utils/single_cell_rna_seq/scrnaseq_utils/' # Stefan's utils sys.path.insert(1, utils_path) from scrnaseq_util_functions import * ``` -------------------------------- ### Calculate similarity metrics for guide targets Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Computes average similarity scores for guides targeting the same gene compared to those targeting different genes. ```python # dict of guides : [guides with at least one same target] A = {} for t in targets: A[t] = list(sim.columns[[t in c for c in sim.columns]]) # Average similarity across guides with same target vs different target tab = pd.DataFrame(index=targets, columns=['same', 'different']) for t in targets: a = A[t] tab.loc[t, 'same'] = np.mean(sim.loc[a, a].values) tab.loc[t, 'different'] = np.mean(sim.loc[~np.isin(sim.index, a), ~np.isin(sim.index, a)].values) ``` -------------------------------- ### Calculate Average Similarity Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Calculates the average similarity between guides targeting the same gene versus guides targeting different genes. Requires pandas and numpy. ```python tab = pd.DataFrame(index=dups, columns=['same', 'different']) for dup in dups: g1, g2 = A[dup] tab.loc[dup, 'same'] = sim.loc[g1, g2] tab.loc[dup, 'different'] = np.mean(sim.loc[sim.index!=g1, sim.columns!=g2].values) ``` -------------------------------- ### Install scperturbR R Package Source: https://github.com/sanderlab/scperturb/blob/master/README.md Install the scperturbR R package from CRAN. This package is designed to work with Seurat objects for analyzing single-cell perturbation data. ```r install.packages('scperturbR') ``` -------------------------------- ### Initialize scperturb environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/parallel.ipynb Imports core libraries and configures Jupyter notebook settings for analysis. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.auto import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * ``` -------------------------------- ### Initialize scANVI Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/scANVI.ipynb Imports necessary libraries and appends the project directory to the system path to enable module loading. ```python import scvi import numpy as np import pandas as pd import scanpy as sc import os from sklearn.metrics import confusion_matrix import importlib import sys sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '../../'))) import scanvi ``` -------------------------------- ### Initialize Environment and Paths Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/e-testing.ipynb Configures the Python path and defines global variables for data directories and perturbation color mappings. ```python sys.path.insert(1, utils_path) from scrnaseq_util_functions import * ``` ```python colors_perturbation_types = { 'CRISPRi': 'tab:blue', 'CRISPRa': 'tab:red', 'CRISPR': 'tab:orange', 'drug': 'tab:green', 'cytokine': 'tab:olive' } SDIR = '/fast/scratch/users/peidlis_c/perturbation_resource_paper/' table_path = '/fast/work/users/peidlis_c/projects/perturbation_resource_paper/single_cell_perturbation_data/code/notebooks/data_analysis/analysis_screens/tables/' ``` -------------------------------- ### Install scperturb Python Package Source: https://github.com/sanderlab/scperturb/blob/master/README.md Install the scperturb Python package using pip. This package is used for computing E-distances and performing E-tests on single-cell perturbation data. ```bash pip install scperturb ``` -------------------------------- ### Map Duplicate Targets to Guides Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Creates a dictionary where keys are duplicate target names and values are lists of corresponding guide identifiers found in the similarity matrix columns. ```python # dict of duplicate targets : [guides] A = {} for dup in dups: A[dup] = [x for x in sim.columns if dup+'_pDS' in x] ``` -------------------------------- ### Calculate and Plot Guide Similarity Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/aggregate_results.ipynb Calculates and plots the average similarity between guides targeting the same gene versus different genes. Requires pandas and numpy for data manipulation and matplotlib for plotting. ```python N = len(modes) fig, axs = pl.subplots(1,N,figsize=[4*N, 3]) tabs = {} for ax, mode in zip(axs, modes): sim = pd.read_csv(f'./analysis_screens/tables/{mode}_AdamsonWeissman2016_GSM2406681_10X010_tables.csv', index_col=0) # we do have dups! Are they more similar to each? targets = [x.split('_')[0] for x in sim.columns] # print(f'Found {len(sim) - len(pd.unique(targets))} duplicate targets.') perts, cts = np.unique(targets, return_counts=True) dups = perts[cts>1] # dict of guides : [guides with at least one same target] A = {} for t in targets: A[t] = list(sim.columns[[t in c for c in sim.columns]]) # Average similarity across guides with same target vs different target # dict of duplicate targets : [guides] A = {} for dup in dups: A[dup] = [x for x in sim.columns if dup+'_pDS' in x] # Average similarity across guides with same target vs different target tab = pd.DataFrame(index=dups, columns=['same', 'different']) for dup in dups: g1, g2 = A[dup] tab.loc[dup, 'same'] = sim.loc[g1, g2] tab.loc[dup, 'different'] = np.mean(sim.loc[sim.index!=g1, sim.columns!=g2].values) tabs[mode] = tab ax.bar([0,1], [np.mean(tab.same), np.mean(tab.different)], yerr=[np.std(tab.same), np.std(tab.different)]) ax.set_ylabel(f'Average similarity') ax.set_xticks([0,1]) ax.set_xticklabels(['same target', 'different target']) ax.set_title(mode) pl.tight_layout() pl.suptitle('Similarity between cells that got a guide with same and different target\n(AdamsonWeissman2016_GSM2406681_10X010)', y=1.15, fontsize=14) pl.show() ``` -------------------------------- ### Calculate Guide Similarity Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/aggregate_results.ipynb Iterates through different similarity modes, reads similarity data, identifies duplicate targets, and calculates average similarity for guides targeting the same or different genes. Requires pandas, numpy, and matplotlib.pyplot. ```python fig, axs = pl.subplots(1,5, figsize=[4*5, 3]) tabs = {} SDIR = "/fast/scratch/users/peidlis_c/perturbation_resource_paper/" for ax, mode in zip(axs, modes): sim = pd.read_csv(f'./analysis_screens/tables/{mode}_NormanWeissman2019_filtered_tables.csv', index_col=0) # we do have dups! Are they more similar to each? targets = [y for x in sim.columns for y in x.split('_')] # print(f'Found {len(sim) - len(pd.unique(targets))} duplicate targets.') perts, cts = np.unique(targets, return_counts=True) dups = perts[cts>1] # dict of guides : [guides with at least one same target] A = {} for t in targets: A[t] = list(sim.columns[[t in c for c in sim.columns]]) # Average similarity across guides with same target vs different target tab = pd.DataFrame(index=perts, columns=['same', 'different']) for t in targets: a = A[t] # list of guides with overlapping targets X_same = sim.loc[a, a].values # submatriks for this list onli tab.loc[t, 'same'] = np.sum(X_same-np.diag(np.diag(X_same))) / (len(a)**2 - len(a)) # mean of these similarities without diagonal X_different = sim.loc[~np.isin(sim.index, a), np.isin(sim.index, a)].values # submatriks for these with all other # X_different = sim.loc[~np.isin(sim.index, a), ~np.isin(sim.index, a)].values # submatriks for all other among them tab.loc[t, 'different'] = np.mean(X_different) # mean of these sims without diagonal tabs[mode] = tab ``` -------------------------------- ### Initialize scperturb environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/parallel.ipynb Sets up the path for the scperturb package and defines the figure directory. ```python sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../../figures/') ``` -------------------------------- ### Get and Set Form Element Values with jQuery Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/CuiHacohen2023.nb.html The `val()` method is primarily used to get the current value of the first element in the set of matched elements or to set the value of every matched element. It works with input, select, and textarea elements. ```javascript S.fn.extend({val:function(n){var r,e,i,t=this[0];return arguments.length?(i=m(n),this.each(function(e){var t;1===this.nodeType&&(null==(t=i?n.call(this,e,S(this).val()):n)?t="":"number"==typeof t?t+="":Array.isArray(t)&&(t=S.map(t,function(e){return null==e?"":e+""})),(r=S.valHooks[this.type]||S.valHooks[this.nodeName.toLowerCase()])&&"set"in r&&void 0!==r.set(this,t,"value")||(this.value=t))})):t?(r=S.valHooks[t.type]||S.valHooks[t.nodeName.toLowerCase()])&&"get"in r&&void 0!==(e=r.get(t,"value"))?e:"string"==typeof(e=t.value)?e.replace(yt,""):null==e?"":e:void 0}}) ``` -------------------------------- ### Annotate mitochondrial and ribosomal genes Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/ShifrutMarson2018.ipynb Adds boolean columns 'mt' and 'ribo' to the AnnData object's variable DataFrame to identify mitochondrial and ribosomal genes, respectively. Genes starting with 'MT-' are marked as mitochondrial, and those starting with 'RPS' or 'RPL' are marked as ribosomal. ```python adata.var['mt'] = adata.var_names.str.startswith('MT-') # annotate the group of mitochondrial genes as 'mt' adata.var['ribo']= adata.var_names.str.startswith('RPS') | adata.var_names.str.startswith('RPL') # annotate the group of ribosomal genes as 'ribo' ``` -------------------------------- ### Load Configuration Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/get_obs.ipynb Initializes global directory paths from a YAML configuration file. ```python import yaml with open('../../configuration/config.yaml', 'r') as file: config = yaml.safe_load(file) DOWNDIR = Path(config['DOWNDIR']) TEMPDIR = Path(config['TEMPDIR']) ``` -------------------------------- ### Inspect Guide IDs Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/ShifrutMarson2018.ipynb Examines the structure of the guide_id column. ```python adata.obs['guide_id'].str.split('.') ``` -------------------------------- ### Initialize scperturb Environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/PBMC_comparison_batches.ipynb Imports necessary libraries for single-cell analysis, distance calculations, and plotting. ```python import scanpy as sc from scperturb import edist, pairwise_pca_distances, equal_subsampling import seaborn as sns import matplotlib.pyplot as plt import numpy as np import pandas as pd from tqdm import tqdm from muon import prot as pt import matplotlib import math from sklearn.metrics import pairwise_distances from statsmodels.stats.multitest import multipletests import h5py from scipy.stats import zscore from scipy.cluster.hierarchy import distance, linkage, dendrogram from scipy.cluster import hierarchy import matplotlib as mpl mpl.rcParams['figure.dpi'] = 300 ``` -------------------------------- ### Get AnnData Version Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/JoungZhang2023.ipynb Imports the anndata library and returns its version. ```python import anndata as ad ad.__version__ ``` -------------------------------- ### Configure Project Paths Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/Evaluate_split.ipynb Sets up data and utility paths dynamically based on whether the script is running on a local machine or a cluster. This ensures correct file access. ```python # paths at_home = False if '/fast/work/users/' in os.getcwd() else True data_path = '/extra/stefan/data/perturbation_resource_paper/' if at_home else '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' signatures_path = '/home/peidli/utils/scrnaseq_signature_collection/' if at_home else '/fast/work/users/peidlis_c/utils/scrnaseq_signature_collection/' utils_path = '/extra/stefan/utils/scrnaseq_utils/' if at_home else '/fast/work/users/peidlis_c/utils/single_cell_rna_seq/scrnaseq_utils/' # # Stefan's utils # sys.path.insert(1, utils_path) # from scrnaseq_util_functions import * ``` -------------------------------- ### Get Perturbations from Index Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/Graph_label_entropy.ipynb Extracts the perturbation names from the index of the similarity matrix. ```python perturbations = sim.index ``` -------------------------------- ### Initialize scperturb environment Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/deprecated_correct_for_bias.ipynb Imports necessary libraries, configures Jupyter display settings, and sets up paths for custom modules and figures. ```python import subprocess import os import sys import matplotlib.backends.backend_pdf import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 from pathlib import Path # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../..') %load_ext autoreload %autoreload 2 from utils import * # scperturb package sys.path.insert(1, '../../package/src/') from scperturb import * from pathlib import Path figure_path = Path('../../figures/') ``` -------------------------------- ### Example console output Source: https://github.com/sanderlab/scperturb/blob/master/package_r/notebooks/scperturbr.ipynb Expected console output when verbose mode is enabled. ```text [1] "Computing E-test statistics for each group." ``` -------------------------------- ### Import Data Science Libraries Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/check-overlaps-K562.ipynb Initializes the environment with scanpy, numpy, pandas, and itertools. ```python import scanpy as sc import numpy as np import pandas as pd import itertools as it ``` -------------------------------- ### Initialize Paths and Imports for scPerturb Source: https://github.com/sanderlab/scperturb/blob/master/notebooks/Fig3.ipynb Sets up necessary imports and defines paths for data, temporary storage, tables, figures, and supplemental materials. Ensure the 'data_path' is correctly configured for your environment. ```python import os import matplotlib.pyplot as pl import pandas as pd import numpy as np import seaborn as sns # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # path with scPerturb data (replace accordingly) data_path = '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' # temp path SDIR = '/fast/scratch/users/peidlis_c/perturbation_resource_paper/' # output from snakemake (tables) table_path = '/fast/work/users/peidlis_c/projects/perturbation_resource_paper/single_cell_perturbation_data/code/notebooks/data_analysis/analysis_screens/tables/' # path for figures figure_path = '../figures/' # path for supplemental figures and tables supp_path = '../supplement/' ``` -------------------------------- ### Define Data Index Source: https://github.com/sanderlab/scperturb/blob/master/dataset_processing/notebooks/data_processing_JS.ipynb Sets the index for the dataset. No specific setup required. ```python index = "FrangiehIzar2021" ``` -------------------------------- ### Retrieve perturbation counts Source: https://github.com/sanderlab/scperturb/blob/master/revision/notebooks/Complex analysis demo.ipynb Get the raw count values for each perturbation present in the dataset. ```python adata.obs['perturbation'].value_counts().values ``` -------------------------------- ### Initialize Analysis Environment Source: https://github.com/sanderlab/scperturb/blob/master/notebooks/Fig4-updated.ipynb Imports necessary libraries, configures Jupyter display settings, and loads project paths from a YAML configuration file. ```python import subprocess import os import sys import scanpy as sc import matplotlib.pyplot as pl import anndata as ad import pandas as pd import numpy as np import seaborn as sns import scvelo as scv scv.settings.verbosity=1 print('Scanpy version:', sc.__version__) # Jupyter stuff from tqdm.notebook import tqdm from IPython.display import clear_output from IPython.core.display import display, HTML display(HTML("")) %matplotlib inline # Custom functions sys.path.insert(1, '../') from utils import * # paths import yaml config = yaml.safe_load(open('../config.yaml', "r")) data_path = config['DOWNDIR'] SDIR = config['DIR'] # output from snakemake (tables) table_path = '/fast/work/users/peidlis_c/projects/perturbation_resource_paper/single_cell_perturbation_data/code/notebooks/data_analysis/analysis_screens/tables/' # path for figures figure_path = '../figures/' # path for supplemental figures and tables supp_path = '../supplement/' ``` -------------------------------- ### Get Unique Datasets Source: https://github.com/sanderlab/scperturb/blob/master/notebooks/add_chembl.ipynb Extract a list of unique dataset names from the drugs dataframe. ```python dsets = drugs['Dataset'].value_counts().index.values ``` -------------------------------- ### Configure Data and Utility Paths Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/data_qc_2.ipynb Sets up environment-specific paths for data, signatures, and utility scripts based on whether the code is running on a local machine or a cluster. This ensures correct data access. ```python # paths at_home = False if '/fast/work/users/' in os.getcwd() else True data_path = '/extra/stefan/data/perturbation_resource_paper/' if at_home else '/fast/work/users/peidlis_c/data/perturbation_resource_paper/' signatures_path = '/home/peidli/utils/scrnaseq_signature_collection/' if at_home else '/fast/work/users/peidlis_c/utils/scrnaseq_signature_collection/' utils_path = '/extra/stefan/utils/scrnaseq_utils/' if at_home else '/fast/work/users/peidlis_c/utils/single_cell_rna_seq/scrnaseq_utils/' ``` -------------------------------- ### Inspect perturbation counts Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/Evaluate_split.ipynb Displays the counts of perturbations and guide IDs in the AnnData object. ```python adata.obs.value_counts(['perturbation', 'guide_id'], sort=False) ``` -------------------------------- ### Get Number of Perturbation Groups Source: https://github.com/sanderlab/scperturb/blob/master/revision/unsolicited_review/notebooks/old/subsampling.ipynb Calculates the number of unique perturbation groups present in the dataset. ```python len(counts_per_group) ``` -------------------------------- ### Load Required Libraries Source: https://github.com/sanderlab/scperturb/blob/master/package_r/notebooks/scperturbr.ipynb Initializes the environment with Seurat, dplyr, and rdist packages. ```R library(Seurat) library(dplyr) library(rdist) ```