### Install copairs Example Dependencies Source: https://github.com/cytomining/copairs/blob/main/examples/README.md This command installs the necessary Python packages, including `copairs` with its demo dependencies, required to run the example notebooks. It ensures all prerequisites are met for a smooth execution of the examples. ```bash pip install copairs[demo] ``` -------------------------------- ### Run copairs Example Notebooks Source: https://github.com/cytomining/copairs/blob/main/examples/README.md These commands navigate into the `examples` directory and then launch the Jupyter Notebook environment. This allows users to interact with the provided `copairs` example notebooks directly in their web browser. ```bash cd examples notebook ``` -------------------------------- ### Install copairs with Demo Dependencies Source: https://github.com/cytomining/copairs/blob/main/README.md Installs the `copairs` package along with additional dependencies required for running example notebooks and demonstrations. ```bash pip install copairs[demo] ``` -------------------------------- ### Install copairs Python Package Source: https://github.com/cytomining/copairs/blob/main/README.md Installs the `copairs` package and its core dependencies using pip, the Python package installer. ```bash pip install copairs ``` -------------------------------- ### Install copairs with test dependencies Source: https://github.com/cytomining/copairs/blob/main/tests/README.md Instructions to install the copairs package with its testing dependencies by checking out the code locally and using pip in editable mode. ```bash pip install -e .[test] ``` -------------------------------- ### Run copairs Tests Source: https://github.com/cytomining/copairs/blob/main/README.md Installs `copairs` in editable mode with testing dependencies and then executes the test suite using pytest to verify functionality. ```bash pip install -e .[test] pytest ``` -------------------------------- ### Load phenotypic activity and raw data Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This code loads two CSV files into pandas DataFrames: `df` contains the main experimental data, and `activity_map` holds pre-calculated mAP scores for phenotypic activity, which are used to filter active compounds. This step assumes the 'Phenotypic activity' example has been run previously. ```python df = pd.read_csv("data/2016_04_01_a549_48hr_batch1_plateSQ00014812.csv") activity_map = pd.read_csv( "data/activity_map.csv" ) # load mAP scores for phenotypic activity ``` -------------------------------- ### Load example datasets for mAP activity analysis Source: https://github.com/cytomining/copairs/blob/main/examples/null_size.ipynb This code loads two critical CSV files, `2016_04_01_a549_48hr_batch1_plateSQ00014812.csv` and `activity_ap.csv`, into pandas DataFrames. These datasets, `df_activity` and `activity_ap`, serve as the input for subsequent calculations and demonstrations of mAP p-value determination and null size estimation. ```python df_activity = pd.read_csv("data/2016_04_01_a549_48hr_batch1_plateSQ00014812.csv") activity_ap = pd.read_csv("data/activity_ap.csv") ``` -------------------------------- ### Filter for phenotypically active compounds Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This snippet filters the main dataset (`df`) to include only compounds identified as phenotypically active. It queries the `activity_map` DataFrame to get a list of active compound IDs based on a corrected p-value threshold (p < 0.05), then uses this list to filter `df`. ```python # only keep active compounds, i.e. those with corrected p-value < 0.05 active_compounds = activity_map.query("below_corrected_p")["Metadata_broad_sample"] df_active = df.query("Metadata_broad_sample in @active_compounds") df_active.head(7) ``` -------------------------------- ### Importing Core Libraries for Profile Matching Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Imports essential Python libraries: `random` for generating synthetic data, `pandas` for efficient data manipulation and DataFrame creation, and `Matcher` along with `MatcherMultilabel` from the `copairs` library, which are central to the profile matching functionality. ```python import random import pandas as pd from copairs import Matcher, MatcherMultilabel ``` -------------------------------- ### Generating Sample Data for `copairs` Matching Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Creates a synthetic pandas DataFrame to simulate experimental data. This DataFrame contains `n_samples` entries with columns for `plate`, `well`, and `label`, representing typical metadata. The code ensures data uniqueness and sorts it for consistent processing. ```python random.seed(0) n_samples = 20 dframe = pd.DataFrame( { "plate": [random.choice(["p1", "p2", "p3"]) for _ in range(n_samples)], "well": [ random.choice(["w1", "w2", "w3", "w4", "w5"]) for _ in range(n_samples) ], "label": [random.choice(["t1", "t2", "t3", "t4"]) for _ in range(n_samples)], } ) dframe = dframe.drop_duplicates() dframe = dframe.sort_values(by=["plate", "well", "label"]) dframe = dframe.reset_index(drop=True) ``` -------------------------------- ### Identifying Valid Pairs Using `Matcher` (Single Label) Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Demonstrates how to use the `Matcher` class to find pairs of samples based on specified metadata criteria. It initializes `Matcher` with the DataFrame and relevant columns, then calls `get_all_pairs` to identify samples that share the same 'label' but originate from different 'plate' and 'well' locations. ```python matcher = Matcher(dframe, ["plate", "well", "label"], seed=0) pairs_dict = matcher.get_all_pairs(sameby=["label"], diffby=["plate", "well"]) pairs_dict ``` -------------------------------- ### Run all unit tests with pytest Source: https://github.com/cytomining/copairs/blob/main/tests/README.md Command to execute all unit tests defined within the copairs project using the pytest framework. ```bash pytest ``` -------------------------------- ### Run specific unit test file with pytest Source: https://github.com/cytomining/copairs/blob/main/tests/README.md Command to execute tests contained within a particular test file, such as 'test_map.py', using the pytest framework. ```bash pytest tests/test_map.py ``` -------------------------------- ### Import core libraries for mAP p-value analysis Source: https://github.com/cytomining/copairs/blob/main/examples/null_size.ipynb This snippet imports essential Python libraries including numpy, pandas, scipy.special.comb, and matplotlib.pyplot. It also imports the `copairs.map` module, providing the foundational tools for statistical analysis, data manipulation, and visualization in the context of mAP p-value calculations. ```python import numpy as np import pandas as pd from scipy.special import comb import matplotlib.pyplot as plt from copairs import map ``` -------------------------------- ### Import necessary libraries for mAP calculation Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This snippet imports the required Python libraries, including `numpy` for numerical operations, `pandas` for data manipulation, `matplotlib.pyplot` for plotting, and the `map` module from the `copairs` library for mean average precision calculations. ```python import numpy as np import pandas as pd import matplotlib.pyplot as plt from copairs import map ``` -------------------------------- ### Performing Multi-label Profile Matching with `MatcherMultilabel` Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Illustrates the use of `MatcherMultilabel` for scenarios where a single entry can have multiple associated labels. It initializes the matcher with the multi-label DataFrame and specifies the column containing the aggregated labels. Subsequently, it retrieves pairs that share at least one common label while differing in 'plate' and 'well'. ```python matcher_multi = MatcherMultilabel( dframe_multi, columns=["plate", "well", "label"], multilabel_col="label", seed=0 ) pairs_multi = matcher_multi.get_all_pairs(sameby=["label"], diffby=["plate", "well"]) ``` -------------------------------- ### Preparing Data for Multi-label Matching with `copairs` Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Transforms the original DataFrame to consolidate multiple labels into a single column, suitable for multi-label analysis. It groups data by 'plate' and 'well', then aggregates all unique 'label' values into a list for each group, optimizing the dataset for `MatcherMultilabel`. ```python dframe_multi = dframe.groupby(["plate", "well"])["label"].unique().reset_index() dframe_multi ``` -------------------------------- ### Cite copairs Research Paper Source: https://github.com/cytomining/copairs/blob/main/README.md BibTeX entry for citing the `copairs` research paper titled 'A versatile information retrieval framework for evaluating profile strength and similarity', published in Nature Communications. ```bibtex @article{kalinin2025versatile, author = {Kalinin, Alexandr A. and Arevalo, John and Serrano, Erik and Vulliard, Loan and Tsang, Hillary and Bornholdt, Michael and Muñoz, Alán F. and Sivagurunathan, Suganya and Rajwa, Bartek and Carpenter, Anne E. and Way, Gregory P. and Singh, Shantanu}, title = {A versatile information retrieval framework for evaluating profile strength and similarity}, journal = {Nature Communications}, year = {2025}, volume = {16}, number = {1}, pages = {5181}, doi = {10.1038/s41467-025-60306-2}, url = {https://doi.org/10.1038/s41467-025-60306-2}, issn = {2041-1723} } ``` -------------------------------- ### Displaying Results of Multi-label Pair Matching Source: https://github.com/cytomining/copairs/blob/main/examples/finding_pairs.ipynb Presents the output of the `MatcherMultilabel` operation. The `pairs_multi` object is a dictionary that contains the identified pairs, structured similarly to the single-label `pairs_dict`, allowing for easy inspection of the multi-label matching results. ```python pairs_multi ``` -------------------------------- ### Aggregate replicate profiles by median Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This code aggregates replicate profiles for each compound by calculating the median of their feature values. It first identifies non-metadata feature columns, then groups the DataFrame by compound and target, applying the median aggregation. Finally, it splits the `Metadata_target` column into a list to handle compounds with multiple targets. ```python # aggregate replicates by taking the median of each feature feature_cols = [c for c in df_active.columns if not c.startswith("Metadata")] df_active = df_active.groupby( ["Metadata_broad_sample", "Metadata_target"], as_index=False )[feature_cols].median() df_active["Metadata_target"] = df_active["Metadata_target"].str.split("|") df_active.head() ``` -------------------------------- ### Identify and List Phenotypically Consistent Targets and Compounds Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This code identifies targets that are deemed 'consistent' based on their corrected p-value. It then filters the active DataFrame to find compounds associated with these consistent targets, handling scenarios where a single compound might have multiple targets. Finally, it prints the lists of consistent targets and their corresponding compounds. ```python consistent_targets = target_maps.query("below_corrected_p")["Metadata_target"] consistent_compounds = df_active[ df_active["Metadata_target"].apply( lambda x: any(t in x for t in consistent_targets) ) ]["Metadata_broad_sample"] print(f"Phenotypically consistent targets: {consistent_targets.str.cat(sep=', ')}") print(f"Phenotypically consistent compounds: {consistent_compounds.str.cat(sep=', ')}") ``` -------------------------------- ### Calculate Multilabel Average Precision for Targets Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This snippet defines criteria for negative pairs (compounds not sharing a target) and then computes the average precision scores for each target. It uses the `map.multilabel.average_precision` function to assess the performance of profiles in distinguishing between positive and negative pairs based on metadata and profile data. ```python neg_diffby = ["Metadata_target"] metadata = df_active.filter(regex="^Metadata") profiles = df_active.filter(regex="^(?!Metadata)").values target_aps = map.multilabel.average_precision( metadata, profiles, pos_sameby=pos_sameby, pos_diffby=pos_diffby, neg_sameby=neg_sameby, neg_diffby=neg_diffby, multilabel_col="Metadata_target", ) target_aps ``` -------------------------------- ### Visualize mAP vs. P-value for Target Consistency Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This snippet generates a scatter plot to visualize the relationship between mean average precision (mAP) and the negative log10 of the p-value. It uses color to indicate whether a target group's p-value is below a significance threshold (0.05), thereby identifying 'consistent' groups. An annotation displays the overall ratio of phenotypically consistent compounds. ```python consistent_ratio = target_maps.below_corrected_p.mean() plt.scatter( data=target_maps, x="mean_average_precision", y="-log10(p-value)", c="below_corrected_p", cmap="tab10", s=10, ) plt.xlabel("mAP") plt.ylabel("-log10(p-value)") plt.axhline(-np.log10(0.05), color="black", linestyle="--") plt.text( 0.5, 1.5, f"Phenotypically consistent = {100 * consistent_ratio:.2f}%", va="center", ha="left", ) plt.show() ``` -------------------------------- ### Calculate mAP Significance Across Varying Null Sizes in Python Source: https://github.com/cytomining/copairs/blob/main/examples/null_size.ipynb This Python script calculates the mean Average Precision (mAP) for a given activity dataset (`activity_ap`) across a range of `null_size` values, from 10 to 5 million. It uses a specified `cache_dir` for null distributions and collects results for further analysis and plotting. The script demonstrates how to iterate through different `null_size` values to observe their effect on mAP and corrected p-values, highlighting the importance of `null_size` for accurate significance estimation. ```python seed = 0 null_cache_dir = "cache" # default is Path.home() / ".copairs" activity_maps = [] for ns_pow in range(1, 7): null_size = 10**ns_pow replicate_map = map.mean_average_precision( activity_ap, ["Metadata_broad_sample"], null_size=null_size, threshold=0.05, seed=seed, cache_dir=null_cache_dir, ) replicate_map["null_size"] = null_size activity_maps.append(replicate_map) replicate_map = map.mean_average_precision( activity_ap, ["Metadata_broad_sample"], null_size=5 * null_size, threshold=0.05, seed=seed, cache_dir=null_cache_dir, ) replicate_map["null_size"] = 5 * null_size activity_maps.append(replicate_map) activity_maps = pd.concat(activity_maps) activity_maps.rename(columns={"mean_average_precision": "mAP"}, inplace=True) activity_maps["-log10(p-value)"] = -activity_maps["corrected_p_value"].apply(np.log10) plot_scatter_grid(activity_maps) ``` -------------------------------- ### Define criteria for positive and negative pairs Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This snippet defines the criteria for identifying positive and negative pairs for mAP calculation. Positive pairs are defined as compounds sharing the same `Metadata_target`. `pos_diffby` is left empty, meaning no additional differentiation is required for positive pairs. `neg_sameby` is also empty, implying negative pairs are those that do not share a target. ```python # positive pairs are compounds that share a target pos_sameby = ["Metadata_target"] pos_diffby = [] neg_sameby = [] ``` -------------------------------- ### Plot mAP p-value distribution across varying null sizes Source: https://github.com/cytomining/copairs/blob/main/examples/null_size.ipynb This Python function, `plot_scatter_grid`, generates a multi-panel scatter plot to visualize mAP scores against their corresponding p-values. It allows for grouping by `null_size` and highlights the 'active' ratio, providing insights into the statistical significance and distribution of results across different null distribution approximations. The function includes options for customization of axes, colors, and figure size. ```python def plot_scatter_grid( data, null_size_col="null_size", x_col="mAP", y_col="-log10(p-value)", color_col="below_corrected_p", cmap="tab10", figsize=(12, 6), ): """Plot a grid of scatter plots for different values of a given column. Args: data (pd.DataFrame): Input DataFrame containing the data. null_size_col (str): Column to split data into subplots. Defaults to "null_size". x_col (str): Column to use for the x-axis. Defaults to "mean_average_precision". y_col (str): Column to use for the y-axis. Defaults to "-log10(p-value)". color_col (str): Column for coloring points. Defaults to "below_corrected_p". cmap (str): Colormap for the scatter plot. Defaults to "tab10". figsize (tuple): Figure size. Defaults to (12, 6). """ unique_null_sizes = sorted(data[null_size_col].unique()) # Get unique values n_rows, n_cols = 3, 4 # Define grid shape fig, axes = plt.subplots(n_rows, n_cols, figsize=figsize, sharex=True, sharey=True) axes = axes.flatten() # Flatten for easy iteration for i, null_size in enumerate(unique_null_sizes): ax = axes[i] subset = data[data[null_size_col] == null_size] # Filter data for current panel # Compute active ratio for the subset active_ratio = subset[color_col].mean() # Scatter plot _ = ax.scatter( subset[x_col], subset[y_col], c=subset[color_col], cmap=cmap, s=10 ) ax.axhline( -np.log10(0.05), color="black", linestyle="--" ) # Significance threshold ax.set_title(f"{null_size_col} = {null_size}") # Display active ratio per panel ax.text( 0.4, 5, f"Active = {100 * active_ratio:.2f}%", va="center", ha="left", fontsize=9, ) if i % n_cols == 0: # Leftmost column ax.set_ylabel(y_col) if i >= (n_rows - 1) * n_cols: # Bottom row ax.set_xlabel(x_col) fig.suptitle(f"Scatter plots across different {null_size_col} values", fontsize=14) plt.tight_layout(rect=[0, 0.05, 1, 0.95]) # Adjust layout plt.show() ``` -------------------------------- ### Compute Mean Average Precision (mAP) and P-values Source: https://github.com/cytomining/copairs/blob/main/examples/phenotypic_consistency.ipynb This code calculates the mean average precision (mAP) for target groups based on the previously computed average precision scores. It performs a statistical test by comparing observed mAP values against a null distribution (simulated with `null_size`) to derive corrected p-values, which are then transformed to a -log10 scale for better interpretability. ```python target_maps = map.mean_average_precision( target_aps, pos_sameby, null_size=1000000, threshold=0.05, seed=0 ) target_maps["-log10(p-value)"] = -target_maps["corrected_p_value"].apply(np.log10) target_maps.head(10) ``` -------------------------------- ### Calculate theoretical complete null size for permutation testing Source: https://github.com/cytomining/copairs/blob/main/examples/null_size.ipynb This snippet calculates the theoretical 'complete null size' (`d`) for permutation testing, which represents the total number of possible rank list re-shuffles. It determines `m` (perturbation replicates) and `n` (control profiles) from the `df_activity` DataFrame and uses `scipy.special.comb` for a numerically stable binomial coefficient calculation. This value is crucial for understanding the scale of the null distribution. ```python # almost all perturbations have 6 replicates m = ( df_activity.query("Metadata_broad_sample != 'DMSO'") .groupby("Metadata_broad_sample") .size() .mode()[0] ) # the number of control profiles is 24 n = df_activity.query("Metadata_broad_sample == 'DMSO'").shape[0] # using SciPy's comb function for numerical stability d = comb(m - 1 + n, m - 1, exact=True) print(f"{m=}, {n=}, {d=}") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.